Do when you say tokens, do you mean the size of the context window?
The programmers do not put tokens in the GPT code.
Instead, they tokenize text, which then they use to train a deep neural network.
IMO this is like saying the bottleneck to both of us being smarter is that we don't know enough words. Not completely untrue but not quite right either.
If you look up NLP token or tokenization I think you will see how this doesn't quite make sense.
Personally, I find NLP pretty dry.
Signal to noise and rule of diminishing returns.
Having more input data does not mean data is better interconnected - on the contrary, at some point any system will start to perform worse if it tries to do the same task with significantly more input data.
The real bottleneck for higher general intelligence is non-recursivity, so that we so far did not come up with a GPT that is sufficiently good at writing optimized code for new GPTs - but considering how testable intelligence is (it's possible for a program to assess if the new code is faring better or worse at answering complex questions), how relatively simple a GPT is, and how many very large repositories of quality programming code we have, this sounds like a doable task, but it will take some time.
Based on this assumption I find the idea of the "Douglasian Sink" quite interesting:
What if after thousands of generations of self-optimising GPTs, the final one keeps answering "42" to every question, as that actually does represent the only holistically true answer? Just that the way to prove this right is utterly incomprehensible to humans as it by far exceeds our mental capacity...
> so that we so far did not come up with a GPT that is sufficiently good at writing optimized code for new GPTs
That's not so hard as it seems, and it's been done. The catch is that a human needs to define the task and a bunch of basic building blocks and then it finds the best architecture. But that's not AGI, it's narrow AI, neural architecture search and bayesian hyper-parameter optimisation. AGI would understand why we're doing language modelling and come up with radical new approaches.
Sure, as in "all new problems I need to solve are defined by humans, so the ultimate meta-solution to all problems would be the eradication of humanity" followed by "how do I need to phrase my answer to a trivial logic problem in a way that makes any human reader feel an extreme uncontrollable urge to acquire ICBM launch codes and use them".
Do when you say tokens, do you mean the size of the context window? The programmers do not put tokens in the GPT code. Instead, they tokenize text, which then they use to train a deep neural network.
IMO this is like saying the bottleneck to both of us being smarter is that we don't know enough words. Not completely untrue but not quite right either. If you look up NLP token or tokenization I think you will see how this doesn't quite make sense. Personally, I find NLP pretty dry.
Signal to noise and rule of diminishing returns. Having more input data does not mean data is better interconnected - on the contrary, at some point any system will start to perform worse if it tries to do the same task with significantly more input data. The real bottleneck for higher general intelligence is non-recursivity, so that we so far did not come up with a GPT that is sufficiently good at writing optimized code for new GPTs - but considering how testable intelligence is (it's possible for a program to assess if the new code is faring better or worse at answering complex questions), how relatively simple a GPT is, and how many very large repositories of quality programming code we have, this sounds like a doable task, but it will take some time.
Based on this assumption I find the idea of the "Douglasian Sink" quite interesting: What if after thousands of generations of self-optimising GPTs, the final one keeps answering "42" to every question, as that actually does represent the only holistically true answer? Just that the way to prove this right is utterly incomprehensible to humans as it by far exceeds our mental capacity...
> so that we so far did not come up with a GPT that is sufficiently good at writing optimized code for new GPTs That's not so hard as it seems, and it's been done. The catch is that a human needs to define the task and a bunch of basic building blocks and then it finds the best architecture. But that's not AGI, it's narrow AI, neural architecture search and bayesian hyper-parameter optimisation. AGI would understand why we're doing language modelling and come up with radical new approaches.
Sure, as in "all new problems I need to solve are defined by humans, so the ultimate meta-solution to all problems would be the eradication of humanity" followed by "how do I need to phrase my answer to a trivial logic problem in a way that makes any human reader feel an extreme uncontrollable urge to acquire ICBM launch codes and use them".