### [GPTs are Predictors, not Imitators](https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators) by Eliezer Yudkowsky

>Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.
>
>Koan:  Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text?  What factors make that task easier, or harder?  (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)

##### My prediction:
- I think, that language modeling is AGI-complete task, because thought process can be described in language, including thought process of smartest humans.
- However, internet is dirty. This makes the task harder, since model learns some gibberish, popular misconceptions and things like that.
- On the other hand, the amount of good data available to a model is still much more than a human can process, the problem is that today's models aren't efficient learners.
- Guy will probably say, that internet text is not as good, and you don't need to be AGI to model internet.

Offtopic thoughts:
** I think, models lack a mechanism of enforcing consistency, which would be able to find and resolve conflicting statements.
** Today's workaround seems to be to forbid the model to have an opinion (otherwise it could promote bad things).

##### Results:
- First point supported:

>As Ilya Sutskever compactly put it, to learn to predict text, is to learn to predict the causal processes of which the text is a shadow.

- Last prediction failed. Why? I was thinking about GPT like about a GAN generator, than a predictor. And I interpreted "modeling internet" as "suggesting plausible continuations", rather than "predict exact continuation".
- Guy (turned to be Eliezer -- I didn't knew that when I was making prediction) says it is much easier to generate realistic continuations than actually correctly __predict__ next token. And he gives examples: try to predict <hash, plaintext> pairs, books which took years to write, how exactly will somebody err. In a sense, he says, that to model internet very well, you need to be __extremely__ smart.
- Point about internet gibberish countered -- Eliezer argues that it's hard to predict, so __if__ a Mind is able to predict it, it is scary.
- Learning efficiency not mentioned.
---
##### My reflections:
- I agree, that __if__ somebody models Internet nearly perfectly, they are immensely smart. Except they might be blatantly overfitted, but suppose we don't talk about it.
- I think that we are really far from developing a system, which can achieve that level of smartness. Current learning proccess is too inefficient for that, we need conceptual changes. Good humans are better than GPT-4, and they trained much less in term of compute (actually hard to compare, need to verify this intuition).

** Upon more thinking, I want to refine the notion of "immensely smart". Imagine tossing a coin. Given initial force, speed and position, one can use laws of mechanics and predict, which side we will see. It requires hard-to-get initial data and hard computation, but the principle is very simple. So, the predictor is not some-strange-alien-way smart, but just thinks fast and knows a lot. And this kind of mind is less scary to me, because it is interpretable -- I just need more time to do the same.

##### Why does this matter?
Yes, simulation and prediction are different objectives. Probably, the point Eliezer is making that the task of internet modeling is, like, super-AGI-complete -- if we model internet nearly perfectly, we have an exceptionally smart entity.
However, I think, modeling internet to such extent is hard in two ways.
- First, as I said, I think our current learning methods are suboptimal in terms of efficiency, so it is practically hard.
- Second, I believe, there is some information-theoretic bound on accuracy on predictions. Some events are inherently random, acccording to our understanding of the world -- for example, some quantum events. So, if we have a sentence, which describes the outcome of a quantum experiment, it is theoretically impossible to predict it accurately.

Currently I do not see other implications of the Eliezer observation.

##### Notes on examples:
tl;dr: some are not comparable, some are unachievable (as I think)
- Point about books which took years to write -- we need to compare not the time, but the amount of computations.
- Point about how exactly will somebody err -- well, popular misconceptions are __popular__, so predicting them will already give you good accuracy.
- Predicting exact answer may be mathematically impossible. (I think) it is mathematically impossible to learn hash function only from examples.
- Predicting exact answer may be statistically impossible. Errors of random person without context is just a random variable with significant irreducible variance.
- Errors of person for which we know a lot of side information may seem more predictable. But to utilize side informations, we need to learn relations of random variables (next token from this person, their side info). And here we will fall victim of the curse of dimensionality -- to reliably learn such relations, (I think) we need much more data than it is available in the internet. So, in practice we need to use heavy regularization, and (I think) the usefullness of side info is therefore quite limited.