Fun with LLMs

Every now and then I test LLMs to see if they have actually achieved human-level intelligence. Or animal-level intelligence. or if the world is just delusional and easily tricked by the Wizard of Oz. I’ve been doing this for many years. Here was the last test I ran. I didn’t want to run the same test this time.

First, in order to have human-level intelligence an LLM has to be able to ask the question “What If?” Humans have taught apes how to use sign language but in 40 years of doing this no ape has ever formulated a question to the researchers. How you use the “What If?” question is a sign of higher human-level intelligence as well. It is the challenge that expands the domain of existing knowledge.

Let me give you an example.

Now an smart human would clarify at each step without being prompted. If you have children you understand this. When children learn something they always come back and ask questions when they get stuck. Or in a situation they think they may be making a mistake.

Which is often when you make assumptions and children figure this out very young.

But, the “What If?” question not being asked in this case is acceptable. Only a very smart person would think to clarify here. Only a smart person would go through the 1945 text and see there were references to earlier texts in it.

This seems impressive. Especially because it continually prompts you to use more tokens. That prompt-induction feels very human but is this a sign of human-level intelligence?

Let’s look at a simpler and more directly expository one. A series of questions that an elementary school child would not need to ask questions to answer.

And a bird would have no problem solving this problem either. Try it at home.

Any sort of animal-level intelligence or higher would not require additional information to understand you just move the Post-It out of the way.

-AJ