Oblique Ways of Measuring Intelligence

If you want to measure someone’s intelligence, the standard approach is to give them an intelligence test, or some reasonably accurate proxy: a standardized test, an admissions test, a rationality test, etc. But these are obviously measures of intelligence; the person taking them knows what you’re doing. What if you want to estimate someone’s intelligence without making it obvious that this is what you’re measuring?

Eye tracking during Raven’s Matrices

One approach is to give them a Raven’s Matrices test, which, to be fair, is still an intelligence test. But instead of scoring the test-taker based on how many items they answer correctly, you can look at their eyes while they take the test. In Using a multi-strategy eye-tracking psychometric model to measure intelligence and identify cognitive strategy in Raven’s advanced progressive matrices, the researchers use eye-tracking data to infer which strategy participants are using on each item.

There are two common strategies people use when solving matrix questions: a constructive matching strategy, in which participants look at the matrix, mentally construct the answer, and then choose the matching option, and a response elimination strategy, in which participants look back and forth between the matrix and the possible answers, iteratively eliminating options.

For example, in the figure below, on the first item, Participant 1 mostly looked at the matrix interest area: they were constructing the correct answer in their head, then quickly glanced around the response-options interest area, found two plausible answers, and picked one. On the second item, the same participant kept looking back and forth between the matrix and the response options, apparently fitting the symbols into the diagram and eliminating them one by one until they made a choice.

It turns out that people adjust their strategies based on both item difficulty and their own ability. As items become harder, more intelligent participants become more likely to use constructive matching, while less intelligent participants become more likely to use response elimination. Using only the information about which strategies participants use for each item, researchers were able to calculate intelligence scores that correlated an astonishing 0.986 with the intelligence scores calculated from which items participants answered correctly.

Predicting intelligence from the MMPI

What if we don’t want them to take a test that looks so much like an intelligence test? We could instead have them take the Minnesota Multiphasic Personality Inventory (MMPI), a psychometric test designed to measure personality and psychopathology. The MMPI consists of 567 short statements, each of which the test-taker answers True or False. In Can Intelligence Be Predicted from the MMPI? An Out-of-Sample Validation Study, Kirkegaard uses data from the Vietnam Experience Study, in which participants took both the MMPI and an intelligence test, to estimate how well intelligence can be predicted from MMPI item responses.

He finds that out-of-sample predictions of intelligence from MMPI responses, using an elastic net model, correlate with intelligence test scores at r = 0.84. This is very high, about as good as an actual intelligence test.

And you don’t even need that many items: the correlation reaches 0.80 with roughly 100 items and 0.70 with roughly 25 items.

Response consistency and response styles

There’s a third method: instead of looking directly at people’s responses, we can look at how their responses differ from a model-predicted “ideal.” This method, developed in Can You Tell People’s Cognitive Ability Level from Their Response Patterns in Questionnaires?, makes use of two ideas. The first is the worst performance rule, which says that people’s worst performance across multiple sequential tasks is more indicative of their cognitive ability than their average or best performance. The second is the task complexity hypothesis, which says that relationships between cognitive ability and performance become stronger as task complexity increases.

In the paper, answering questionnaire items is treated as a series of cognitively demanding tasks. The researchers use an IRT model to estimate each participant’s level of the relevant latent traits, then calculate the difference between each participant’s observed response and their model-predicted response for each item. These differences are called “response error” scores. The latent variable estimated from response errors on the most complex items correlated 0.50 with a latent cognitive ability factor. In other words, people who give more internally consistent answers tend to be smarter as well.

There are also many other kinds of response styles with distinctive relationships to intelligence. For example, acquiescence bias (the tendency to agree with items regardless of their content) and extreme responding (the tendency to choose the extremes of a response scale) both show negative correlations with intelligence. In principle, one could measure respondents’ intelligence by measuring the extent to which they exhibit these response styles.

Takeaway

None of these methods are magic. They work because intelligence isn’t an isolated trait that only appears in the answers of intelligence tests. It leaks into other behaviors: how people narrow down answer spaces, how internally consistent their responses are, and even what kind of biases they exhibit.

This is both interesting and kind of uncomfortable. Eye movements, questionnaire responses, and response styles aren’t usually thought of as measures of intelligence, but they can be used for the same purpose. This raises an obvious question: how far does this generalize? If eye movements, questionnaire responses, and response styles contain this much signal, what about writing style, speech patterns, mouse movements, or the thousands of other small quirks that people have?

Most of these signals are weak. But our world is inferentially entangled: traits leave traces, traits correlate with other traits, and enough weak signals can add up to a strong proxy. If one is smart enough, or simply measures precisely enough, one can infer a great deal of hidden knowledge.