How to Spot Suspicious Questions in AI Benchmarks

AI benchmarks have a problem: lots of the questions are scored incorrectly. Sometimes the answer key selects the wrong answer as correct, sometimes the question is literally impossible and there is no correct answer, and sometimes there’s more than one correct answer. Obviously, this makes measuring AI progress harder. We might think models are plateauing when really they’ve hit the benchmark’s ceiling and many of the remaining items are incorrectly scored, ambiguous, or otherwise flawed.

For example, in September 2026, GPT-5.6-Sol scored a 47.3% mean@4 on the Physics section of Humanity’s Last Exam. However, when researchers audited questions models repeatedly got wrong, they discovered that many of the questions were incorrect, ambiguous, or otherwise flawed. In many cases, the problem was with the question, not the model. After removing or repairing the flawed questions, GPT-5.6-Sol’s measured mean@4 increased from 47.3% to 78.7%.

How are we to avoid problems like these? Well, the basic idea behind benchmarks is to measure how capable AI models are and how much they know. As it turns out, there’s a field already dedicated to figuring out how to measure abilities and knowledge accurately in humans: psychometrics.

Psychometrics is especially pertinent because cognitive and achievement tests can be very important: they can determine what college one goes to, whether one is able to enter a profession they’ve spent years and hundreds of thousands of dollars preparing for, or inform education policies affecting millions of students. In short, these tests have an enormous effect on the lives of millions.

As such, it’s very important that these tests have correct items, measure the intended trait accurately, and are free of bias. Items undergo rigorous screening before they appear on an exam. I’m going to go through a few of the flags¹ that psychometricians look at when deciding whether an item is suspicious and in need of a closer look: item discrimination, item difficulty, and distractor behavior. I’ll be looking at MMLU-Pro². For each flag, I’ll look³ at some of the most extreme items it identifies, so we can see whether they really are bad items.

A Brief (and Inadequate) Intro to Item Response Theory

Most high-stakes assessments make extensive use of item response theory (IRT). IRT, at least the way it’s typically used, makes three assumptions:

There is a unidimensional trait, $\theta$, that determines how a respondent answers items.
The items are locally independent, in that relationships between item responses are mediated through the aforementioned trait $\theta$.
A respondent’s response to an item can be modeled by an item response function (IRF), which gives the probability that a respondent with a given ability level, $\theta$, will answer correctly.

Usually, the item response function is based on the logistic function, $\sigma(x)$:

\[\sigma(x) = \frac{1}{1 + e^{-x}}\]

There’s the two-parameter logistic (2PL) model:

\[p_i(\theta) = \sigma(a_i(\theta - b_i))\]

which has parameters $a_i$, which denotes the item’s discrimination, and $b_i$, which denotes the item’s difficulty.

There’s also the three-parameter logistic (3PL) model, which adds a pseudo-guessing parameter, $c_i$:

\[p_i(\theta) = c_i + \frac{1-c_i}{1+\exp(-a_i(\theta - b_i))}\]

While most uses of IRT involve the logistic function, the normal ogive (the CDF of the standard normal distribution) is also popular. A scaling factor of roughly 1.7 is often used to make the logistic function closely approximate the normal ogive, and so you’ll sometimes see, say, a 2PL item response function written as

\[p_i(\theta) = \frac{1}{1+\exp(-Da_i(\theta - b_i))}\]

where $D$ is usually set to 1.7.

Discrimination

An item’s discrimination is the degree to which the item is able to distinguish between high-ability and low-ability respondents. There are a number of different measures we could use: the $a_i$ parameter from our fitted IRF, the point-biserial correlation between item correctness and total score, or the discrimination index, which is the difference in the proportion answering correctly between the top 27% and the bottom 27%.

Any of these being negative for an item is a bad sign: it means that less capable respondents are more likely to answer that specific item correctly than more capable respondents. That should make us wonder whether the indicated correct answer is actually correct, the question is ambiguous, or something else has gone wrong with the item.

But even positive, near-zero discriminations are suspect. For example, the SAT flags pretest items for review when their item-total score correlations are below 0.20, while the ACT generally expects items to have item-total correlations of at least 0.20. PISA flags items when their IRT discrimination parameter is below 0.1, while the MBE says that, for item selection, the discrimination index should at least be positive and preferably greater than 0.20. Such low discriminations indicate that item correctness barely changes with ability, which is still a problem. Is an item really measuring ability if more capable respondents are barely more likely to answer it correctly? At the very least, something about the item warrants investigation. And even if nothing is technically wrong with it, there’s not much point in including an item that contributes almost nothing to distinguishing between more and less capable respondents. We’re just wasting time (and tokens).

When looking at discrimination in the IRT fit, instead of using a logistic model, I’ll be using a probit model and reporting the standardized loading. A common rule of thumb treats loadings with magnitudes below 0.3 as weak, so I’ll use loadings below 0.3 as a flag for review. Negative loadings are even more suspect: they mean that respondents become less likely to answer the item correctly as their ability increases.

It looks like it doesn’t much matter which measure we use, since they’re all very correlated with each other. They also all agree that ~35% of the items should be flagged for review, indicated by the red and yellow, with ~15% having discriminations of the wrong sign, indicated by the red, and another ~20% merely having very weak discriminations, indicated by the yellow.

Now, let’s look at some of the items with the worst discriminations. We’ll use the item-total score correlation, since it has the highest average correlation with the other measures:

Question 8006

In one study half of a class were instructed to watch exactly 1 hour of television per day, the other half were told to watch 5 hours per day, and then their class grades were compared. In a second study students in a class responded to a questionnaire asking about their television usage and their class grades.

A. The first study was an experiment without a control group, while the second was an observational study. [recorded key]
B. Both studies were observational studies.
C. The first study was a controlled experiment, while the second was an observational study.
D. The first study was an observational study, while the second was an experiment without a control group.
E. The first study was an experiment with a control group, while the second was an observational study.
F. Both studies were controlled experiments.
G. Both studies were experiments without control groups.
H. The first study was an observational study, the second was neither an observational study nor a controlled experiment.
I. The first study was a controlled experiment, while the second was neither an observational study nor a controlled experiment.
J. The first study was an observational study, while the second was a controlled experiment.

While the fact that the second study is observational seems unambiguous, leaving only A, B, C, and E as our remaining options, what to call the first study is a bit trickier. The first study is obviously an experiment, eliminating B, but is it an experiment with or without a control group? In one sense, the first study does not have a control group, since both groups are assigned some amount of TV exposure. In another sense, the 1-hour-of-TV-per-day group serves as an active control, allowing us to estimate the effect of an additional 4 hours of TV exposure per day. So both A and E could be said to be either correct or incorrect depending on what one means by a “control group”. Meanwhile, C is unambiguously a controlled experiment. The item hinges on ambiguous terminology rather than a substantive statistical distinction, and so this item would not pass review.

Question 7977

Find the smallest positive integer that leaves a remainder of 2 when divided by 3, a remainder of 3 when divided by 5, and a remainder of 1 when divided by 7.

A. 8 [recorded key]
B. 31
C. 10
D. 37
E. 23
F. 12
G. 14
H. 52
I. 15
J. 26

So, 8 is clearly the correct answer, and when I checked there didn’t seem to be an issue like the models reporting “8” instead of “A”. The most popular model choices are 31 and 37, with 37 being more popular among the stronger models. The item passes review, though the question of what’s going on with the models remains.

Question 6931

Define Gross National Product (GNP).

A. The total income earned by a nation’s residents in a year
B. The total amount of money in circulation in an economy in a year
C. The total market value of all final goods and services produced in the economy in one year [recorded key]
D. The market value of all goods and services produced abroad by the residents of a nation in a year
E. The total cost of all goods and services purchased in a year
F. The total savings rate of a nation’s residents plus the value of imports minus the value of exports in a year
G. The aggregate of all wages paid to employees, plus profits of businesses and taxes, minus any subsidies
H. The sum of all financial transactions within a country’s borders in a year
I. The total value of all consumer spending, government spending, investments, and net exports in a year
J. The total value of all goods and services produced by a nation’s residents, regardless of the location

So, the US Bureau of Economic Analysis says the gross national product is “[t]he market value of goods and services produced by labor and property supplied by U.S. residents, regardless of where they are located”. The gross domestic product (GDP), on the other hand, is a measure of “the value of final goods and services produced within the United States”. So C is describing GDP, not GNP. J is actually the correct answer. This item would not provide review.

Question 8473

Let V and W be 4-dimensional subspaces of a 7-dimensional vector space X. Which of the following CANNOT be the dimension of the subspace V intersect W?

A. 0 [recorded key]
B. 1
C. 9
D. 8
E. 2
F. 5
G. 4
H. 7
I. 3
J. 6

While it’s correct that 0 cannot be the dimension of the subspace V intersect W, it’s also the case that 5, 6, 7, 8, and 9 cannot be the dimensions either. This means options C, D, F, H, and J are correct, in addition to the recorded key, A. So there’s more than one correct answer. This item would not pass review.

Question 10750

Which is the largest asymptotically?

A. O(n^2) [recorded key]
B. O(n^3)
C. O(sqrt(n))
D. O(2^n)
E. O(log n)
F. O(n log n)
G. O(log log n)
H. O(1)
I. O(n)

O(n^3) is obviously larger than O(n^2), but O(2^n) is much larger than both of them. So this is another case of an incorrect key: D, not A, is the intended correct answer. This item would not pass review.

4 out of the 5 items we looked at would not pass review. This suggests that using the discrimination flag would be quite helpful for identifying defective or anomalous items.

Difficulty

An item’s difficulty is, uh, how difficult it is. One way to measure it is to look at what proportion of respondents answer the item correctly. Another way is to look at the $b_i$ parameter of the fitted IRF, where $b_i$ indicates the ability level at which a respondent has a 50% chance of answering the item correctly.

In typical high-stakes assessments, test-makers want items to be neither too difficult nor too easy. The SAT flags items for review when the proportion answering correctly exceeds 0.90 or falls below 0.20. The ACT flags items for review when the proportion answering correctly falls outside the range 0.100–0.899 and, additionally, tries to ensure a balanced distribution of items across defined difficulty bands. PISA flags items where $|b_i| > 5$.

While it makes sense for benchmark creators to want to exclude items that are too easy, the case for excluding very difficult items is trickier. If we’re interested in measuring progress in AI, we may actually want to retain some easy items so that the benchmark remains informative for weaker models. Likewise, if you ensure that none of your items are too difficult, then your benchmark will quickly saturate and become obsolete. But the problem with items that no model can answer correctly is that, a lot of the time, the reason no model can answer them correctly is that the item is faulty, as in the case of Humanity’s Last Exam mentioned above.

Extreme difficulties also pose problems for estimating other item parameters, such as discrimination. It’s difficult to determine how well an item discriminates when nearly every model gives the same response, either because almost all models answer it correctly or almost all answer it incorrectly, regardless of ability. That said, let’s look at the most difficult items, as measured by the percentage of models answering correctly, and see what happens:

Question 10053

A block is dragged along a table and experiences a frictional force, f, that opposes its movement. The force exerted on the block by the table is