Putting a number on 328 olympiad problems

The most important decision in teaching olympiad maths is which problem to put in front of a student. One problem at the edge of what they can do teaches more than ten they can already handle, but that edge is narrow, it is different for every student, and it moves every few weeks. Hitting it takes more than knowing the papers well. It takes knowing, for every problem in thirty years of them, which techniques it uses and how hard it will be for this student now.

So I built a catalogue that holds exactly that: every question solved, classified by the techniques it uses, and rated for difficulty, so that the problems a student practises on can come from the whole of BMO's history. This piece is about the last of those three — how to put a number on a problem at all.

What "difficulty" should mean

The obvious approach is to label problems easy, medium and hard. This is useless within a week: a student who finds "medium" comfortable has no next step, and two mediums can be half an hour apart in reality.

Chess has a better answer. Ratings are not labels, they are predictions: the gap between two ratings tells you the probability of each result. Treat a problem as the opponent and the same idea works, which is exactly how chess sites rate their puzzles. A rating then answers the question I actually care about: what are the chances this student cracks this problem?

That gives every question a number on the same scale as every student, and it makes the tool's job precise. "Three geometry questions I have an even chance at" becomes a search, not a judgement call.

The question's rating moves just as the student's does. Every attempt is a game, and afterwards both update: a question that keeps getting solved drifts down, and one that keeps beating people drifts up — though only by a nudge each time, since a question meets many students and a student meets it once. The starting numbers below are only where each question begins; from then on it is rated by the attempts made on it.

Where the starting numbers came from

Ratings are only useful once they are calibrated, and a new system has no attempts to learn from. The British Mathematical Olympiad has thirty-three years of papers but almost no published statistics, so the starting difficulties had to come from what the UK Mathematics Trust does release: the markers' reports.

Nine of those reports exist online. Two of them, from 2016/17 and 2017/18, give the average mark for each question, which is close to a direct difficulty measurement. The other seven give a histogram of total scores, drawn as a chart. Those charts turned out to be vector graphics rather than images, so the exact bar heights could be read out of the PDF: for 2019/20 the extracted counts sum to precisely the 1,475 candidates the report states.

A total-score histogram seems like weak evidence about individual questions, but it is not. BMO is marked "10 minus / 0 plus": a candidate who cracks a question scores 7 to 10, and one who does not scores 0 to 3. Totals therefore cluster near multiples of ten, and each year's histogram is effectively a count of how many questions each candidate solved. Fit a model where ability varies across candidates and difficulty rises through the paper, and the shape of that distribution pins down how much harder each question is than the last.

The fit says about 1.39 logits per question position, which on the scale I settled on is roughly 173 points. Anchoring it so that a typical modern Round 1 Q1 sits at 1200 and a Round 2 Q4 at 2500 gives a ladder for all 328 questions, adjusted for the year and for how hard each individual paper turned out to be.

Three things the data said that I did not expect

Question order is only a rough guide. In 2017/18 the average mark on Q3 was 7.2, higher than Q1's 6.8 and far above Q2's 4.2. The paper's intended order and the order candidates experienced were different, which is worth remembering before assuming Q5 is harder than Q4.

Papers vary far more year to year than over decades. Older papers are easier, but mildly: the long trend is about nine points a year. Individual papers swing several times that. The qualifying mark for Round 2 was 43 in two consecutive years and then 31, a twelve-mark collapse at the top of the cohort in a single year.

Round 2 looks easier than its reputation. Each year, 70% of qualified entrants reach the Merit threshold on Round 2, and those same students mostly did not solve Round 1's last question. Taken literally, Round 2 Q1 behaves like a Round 1 Q4 under exam conditions. I overrode that with judgement, because a Merit mark of 6 to 10 can be assembled from partial credit rather than one solved problem, but it was not what I expected to find.

Why the whole scale drifts

The model only ever sees the gap between a student and a question, never either number on its own, so the entire scale could slide up or down without a single prediction changing. Nothing in the data says where it should sit.

Chess has exactly this blind spot and gets away with it, because a rated chess game is zero-sum: every point the winner gains is a point the loser loses. The pool's total is fixed and its average has nowhere to go. Chess ratings do drift slowly as players join and leave, but they do not run away.

Here we break that balance on purpose. Chess opponents are usually close in strength, but a student can open any question, so lopsided attempts are routine, and so are big upsets. When one happens, it would not be fair to penalise the loser by as much as the winner is rewarded, so the rule limits the penalty: a gap can count for up to twice as much for the underdog as for the favourite.

That fairness has a price. In a big upset the winner now gains more than the loser loses, and the difference is rating created from nothing. It happens more quietly on every lopsided attempt, upset or not: the two sides' expected scores add up to about 0.92 rather than one, and exactly one result always happens. It is not the only thing here that fails to balance — a question is also meant to move far less than a student — but it is the easiest to see.

Left alone, the scale does not settle. In simulation, with every student's true ability held fixed so that nothing real was changing, the catalogue's average rating rose 12 points over 6,000 attempts, 36 over 30,000 and 146 over 150,000. With nothing to pull it back, a drift like that has no limit: left untreated, every rating in the system diverges to infinity.

The fix is to notice that a question is a fixed object. Its difficulty never changes, so any common movement across the whole catalogue is the population changing, not the questions, and it belongs in student ratings. So the catalogue is recentred continually: after every update, every question shifts by the same small amount to hold the average where the calibration put it. Each correction is tiny, well under a percentage point on anyone's predicted chance, and questions still move relative to each other. The yardstick stays still and the students move against it.

What a rating cannot tell you

None of this will be perfect. The starting ratings are an educated guess from nine markers' reports, attempts are self-reported, and some questions will sit in the wrong place for a while before enough students correct them.

And a rating only ever says how likely a student is to solve a question. Two problems with the same rating can teach completely different things, which is why every question in the catalogue is also classified by the techniques its solution uses, so I can look for the technique a student needs and not just a level. Which of those problems will teach this student the most right now, and what they should take from it, is still my call. That is what teaching is built from, and it is not something you can put a number on.

More: Mathelo