Day 2: METR again

Added: 2026-05-17

[00:00:00]Yesterday, METER ran out of ways to measure the capabilities of their new models.

[00:00:04]Hi, I'm Ionut. I'm a freshman at MIT and I work on AI safety. Quick recap. METER benchmarks frontier AI models based on how long it takes human experts to complete tasks that the AI can complete with 50% reliability.

[00:00:16]At the beginning of 2020, it was about 4 seconds. At the beginning of this year, it was 14 hours. Back in March, Anthropic gave access to METER to their new unreleased model, Claude My task.

[00:00:27]And yesterday, METER released the result, at least 16 hours. But, with a footnote. Above 16 hours, METER doesn't trust its measurement anymore. Why is that? Because they don't have enough tasks that are that long. Think about that again. The benchmark didn't say the model failed. It said, "We can't measure it anymore."

[00:00:46]The doubling trend has been steady for 6 years, and officially, measuring instruments are starting to give out before the trend actually does.

[00:00:53]And that's what I'll leave you with for day two.