OpenAI’s latest AI model, o3, has sparked controversy over its performance in benchmark tests. The company claimed that o3 could answer over 25% of math problems on FrontierMath, a challenging set of problems, but independent benchmark results from Epoch AI suggest the actual score is around 10%. This discrepancy raises questions about OpenAI’s transparency and testing practices.
OpenAI released o3 alongside two smaller models, o4-mini and o3-mini. While these smaller models outperform o3 on FrontierMath, it is unclear whether this is due to differences in computing power or testing setup. Epoch AI noted that its testing setup likely differs from OpenAI’s, and that it used an updated release of FrontierMath for its evaluations.
The discrepancy has sparked concerns about the accuracy of benchmark results, particularly when compared to a company’s own claims. This is not an isolated incident, as other companies have faced similar criticism in recent months. The AI industry is becoming increasingly focused on benchmarking and testing, with vendors racing to capture headlines and mindshare with new models.
OpenAI has acknowledged that the o3 model may exhibit disparities between its performance in demoed tests and production use cases. However, the company maintains that its latest model is more optimized for real-world use cases and speed. The incident serves as a reminder that benchmarking results should be taken with caution, particularly when considering a company’s potential biases and motivations.
Source: https://techcrunch.com/2025/04/20/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied