A new study has proposed a novel way to measure the capabilities of artificial intelligence (AI) systems, focusing on how fast they can complete tasks compared to humans. The researchers found that AI models can outperform humans in short tasks but struggle with longer ones, where their “attention span” seems to decrease.
The study used a range of AI models and tasks, including text prediction, programming, and software engineering, to assess their capabilities. The results showed that AI models could complete tasks that would take humans less than four minutes with a near-100% success rate. However, this dropped to 10% for tasks taking more than four hours.
The researchers found that the length of tasks generalist AIs can complete has been doubling roughly every seven months over the past six years. This trend is expected to continue, with the study projecting that AI can automate a month’s worth of human software development by 2032.
Experts believe that this advancement could lead to the emergence of generalist AI agents capable of handling various tasks across an entire day or week. Such systems could take on substantial portions of professional workloads, reducing costs and improving efficiency for businesses. For consumers, AI may evolve into a dependable personal manager capable of handling complex life tasks with minimal oversight.
The study’s findings have significant implications for society, as they highlight the need for a new benchmark metric to understand AI capabilities. Experts agree that measuring AI performance on the length of time it takes humans to complete tasks is a valuable and intuitive approach, reflecting real-world complexity and capturing AI’s proficiency in maintaining coherent goal-directed behavior over time.
Source: https://www.livescience.com/technology/artificial-intelligence/ai-can-handle-tasks-twice-as-complex-every-few-months-what-does-this-exponential-growth-mean-for-how-we-use-it