New AI scaling method faces expert skepticism

Researchers claim to have uncovered a new method for improving artificial intelligence (AI) performance, but experts are cautious about its effectiveness. This buzz originates from discussions on social media regarding a potential new “scaling law” for AI. Scaling laws describe how AI performance increases with larger datasets and more computing power during training. Until recently, the focus was mainly on “pre-training,” which involves training models on more extensive data sets. Now, two additional methods, called post-training scaling and test-time scaling, have started to gain attention. A paper released by researchers from Google and UC Berkeley introduces a concept known as “inference-time search.” This method allows a model to create numerous answers simultaneously and choose the best one. The researchers believe that this could enhance the performance of older models, like Google’s Gemini 1.5 Pro, making them more competitive against newer models from other companies. Despite the excitement, some experts are skeptical. They argue that inference-time search may not be practical for most situations. Matthew Guzdial, an AI researcher, points out that it works best when there is a clear best answer. He notes that many queries are complex and cannot be easily evaluated. Mike Cook, a researcher at King’s College London, agrees. He emphasizes that this method does not improve the model's reasoning but merely helps to identify mistakes. He mentions that checking multiple attempts could help find errors, but it is not a true solution to the limitations of AI reasoning. As the AI industry continues to seek efficient techniques for scaling model performance, the limitations of inference-time search may pose challenges. Current reasoning models can be very costly to operate, encouraging ongoing exploration for new methods.