OpenAI and other leading companies have difficulty creating better AI models

The three leading AI companies – OpenAI, Google, and Anthropic – are having difficulty developing new large-scale language models. The biggest challenge is to create a model that will significantly outperform the previous one. This was reported by Bloomberg, citing anonymous sources within the companies.

In September 2024, OpenAI completed training a new model known as Orion, but the final result was not very encouraging. The model failed to significantly improve the results compared to the previous ones. As of the end of the summer, Orion was unable to answer questions about coding that it had not been trained on.

Overall, the leap from GPT-4 to Orion was much smaller than from GPT-3.5 to GPT-4. Despite previous statements, one of the people close to the development says that the model is unlikely to be released before next year.

Google and Anthropic, two other leaders in the AI industry, are facing similar problems. Google’s new version of Gemini is not meeting the company’s internal expectations. Anthropic, in turn, postponed the release of the long-awaited Claude model called 3.5 Opus.

The main problem with the development of both Orion and other models is finding information to train them. While Wikipedia articles or YouTube videos were enough for the first versions, more powerful models require more scientific data, which is much harder to obtain.

Derived from the previous problem is the problem of money. Developing, training, and maintaining new models requires more resources and constantly requires more funding. At the same time, not being able to show better results has a negative impact on potential future investments.

Despite these challenges, OpenAI, Google, Anthropic, and others continue to develop their large language models in an attempt to create a general AI that could potentially match the power of the human brain. Companies are finding new ways to improve AI, such as post-training: before releasing a model, they add user feedback to improve responses and tone. However, this is still not enough.