A group of researchers from the Hao AI Lab at the University of California, San Diego "asked" various artificial intelligence models to play Super Mario Bros. The experiment presented AI with a new challenge in the form of another gaming benchmark.
This is reported by TechCrunch.
The 1985 classic was ported using a proprietary emulator called GamingAgent and a framework that allowed the AI to control the game's character. Different models were given a set of instructions, such as "if there's an obstacle or enemy nearby, move/jump left to dodge" and corresponding screenshots. The AI was then asked to generate actions using Python and attempt to play the game.
As the researchers say, the AI had to learn to plan actions and develop a game strategy.
Among the four "contestants", Claude 3.7 and Claude 3.5 coped with the game best, while Gemini 1.5 Pro and GPT-4o had some difficulties.
It is noted that while models like OpenAI o1, which consider actions step by step and outperform competitors in some other texts, may fare worse on such a benchmark. Super Mario Bros. is a game in which timing is a very important element, which becomes a problem for AI, which needs seconds to "make decisions."