Nearly two weeks after Elon Musk’s xAI startup opened the AI model behind Grok to the public, its AI chatbot is set to get an update.
The company announced Grok-1.5 on Thursday and said its latest model can understand longer documents, handle more complex requests and perform more advanced reasoning.
While Grok-1.5 appears to be a step ahead of the original 1.0 with improvements in coding and math capabilities, its announcement post shows that it still lags behind Google’s Gemini Pro 1.5 AI, OpenAI’s GPT-4, and Anthropic’s Claude 3 Opus in some benchmark tests, beating OpenAI in a key HumanEval test.
Related: Meet Grok: Elon Musk unveils ‘spicy’ AI chatbot full of ‘sarcasm’ and ‘humor’
Grok-1.5 scored higher than GPT-4 on the HumanEval benchmark, which consists of 164 challenging programming problems not included in the AI model’s training data. GPT-4 scored 67% and Gemini Pro 1.5 scored 71.9%, while Grok-1.5 received 74.1%.
Elon Musk’s xAI company is set to release a new version of the Grok AI chatbot, a competitor to ChatGPT. Photo by Jaap Arriens/NurPhoto via Getty Images.
With a score of 81.3% on the MMLU test, which covers knowledge of 57 subjects from elementary to advanced level, Grok-1.5 came close to the Google Gemini score (83.7%).
He also scored close to the GPT-4 score of 52.9% with a score of 50.6% on the MATH test, a benchmark that covers elementary and high school math competition problems.
Related: Elon Musk sues ChatGPT-Maker OpenAI, accusing the company of working to “maximize profits for Microsoft, rather than benefit humanity”
Musk said Friday posts on social media that Grok 1.5 should be available on X, formerly Twitter, within the next week.
The owner of Grok 2 is “in training now,” he wrote in the post.
Grok AI is currently only available to those who have a $16 per month or higher Premium+ subscription on X.
Musk sued OpenAI, a competitor to xAI, earlier this month and asked for a court ruling that would force OpenAI to make public the research and technology behind its artificial intelligence.