Did Mixtral 8x7B Really Beat GPT 3.5 Turbo? Performance Comparison on 15 Questions for Reasoning, Logic and Coding Tasks

A year later, after GPT-3.5 was released, social media is buzzing with an open-source LLM that matches and even outperforms GPT-3.5!

Even better, we can run it up to 100 token/s for $0.0002/1K tokens, or run entirely within two 80GB GPUs.

Everybody says the numbers don’t lie, but is that always the case?

We will try to solve 15 questions/tasks from Reasoning, Logic, and Coding categories, and this will show us whether the recent news about Mixtral is a few steps ahead of reality or not!

Join our next cohort: Full-stack GenAI SaaS Product in 4 weeks!

Let’s go through it together:

• Setting up API access for Mixtral 8x7B, GPT-3.5, and GPT-4
• Running initial tests with API endpoints from your local environment
• Generating answers to questions in Reasoning, Logic, and Coding, and saving to a CSV for further analysis
• Comparing Mixtral 8x7B in Reasoning, Logic, and Coding to GPT-4 and GPT-3.5-Turbo (Google Sheet provided)
• Comparing pricing between providers
• Integration with LangChain and chat history management
• Resources to dig deeper

Roll up your sleeves, this will be a fun ride!