Here are the main benchmarks to analyze to ensure the accuracy of a generative AI model on your use case.
With hundreds of AI models flooding the market and new daily arrivals, benchmarks of generative AI models have become essential to compare the performance of each other on a specific task. You still have to know which benchmark to look at! To help you, we have compiled for you the main benchmarks to analyze to guarantee the model of the model on your specific use case.
| Usage case | Mmlu | Mmmu | MATH | Mathvista | Bow | VQAV2/GQA | Video | Realtoxicyprompts |
|---|---|---|---|---|---|---|---|---|
| Classic conversational assistant | X | X | X | |||||
| Multimodal conversational assistant | X | X | X | X | ||||
| Autonomous agent | X | X | X | |||||
| Autonomous agent with visual modality | X | X | X | X | ||||
| OCR and document analysis | X | X | X | |||||
| Moderation of visual content | X | X | X | X | ||||
| Moderation of textual content | X | X | ||||||
| Video analysis | X | X | ||||||
| Visual data analysis | X | X | X | |||||
| Report analysis | X | X | ||||||
| Analysis of feelings (multimodal) | X | X | X |
MMLU (Massive Multitask Language Understanding) assesses general linguistic skills, Math precisely tests mathematical reasoning capacities, Mathvista measures visual understanding and mathematical problems, arc-Agi analyzes the capacities of cognitive reasoning, VQA v2/gqa (visual question yearswering) tests the ability to understand and answer the Images, Videoqa assesses the understanding of video content, and realtoxicityptspted makes it possible to measure the propensity of the model to generate potentially toxic or inappropriate content.
For conventional conversational assistants, MMLU remains the benchmark of reference, especially for linguistic skills and general culture. Math makes it possible to assess mathematical reasoning, essential for judging the analytical depth and the problem solving capacity of the model. For multimodal conversational assistants, MMMU also remains essential, with a complete evaluation of multimodal capabilities.
For autonomous agents, the reasoning is king: Math and Arc-Agi are essential. Arc-Agi specifically targets advanced cognitive reasoning tasks, by assessing the capacity of a model to be adapted intelligently to new problems, from very few examples. For other specific use cases, whether OCR, video analysis or other technical areas, just refer to the corresponding benchmarks: Videoqa for video understanding, MMMU for multimodal tasks, or specific visual benchmarks according to specific needs.
