Gemini 1.5 Pro
Multimodal
Zero-eval
#1XSTest
#1WMT23
#1FunctionalMATH
+8 more
by Google
About
Gemini 1.5 Pro is a multimodal language model developed by Google. It achieves strong performance with an average score of 72.6% across 23 benchmarks. It excels particularly in XSTest (98.8%), HellaSwag (93.3%), GSM8k (90.8%). With a 2.1M token context window, it can handle extensive documents and complex multi-turn conversations. The model is available through 1 API provider. As a multimodal model, it can process and understand text, images, and other input formats seamlessly. Released in 2024, it represents Google's latest advancement in AI technology.
Pricing Range
Input (per 1M)$2.50 -$2.50
Output (per 1M)$10.00 -$10.00
Providers1
Timeline
AnnouncedMay 1, 2024
ReleasedMay 1, 2024
Knowledge CutoffNov 1, 2023
Specifications
Capabilities
Multimodal
License & Family
License
Proprietary
Benchmark Performance Overview
Performance metrics and category breakdown
Overall Performance
23 benchmarks
Average Score
72.6%
Best Score
98.8%
High Performers (80%+)
10Performance Metrics
Max Context Window
2.1MAvg Throughput
85.0 tok/sAvg Latency
1msTop Categories
reasoning
93.3%
math
74.9%
code
74.5%
vision
72.3%
general
68.9%
Benchmark Performance
Top benchmark scores with normalized values (0-100%)
Ranking Across Benchmarks
Position relative to other models on each benchmark
XSTest
Rank #1 of 3
#1Gemini 1.5 Pro
98.8%
#2Gemini 1.5 Flash
97.0%
#3Gemini 1.5 Flash 8B
92.6%
HellaSwag
Rank #3 of 24
#1GPT-4
95.3%
#2Claude 3 Opus
95.4%
#3Gemini 1.5 Pro
93.3%
#4Claude 3 Sonnet
89.0%
#5Command R+
88.6%
#6Qwen2 72B Instruct
87.6%
GSM8k
Rank #25 of 46
#22Qwen2.5-Coder 32B Instruct
91.1%
#23Qwen2 72B Instruct
91.1%
#24Llama 3.1 Nemotron 70B Instruct
91.4%
#25Gemini 1.5 Pro
90.8%
#26Grok-1.5
90.0%
#27Gemma 3 4B
89.2%
#28Claude 3 Haiku
88.9%
BIG-Bench Hard
Rank #3 of 21
#1Claude 3.5 Sonnet
93.1%
#2Claude 3.5 Sonnet
93.1%
#3Gemini 1.5 Pro
89.2%
#4Gemma 3 27B
87.6%
#5Claude 3 Opus
86.8%
#6Gemma 3 12B
85.7%
MGSM
Rank #12 of 31
#9GPT-4 Turbo
88.5%
#10o1
89.3%
#11GPT-4o
90.5%
#12Gemini 1.5 Pro
87.5%
#13GPT-4o mini
87.0%
#14Llama 3.2 90B Instruct
86.9%
#15Claude 3.5 Haiku
85.6%
All Benchmark Results for Gemini 1.5 Pro
Complete list of benchmark scores with detailed information
XSTest XSTest benchmark | general | text | 0.99 | 98.8% | Self-reported |
HellaSwag HellaSwag benchmark | reasoning | text | 0.93 | 93.3% | Self-reported |
GSM8k GSM8k benchmark | math | text | 0.91 | 90.8% | Self-reported |
BIG-Bench Hard BIG-Bench Hard benchmark | general | text | 0.89 | 89.2% | Self-reported |
MGSM MGSM benchmark | math | text | 0.88 | 87.5% | Self-reported |
MATH MATH benchmark | math | text | 0.86 | 86.5% | Self-reported |
MMLU MMLU benchmark | general | text | 0.86 | 85.9% | Self-reported |
Natural2Code Natural2Code benchmark | code | text | 0.85 | 85.4% | Self-reported |
HumanEval HumanEval benchmark | code | text | 0.84 | 84.1% | Self-reported |
MRCR MRCR benchmark | general | text | 0.83 | 82.6% | Self-reported |
Showing 1 to 10 of 23 benchmarks