Key Highlights:
- The IndQA benchmark tests all models in 12 Indian languages.
- It consists of 2,278 questions carefully designed by a team of 261 domain experts.
- OpenAI’s IndQA benchmark shows that GPT-5 offers the best understanding of Indian languages compared to Google’s Gemini and xAI’s Grok models.
OpenAI has invented a new benchmark named IndQA, to test the performance of any AI model in understanding Indian languages. The new benchmark not only understands the context of Indian languages, but can also analyse the culture, references, and other local aspects. It also tests the model for Hinglish, a popular combination of Hindi and English which is widely used by Indians in regular conversations naturally. IndQA tests the AI model against 12 Indian languages covering all regions.
IndQA for India: OpenAI’s Benchmark for Bharat
OpenAI says that existing benchmarks focus highly on English, even though 80% of the world speaks non-English languages. It also mentioned that tests like MMMLU are becoming saturated as they are not able to measure the progress made by models. The company also mentioned that most language benchmarks test mainly on translation, whereas a language contains emotions, context, and culture.
Based on these observations, OpenAI built the IndQA benchmark which evaluates any AI model for its relevance with Indian languages. It tests the model with 2,278 questions across 12 languages and 10 cultural domains. This includes Marathi, Hindi, Hinglish, Gujarthi, Kannada, Bengali, Malayalam, Odia, Punjabi, Tamil, and Telugu. The questions for the benchmark were designed in partnership with 261 domain experts across India, to have an accurate understanding of the context and culture of each Indian language.
Introducing IndQA — a new benchmark that evaluates how well AI systems understand Indian languages and everyday cultural context.https://t.co/MWbRDFQQup
— OpenAI (@OpenAI) November 5, 2025
In addition to testing just the language, IndQA also checks for cultural topics like design, architecture, food and cuisine, entertainment, law, ethics, religion, sports, and more. In simple words, the IndQA benchmark tests the AI model to see how an Indian individual will use the service with respect to local culture.
IndQA’s Superior Context Based Understanding
OpenAI also shared an example showcasing how IndQA understands the context of a statement, instead of just trying to decode the meaning of the words.
Using a normal translator in such statements would result in vague outcomes, as many of the words like ‘Vigyaan’ have multiple meanings, based on the sentence. Here’s where IndQA excels as it tests the model for the meaning of the statement, instead of just judging how accurate the transition was.
IndQA was also designed using powerful reasoning and logic abilities of GPT-4o, GPT-4.5, and even o3. This also reflects that IndQA was in the making for a long time, and OpenAI took its own time to fine tune the benchmark as a long-term project.
OpenAI’s Expertise in Indian Languages
OpenAI also tested its own in-house GPT models with the IndQA benchmark to evaluate their performance. The tests show thatGPT-5 Thinking High mode understands Indian languages just as good as Gemini 2.5 Pro Thinking Mode. GPT-5 also does better than Grok 4, and previous GPT models. However, the company did not share any comparison with Anthropic’s Claude models, or any other competitor.
They further mention that IndQA is still under development, and OpenAI will further improve this benchmark for higher standards. It is also planning to create similar benchmarks for help AI research labs understand more about languages and other domains.
India is among the biggest markets for OpenAI, and the company is investing heavily in the region. Recently, the company opened its new officesOpenAI IndQA benchmark example in New Delhi and Bengaluru for its IndiaAI Mission. OpenAI is also hiring local talent to expand in the South East Asia region. Interestingly, Anthropic also opened its new office in the vicinity with a similar mission, which reflects how important India is for AI companies.
