Localizing AI in the global south

Countries in the global south stand to benefit considerably from AI developments and are taking the lead in determining the direction of inclusive AI research efforts.

Much attention is directed at the race in developing large language models (LLMs) such as GPT-4, Gemini, Claude and DeepSeek, which are competing to outperform each other in language generation and content creation. However, these models, trained mainly on English and Western culture-centric data, perform badly in non-Western contexts and languages.

This is an unfortunate development, as artificial intelligence (AI) technology, with the right focus, has the potential to address pressing societal challenges. For example, in a recent Correspondence¹ in this journal, Chakraborty et al. highlight the potential of generative AI to support public mental health care in India, particularly in suicide prevention. LLM-based systems could alleviate challenges posed by the country’s vast population, cultural diversity, shortage of trained professionals, and the stigma that surrounds mental health. However, although current systems perform well on specialized tasks in English or basic tasks across many languages, they remain inadequate for specialized, culturally sensitive applications. Chakraborty et al.¹ call for the development of cost-effective models tailored to the Indian context, trained on native languages and capable of operating with minimal user IT infrastructure.

A recent News Feature in Nature India by Sibusiso Biyela, Amr Rageh and Shakoor Rather explores the rise and promise of generative AI in the global south, while highlighting the limited multilingual and multicultural capabilities of widely used LLMs. The authors point out that although vast amounts of English-language data are available on the internet, data in languages such as Hindi, Arabic or Xhosa remain scarce. As a result, testing LLM performances in other languages leads to unimpressive results, as the models not only struggle with grammar and vocabulary but miss cultural nuances. To address similar challenges in the context of African languages, researchers led by David Ifeoluwa Adelani developed AfroBench, a multi-task benchmark designed to evaluate the performance of LLMs in 64 African languages². The study identified major gaps in tasks such as reasoning, summarization and text classification for nearly all languages tested. In other work, researchers used a benchmark dataset of 670 languages spoken across five continents to assess ChatGPT’s language identification abilities, and found it had poor performance, specifically on African languages³.

Better data representation for native languages is urgently needed. In a digital-first world, with most content and AI tools limited to English-language culture and context, there is a real risk of cultural erasure, with younger generations growing up less aware of local history and culture, as Balaraman Ravindran writes in a Comment in Nature India about bridging the AI divide. Moreover, it is important for AI researchers and communities from the global south to take charge of the development of AI systems that can serve localized societal needs.

Several grassroots initiatives are working to shift the focus, challenging the concentration of AI power within technology companies. In a recent Comment in Nature Africa, Georgina Curto et al. discuss AI developments in Africa. Several of the authors are involved in the Deep Learning Indaba, a non-profit organization founded in 2017 to build a machine learning community in Africa and ensure that African researchers are involved in global AI conversations (Indaba means ‘gathering’ in Zulu). Another initiative, Masakhane, is a grassroots community focused on natural language processing (NLP) research in African languages and is the driving force behind the AfroBench initiative.

Curto et al. highlight the development of home-grown small language models (SLMs), typically having fewer than a billion parameters (in comparison, mainstream LLMs are hitting the trillion-parameter mark). These models are optimized for the context of tasks that can make a difference to individual communities. The authors point out that these smaller models can outperform larger ones in tasks such as mathematical problem solving and code generation. The development and adoption of SLMs promotes an entrepreneurial AI ecosystem focussed on applying LLMs to practical regional challenges in education, health and environment, benefitting local talent.

There is a real opportunity for the global south to take charge of the AI agenda, as they have the ideas, talent and data to develop AI solutions in local contexts, as the authors write in the News Feature and Comments mentioned above. Government input is needed, with accompanying ethical and governance frameworks, to ensure that countries develop an independent AI infrastructure. Instead of continuing the global AI race, researchers and developers in the global south can take charge of highly valuable curated datasets in the native languages they are creating and determine the directions of AI research that will benefit their own communities.