Kilkari pregnancy & infant care bot
Human-centred, conversational Q&A service to guide expectant and new mothers
A significant proportion of maternal and child deaths are preventable, with a critical gap being a lack of access to preventive care information, leading to delayed care seeking and worsening of complications and risk factors.
Along with nonprofit ARMMAN, we are building an LLM-based conversational agent that integrates into India’s Kilkari mHealth program which empowers 3M+ mothers with critical preventive care information. Our Q&A bot will provide on-demand, real-time, accurate, actionable and personalised information on pregnancy and infant care in an effort to improve health-seeking behaviour that could lead to reduced maternal and child mortality and morbidity.
Recognition
Understanding the Problem
A woman dies in childbirth every twenty minutes in India and for every woman who dies, 20 more suffer lifelong ailments. Two children under age five die every minute while four out of ten children don't realise their full potential due to chronic undernutrition/stunting.
Access to trustworthy information can improve women’s agency, self-efficacy and decision-making capacity regarding their own health. Kilkari is a free national mHealth service focused on preventive care (covering antenatal care, family planning, infant care & immunisation). It delivers weekly pre-recorded, stage-specific audio messages to 3M+ pregnant women and mothers with children under one year across 20 states via IVR, and is projected to reach 7M+ women by 2025-26. Since 2019, Kilkari is being implemented by ARMMAN in partnership with the Ministry of Health & Family Welfare.
Women in India still face barriers to accessing reliable information due to poor awareness and education, cultural norms, and lack of high-quality information sources. This is an opportunity to use LLMs to serve an underserved population of 7M+ women in rural and peri-urban India, including vulnerable and marginalised groups, and bridge key knowledge gaps.
Designing the Solution
In a Whatsapp pilot with 500 rural pregnant women, 73% engaged in some two-way communication. 24% initiated a conversation and 8% participants asked health queries. Queries covered stage-specific symptoms, diet and exercise guidance, and pregnancy myths / misconceptions. While ARMMAN program executives could respond directly to these questions in the pilot phase, such manual interventions will not be possible at scale. We are excited to explore LLMs to build a Q&A service within Kilkari which can answer queries about pregnancy and infant care with clinically-validated, stage-specific information that also reflects location-specific knowledge and practices.
When a woman sends a question, the LLM will provide an answer from the Kilkari knowledge corpus (including expert-vetted FAQs) that is personalised to the user’s pregnancy/infancy stage and location. It will refuse to respond if a question is outside the Kilkari scope. A human will respond where the model abstains or the user rates a response as unsatisfactory.
We plan to work with Kilkari subscribers in Delhi and Jharkhand (two Hindi-speaking states), of which approximately 20k-30k will be on WhatsApp, and eligible for a chat-based Q&A service. Our evaluation approach will focus on rapid iteration and product improvement as well as driving user impact.
Key Objectives
The main objectives of the project are as follows:
Develop an LLM that can generate accurate, relevant, concise and non-harmful responses to user queries about pregnancy and infant care
Embed the LLM into the Kilkari Hindi Whatsapp user flow for a small number of users in a pilot setting to validate usability and technical feasibility
Measure the impact of Q&A service on knowledge & attitudes through a deployment in select geographies
Implementation & Deployment
Our project has 5 broad phases:
1. Prepare Kilkari knowledge corpus for model training, create FAQ banks and evaluation question sets
2. Develop a proof of concept model that provides accurate & validated answers to questions about pregnancy and infant care in two modes: FAQ match and Generation of answers with citation
3. Expand scope of LLM to multilingual and multimodal queries
4. Pilot LLM-powered Q&A service with a subset of Kilkari users through Whatsapp
5. Deployment in select geographies
In the first three phases, the focus will be on model performance: as defined by model accuracy, relevance, quality scores, rates of model abstention on standard evaluation sets. There will be a mix of objectives evaluation (i.e., FAQ match) and expert evaluations / subjective evaluations (i.e., grading generated answers)
In the pilot phase, the Q&A bot will have in-built user satisfaction surveys following the response. Public health experts will validate the correctness and completeness of answers provided by the bot in the pilot phase. We will follow this up with in-depth interviews with 10-15 pilot participants to understand user experience and challenges.
In the deployment phase, we will use telephonic surveys to understand the impact of the Q&A service, with a focus on user satisfaction and impact on knowledge and attitudes.
We also plan to conduct user research to understand how LLM interfaces will be perceived, used, and interpreted by users. We will focus on user expectations, trust, and appropriate guardrails needed to prevent harm as well as factors that mediate comfort with LLM interfaces (i.e, user types and personas likely to be left out) and equity implications at scale. Through these qualitative studies, we will identify additional risks and challenges early before actual scaled deployment.
Path to Scale
The Kilkari program, which has been live since 2016, currently has 3.2 million subscribers on IVR across 20 states & UTs in India, which is expected to double to 6 million subscribers by 2024-25 as the service coverage expands to all states & UTs in India.
In 2-3 years, the Kilkari service will have 3-5 million subscribers on Whatsapp. After the validation phase, the LLM-powered Q&A service will be available to all Kilkari subscribers on Whatsapp. With advances in natural language interfaces, it may also be possible to integrate the service into IVR.
A major output of this study will be the creation of a set of validated strategies and design guidelines for a scaled LLM-powered Q&A service. We will address questions of usability, reliability and trust, safety and harm and equity. While the focus of this study will be pregnant women and mothers of infants in India, we expect that the strategies will be broadly applicable to applying LLMs in other digital health services in developing countries, especially in maternal messaging applications.