Evaluating mobile health initiatives in India

BBC Media Action is implementing two large scale mobile health (mHealth) initiatives supported by the Government of India: Kilkari, and Mobile Academy. Kilkari is an outbound service that delivers weekly, time-appropriate audio messages about pregnancy, childbirth, and childcare directly to families on their mobile phones, starting from the second trimester of pregnancy until the child is one-year-old. Mobile Academy is a mobile-based training course for the Accredited Social Health Activist (ASHA) community health workers. ASHAs access the course, offering four hours of audio content, by dialling a number from phone. We have evaluated impact of these two initiatives using an experimental framework.

We partnered with Johns Hopkins Bloomberg School of Public Health to provide inputs in the evaluation method, assignment of treatment and control, data collection and data analysis towards assessing the impact of the Kilkari and Mobile Academy programmes for the state of Madhya Pradesh in central India.

Challenges

During implementation of fieldwork, the listing exercise had to be expanded substantially to reach the desired number of sample, especially as the sample included pregnant women with 12-34 weeks of gestation, above the age of 18 years, could speak and understand Hindi and had access to a mobile phone during daytime with a private telecom service provider connection. We also increased listing team size and extended timeline.

Our approach

The evaluation design followed an individually randomised controlled trial (RCT) with parallel and unblinded assignment of treatment and control groups. Under this study, we first collected data from around 3800 villages (covering >4,00,000 households) from four districts of rural Madhya Pradesh. A total of 5200 women were selected for the study as part of the RCT evaluation. During baseline, we collected data from 5200 pregnant women and their spouses in 2018. The ASHAs in the sampled villages were also interviewed. As a follow up survey, the same 5200 women, their spouses and ASHAs were revisited (as part of panel survey) in 2019-2020. The listing survey, baseline and endline were conducted using CAPI, which our in-house experts developed. In addition to that, we also collected qualitative data from the target study population. Our researchers collaborated with the JHU and BBC Media Action teams for the analysis focusing on evaluation of Kilkari and Mobile Academy programmes.

Outcomes

Both qualitative and quantitative analytical research work, have been published in peer reviewed journals focusing on the research protocol, usage of machine learning and mobile messages during data collection for data quality assurance, impact of the Kilkari and Mobile Academy programmes and the underlying pathways of the impact.

Some important references:

Using machine learning to optimise the quality of survey data: Protocol for a use case in India

Development of a phone survey tool to measure respectful maternity care during pregnancy and childbirth in India: Study Protocol

Are stage-based health information messages effective and good value for money in improving maternal newborn and child health outcomes in India? Protocol for an individually randomised controlled trial

Can health information through mobile phones close the divide in health behaviours among the marginalised? An equity analysis of Kilkari in Madhya Pradesh, India

SMS feedback system as a quality assurance mechanism: experience from a household survey in rural India

Does exposure to health information through mobile phones increase immunisation knowledge, completeness and timeliness in rural India?

Examining the reach and exposure of a mobile phone-based training programme for frontline health workers (ASHAs) in 13 states across India

Another voice in the crowd: the challenge of changing family planning and child feeding practices through mHealth messaging in rural central India

Freedom within a cage: how patriarchal gender norms limit women’s use of mobile phones in rural central India

The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India