Effective eHealth and mHealth Evaluations in Low Resource Environments

Hamish Frasera, Shariq Khojab, Jan Talmonc, Janise Richardsd

a Division of Global Health Equity, Brigham and Womens Hospital and Harvard Medical School, Boston, MA, USA

b mHealth Alliance, United Nations Foundation, Washington, DC, USA

c Center for Research Innovation, Support and Policy, Maastricht University, Maastricht, The Netherlands

d Division of Global HIV/AIDS, Center for Global Health, Centers for Disease Control and Prevention, Atlanta, GA, USA


Note: This is a two-session tutorial



Evaluation of eHealth and mHealth (e/mHealth) projects is necessary to understand the impact of electronic health information systems on health outcomes. Currently there is a shortage of well-designed and implemented evaluations in the published literature. The aim of this tutorial is to provide participants with the knowledge and skill to critique evaluation designs, recognize potential barriers, and select suitable evaluation methodologies to ensure meaningful and generalizable results that will assist in building the needed evidence-base. Divided into two sessions, the tutorial first concentrates on describing the fundamental principles of evaluation. Within this session the participant will learn about the importance of evaluation, evaluation methods, and evaluation environments. During the second session, the tutorial focuses on the existing evidence base on e/mHealth in resource poor environments and how to select suitable evaluation designs, and plan and implement evaluations. Participants will work in small groups to critique and redesign evaluation studies and are encouraged to bring a current evaluation project for the discussion. The faculty have over 50 years of combined experience of informatics and evaluation.

eHealth and mHealth projects are growing rapidly in scope and scale worldwide. Over the last decade many healthcare organizations in low resource environments have started to implement electronic medical records, pharmacy and supply chain systems, and electronic information system based tools for disease surveillance and reporting healthcare processes and outcomes. In the last five years there has also been rapid growth in the use of mobile phones for such projects and to support community healthcare workers.

Calls for development of an evidence-base for effective and efficient eHealth and mHealth projects have come from ministries of health, system developers and funders.  In 2011 a WHO-sponsored meeting was held in Bellagio, Italy where health informatics (eHealth and mHealth) leaders discussed barriers to health system evaluation in low-resource environments and developed a “Call to Action on Global eHealth Evaluation”. A significant barrier discussed during this meeting was the need for trained evaluators and a key recommendation within the Call to Action was the need for increased numbers and improved quality of eHealth research studies. Additionally, IMIA’s STARE-HI initiative in multiple publications, has called for rigorous evaluation methodology for health information systems and publication of the results.

Despite this growth in electronic health information systems implementation and the call from leaders for a better evidence base for improving health outcomes, very few eHealth and mHealth projects have been effectively evaluated and the results published. Those studies, which have been published, are usually small scale, narrowly focused, and, frequently, not rigorously designed.


Outline of topics

This tutorial is designed into two sessions.  The first session will consist of didactic presentations that provide the foundational principles for effective eHealth and mHealth evaluation. The second session will provide participants opportunities to apply the foundational evaluation principles on example evaluation cases, or their own evaluation protocols.


Session One: Principles of e/mHealth Evaluation

This session focuses on the issues, principles and methodologies associated with effective e/mHealth evaluation. More specifically, the presentations and group discussions will address the following:

  • Importance and usefulness of evaluation

  • Barriers to evaluation

  • Types of evaluation most relevant to system, environment and research questions being asked (performance, clinical process and outcomes, economics, formative, summative, etc.)

  • Types of studies from small and local, to large, multi-site, RCTs etc.

  • Utilization focused evaluation and the importance of local ownership

  • Evaluation frameworks, with examples and uses

  • Qualitative methods, case histories and user experience


Session Two: Application of Effective e/mHealth Evaluation Principles

This session provides a hands-on analysis of specific evaluation problems and creation of effective study designs. The content focuses generally on the application of the principles and specifically on how to improve evaluations of e/mHealth projects and develop publishable reports of the findings. Breakout groups will cover the following:  formative/system optimization studies, summative impact studies, economic and cost benefit studies, qualitative evaluation, Utilization Focused Evaluation and evidence needs of policy makers. Participants in this session are encouraged to bring examples of studies that they are planning or implementing for discussion with the group and faculty. Participants who have attended Session one can apply the principles covered there to this session. Those with existing evaluation experience, particularly in study design in LMIC environments, may wish to attend session two only.


Learning objectives

By the completion of the tutorial the participant will be able to:

  • Describe evaluation principles related to e/mHealth systems

  • Summarize the importance of evaluation of e/mHealth systems to major stakeholders and for the development of the evidence-base for the impact of the systems on health outcomes in resource poor environments

  • Explain their own specific needs and barriers for the implementation of e/mHealth evaluation

  • Critique the application of different evaluation methodologies to a specific setting

  • Select effective evaluation designs for different systems and environments

  • Plan and implement an effective evaluation


Target audience

Practitioners implementing eHealth and mHealth projects, researchers studying informatics performance and impact, and policy makers planning or leading such projects, in low and middle income countries (LMIC).


Prerequisite knowledge

Global eHealth and mHealth is at the intersection of Healthcare, Development and Informatics, and evaluation of these projects requires additional skills in public health, study design and health systems research. Practitioners usually have a base in one field and additional knowledge in one or both of the others. Previous experience in evaluation of  information systems is not essential but an those with at least outline plan for such a study are likely to get the most out of this tutorial. Experience levels: novice and intermediate