Evaluating Health Information and Communications Technologies: Why, How, Challenges

Caitlin M. Cusacka,b, Brian E. Dixonc,d,e, Adam Wrightf

a Insight Informatics, Manchester, NH, USA

b Health Systems, Office of Health Informatics, Office of Informatics and Analytics,

Veterans Health Administration, Silver Spring, MD, USA

c School of Informatics and Computing, Indiana University-Purdue University Indianapolis, Indianapolis, IN, USA

d Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, USA

e Center for Implementing Evidence-Based Practice, Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service, Indianapolis, IN, USA

f Division of General Medicine and Primary Care, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA,USA



This tutorial will present a practical approach to planning and executing an evaluation of health information and communications technologies (ICT).Participants will learn about the spectrum of evaluation methods available, techniques for developing evaluation plans, available tools for planning and conducting evaluations, common challenges when evaluating health ICT, and methods for disseminating findings from health ICT evaluations. Emphasis will be placed on guiding attendees through the use of practical tools available for free on the Internet, which can be used after attendees return home. The session will be led by experienced health ICT evaluators who have worked on measuring the impact of health ICT across a wide spectrum of inpatient, ambulatory, public and population health settings. The session will largely be didactic teaching around a range of evaluation topics with the opportunity for attendees to share their experiences and ask advice on their own projects. Tutorial material will be drawn from real-world projects, with textbook material used to supplement and fill in gaps.


Outline of topics

  • motivations behind health informatics evaluations: why evaluation is important

  • choosing evaluation measures

  • study design and analysis

  • evaluation tools

  • A case study

  • common challenges and dissemination.


Target audience

The target audience includes individuals seeking guidance on how to evaluate clinical and public health ICT projects. Individuals may be in the earliest stages of planning an evaluation, or have already begun to evaluate a project and are experiencing challenges. Attendees may include but are not limited to junior faculty/researchers, project managers, ICT directors, chief health informatics officers (CMIO/CHIO/CNIO), epidemiologists, public health informatics professionals, or clinical leaders looking to measure the impact of health ICT on process, patient, or population outcomes.

Learning objectives

At the end of the tutorial, participants will have the knowledge and tools needed to develop a realistic, feasible evaluation plan for their health ICT project and will understand the challenges they may encounter. Attendees will more specifically be able to:

  1. Identify realistic evaluation measures for various health ICT projects;
  2. Distinguish practical metrics given a context;
  3. Design a feasible approach for measuring the impact of ICT on health care processes, quality, safety, or costs;
  4. Recognize challenges when planning or conducting a health ICT evaluation;
  5. Utilize available tools and resources for developing or conducting a health ICT evaluation; and
  6. Prepare a summary of a health ICT evaluation for dissemination to stakeholders or other health systems.


Prerequisite knowledge

Basic knowledge of biomedical informatics is assumed. Detailed descriptions of specific health ICT, reasons to implement health ICT, implementation challenges and barriers, and adoption challenges and barriers will not be covered. Knowledge of basic statistics, such as mean, standard deviation, and paired T-test, is preferable but not required. Familiarity with health care and public health settings and workflows is also assumed as the instructors will use examples drawn from real-world settings.