Evaluating the Impact of Learning Networks: Insights from the JLN

Author: Donnelly Mwachi, Amanda Folsom, Mahlet Gizaw and Rahul Kadarpeta

The JLN is a country-driven network of practitioners and policymakers from 40 countries across the globe who come together to problem solve, co-develop global knowledge products, and implement solutions that help bridge the gap between theory and practice. This collective wisdom of network members is harnessed to address complex health systems challenges ultimately accelerating progress towards Universal Health Coverage (UHC). The joint learning approach evolved over a period since 2010, when JLN was launched, drawing on several global best practices in action-oriented adult learning. It emphasizes a locally led approach, where country practitioners determine priorities, set the learning agenda, and co-develop effective strategies and promising practices. Knowledge exchange among countries is organized into learning exchanges (3-6 months) and collaboratives (18-24 months). Technical facilitators play a critical role, providing organizational capacity and analytical rigour to help countries frame issues and articulate their insights in a structured manner. The JLN encourages flexible thinking, enabling practitioners to synthesize new knowledge into knowledge products – including tools, assessments, policy analysis frameworks, decision-making tools, implementation guidance, and case studies – that serve the needs of the country participants who co-created them and become global public goods for the global health community.

Challenges in evaluating learning networks, such as JLN

Practitioner-to-practitioner learning, managed by knowledge exchange networks or platforms, is emerging as a key pathway for building in-country capacities and contextualizing global best practices for strengthening health systems and sustainable health reforms. However, the empirical evidence on influence of these initiatives is scarce because of the challenges faced in measuring their impact. Measuring the impact of such networks can be challenging due to several factors. First, these networks often consist of a diverse range of practitioners with varying levels of expertise, and organisational backgrounds. The fluid nature of membership, with participants joining or leaving, makes it difficult to track and measure consistent outcomes over time. In addition, the ‘intangible’ benefits of developing connections with peers through the network and maintaining them, even after participating at network related activities, are difficult to measure. Further, the impact of knowledge exchange in these networks may not follow a linear path and can be diffuse, with knowledge being applied in varied contexts and at different times. This makes it hard to attribute specific outcomes directly to the network’s activities or interventions. Evaluating their effectiveness requires a comprehensive understanding of their design, functionality, and focus on purpose, membership, and knowledge-sharing mechanisms. Other challenges include the complexity, indirect and distributed nature of networks, including the long time it takes to achieve systemic changes.

JLN’s approach to building its evidence base

JLN’s approach to learning and impact measurement is informed by various established frameworks. For instance, the ‘Theory of Change.’ WHO utilizes Theory of Change (ToC) frameworks to outline the pathways through which knowledge exchange networks are expected to drive improvements in health systems. This involves identifying and illustrating inputs, activities, outputs, outcomes and impacts to understand how and why changes occur. JLN employs an explicit Theory of Change to map out both internal and external pathways of change and causal assumptions. The JLN Theory of Change predicts that JLN outputs will lead to knowledge dissemination, increased advocacy, and knowledge application at the country level. Directly engaging in problem-solving and creating new knowledge with peers in other countries will build the capacity and motivation of local leaders who are well-positioned to implement reforms, leading to changes in policy and practice in their countries. This framework helps elucidate how JLN’s interventions contribute to achieving desired outcomes and, ultimately, to systemic health improvements related to accelerating Universal Health Coverage (UHC) in target countries.

Various approaches are employed to measure individual, organisational and systemic health changes resulting from JLN’s contributions. For instance, JLN utilizes a comprehensive monitoring and evaluation framework that includes both output and outcome indicators to track progress and changes at multiple levels, both internally and externally. Internally, the indicators assess the effectiveness and efficiency of JLN’s model and operations. Externally, the indicators focus on measuring changes at the individual, organizational, and systemic levels. This approach aligns with global best practices from other organisations. For instance, the World Health Organization (WHO) and the United Nations framework are used to track progress across various dimensions of health interventions.

JLN’s use of complexity-aware methods to explore “what works, in which circumstances, and for whom” is in line with best practices in impact evaluation employed by the Institute for Development Studies (IDS), Collaborative Impact (Participatory Impact Assessment and Learning Approach), among other research institutions, to understand the nuances of change processes and the factors influencing outcomes. JLN’s use of outcome harvesting to document the adaptation and implementation of its knowledge products by countries has led to the documentation of over 20 case studies since its inception.

JLN will continue to contribute its experiences in measuring its network’s impact while actively learning from the latest developments in the field.

Measurement of Collaborative Networks is evolving

Efforts to enhance the measurement of collaborative learning networks have gained traction, focusing on improving impact assessment. For example, Results for Development (R4D) and Collaborative Impact have developed and piloted a Collaborative Learning Measurement & Learning (M&L) Framework designed to facilitate open reflection and learning about effective practices and areas for improvement within Collaborative Learning Networks. The framework also aims to guide for generating robust and credible evidence on network performance and its contributions to impact.  In 2023, R4D and CI piloted and stress-tested the framework with two Collaborative Learning Networks, the Linked Immunization Action Network (Linked) and the Strategic Purchasing for Africa Resource Center (SPARC). Linked used the M&L framework to systematically measure whether and how the network was contributing to Gavi’s Middle-Income Country (MICS) strategy, resulting in an impact brief that highlights how the network is enabling, catalyzing, and accelerating country progress toward MICS immunization outcomes and objectives.

The JLN is currently validating its frameworks against these emerging innovations to ensure that its impact measurement approaches are aligned with global best practices. For instance, JLN is in the process of revamping its data analytic dashboard (Tableau) to track and analyze the network’s interaction, efficiency and outcomes in real-time. JLN is also engaging with external experts specialized in impact measurement and collaborative learning network facilitation from Results for Development (R4D) to provide fresh perspectives aimed at ensuring that the JLN framework is robust, comprehensive and in line with the latest global practices.

Monitoring and measuring Networks is an ongoing challenge. The JLN continues to learn and adapt from other similar Networks and supporters facing the same measurement challenges.  The network’s forward-looking strategy, grounded in a well-defined Theory of Change and comprehensive M&E frameworks, highlights its commitment to capturing nuanced outcomes at individual, organizational, and systemic levels. The network’s collaboration with external experts will continue to ensure that its evidence base remains robust and aligned with global best practices. As it continues to pioneer efforts in practitioner-to-practitioner learning, JLN will not only strengthen health systems globally but also contribute valuable insights into the evolving field of collaborative network impact measurement, driving greater accountability and effectiveness across health interventions.

About the Authors:

•  Donnelly Mwachi is the Monitoring, Evaluation, and Knowledge Management Manager at the Joint Learning Network for Universal Health Coverage (JLN), spearheading the network’s initiatives in these areas. He is the author of the JLN Learning and Impact Strategy developed in consultation with the World Bank JLN team and other JLN stakeholders and played a pivotal role in crafting this blog.
• Amanda Folsom is a Practice Lead for Collaborative Learning at Results for Development (R4D), serves as an invaluable advisor to JLN M&E initiatives. She contributed to the review of the first draft of this blog. 
• Mahlet Gizaw is a JLN Health Specialist with the World Bank for reviewing the blog and sharing her valuable inputs.
• Rahul Kadarpeta is the Executive Director of the JLN Network Manager, providing comprehensive oversight and actively engaged in reviewing and shaping this blog.