Review of Development Projects as Policy Experiments: An Adaptive Approach to Development Administration - Uphoff, Norman (Economic Geography, 1985)

Review of current system & portfolio practices in the development sector: Organizational… - Pereverza, Kateryna (Medium, 2024)
Building on interviews with colleagues in development organizations, donor & partner governments and foundations, this article explores the adoption of system and portfolio-based approaches in the development sector focusing on organizational motivations and challenges, changes in the network of the development sector actors, and types of collaborative experiments initiated around the novel approach. The aim is to take steps towards understanding how this type of approach for system change can be institutionalized in the sector and become one of the main ways of operating for delivering transformative impact. Summary While there is a broad consensus that linear, fragmented, projectized ways of doing development & public policy are not a match for the type of context we live in today, there is less consensus about what alternatives look like. In this space, system and portfolio approaches have emerged during the last few years and a new field is beginning to slowly emerge. Currently, there is a diversity in interpretations of system and portfolio-based approaches by the development sector organizations. This is connected to different logics and theories of transformative change in complex systems. Also, the intent with the adoption of the novel approach can vary along the dimension of organizational efficiency vs place-based transformations. Organizations engage with portfolio-based approaches with different intents and in different roles. For example, OECD OPSI takes forward the innovation portfolio management for partners; the European Commission funds place-based portfolios e.g. within the M4EG program; while UNDP facilitates place-based and thematic portfolios for system transformations. When it comes to the motivations of development sector partners to experiment with system and portfolio-based approaches, these include increased effectiveness & efficiency of the use of funds for place-based transformations; response to the post-colonial critique; creation of visibility as a frontrunner; the potential of joint risk-taking; the opportunity of collective learning from an entire portfolio (coherence & dynamic evolution); and directionality added to internal and external activities. Challenges encountered by partners engaged in the experimentation with system and portfolio-based approaches are connected to resources and time constraints that emerge from the prevailing logics & constraints of current organizational structures premised on fragmentation & projectization; organizational leadership dynamics; the influence of dominant organizational culture and mindsets; the difficulties of adopting the ‘portfolio language’/jargon; prevailing funding structures and cycles; and a shortage of both time and capacity to engage with numerous local partners. The structure of the development sector ecosystem is undergoing changes driven by a rapidly changing context. The experimentation with system and portfolio-based approaches has led to new actors joining the space, such as innovative consultancies developing the logic of transformative change in systems and facilitating portfolios on the ground, and the emergence of new roles and relationships, for example, ‘learning partners’. Since 2020, three noticeable types of collaborative experiments emerged in the development sector ecosystem: systemic investments; thematic and place-based portfolio interventions; and monitoring, evaluation and learning (MEL) for the context of system change. This study revealed areas for further exploration and analysis. One is a need to co-create institutional structures to enable a co-ownership of place-based portfolios by several international partners as well as local stakeholders. Another area connects to the need to create and implement spaces and tools for collective learning from portfolios. Finally, there is a need to understand what mechanisms can enable the institutionalization of novel approaches on the level of the entire sector and their interconnection with organizational changes needed. This study is based on the interviews with representatives of ten development sector organizations conducted in spring 2023 and insights shared within portfolio practitioner gatherings and the M&E Sandbox webinars run during 2023. Introduction: The article emphasizes the importance of (re)framing challenges in portfolio-based approaches for steering system transformations. It acknowledges that the way we frame challenges significantly influences the solutions we develop, and therefore, it’s crucial to address the right problems rather than wasting resources on incorrect issues. Study Insights: The insights come from a study initiated in autumn 2021, focusing on how framing processes unfold, what needs to be framed, the open-ended nature of framing, and the roles international organizations like UNDP can play in developing inclusive and system transformation-oriented framings of urban challenges. Key Points: The study involved interviews with experts from the UNDP Istanbul and Thailand Regional hubs. It explored the experimentation with the portfolio approach in collaboration with local and national governments and partners. The article presents insights related to the framing process and highlights open questions for researchers and practitioners. System Transformations: System transformations require a deep understanding beyond surface-level issues, as existing systems often limit our imagination and prevent us from exploring the root causes of complex societal challenges. Portfolio Approaches: The article references the collaboration between the Swedish Strategic Innovation Programme “Viable Cities” and the Strategic Innovation Unit of UNDP. They focus on portfolio-based approaches for urban low-carbon transitions in Sweden, targeting the mission: “Climate-neutral cities 2030 with a good life for all within planetary boundaries.” Conclusion: The blog post concludes by highlighting the need for further research and practice in developing effective portfolio approaches to address complex societal challenges.

Review Article: Albert O. Hirschman: Development Projects Observed - Singer, Dr H. W. (1969)

Revealing the Relational Mechanisms of Research for Development Through Social Network Analysis - Apgar, Marina; Fournie, Guillaume; Haesler, Barbara; Higdon, Grace Lyn; Kenny, Leah; Oppel, Annalena; Pauls, Evelyn; Smith, Matthew; Snijder, Mieke; Vink, Daan; Hossain, Mazeda (The European Journal of Development Research, 2023)
Achieving impact through research for development programmes (R4D) requires engagement with diverse stakeholders across the research, development and policy divides. Understanding how such programmes support the emergence of outcomes, therefore, requires a focus on the relational aspects of engagement and collaboration. Increasingly, evaluation of large research collaborations is employing social network analysis (SNA), making use of its relational view of causation. In this paper, we use three applications of SNA within similar large R4D programmes, through our work within evaluation of three Interidsiplinary Hubs of the Global Challenges Research Fund, to explore its potential as an evaluation method. Our comparative analysis shows that SNA can uncover the structural dimensions of interactions within R4D programmes and enable learning about how networks evolve through time. We reflect on common challenges across the cases including navigating different forms of bias that result from incomplete network data, multiple interpretations across scales, and the challenges of making causal inference and related ethical dilemmas. We conclude with lessons on the methodological and operational dimensions of using SNA within monitoring, evaluation and learning (MEL) systems that aim to support both learning and accountability.

Retrospective ‘Outcome Harvesting’: Generating robust insights about a global voluntary environmental network - Rassmann, Kornelia; Smith, Richard; Mauremootoo, John; Wilson-Grau, Ricardo (Better Evaluation, 2013)

Rethinking value for money for adaptive, politically smart programmes - Lessons from Institutions for Inclusive Development in Tanzania - Laws, Ed (ODI, 2020)
- This short paper draws out lessons for working effectively with and through partners, based on the experience of the Institutions for Inclusive Development (I4ID) programme – an adaptive, politically smart governance programme in Tanzania. • Cultivating effective partnerships can be a key part of delivering locally legitimate projects that have the potential to create sustainable change. Adaptive and politically informed ways of working create specific opportunities and challenges for doing this well. • Flexible and adaptive programmes are deliberately designed to experiment and to make small investments in different areas, to see what will work. While this is often important for making headway on complex challenges, it can also leave partners exposed and can undermine trust. • It can also be challenging to balance the need to meet accountability commitments to donors while allowing local partners to take the lead in pursuing their own objectives. • Co-creating plans, priorities and activities with partners has the potential to resolve some of these tensions. But the time and patience required to do this successfully should not be underestimated, and can be difficult to maintain in the face of pressure from donors to see results within a confined timeframe.

Rethinking systemic change in practice - perspectives from NUTEC-MD in northern Uganda - Koleros, Andrew (BEAM Exchange, 2016)
Document description This document is currently being finalised and will be published shortly. Please try again soon. Recent research by The BEAM Exchange seeks to understand theoretical perspectives on how market systems approaches can contribute to inclusive economic development through systemic change. It produced three main insights. Economies are evolving systems, building on the mechanisms of variety creation, selection and amplification. Current economic performance, including aspects like the inclusiveness of growth and economic evolution, are shaped by the ability of a society to explore different options for institutional arrangements and adjust them over time. This process of evolution is complex. While some aspects can be designed and managed, others need to be explored through a process of learning and adjustment. This paper responds to these insights by providing a practitioner perspective through the lens of one programme: “Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD)”, implemented by Palladium Group. It begins with a brief introduction to the programme and key aspects of its design phase. It then provides a summary of the three main research insights, with reflections on their applicability for programme design based on the NUTEC-MD experience. Finally, it provides some conclusions for the wider practitioner community based on this case.

Rethinking systemic change: economic evolution and institutions. Technical Paper - Cunningham, Shawn; Jenal, Marcus (BEAM Exchange, 2016)
The primary aim of this research project was to find a conceptually sound definition of systemic change. To do so, it was essential to gain a better understanding of how economies change. The central part of the research work, therefore, was an extended literature review on three bodies of knowledge: evolutionary economics, new institutional economics and complexity theory. There is a growing interest in these bodies of knowledge, combined often called New Economic Thinking, and how they affect economic development. Hence, while rethinking systemic change, this work also contributes to answering the broader question of how market systems approaches can contribute to inclusive economic development. The answer, in short, is to shift the focus away from improving transactions at the micro level towards enabling actors to continuously shape an institutional landscape that supports inclusive economic evolution. This technical paper provides an in-depth review of the fields of evolutionary economics, new institutional economics and complexity and social change. It is argued that economic development is a complex, non-linear and continuous evolutionary process. Both market and non-market institutions matter greatly in shaping economic performance. The paper then explores the consequences of this understanding for market development practice. It discusses how market development practitioners can engage in and shape an intentional change processes. To translate the theory into practice, seven principles are suggested that can be applied to market systems development practice. The paper concludes with a reframed look at systemic change in market systems development. A discussion paper presents the key messages from the literature review and seven principles drawing from this literature, and a case study offers a practitioner perspective through the lens of one market systems development programme: DFID-funded Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD).

Rethinking systemic change: economic evolution and institutions. Discussion paper - Cunningham, Shawn; Jenal, Marcus (BEAM Exchange, 2016)
The primary aim of this research project was to find a conceptually sound definition of systemic change. To do so, it was essential to gain a better understanding of how economies change. The central part of the research work, therefore, was an extended literature review on three bodies of knowledge: evolutionary economics new institutional economics complexity theory There is a growing interest in these bodies of knowledge, combined often called New Economic Thinking, and how they affect economic development. Hence, while rethinking systemic change, this work also contributes to answering the broader question of how market systems approaches can contribute to inclusive economic development. The answer, in short, is to shift the focus away from improving transactions at the micro level towards enabling actors to continuously shape an institutional landscape that supports inclusive economic evolution. This discussion paper briefly presents the key messages from the literature review and seven principles drawing from this literature. The principles can be used by market development practitioners, including technical advisers in donor organisations, programme designers and team leaders, to shape programmes and become more in line with how change happens in the economy. A list of selected references is presented at the end. A technical paper contains a much more detailed discussion of the findings and the principles and an extensive list of references. A case study offers a practitioner perspective through the lens of one market systems development programme: DFID-funded Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD).

Rethinking systemic change: economic evolution and institutions - Cunningham, Shawn; Jenal, Marcus (BEAM Exchange, 2016)
Resources that contain evidence of market systems interventions. Recently updated.

Rethinking Rigor - Increasing Credibility and Use - Lynn, Jewlya; Preskill, Hallie (Spark Policy Institute & FSG, 2016)
Evaluations of complex systems change strategies or adaptive, innovative programs cannot use the traditional “rigor” yardstick to measure quality. This paper proposes a new definition of rigor, one that applies in any setting, but particularly in complex, adaptive ones. Top Takeaways: 4 criteria for the new definition of rigor: - Quality of thinking - Credible and legitimate claims - Cultural context and responsiveness - Quality and value of the learning process Regardless of its other positive attributes, an evaluation of a complex, adaptive program that fails to take into account systems thinking will not be responsive to the needs of that program.

Rethinking Participation and Re-enacting Its Dilemmas? Aarhus 2017 and “The Playful Society” - Eriksson, Birgit; Stephensen, Jan Løhmann (Conjunctions. Transdisciplinary Journal of Cultural Participation, 2016)
In 2012 the Danish city of Aarhus was appointed European Capital of Culture for 2017. The appointment was based on an ambitious programme that – under the headline Rethink – tried to set an agenda of societal transformation, mainly by seeking to increase the impact of art and culture, and to enhance civic participation at all levels of society. In this article we examine one of the first attempts of Aarhus 2017 to realize these grand ambitions: ‘The Playful Society’, a series of micro grants aimed at enabling young people to make their own art/culture projects and participate in the overall Rethink project. Informed by theoretical distinctions between different forms of participation, and the diverse interests invested in participatory processes, we investigate how the young cultural entrepreneurs and the artistic administrators of Aarhus 2017 separately, in conjunction, and sometimes even in opposition to each other, translated these overall ambitions into practice. We argue that they illuminate some of the dilemmas of contemporary cultural participation, including the importance of question- ing who participates in what, how they do it, and in what context.
Keywords: Cultural Entrepreneurship; Participation; youth

Rethinking monitoring and evaluation in complex systems — when learning is a result in itself - Innovation, UNDP (Medium, 2022)
By Søren Vester Haldrup, UNDP’s Strategic Innovation Unit

Responding to complexity: A Case Study on the Use of “Developmental Evaluation for Managing Adaptively” - Boisvert, Kayla (University of Massachusetts, 2017)

Responding to change: Learning to adapt in development cooperation - Engel, Paul; Keijzer, Niels; Ørnemark, Charlotte (Policy Management Brief 19, 2007)

Resilience Thinking: Sustaining Ecosystems and People in a Changing World - Walker, Brian; Salt, David; Reid, Walter V. (Island Press, 2006)
Increasingly, cracks are appearing in the capacity of communities, ecosystems, and landscapes to provide the goods and services that sustain our planet's well-being. The response from most quarters has been for "more of the same" that created the situation in the first place: more control, more intensification, and greater efficiency. "Resilience thinking" offers a different way of understanding the world and a new approach to managing resources. It embraces human and natural systems as complex entities continually adapting through cycles of change and seeks to understand the qualities of a system that must be maintained or enhanced in order to achieve sustainability. It explains why greater efficiency by itself cannot solve resource problems and offers a constructive alternative that opens up options rather than closing them down. In "Resilience Thinking", scientist Brian Walker and science writer David Salt present an accessible introduction to the emerging paradigm of resilience. The book arose out of appeals from colleagues in science and industry for a plainly written account of what resilience is all about and how a resilience approach differs from current practices. Rather than complicated theory, the book offers a conceptual overview along with five case studies of resilience thinking in the real world. It is an engaging and important work for anyone interested in managing risk in a complex world.

Requisite Variety and Its Implications for the Control of Complex Systems - Ashby, W. Ross (Cybernetica, 1958)

Representing theories of change: technical challenges with evaluation consequences - Davies, Rick (Journal of Development Effectiveness, 2018)
This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation. The paper then outlines a range of different ways of addressing these problems that could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.
Keywords: Evaluation; evaluability; representation; theory of change

Report templates for evaluability assessment - Methods Lab (2015)
An evaluability assessment aims to assess the extent to which, and how best, a project can be evaluated in a reliable and credible fashion. These templates are intended to help anyone conducting an evaluability assessment to structure the final report. This tool was developed by Anne Buffardi and Bronwen McDonald for use in Methods Lab projects. It accompanies The Methods Lab publication ‘Evaluability assessment for impact evaluation: guidance, checklists and decision support’.

Report template on integrating impact into an existing monitoring and evaluation system - Methods Lab (2015)
Many development programme staff will commission an impact evaluation towards the end of a project or programme, only to find that the monitoring system did not provide adequate data about implementation, context, baselines or interim results. This tool provides a template outline for a report making recommendations on how to integrate a focus on impact into a programme’s existing monitoring and evaluation system, as the programme moves into a new phase. This template was developed by Anne Buffardi and Tiina Pasanen for use in Methods Lab projects.

Report on outcomes and get everyone involved: The Participatory Performance Story Reporting Technique - Dart, J. (Presented at the AES conference in Perth, September 2008, 2008)
This paper outlines the background and philosophy of Collaborative Outcomes Reporting (COR) and Performance Story Reporting (PSR), providing an overview of these emerging techniques and showing how they can be used as a framework for reporting on contribution to long-term outcomes (or targets) using mixed methods and participatory process. The report serves as an introduction to these approaches in evaluation and discusses their bias, limitations and where they might best be applied.

Release the pressure on governance practitioners - de Weijer, Frauke; Hauck, Volker (A Governance Practitioner’s Notebook: Alternative Ideas and Approaches, 2015)
The Governance Practitioner’s Notebook takes an unusual approach for the OECD-DAC Network on Governance (GovNet). It brings together a collection of specially written notes aimed at those who work as governance practitioners within development agencies. It does so, however, without attempting to offer definitive guidance – instead aiming to stimulate thinking and debate. To aid this process the book is centred on a fictional Governance Adviser. The Notebook’s format provides space for experts to speak on today’s governance issues: politics, public sector reform and stakeholder engagement. It encourages debate, charts the evolution of donor thinking, and highlights future challenges in the age of the Sustainable Development Goals. Each section introduces both technical issues and major areas of debate, providing ideas for future development support to institutional reform.
Keywords: IMPORTANT

Regional Healthcare Ecosystem Analyst for Healthcare Associated Infections - Lee, Bruce Y. (Grantome, 2018)
Understanding and preventing the spread of both endemic and emerging healthcare-associated infectious diseases throughout hospitals and nursing homes is a national priority. Our work has shown that the many disparate inpatient healthcare facilities in a region can form a complex healthcare ecosystem connected by both direct and indirect patient sharing allowing pathogens in one health care facility to readily spread to other facilities. Our goal is to further develop RHEA (Regional Healthcare Ecosystem Analyst) into a user-friendly software tool that decision makers (e.g., policy makers, funders, product developers and manufacturers, healthcare administrators, infection prevention specialists, researchers, and educators) can readily use to help healthcare ecosystems prevent and control the spread of an endemic pathogen, methicillin- resistant Staphylococcus aureus (MRSA), and an emerging pathogen, carbapenem-resistant Enterobacteriaceae (CRE). This next generation of RHEA will bring multiple innovations by: 1) further elucidating the interconnectedness of the ecosystem and showcasing the value of cooperation among facilities versus the current individual facility approach to infectious disease control, 2) incorporating new healthcare ecosystem responses for endemic MRSA and emerging CRE, 3) continuing our work integrating economic and operational models into a framework of infectious disease epidemiological models, 4) imbuing virtual patients with characteristics linked to MRSA and CRE transmission and outcome risk, 5) building a comprehensive """"""""virtual laboratory"""""""" to help address many existing and future healthcare ecosystem infectious disease questions, and 6) building a user-friendly modeling tool that decision makers can use. The project will continue our team's longstanding modeling work and developing computational tools for decision makers via three specific aims. First, completing Specific Aim 1 will expand RHEA by adding the following capabilities: i) integrated clinical outcome and economic models, ii) more extensive and detailed patient characteristics linked to changes in infection risk and costs, iii) expanded HAI control measures, and iv) stochastic and adaptive disease parameters to represent emerging and evolving diseases. Next, Specific Aim 2 will use our newly expanded RHEA framework to model a sample endemic pathogen, MRSA, and a sample emerging pathogen, CRE to identify optimal control strategies even when pathogen characteristics are evolving. Finally, Specific Aim 3 will involve developing a user-friendly interface for RHEA and deploying it as a healthcare ecosystem computational modeling tool that various stakeholders (e.g., policy makers, healthcare administrators, infection control specialists, funders, and product manufacturers) can readily use to make decisions regarding the control of healthcare-associated infections. Public Health Relevance Understanding and preventing the spread of healthcare-associated infectious diseases throughout hospitals and nursing homes is a national priority. Simulation models can serve as virtual laboratories to help identify best practice solutions for containing common [such as methicillin-resistant Staphylococcus aureus (MRSA)] and emerging [e.g., carbapenem-resistant Enterobacteriaceae (CRE)] causes of healthcare associated infections (HAIs). Our goal is to develop our software RHEA (Regional Healthcare Ecosystem Analyst) into a computational modeling tool that can be used directly by decision makers to identify, develop, and evaluate strategies and interventions to reduce HAIs across a large geographic region.

Reframing, refining, and reconceptualising the worst forms of child labour through participatory adaptive programming - Apgar, Marina; Burns, Danny (Journal of modern slavery, 2021)
This article explores the potential of using participatory action research as an adaptive programming modality to drive learning and innovation to tackle the drivers of (and seek to eliminate) the Worst Forms of Child Labour. We draw on our experience from early phases of implementation of a large-scale action research programme, which despite the constraints covid-19 posed in moving to full implementation and participatory engagement with children and other stakeholders on the ground, is already generating rich learning about the opportunities and challenges of designing programmes that respond to the complex reality of WFCL. We share early learning about what it takes to be fully open to using the lived experience of programme development, and early findings from scoping and mapping of the dynamics of social norms, business practices and urban neighbourhoods and supply chains influencing WFCL in Bangladesh and Nepal, to frame and reframe the questions and response strategies and operationalise a participatory adaptive intent to work with hidden and complex dynamics that characterise the WFCL.

Reframing Complexity: A Four-Dimensional Approach to Organizational Diagnosis, Development and Change - Gallos, Joan V. (Organization Development: A Jossey-Bass Reader, 2006)
This is the third book in the Jossey–Bass Reader series, Organization Development: A Jossey–Bass Reader. This collection will introduce the key thinkers and contributors in organization development including Ed Lawler, Peter Senge, Chris Argyris, Richard Hackman, Jay Galbraith, Cooperrider, Rosabeth Moss Kanter, Bolman & Deal, Kouzes & Posner, and Ed Schein, among others. "Without reservations I recommend this volume to those students of organizational behavior who want an encyclopedia of OD to gain a perspective on the past, present, and future...." Jonathan D. Springer of the American Psychological Association.

Reforming solid waste management in Phnom Penh - Denney, Lisa (Working Politically in Practice, 2016)
This paper tracks the efforts of an Asia Foundation team and local stakeholders as they worked to support improvements in the solid waste management sector in Phnom Penh, Cambodia. The team worked in a flexible way with a range of partners, and with particular focus on understanding the incentives and politics affecting service delivery. While reform of the sector remains in progress, steps have been taken to introduce more competition and better public sector management of solid waste collection in the city. This case study lays out the real-time decisions and processes which drove the strategy and implementation of this project, providing useful insights into how politically astute and flexible programs can be successfully implemented. This case has emerged from an action research process, which was led by a researcher from the Overseas Development Institute (ODI) and conducted over the course of almost two years. By capturing and analyzing the experiences of the program team in Phnom Penh, the paper intends to provide practical insights for others in the development community aiming to implement similar kinds of programming.

Reforming donors in fragile states: using public management theory more strategically - Gulrajani, Nilima; Honig, Dan (Overseas Development Institute, 2016)
This paper identifies ways in which donors can be more effective in fragile and conflict-affected states by exploiting theories and concepts drawn from public management.

Reflections on the Utilization-Focused Evaluation (UFE) Process - Lim, Yvonne; Mizumoto, Ann (Strengthening ICTD Research Capacity in Asia (SIRCA) Programme, 2010)
This presentation from the Strengthening ICTD Research Capacity in Asia (SIRCA) provides an overview of how UFE was used in their SIRCA programme. It was presented at the Evaluation Conclave 2010, New Delhi, India The key objectives of the program are to: Enhance research capacity in Asia through rigorous academic research Create a space for dialogue on ICT4D social science research issues in Asia Create linkages through a mentorship program Disseminate findings in publications and conferences Contents SIRCA Programme SIRCA Key Objectives SIRCA Evaluation UFE Learnings UFE Challenges Evaluation is over…but there’s a lasting outcome...

Reflections on Ten Years of USAID’s Experience with Political Economy Analysis and Thinking and Working Politically - Brinkerhoff, Derick; Cassidy, Marc (TWP CoP, 2023)
The global Thinking and Working Politically (TWP) Community of Practice (CoP), the Washington DC TWP CoP, and USAID organised a webinar on 5 December 2022 to take stock of how USAID and its partners have used PEA to inform programme strategy, design and implementation, and support TWP. This paper synthesises the key points arising from the webinar, including observations on the impacts, opportunities, challenges, and prospects for PEA/TWP to become more deeply adopted and sustained as a development methodology and approach across sectors. It starts by defining key concepts. It then highlights insights from the discussions of the impact of the application of PEA and TWP principles across sectors. The paper concludes by looking at progress achieved to date, as well as constraints and opportunities to increase the uptake of both thinking and working politically in USAID-sponsored programming.

Reflections on implementing politically informed, searching programs: lessons for aid practitioners and policy makers - Cole, William; Ladner, Debra; Koenig, Mark; Tyrrel, Lavinia (Working Politically in Practice, 2016)

Reflections on if and how our partnership is working in Bangladesh - Paul, Sukanta; Snijder, Mieke; Apgar, Marina (CLARISSA, 2020)
Effective partnership working is crucial to producing quality results, especially when working with complex problems such as the worst forms...

Reflections on applying iterative and incremental software development methodologies to aid and development work in developing countries - Haikin, Matt (MattHaikin.com, 2013)

Reflections from a Realist Evaluation in Progress: Scaling Ladders and Stitching Theory - Punton, Melanie; Vogel, Isabel; Lloyd, Rob (CDI Practice Paper 18, 2016)
Realist evaluation provides valuable insights into how and why programmes lead to change, and can generate transferable lessons to help practitioners roll out or scale up an intervention. However, as ...

Reflection methods: Tools to make learning more meaningful - Gordijn, Femke; Eernstman, Natalia; Helder, Jan; Brouwer, Herman (Wageningen Centre for Development Innovation, 2018)
This handbook summarises methods that can be used to facilitate the process of reflection on the knowledge and experiences people acquire during a capacity development trajectory or training event. We believe that by explicitly integrating reflection in the learning process the learning will become clearer and better articulated and will contribute more strongly to meaningful change. Therefore we advise facilitators to deliberately include reflective learning sessions in their process design and implementation. This handbook can inspire you to do so and provides many methods which help to facilitate this.

Reflect Action Methods - ActionAid (2009)
Reflect is an innovative approach to adult learning and social change, which fuses the theories of Brazilian educator Paulo Freire with participatory methodologies. It was developed in the 1990s through pilot projects in Bangladesh, Uganda and El Salvador and is now used by over 500 organisations in over 70 countries worldwide. Organisations working with Reflect won UNESCO literacy prizes in 2003, 2005, 2007, 2008 and 2010. This site is aimed at Reflect practitioners and others who want to know more about the approach. We want it to be truly interactive and the more stories, graphics and photos are submitted, the better the site will be. Please add your comments to tell us about the work you are doing with Reflect and join the discussion forum to share your ideas and experiences of working with the approach.

Refining advocacy assessment: reflections from practice - Buffardi, Anne; Hearn, Simon; Tilley, Helen (Working Paper, 2017)
• This paper revisits how existing advocacy evaluation frameworks classify people and activities, and define and assess outcomes. We discuss how assessments could be more specific and propose bounding the scope of inquiry in one of four ways: strategy specific, outcome oriented, actor-centric or system-wide. • In classifying activities, the same action or event may be situated at different phases of the change pathway – in some cases used as a tactic to influence a policy outcome, and in others an intended outcome itself. • Because advocacy is more relational than other types of more technical development interventions, there will be fewer sources of directly observable data, and the direction of potential bias may be unknown. • In terms of learning, advocacy initiatives are contextually dependent, therefore lessons may be less directly transferable to subsequent phases of an initiative or to other settings. Organisations have bounded repertoires and the transferability of skill sets is limited, so advocates adapt how and with whom they engage more than what they do

Rediscovery of Traditional Ecological Knowledge as Adaptive Management - Berkes, Fikret; Colding, Johan; Folke, Carl (Ecological Applications, 2000)
Indigenous groups offer alternative knowledge and perspectives based on their own locally developed practices of resource use. We surveyed the international literature to focus on the role of Traditional Ecological Knowledge in monitoring, responding to, and managing ecosystem processes and functions, with special attention to ecological resilience. Case studies revealed that there exists a diversity of local or traditional practices for ecosystem management. These include multiple species management, resource rotation, succession management, landscape patchiness management, and other ways of responding to and managing pulses and ecological surprises. Social mechanisms behind these traditional practices include a number of adaptations for the generation, accumulation, and transmission of knowledge; the use of local institutions to provide leaders/stewards and rules for social regulation; mechanisms for cultural internalization of traditional practices; and the development of appropriate world views and cultural values. Some traditional knowledge and management systems were characterized by the use of local ecological knowledge to interpret and respond to feedbacks from the environment to guide the direction of resource management. These traditional systems had certain similarities to adaptive management with its emphasis on feedback learning, and its treatment of uncertainty and unpredictability intrinsic to all ecosystems.
Keywords: Adaptive Management; Resource management; Social learning; Traditional Ecological Knowledge; human ecology; resilience

Redesigning Field Papers’ User Interface - Jacks, Lindsey (Cadasta, 2016)
For the past three months, I've been working with Cadasta and Stamen on the atlas creation pages of the Field Papers map annotation tool. The end result is now live on the Field Papers website. We took the four step “wizard” process for creating an atlas, combined it all into a single-page, and...
Keywords: IMPORTANT

Redefining rigour: using stories to evaluate systems change? - Lowther, Keira (Centre For Public Impact (CPI), 2022)
What might a different way of understanding rigour for work in complex adaptive systems look like?

Redefining Rigor: Describing Quality Evaluation in Complex, Adaptive Settings - Preskill, Hallie; Lynn, Jewlya (FSG, 2016)
Traditionally, evaluation has focused on understanding whether a program is making progress against pre-determined indicators. In this context, the quality of the evaluation is often measured in part by the “rigor” of the methods and scientific inquiry.

Reciprocity in International Volunteer Cooperation - Lough, Benjamin J. (Fredskorpset Norway, 2016)

Recipes for Public Spheres: Eight Institutional Design Choices and Their Consequences - Fung, Archon (Journal of Political Philosophy, 2003)

Realist Impact Evaluation: An introduction - Westhorp, Gill (Methods Lab, 2014)
• Realist evaluation is a member of a family of theory-based evaluation approaches which begin by clarifying the ‘programme theory’: the mechanisms that are likely to operate, the contexts in which they might operate and the outcomes that will be observed if they operate as expected. • Realist approaches assume that nothing works everywhere for everyone: context makes a big difference to programme outcomes. A realist evaluation asks not ‘what works?’ but ‘how or why does this work, for whom, in what circumstances?’ • Realist impact evaluation is most appropriate for evaluating new initiatives or programmes that seem to work but where ‘how and for whom’ is not yet understood; programmes that have previously demonstrated mixed patterns of outcomes; and those that will be scaled up, to understand how to adapt the intervention to new contexts.

Realist Evaluation - Better Evaluation (Better Evaluation)
Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings. Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?. The complete realist question is: “What works, for whom, in what respects, to what extent, in what contexts, and how?”. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.

Realist Evaluation - Pawson, Ray; Tilley, Nick (2004)
Contents: Introduction 1. The nature of programmes and how they work 2. Basic concepts in the explanation and understanding of programmes 3. Strategies and methods of realist evaluation 4. Realism’s place in the policy cycle: formative, summative and synthetic approaches 5. The nature, presentation and use of findings from realist evaluation 6. Conclusion: strengths, limitations and relationships with other approaches Appendix I – ‘Thinking it through’: an exercise in realist hypothesis making. Appendix II – ‘Varieties of realist evaluation’: pocket illustrations of quantitative, qualitative, formative and synthetic applications. Appendix III – ‘Would it work here?’: a grid to help decide on the feasibility of mounting a programme ‘on your patch’.

Real World SAFe – Leapfrogging a successful waterfall company into Scaled Agile - Skarin, Mattias (CRISP, 2017)

Real Time Monitoring and the New Information Technologies - Lucas, Henry; Batchelor, Simon; Berdou, Evangelia (IDS Bulletin, 2013)
Debates as to the potential role of new Information and Communication Technologies (ICTs) in monitoring the wellbeing of vulnerable groups is often bedevilled by the failure of two principal actors – social researchers and technical experts – to address the other’s concerns or even to use language that is comprehensible to the other side. The aim here is to unpick some of the technical language relevant in this context and provide a brief introductory guide to some aspects of the current, rapidly changing and highly diverse ICT environment.

Real-Time Data for Adaptive Management at USAID - Doshi, Samir (Health Communication Innovation Webinar, 2016)

“Real”​ process tracing: part 1 — context - Aston, Thomas (Medium, 2020)
When asserting the value of theory-based methods, you often here words like “black boxes” and “causal mechanisms.” These are commonly…

Reading the Local Context: A Causal Chain Approach to Social Accountability - Joshi, Anuradha (IDS Bulletin, 2014)
There is a general consensus that ‘context’ matters for development outcomes, yet we have little understanding of how exactly ‘context’ affects outcomes. This article focuses on the question of ‘context’ in social accountability (SA) initiatives by separating macro- and micro-contextual factors. On the macro side (country level), accountability processes need to take into account broad factors such as national histories of citizen–state engagement. On the micro side, local factors can drive the extent to which SA initiatives are successful, even within otherwise broadly similar national contexts. The article outlines the basic components of accountability and proposes a ‘causal chain'strategy to better understand the micro-context. This would allow existing evidence to be reorganised to assess the promise of existing and new initiatives by deconstructing the various mini-causal pathways (i.e. in the micro-context) and understanding the contextual conditions that make them work.
Keywords: IMPORTANT

READ ME - HOW TO USE THIS AM LIBRARY - Prieto Martin, Pedro (Institute of Development Studies, 2019)

Reacting or adapting? Purposeful adaptation and response to contextual change. - Wood, Nik (Rebel with Causation, 2020)
Adaptive management is increasingly on the ticket for development programming, and has been crucial in the wake of covid-19. I’ve been working with adaptive programmes for most of my career, …

Re-Perceiving Results: Aid Programs and Change in Fragile Societies - van Veen, Erwin; Rijper, Alies (Policy Brief, 2017)
Aid programmes need to be able to adapt their objectives and operations to changes in their political environment, since development processes are subject to political contestation. Change takes time and is often a matter of seizing the opportunity.
Keywords: IMPORTANT

(Re)making the case for adaptive management - Aston, Thomas (Medium, 2022)
Great overview of what to read on adaptive management. It’s a long one, so I’ve split it into two – second installment tomorrow. Christian Aid Ireland’s recent publication The Difference Learning Makes by Stephen Gray and Andy Carl made a bit of a splash. The study found that Christian Aid Ireland’s application of adaptive programming contributed to better development outcomes and supported more flexible delivery. The much vaunted MUVA programme in Mozambique is also coming to a close and presenting its results from using and adaptive approach. So, it struck me that we might be at a critical juncture in the conversation on adaptive management. We’ve had the crashing to earth of inflated expectations in recent misanthropic reflections, [misanthropic, moi? – Duncan] alongside a fragile institutionalisation of adaptive management in donor agencies, NGOs, and private sector organisations. However, I’d argue that we’ve reached the point where adaptive management has passed the proof-of-concept stage.

Re-institute website - RE!NSTITUTE (Re!nstitute, 2023)
We are a dynamic non-profit organization with a unique approach to managing change and catalyzing innovation to support global systems transformation. We know that to achieve equal, just, and safe communities, we must collectively challenge and change the systems that people rely on — especially those most vulnerable. Together, we must RE!BUILD a system's resilience and capacity to take on our biggest generational challenges. Justice, housing, health, and other systems in every country are often struggling to come together, across institutions and organizations, to unite under a common vision for change, and implement those changes in effective and innovative ways. Our work and methodology — the 100-Day Challenge — are designed to support these systems to achieve extraordinary results. Our approach is rooted in the philosophy that frontline staff, supported by leadership, can unleash innovation and achieve incredible results by creating new relationships, experimenting with fresh ideas from across their system, and building authentic engagement with those with lived experience.

Re-imagining Inclusive Urban Futures for Transformation - Friend, Richard M; Anwar, Nausheen H; Dixit, Ajaya; Hutanuwatr, Khanin; Jayaraman, Thiagarajan; McGregor, J Allister; Menon, Meena R; Moench, Marcus; Pelling, Mark; Roberts, Debra (Current Opinion in Environmental Sustainability, 2016)

Rapid Results!: How 100-Day Projects Build the Capacity for Large-Scale Change - Schaffer, Robert H.; Ashkenas, Ron (Jossey-Bass, 2005)
Rapid Results! shows how to make large-scale changes succeed  by using 100-day results-producing projects to develop this vital implementation capability. Written by Robert H. Schaffer, Ronald N. Ashkenas, and their associates—leaders in the field of change management—Rapid Results! describes an approach that has been field-tested by real organizations of every size and description to improve performance and speed the pace of change. Rapid results projects produce results quickly, introduce new work patterns, and enable participants to learn a variety of lessons about managing change. Step by step, the book describes how the use of rapid-cycle, or 100-day, projects   will multiply your organization’s power to succeed at large-scale change. Schaffer and Ashkenas specifically outline the concept behind 100-day projects and show you how to Set up the architecture to implement rapid results projects Improve operational performance and also attain hard results in the soft areas of management Build rapid results into major organizational change such as reorganization, acquisition integration, and international development Use rapid results to drive leadership development and culture change

Rapid outcome mapping approach: a guide to policy engagement and influence - Young, John; Shaxson, Louise; Jones, harry; Hearn, Simon; Datta, Ajoy; Cassidy, Caroline (ODI, 2014)

Rapid experiments with a purpose - Moses, Michael (BEAM Exchange, 2017)
New insights, opinions and perspectives on market systems development, from experts and practitioners.
Keywords: IMPORTANT

Rapid Action Learning for Sanitation and Hygiene Programming - Chambers, Robert; Myers, Jamie; Vernon, Naomi (Frontiers of Sanitation, 2020)
There is a glaring gap and compelling need for approaches and methods that realign to new rigour through timeliness, cost-effectiveness, relevance and being actionable. Over the past few years, the Sanitation Learning Hub, in collaboration with the Government of India, Praxis, WSSCC and WaterAid India, have been developing Rapid Action Learning approaches. Multiple approaches have been trialled, with flexible formats, but the essential criteria is that learning is timely, relevant and actionable. These learning approaches are the focus of the latest edition of the Frontiers of Sanitation series. This Frontiers explains the advantages and disadvantages of the approaches trialled and sets out a challenge to those working in the water, sanitation and hygiene sector to: Reflect on what, for you, constitutes rigour. Adopt and adapt approaches to fit your context and needs. Develop your own approaches. Record your experiences and lessons learnt. Take the time to share your experiences with us

Rapid Action Learning and COVID-19 - Chambers, Robert (Sanitation Learning Hub, 2020)
A-ha! A moment etched in my memory: 20 or so researchers were...

Randomised Controlled Trial - Better Evaluation (Better Evaluation)
Randomised controlled trials (RCTs), or randomised impact evaluations, are a type of impact evaluation which uses randomised access to social programmes as a means of limiting bias and generating an internally valid impact estimate.

RAND Program Evaluation Toolkit for Countering Violent Extremism - Helmus, Todd; Matthews, Miriam; Ramchand, Rajeev; Beaghley, Sina; Stebbins, David; Kadlec, Amanda; Brown, Michael; Kofner, Aaron; Acosta, Joie (RAND Corporation, 2017)

QUIP: Understanding clients through in-depth interviews - IMP-ACT (IMP-ACT PRACTICE NOTES, 2004)
This Practice Note by Imp-Act gives a step-by-step guide to developing and conducting in-depth interviews using the QUIP approach, and analysing the information and making conclusions based on what you have learned.

Quality standards for development evaluation - OECD; Development Assistance Committee (OECD, 2010)
Based on a broad international consultative process, the DAC Quality Standards for Development Evaluation are a reference guide to good practice in development evaluation. With a view to improving the quality of evaluation processes and products, and facilitating collaboration, this reference guide lays out standards for each phase of a typical evaluation process: from defining purpose, to planning, designing, implementing, reporting, and learning from and using evaluation results.

Quality Flaws: Issues and Challenges in Software Development - Mohan, P.; Shankar, A. Udaya; JayaSriDevi, K. (Computer Engineering and Intelligent Systems, 2012)
A statement “Prevention is better than cure” for illnesses in medical sciences also applies to the software development life cycle in terms of software defects. A defect is a deviation from actual functionality of the application in terms of the correctness and completeness of the specification of the customer requirements. Defective software fails to meet its customer requirements leading to the development of applications with poor quality. Quality is a top priority in every enterprise these days. Organizations struggle in a treadmill race to deliver quality software to stay ahead with new technology, deal with accumulated development backlogs, handle customer issues as software teams work as hard as they can make their organizations stay alive and competitive in the market place. Software companies face an immense pressure to virtually release a bug-free product or a software package. The culture of an organization is a critical success factor in the efforts of process improvement. The paper aims at assessing quality as a function for monitoring and measuring the strength of development processes and any successful application development enterprise requires an unambiguous understanding of customer expectation and maximizing participation of customers in the development activities thereby ensuring that people involved in development activities do the right thing and do the thing right for delivering high quality software . Keywords: Software development, process improvement, software defect, bug-free product, software package

Qualitative Impact Assessment Protocol (QUIP) - Better Evaluation (Better Evaluation)
The QUIP sets out to generate differentiated evidence of impact based on narrative causal statements elicited directly from intended project beneficiaries without use of a control group. Evidence of attribution is sought through respondents’ own accounts of causal mechanisms linking X to Y alongside Z rather than by relying on statistical inference based on variable exposure to X. This narrative data is intended to complement quantitative evidence on changes in X, Y and Z obtained through routine project monitoring.

Qualitative comparative analysis: A valuable approach to add to the evaluator’s toolbox? Lessons from recent applications - Schatz, Florian; Welle, Katharina (CDI Practice Paper, 2016)

Qualitative Comparative Analysis - A Rigorous Qualitative Method for Assessing Impact - Baptist, Carrie; Befani, Barbara (Coffey, 2015)

Q&A: Complexity-Aware Planning for Stabilization Programming - Martins, Christy (DAI - Checkpoint, 2018)

Putting vulnerability to climate change on the map: a review of approaches, benefits, and risks - Preston, Benjamin L.; Yuen, Emma J.; Westaway, Richard M. (Sustainability Science; Dordrecht, 2011)
There is growing demand among stakeholders across public and private institutions for spatially-explicit information regarding vulnerability to climate change at the local scale. However, the challenges associated with mapping the geography of climate change vulnerability are non-trivial, both conceptually and technically, suggesting the need for more critical evaluation of this practice. Here, we review climate change vulnerability mapping in the context of four key questions that are fundamental to assessment design. First, what are the goals of the assessment? A review of published assessments yields a range of objective statements that emphasize problem orientation or decision-making about adaptation actions. Second, how is the assessment of vulnerability framed? Assessments vary with respect to what values are assessed (vulnerability of what) and the underlying determinants of vulnerability that are considered (vulnerability to what). The selected frame ultimately influences perceptions of the primary driving forces of vulnerability as well as preferences regarding management alternatives. Third, what are the technical methods by which an assessment is conducted? The integration of vulnerability determinants into a common map remains an emergent and subjective practice associated with a number of methodological challenges. Fourth, who participates in the assessment and how will it be used to facilitate change? Assessments are often conducted under the auspices of benefiting stakeholders, yet many lack direct engagement with stakeholders. Each of these questions is reviewed in turn by drawing on an illustrative set of 45 vulnerability mapping studies appearing in the literature. A number of pathways for placing vulnerability mapping on a more robust footing are also identified. [PUBLICATION ABSTRACT]
Keywords: Climate change; Mapping; Risk assessment; Sustainable development

Putting theory into practice: how DFID is doing development differently - Wild, Leni; Booth, David; Valters, Craig (ODI, 2017)
Keywords: IMPORTANT

Putting learning at the centre: Adaptive development programming in practice - Valters, Craig; Cummings, Clare; Nixon, Hamish (ODI, 2016)
Adaptive programming suggests, at a minimum, that development actors react and respond to changes in the political and socio-economic operating environment. It emphasises learning and the development practitioner is encouraged to adjust their actions to find workable solutions to problems that they may face. Being prepared to react to change may seem like common sense – and indeed it is. However much development thinking and practice remains stuck in a linear planning model which discourages learning and adaptation, in part because projects are seen as ‘closed, controllable and unchanging systems’ (Mosse, 1998: 5). This paper critically engages with this problem and makes clear why and how learning needs to be at the centre of adaptive development programming. It begins by clarifying why and what kind of learning matters for adaptive programming. The paper then turns its focus to how strategies and approaches applied throughout a programme’s conception, design, management and M&E can enable it to continually learn and adapt.
Keywords: IMPORTANT

Putting Gender into Political Economy Analysis: why it matters and how to do it - Brown, Emily; Haines, Rebecca; O’Neil, Tam (From Poverty to Power, 2018)
Gender specialists from Oxfam and Care introduce a new guide that covers both theory and practice of including gender in political economy analysis.
Keywords: Gender

Purposeful Program Theory: Effective Use of Theories of Change and Logic Models - Funnell, Sue C.; Rogers, Patricia J. (Wiley, 2011)
Program Theory in Evaluation Practice is a ground-breaking reference that teaches how to develop an explicit causal model that links an intervention (project, program or policy) with its intended or observed impacts and using this to guide monitoring and evaluation. Peerless in its explanation of why and how to use and develop program theory, the book is rich with examples and alternative approaches. The book is an invaluable resource to faculty and students as well as professionals in professional development programs, education, social work, and counseling.

Purposeful Program Theory: Effective Use of Theories of Change and Logic Models - Funnell, Sue C.; Rogers, Patricia J. (2011)
Between good intentions and great results lies a program theory—not just a list of tasks but a vision of what needs to happen, and how. Now widely used in government and not-for-profit organizations, program theory provides a coherent picture of how change occurs and how to improve performance. Purposeful Program Theory shows how to develop, represent, and use program theory thoughtfully and strategically to suit your particular situation, drawing on the fifty-year history of program theory and the authors' experiences over more than twenty-five years."From needs assessment to intervention design, from implementation to outcomes evaluation, from policy formulation to policy execution and evaluation, program theory is paramount. But until now no book has examined these multiple uses of program theory in a comprehensive, understandable, and integrated way. This promises to be a breakthrough book, valuable to practitioners, program designers, evaluators, policy analysts, funders, and scholars who care about understanding why an intervention works or doesn't work." —Michael Quinn Patton, author, Utilization-Focused Evaluation"Finally, the definitive guide to evaluation using program theory! Far from the narrow 'one true way' approaches to program theory, this book provides numerous practical options for applying program theory to fulfill different purposes and constraints, and guides the reader through the sound critical thinking required to select from among the options. The tour de force of the history and use of program theory is a truly global view, with examples from around the world and across the full range of content domains. A must-have for any serious evaluator." —E. Jane Davidson, PhD, Real Evaluation Ltd.Companion Web site: josseybass.com/go/funnellrogers

Propel - Strhive (2023)
Learn from experience. Together. In the complex landscape of international development, organisations need a way to learn from their experiences and build on what works. Propel is the software solution that revolutionises the way organisations capture, access, and reuse learnings to adapt, innovate, and create lasting change on a global scale. Let's stop reinventing the wheel and build on what works, together.

Projects as instruments of development administration: A qualified defence and suggestions for improvement - Rondinelli, Dennis A. (Public Administration and Development, 1983)
Projects have become an important instrument of international assistance and of development administration because they seem to offer major advantages over other forms of planning and management. But projects have also come under increasing criticism in recent years. The benefits they offer to various interests involved in development ensure, however, that they are unlikely to be abandoned or to diminish in importance in the near future. Thus, ways must be found to make them more flexible and responsive methods of planning and managing social and human development activities. This can be done by planning, appraising and implementing projects as policy experiments, making their design and administration more learning-oriented, and using them as instruments of strategic planning and management.

Project Evaluation and the Project Appraisal Reporting System - Rosenberg, L.J.; Posner, L.D.; Hanley, E.J. (Fry Consultants Inc., 1970)

Project Cycle Hacker's Toolkit - changing the conversation around your projects - UNDP (UNDP, 2017)
The Istanbul Regional Hub partnered with Nesta and 4 country offices (fYR Macedonia, Moldova, Georgia and Kyrgyzstan) to develop a program that can help us bring innovation closer to the center of the organization by embedding it in the key project management business processes - The Project Cycle Hackers Toolkit.

Program Logic Foundations: Putting the Logic Back into Program Logic - Hawkins, Andrew J. (Journal of MultiDisciplinary Evaluation, 2020)
Abstract Background: Program logic is one of the most used tools by the public policy evaluator. There is, however, little explanation in the evaluation literature about the logical foundations of program logic or discussion of how it may be determined if a program is logical. This paper was born on a long journey that started with program logic and ended with the logic of evaluation. Consistent throughout was the idea that the discipline of program evaluation is a pragmatic one, concerned with applied social science and effective action in complex, adaptive systems. It gradually became the central claim of this paper that evidence-based policy requires sound reasoning more urgently than further development and testing of scientific theory. This was difficult to reconcile with the observation that much evaluation was conducted within a scientific paradigm, concerned with the development and testing of various types of theory. Purpose: This paper demonstrates the benefits of considering the core essence of a program to be a proposition about the value of a course of action. This contrasts with a research-based paradigm in which programs are considered to be a type of theory, and in which experimental and theory-driven evaluations are conducted. Experimental approaches focus on internal validity of knowledge claims about programs and on discovering stable cause and effect relationships—or, colloquially, ‘what works?’. Theory-driven approaches tend to focus on external validity and in the case of the realist approach, the search for transfactual causal mechanisms—extending the ‘what works’ mantra to include ‘for whom and in what circumstances’. On both approaches, evaluation aspires to be a scientific pursuit for obtaining knowledge of general laws of phenomena, or in the case of realists, replicable context-mechanism-outcome configurations. This paper presents and seeks to justify an approach rooted in logic, and that supports anyone to engage in a reasonable and democratic deliberation about the value of a course of action. It is consistent with systems thinking, complexity and the associated limits to certainty for determining the value of a proposed, or actual, course of action in the social world. It suggests that evaluation should learn from the past and have an eye toward the future, but that it would be most beneficial if concerned with evaluating in the present, in addressing the question ‘is this a good idea here and now? Findings: In seeking foundations of program logic, this paper exposes roots that extend far deeper than the post-enlightenment, positivist and post-positivist social science search for stable cause and effect relationships. These roots lie in the 4th century BCE with Aristotle’s ‘enthymeme’. The exploration leads to conclusions about the need for a greater focus on logic and reasoning in the design and evaluation of programs and interventions for the public good. Science and research are shown to play a crucial role in providing reasons or warrants to support a claim about the value of a course of action; however, one subordinate to the alpha-discipline of logical evaluation and decision making that must consider what is feasible given the context, capability and capacity available, not to mention values and ethics. Program Design Logic (PDL) is presented as an accessible and incremental innovation that may be used to determine if a program makes sense ‘on paper’ in the design stage as well as ‘in reality’ during delivery. It is based on a configurationalist theory of causality and the concepts of ‘necessary’ and ‘sufficient’ conditions. It is intended to guide deliberation and decision making across the life cycle of any intervention intended for the public good.

Program Cycle Road Map - USAID (2016)

Program Cycle - How-To Note: Strategy-Level Portfolio Review - USAID (USAID PPL, 2017)

Program Cycle - Discussion Note: Managing Monitoring, Evaluation, and Learning Platforms - USAID (USAID, 2018)
This Discussion Note complements ADS 201.3.1.2 Program Cycle Principles by elaborating on Principle 2: Manage Adaptively through Continuous Learning. This Discussion Note is intended for USAID staff interested in learning about recent and promising practices in adaptive management across the Program Cycle.

Program Cycle - Discussion Note: Designing Monitoring, Evaluation, and Learning Platforms - USAID (USAID, 2018)
This Discussion Note complements ADS 201.3.1.2 Program Cycle Principles by elaborating on Principle 2: Manage Adaptively through Continuous Learning. This Discussion Note is intended for USAID staff interested in learning about recent and promising practices in adaptive management across the Program Cycle.

Program Cycle - Discussion Note: Adaptive Management - USAID (USAID, 2018)
This Discussion Note complements ADS 201.3.1.2 Program Cycle Principles by elaborating on Principle 2: Manage Adaptively through Continuous Learning. This Discussion Note is intended for USAID staff interested in learning about recent and promising practices in adaptive management across the Program Cycle.

Profiling the international development mutants - Quaggiotto, Giulio (Development Impact and You, 2017)
As part of a new series, we talk to Giulio Quaggiotto about the development ‘mutants’ – innovators working at the edge of the field.

Producer-led value chain analysis: The missing link in value chain development - Ghore, Yogesh (Coady, 2015)
Introduction and rationale The concept of a value chain is increasingly being applied in the design and implementation of development programs aimed at poverty reduction. As an analytical tool, it provides a useful framework for understanding key activities, relationships, and mechanisms that allow producers, processors, buyers, sellers, and consumers—separated by time and space—to gradually add value to products and services as they pass from one link of the chain to another, making it a “value chain” (UNIDO, 2009). While it has been popular in the private sector ever since it was conceptualized by Michael Porter in the 1980s, more recently various donors and governments have shown interest in its use and have applied it to a range of development interventions, particularly in the area of sector development, livelihoods promotion, small and medium enterprise (SME) development, and rural and economic development. Academics and development organizations have designed numerous instruments for value chain analysis (VCA) and implementation. Beginning in the early 2000s, international organizations and donor agencies have sponsored the development of these tools, including the United States Agency for International Development (USAID) Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), the World Bank, the International Finance Corporation (IFC), and the International Labour Organization (ILO). These guides and tools have been used in many development programs to address organizational, donor and local and regional priorities for development. While VCA guides and tools are important in understanding markets and relationships among key stakeholders, a recent comparative review of popularly used guides by Donovan et al. (2013) suggests that most of these guides are designed to be implemented independently of the local context and do not sufficiently focus on mutual learning, whether related to tool design or to the outcomes and impacts of the designed project activities. VCA is often conducted by external experts and the knowledge generated in the process is often confined to reports. Without the capacity building and effective participation of women and men producers—smallholder farmers who hold critical knowledge about the local context—an important link is missing in the entire process of VCA. The insufficient attention to the human, social, and other contextual factors undermines the full potential of the value chain systems approach, not only for economic outcomes, but for the long term sustainability of the intended benefits. So what are the ways to effectively engage producers in the community in the VCA? The producer-led process described here tries to address this basic question. It introduces simplified tools and an approach that ensures farmer participation in data collection, analysis, and identification of opportunities and constraints, and design of value chain interventions. This participation requires: a) an environment in which they feel comfortable to share their knowledge and insights, which is often not the case when extractive surveys and questionnaires are administered to collect information; and b) the use of simple and participatory tools that will allow them to provide inputs into the process of VCA and also help them to understand complex value chain systems and use this understanding for making livelihood decisions. The key steps and tools described are aimed at empowering the women and men farmers to make informed decisions about their own enterprises and how they relate to the value chain, thereby directly contributing to, and influencing, the overall process of value chain development. The process of VCA involving these steps and tools was developed at Coady Institute and first tested with Oxfam Canada and its local partners in Ethiopia in August 2012.

Procuring and Managing Adaptively: 5 Case Studies of Adaptive Mechanisms - Pryor, Tony (USAID Learning Lab, USAID, 2018)
Flexibility in program management is essential in all of the countries where USAID works. This is especially true in non-permissive environments (NPEs), where the ability to learn and adapt quickly to changing circumstances can help USAID staff members achieve their desired outcomes.

Process Tracing Innovations in Practice: Finding the Middle Path - Aston, Thomas; Wadeson, Alix (CDI Practice Paper, 2023)
Evaluation practitioners in the international development sector have given considerable attention in recent years to process tracing as a method for evaluating impact, including discussion of how to assess the relative importance of causal factors. Despite the increasing interest, there is a relative dearth of examples of practical learning and evidence of applying process tracing in practice. This CDI Practice Paper draws on comparative learning from applying three different types of process tracing in international development initiatives. It argues in favour of a ‘middle path’ of applying evidence tests and rubrics to structure evaluative judgements rather than formal Bayesian updating or looser forms of process tracing. It also calls attention to the potential added value of taking a participatory approach, offering practical recommendations for how to do this effectively.

Process Tracing as a Practical Evaluation Method: Comparative Learning from Six Evaluations - Wadeson, Alix; Monzani, Bernardo; Aston, Tom (2020)

Process Monitoring of Impacts: Towards a new approach to monitor the implementation of Structural Fund Programmes - Hummelbrunner, Richard; Huber, Wolf; Arbter, Roland (ÖAR Regionalberatung, 2005)

Process and Contextual Factors Supporting Action-Oriented Learning: A Thematic Synthesis of Empirical Literature in Natural Resource Management - Suškevičs, Monika; Hahn, Thomas; Rodela, Romina (Society & Natural Resources, 2019)
Despite a long-term focus on learning in natural resource management (NRM), it is still debated how learning supports sustainable real-world NRM practices. We offer a qualitative in-depth synthesis of selected scientific empirical literature (N = 53), which explores factors affecting action-oriented learning. We inductively identify eight key process-based and contextual factors discussed in this literature. Three patterns emerge from our results. First, the literature discusses both facilitated participation and self-organized collaboration as dialogical spaces, which bridge interests and support constructive conflict management. Second, the literature suggests practice-based dialogs as those best able to facilitate action and puts a strong emphasis on experimentation. Finally, not emphasized in existing reviews and syntheses, we found multiple evidence about certain contextual factors affecting learning, including social-ecological crises, complexity, and power structures. Our review also points at important knowledge gaps, which can be used to advance the current research agenda about learning and NRM.
Keywords: Ecosystem governance; Social learning; intermediaries; social-ecological systems; structural constraints

Problem-driven iterative approaches and wider governance reform - Rao, Sumedh (Helpdesk Research Report 1099, 2014)
Identify evidence which suggests that problem-driven, iterative approaches to public sector reform can deliver more substantial, wider, long-term governance reform.

Probes, toolkits and prototypes: three approaches to making in codesigning - Sanders, Elizabeth B.-N.; Stappers, Pieter Jan (CoDesign, 2014)
The role of making in the design process has been growing, taking on new forms and involving new players over the past 10 years. Where we once primarily saw designers using making to give shape to the future, today we can see designers and non-designers working together, using making as a way to make sense of the future. In this paper, we describe the landscape of design research and practice at the end of 2013 with special attention to the role of making across these perspectives: approach (cultural probes, generative toolkits and design prototypes), mindset (designing for people and designing with people), focus in time (the world as it is, the near future and the speculative future) as well as variations in design intent (provoking, engaging and serving).

Prioritizing and Selecting Strategic Approaches in USAID Biodiversity Programming - Stem, Caroline; Swaminathan, Vinaya; Flores, Marco (Measuring Impact, 2018)
This guide supplements Biodiversity How-To Guide 2: Using Results Chains to Depict Theories of Change in USAID Biodiversity Programming. Step 4, Brainstorming Strategic Approaches, involves generating a suite of potential strategic approaches for a design team to consider. Step 5 introduces the practice of prioritizing and selecting among these draft strategic approaches but does not provide in-depth guidance on the process and criteria for doing so.

Principles for Monitoring, Evaluation, and Adaptive Management of Environmental Water Regimes - Webb, J. Angus; Watts, Robyn J.; Allan, Catherine; Warner, Andrew T. (Water for the Environment, 2017)
Environmental water is often a contentious investment in the environment that must be delivered under uncertainty regarding the ecological and social benefits they deliver. Adaptive management can be used to facilitate decision making under uncertainty, and use new knowledge and understanding to improve management decisions and outcomes over time. However, there is a perception that adaptive management has failed to deliver improvements in decision making and learning. Monitoring and evaluation are an essential feature of adaptive management. However, past monitoring for river protection and restoration has too often been of insufficient quality to facilitate adaptive learning. Moreover, environmental water represents a class of protection and restoration that is inherently difficult to monitor, and some of the learnings are not evident in part because fragmented assessment, documentation, and reporting of adaptive management can obscure successes. We outline a set of general principles for improving monitoring, evaluation, and adaptive management of environmental water regimes, and provide several examples from Australia and North America. Successful adaptive management depends upon the formation and maintenance of strong partnerships and an appreciation of the importance of individuals in teams. Through such partnerships, it is possible to coordinate monitoring programs over large scales, create programs of requisite simplicity, and use innovative approaches to evaluation. Finally, we note that although adaptive management has occurred in the past, too often it has been poorly documented and reported. The inclusion of reflectors in monitoring and evaluation teams will better capture the lessons learned by individual programs, improving adaptive management into the future.
Keywords: Environmental water management; Knowledge exchange; Legitimacy; implementation challenge

Principles for managing in complexity - Proud, Emma (LearnAdapt, 2020)
Written by Toby Lowe and Shaheen Warren (Centre for Public Impact) and Sam Sharp (Overseas Development Institute), with input from Jamie…

Principles for Digital Development - digitalprinciples.org (2018)

Principles-Focused Evaluation: The GUIDE - Patton, Michael Quinn (Guilford Press, 2017)
How can programs and organizations ensure they are adhering to core principles--and assess whether doing so is yielding desired results? From evaluation pioneer Michael Quinn Patton, this book introduces the principles-focused evaluation (P-FE) approach and demonstrates its relevance and application in a range of settings. Patton explains why principles matter for program development and evaluation and how they can serve as a rudder to navigate the uncertainties, turbulence, and emergent challenges of complex dynamic environments. In-depth exemplars illustrate how the unique GUIDE framework is used to determine whether principles provide meaningful guidance (G) and are useful (U), inspiring (I), developmentally adaptable (D), and evaluable (E). User-friendly features include rubrics, a P-FE checklist, firsthand reflections and examples from experienced P-FE practitioners, sidebars and summary tables, and end-of-chapter application exercises.

Primer: Technology to monitor and share information on rainforests and forest people's rights - The Engine Room (The Engine Room & Rainforest Foundation Norway, 2016)

Practitioners Guidance to Assessing Systems Change: Co-Authors Preview - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (USAID, 2023)
Check out this video to see what’s inside our new resource: Practitioners' Guidance to Assessing Systems Change, developed by MEL Managers for MEL Managers. (Check out the Guidance here https://bit.ly/MSPMELClinics.) Hear from the authors about which parts they love the most and how this guide challenges MEL managers to assess systems change as an ongoing aspect of implementation, generating feedback that teams need to better understand and catalyze change, for more impact.

Practising change together – where nothing is clear, and everything keeps changing - Sharp, Cathy (Action Learning: Research and Practice, 2020)
This paper explores the thinking and practice of ‘action inquiry’ an embedded learning practice that can help navigate complexity when practising change together. The paper uses examples from social contexts where there are concerns about community wellbeing and health care. These are drawn from collaborative or collective leadership development programmes within public services that seek to bring new attention to the qualities of how people think, converse and interact, as part of their collective professional practice. This treats social action as a relational and dialogical practice, something that we do together as professionals by engaging in reflective inquiry and action. The paper suggests that action inquiry offers a prospect of rekindling the links between ‘action learning’ and collaborative leadership by developing a co-mission and a mutual commitment to a new type of learning partnership. Action inquiry can be wrapped around and enmeshed within initiatives and programmes that work with complexity, anywhere where effective social action will depend on the quality of relationships that can be developed. This research was funded by two separate Scottish Government commissions, where the author was a learning partner. The paper also draws on the further reflections of some of the practitioners most centrally involved.

Practioners Guide to Assessing Systems Change - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (MSD in MEL Brief, 2023)

Practicing what we preach — our experience rethinking M&E in complex settings - Bina, Andrea; Vester, Søren (UNDP Strategic Innovation, 2023)
The UNDP has been exploring new approaches to M&E to better understand and address complex systems challenges. Traditional linear project-based planning and evaluation methods are insufficient for dealing with the unpredictable nature of such systems. The UNDP’s new Strategic Plan calls for a shift in M&E practices to align with the complexity of today’s global challenges. Key points from the blog post include: Learning and Adapting: There’s a need for continuous learning and adaptation in M&E practices to effectively address complex problems like transitioning to a circular economy. Longer Time Horizons: Substantive change takes time, and it’s often unclear what the results will look like, making it difficult to track progress. Aggregate Impact: Evaluating interconnected interventions as a portfolio is crucial because they cannot be assessed in isolation. Contribution vs. Attribution: Focusing on the contribution to broader change processes is more important than attributing specific outcomes to individual actions. The UNDP has initiated the M&E Sandbox to experiment with and learn from innovative M&E practices. This collaborative space has attracted participants from over 180 organizations, highlighting a shared interest in evolving M&E to better understand change in contexts of uncertainty. A partnership with the Bill and Melinda Gates Foundation has also been formed to focus on M&E in food systems. The blog post emphasizes the importance of embracing uncertainty, experimenting, and collaborating to contribute to positive change in complex settings1.

Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective - Chen, Huey (SAGE Publications, Inc, 2014)
The Second Edition of Practical Program Evaluation shows readers how to systematically identify stakeholders’ needs in order to select the evaluation options best suited to meet those needs. Within his discussion of the various evaluation types, Huey T. Chen details a range of evaluation approaches suitable for use across a program’s life cycle. At the core of program evaluation is its body of concepts, theories, and methods. This revised edition provides an overview of these, and includes expanded coverage of both introductory and more cutting-edge techniques within six new chapters. Illustrated throughout with real-world examples that bring the material to life, the Second Edition provides many new tools to enrich the evaluator’s toolbox.

PRA five years later - where are we now? / DRP - Cinco años después, ¿dónde nos encontramos? - Chambers, Robert; Guijt, Irene (Forest, Trees and People Newsletter, 1995)
PRA five years later: where are we now?

Power, Violence, Citizenship and Agency - McGee, Rosemary (IDS Bulletin, 2014)

Power, Empowerment and Social Change - McGee, Rosemary; Pettit, Jethro (2019)
This book uncovers how power operates around the world, and how it can be resisted or transformed through empowered collective action and social leadership. The stakes have never been higher. Recent years have seen a rapid escalation of inequalities, the rise of new global powers and corporate interests, increasing impunity of human rights violations, suppression of civil society, and a re-shaping of democratic processes by post-truth, populist and nationalist politics. Rather than looking at power through the lenses of agency or structure alone, this book views power and empowerment as complex and multidimensional societal processes, defined by pervasive social norms, conditions, constraints and opportunities. Bridging theory and practice, the book explores real-world applications using a selection of frameworks, tools, case studies, examples, resources and reflections from experience to support actors to analyse their positioning and align themselves with progressive social forces. Compiled with social change practitioners, students and scholars in mind, Power, Empowerment and Social Change is the perfect volume for anyone involved in politics, international development, sociology, human rights and environmental justice who is looking for fresh insights for transforming power in favour of relatively less powerful people.

Power Analysis: A Practical Guide - Pettit, Jethro (Sida, 2013)

Power Above and Below the Waterline: Bridging Political Economy and Power Analysis - Pettit, Jethro; Mejía Acosta, Andrés (IDS Bulletin, 2014)

Poverty Unperceived: Traps, Biases and Agenda - Chambers, Robert (IDS Working Paper 270, 2006)

Poverty Stoplight - povertystoplight.org (2018)
What is the Poverty Stoplight? The Stoplight is a tool that seeks to activate the potential of families and communities to lift themselves out of poverty. Using a technology platform, it offers a self-assessment survey and intervention model that enables people to develop practical solutions to overcome their specific needs. We work with organizations from different sectors around the world and bring together a network of powerful actors with one clear mission: eliminate poverty worldwide. The Stoplight Community is comprised of government agencies, private companies, small and medium-sized enterprises, non-profits, cooperatives, microfinance organizations, academic institutions, and sports clubs, among others.

Positive Deviant - Dorsey, David (Fast Company, 2000)
Jerry Sternin’s job was to help save starving children in Vietnam. Faced with an impossible time frame, he adopted a radical approach to making change. His idea: Real change begins from the inside.

Positive Deviance Initiative (2016)

Positive Deviance Initiative - Positive Deviance Initiative (2018)

Positive deviance, big data, and development: A systematic literature review - Albanna, Basma; Heeks, Richard (The Electronic Journal of Information Systems in Developing Countries, 2019)
Positive deviance is a growing approach in international development that identifies those within a population who are outperforming their peers in some way, eg, children in low-income families who are well nourished when those around them are not. Analysing and then disseminating the behaviours and other factors underpinning positive deviance are demonstrably effective in delivering development results. However, positive deviance faces a number of challenges that are restricting its diffusion. In this paper, using a systematic literature review, we analyse the current state of positive deviance and the potential for big data to address the challenges facing positive deviance. From this, we evaluate the promise of “big data-based positive deviance”: This would analyse typical sources of big data in developing countries—mobile phone records, social media, remote sensing data, etc—to identify both positive deviants and the factors underpinning their superior performance. While big data cannot solve all the challenges facing positive deviance as a development tool, they could reduce time, cost, and effort; identify positive deviants in new or better ways; and enable positive deviance to break out of its current preoccupation with public health into domains such as agriculture, education, and urban planning. In turn, positive deviance could provide a new and systematic basis for extracting real-world development impacts from big data.
Keywords: big data; developing countries; machine learning; mobile data; positive deviance; systematic literature review

Positive Deviance - Better Evaluation (Better Evaluation)
Positive Deviance (PD) refers to a behavioral and social change approach which is premised on the observation that in any context, certain individuals confronting similar challenges, constraints, and resource deprivations to their peers, will nonetheless employ uncommon but successful behaviors or strategies which enable them to find better solutions. Through the study of these individuals– subjects referred to as “positive deviants” - the PD approach suggests that innovative solutions to such challenges may be identified and refined from their outlying behavior.

Portfolios of Agile Projects: A Complex Adaptive Systems’ Agent Perspective - Sweetman, Roger; Conboy, Kieran (Project Management Journal, 2018)
While agile approaches can be extremely effective at a project level, they can impose significant complexity and a need for adaptiveness at the project portfolio level. While this has proven to be highly problematic, there is little research on how to manage a set of agile projects at the project portfolio level. What limited research that does exist often assumes that portfolio-level agility can be achieved by simply scaling project level agile approaches such as Scrum. This study uses a complex adaptive systems lens, focusing specifically on the properties of projects as agents in a complex adaptive portfolio to critically appraise current thinking on portfolio management in an agile context. We then draw on a set of 30 expert interviews to develop 16 complex adaptive systems (CAS)-based propositions as to how portfolios of agile projects can be managed effectively. We also outline an agenda for future research and discuss the differences between a CAS-based approach to portfolio management and traditional approaches.

Polycentric Systems of Governance: A Theoretical Model for the Commons - Carlisle, Keith; Gruby, Rebecca L. (Policy Studies Journal, 2019)
Polycentricity is a fundamental concept in commons scholarship that connotes a complex form of governance with multiple centers of semiautonomous decision making. If the decision-making centers take each other into account in competitive and cooperative relationships and have recourse to conflict resolution mechanisms, they may be regarded as a polycentric governance system. In the context of natural resource governance, commons scholars have ascribed a number of advantages to polycentric governance systems, most notably enhanced adaptive capacity, provision of good institutional fit for natural resource systems, and mitigation of risk on account of redundant governance actors and institutions. Despite the popularity of the concept, systematic development of polycentricity, including its posited advantages, is lacking in the commons literature. To build greater clarity and specificity around the concept, we develop a theoretical model of a polycentric governance system with a focus on the features necessary or conducive for achieving the functioning predicted by commons scholars. The model is comprised of attributes, which constitute the definitional elements, and enabling conditions, which specify additional institutional features for achieving functionality in the commons. The model we propose takes the concept a step further toward specificity without sacrificing the generality necessary for contextual application and further development.

Politics, Political Change and International Development - Halloran, Brendan (Politics, Governance and Development, 2013)
As my inaugural post on my new blog, I thought I would re-post a short piece I wrote earlier this year about foreign aid (original here). Thomas Carothers and Diane de Garamont address this issue i…

Politics matters, so what? Time for bigger bets (and more learning) on adaptive programming - Alan (www.globalintegrity.org, 2016)
By Alan Hudson, Executive Director, Global Integrity, July 26, 2016 Politics matters. Context too. And blueprints have limited value. Our strategy is based on these insights, so we’re totally on board. A World Development Report (WDR) that puts power and politics...

Politically smart support to economic development: DFID experiences - Booth, David (ODI, 2016)

Politically smart, locally led development - Booth, David; Unsworth, Sue (Discussion Paper, 2014)
Aid donors have found it hard to move from thinking politically to working differently, but there is evidence that they can do so and that this improves outcomes. This paper presents seven examples of where adopting a politically smart, locally led approach has led to better outcomes.
Keywords: Practice

Politically smart approaches to donor investment climate programming. A Guidance Note - LASER (Synthesis Paper, 1, 2016)
In order to help enhance the effectiveness of donors and development practitioners on the ground, LASER has produced a range of practical guidance and tools. These are primarily intended for the international development community engaged in designing and implementing investment climate programmes, though can also be used more widely by other stakeholders across sectors. General guidance and tools: Politically smart approaches to donor investment climate programming (and annexes) Revised February 2016 - sets out practical advice on how to adopt a politically smart (which requires being both ‘politically informed’ and ‘politically astute’) approach to programming. The annexes include a traffic light matrix for initial high level analysis; a problem diary template with examples; and an example of a flexible theory of change for adaptive, large scale programming
Keywords: Practice

Politically informed, gender aware programming: Five lessons from practice - Derbyshire, Helen; Gibson, Sam; Hudson, David; Roche, Chris (DLP Program, 2018)

Politically informed approaches to gender in fragile and conflict-affected settings - Castillejo, Clare; Domingo, Pilar; George, Rachel; O’Connell, Shannon (ODI, 2020)
This report summarises the discussions at a meeting held in September 2019 of a group of global development research and policy experts and practitioners, convened by ODI and the Westminster Foundation for Democracy, to share experiences and knowledge, reflect on what we already know about working politically on gender in fragile and conflict-affected settings (FCAS), and identify what further evidence would be useful. Key messages: - It is important to identify, document and learn from politically informed and adaptive ways of working in practice on gender in FCAS. - Different analytical and monitoring, evaluation and learning tools are required for politically informed programming on gender. These should be embedded within programme teams and processes, and be both gender-responsive and responsive to the political economy context. - There is a need for more politically smart use of quantitative and qualitative data in order to identify plausible entry points and ways of working on gender in FCAS. This should include increasing the capacity of programme staff to use data to inform, adapt and correct programmes. - Staff promoting politically informed work on gender equality are often isolated, with little opportunity to share experiences or learn from others. Platforms should therefore be created to share experience and knowledge, and to bring together donors and implementers involved in this work.

Political economy analysis: Topic Guide - Mcloughlin, Claire (GSDRC, University of Birmingham, 2014)
Political economy analysis (PEA) aims to situate development interventions within an understanding of the prevailing political and economic processes in society – specifically, the incentives, relationships, and distribution and contestation of power between different groups and individuals. Such an analysis can support more politically feasible and therefore more effective development strategies by setting realistic expectations …

Political Economy Analysis - How to note - DFID (Practice Paper, 2009)
Political economy analysis is a powerful tool for improving the effectiveness of aid. Bridging the traditional concerns of politics and economics, it focuses on how power and resources are distributed and contested in different contexts, and the implications for development outcomes. It gets beneath the formal structures to reveal the underlying interests, incentives and institutions that enable or frustrate change. Such insights are important if we are to advance challenging agendas around governance, economic growth and service delivery, which experience has shown do not lend themselves to technical solutions alone. Political economy analysis is not a magic bullet for the resolution of intractable development problems. However, it can support more effective and politically feasible development strategies, as well as inform more realistic expectations of what can be achieved, and the risks involved. It can also contribute to better results by identifying where the main opportunities and barriers for policy reform exist and how donors can use their programming and influencing tools to promote positive change. This understanding is particularly relevant in fragile and conflict-affected environments where the challenge of building peaceful states and societies is fundamentally political. There are an increasing number of political economy tools available to development agencies for a range of analytical and operational purposes. This note brings together this material with a view to explaining the relevance and uses of political economy analysis. It is intended to be used by a wide range of DFID programme managers and advisers, as well as staff in other HMG departments and partner organisations. The main questions it addresses are: 􀂃 what is political economy analysis? 􀂃 how and why does political economy analysis add value to DFID work? 􀂃 what approaches and tools are available? 􀂃 how should the analysis be prepared, undertaken and applied to DFID’s work? 􀂃 how should we work with other development partners and across HMG on analysis? Key messages include: • Political economy analysis is central to the formulation of sound country plans and sector programmes, and can play a key role in risk mitigation and ensuring that donors avoid harmful practices. • Political economy analysis can help to improve development effectiveness by identifying how and where donors should focus efforts to promote positive change. • There are a growing number of operationally relevant tools which can be used to inform development strategies at the country or sector level, or in relation to particular development problems. • Several DFID country offices have used political economy analysis to improve the quality and impact of aid. This experience provides valuable lessons that should be considered when commissioning and undertaking political economy analysis. • Where possible, analysis should be conducted on an ongoing basis with key partners in HMG and the wider development community to encourage shared understanding and joint action.

Political Economy Analysis – Guidance for legal technical assistance - Denney, Lisa; Domingo, Pilar (ROLE UK, 2017)
Political economy analysis is about understanding how change in relation to rule of law and international development is embedded within and shaped by political and economic relations that interact and are particular to each context. These political economy dynamics determine the distribution of power and resources within any given society and must be taken into account when attempting to achieve change. Political economy analysis is important in the justice sector because rule of law and dispute resolution processes reflect the rules of the game about who wins and who loses in the allocation of power and resources. Using political economy analysis in legal technical assistance can help to ensure more relevant and effective assistance that sustainably addresses concrete problems. It can also help to avoid the risk that poorly targeted assistance might inadvertently exacerbate existing power structures in a way that was not intended.

Policy messes and their management - Roe, Emery (Policy Sciences, 2016)
This paper presents a framework for better managing policy messes and draws implications for bad and good mess management in policy analysis and management. The framework has three foci: (1) the cognitive space in which policy messes develop, particularly in terms of gaps between macro-designers and micro-operators; (2) the unique domain of competence within that space where professionals manage the resulting messes by virtue of their skills in recognizing system-wide patterns, formulating locally specific contingency scenarios and translating both patterns and scenarios in highly reliable services; and (3) the ability of those mess and reliability professionals to be reliable in their domain and with these skills by maneuvering across different performance modes as conditions dictate—just-in-case, just-on-time, just-for-now or just-this-way.

Policy experimentation and innovation as a response to complexity in China’s management of health reforms - Husain, Lewis (Globalization and Health, 2017)
There are increasing criticisms of dominant models for scaling up health systems in developing countries and a recognition that approaches are needed that better take into account the complexity of health interventions. Since Reform and Opening in the late 1970s, Chinese government has managed complex, rapid and intersecting reforms across many policy areas. As with reforms in other policy areas, reform of the health system has been through a process of trial and error. There is increasing understanding of the importance of policy experimentation and innovation in many of China’s reforms; this article argues that these processes have been important in rebuilding China’s health system. While China’s current system still has many problems, progress is being made in developing a functioning system able to ensure broad population access. The article analyses Chinese thinking on policy experimentation and innovation and their use in management of complex reforms. It argues that China’s management of reform allows space for policy tailoring and innovation by sub-national governments under a broad agreement over the ends of reform, and that shared understandings of policy innovation, alongside informational infrastructures for the systemic propagation and codification of useful practices, provide a framework for managing change in complex environments and under conditions of uncertainty in which ‘what works’ is not knowable in advance. The article situates China’s use of experimentation and innovation in management of health system reform in relation to recent literature which applies complex systems thinking to global health, and concludes that there are lessons to be learnt from China’s approaches to managing complexity in development of health systems for the benefit of the poor.

Policy-driven monitoring and evaluation: Does it support adaptive management of socio-ecological systems? - Waylen, Kerry A.; Blackstock, Kirsty L.; van Hulst, Freddy J.; Damian, Carmen; Horváth, Ferenc; Johnson, Richard K.; Kanka, Robert; Külvik, Mart; Macleod, Christopher J. A.; Meissner, Kristian; Oprina-Pavelescu, Mihaela M.; Pino, Joan; et al. (Science of The Total Environment, 2019)
Inadequate Monitoring and Evaluation (M&E) is often thought to hinder adaptive management of socio-ecological systems. A key influence on environmental management practices are environmental policies: however, their consequences for M&E practices have not been well-examined. We examine three policy areas - the Water Framework Directive, the Natura 2000 Directives, and the Agri-Environment Schemes of the Common Agricultural Policy - whose statutory requirements influence how the environment is managed and monitored across Europe. We use a comparative approach to examine what is monitored, how monitoring is carried out, and how results are used to update management, based on publicly available documentation across nine regional and national cases. The requirements and guidelines of these policies have provided significant impetus for monitoring: however, we find this policy-driven M&E usually does not match the ideals of what is needed to inform adaptive management. There is a tendency to focus on understanding state and trends rather than tracking the effect of interventions; a focus on specific biotic and abiotic indicators at the expense of understanding system functions and processes, especially social components; and limited attention to how context affects systems, though this is sometimes considered via secondary data. The resulting data are sometimes publicly-accessible, but it is rarely clear if and how these influence decisions at any level, whether this be in the original policy itself or at the level of measures such as site management plans. Adjustments to policy-driven M&E could better enable learning for adaptive management, by reconsidering what supports a balanced understanding of socio-ecological systems and decision-making. Useful strategies include making more use of secondary data, and more transparency in data-sharing and decision-making. Several countries and policy areas already offer useful examples. Such changes are essential given the influence of policy, and the urgency of enabling adaptive management to safeguard socio-ecological systems.
Keywords: Adaptive Management; Environmental governance; Monitoring and Evaluation; Policy making; Socio-ecological systems; Sustainability

Pocket Guide to Systems Wayfinding - Davidson, Seanna (The Systems School, 2019)

Playbook for innovation learning. 35 diagrams to support talking and thinking about learning for innovation - Leurs, Bas; Roberts, Isobel (Nesta, 2018)

Plans Are Worthless but Planning Is Everything: A Theoretical Explanation of Eisenhower’s Observation - Garcia Contreras, Angel F.; Ceberio, Martine; Kreinovich, Vladik (Decision Making under Constraints, 2020)
The 1953–1961 US President Dwight D. Eisenhower emphasized that his experience as the Supreme Commander of the Allied Expeditionary Forces in Europe during the Second World War taught him that “plans are worthless, but planning is everything”. This sound contradictory: if plans are worthless, why bother with planning at all? In this paper, we show that Eisenhower’s observation has a meaning: while directly following the original plan in constantly changing circumstances is often not a good idea, the existence of a pre-computed original plan enables us to produce an almost-optimal strategy—a strategy that would have been computationally difficult to produce on a short notice without the pre-existing plan.

Planning & Navigating Social Change - Tools for Pacific voyagers - Orr, Douglas Epeli; Cavatore, Maria; Aston, Tom (Oxfam, 2019)
Those working to advance social change in the Pacific must understand, work with and respond to the complex and changing relationships and dynamics of power that exist within such networks, and situate their work in the context of decolonisation and self-determination. But the approaches typically used by NGOs to plan our projects and programs have failed to give due attention to these dynamics. The result is that our projects fall short of achieving their promise, despite being technically sound and logical. In this guide, we aim to provide a process and tools that prioritise and draw out the rich, often implicit, knowledge that Pacific Islanders have about our contexts to plan and manage social change initiatives in a manner that values and responds to this ocean of relationships. Drawing from these relationship-based and voyaging traditions, we emphasise the need to: • frame change in terms of change in people; e.g. change in the situation of groups of people and change in terms of the actions of people; • identify a course towards a particular ‘change destination’; • identify the signs to read that will help you determine where you are on that course; • be constantly attentive to the context as you make the journey, and • be ready to adjust your course in response to changes in your context. To support this way of working, we have selected a number of tools from approaches that we believe fit. In particular, we have drawn from the following approaches: • Thinking and Working Politically and other ‘politically aware’ approaches that recognise that the success of a development initiative will be determined by the interplay of relationships of power within a particular context, not just by having the right technical solution; • Outcome mapping because of its focus on people and relationships, and on identifying and supporting needed changes in the behaviour of specific actors in relation to others. This is key, because it’s individuals who deliver services, individuals who decide whether to allocate budget and resources to address a problem, and individuals who make the choice to believe whether a service is relevant to solving their issues; • Adaptive management because of its recognition that development initiatives take place within complex and changing systems and need to adjust to these and that this requires ongoing learning and adaptation.

Planners versus Searchers in Foreign Aid - Easterly, William (Asian Development Review, 2006)
Only for the recipients of foreign aid is something akin to central planning seen as a way to achieve prosperity. The end of poverty is achieved with free markets and democracy—where decentralized “searchers” look for ways to meet individual needs—not Poverty Reduction Strategy Papers (PRSPs) to achieve Millennium Development Goals (MDGs). The PRSPs and MDGs create lots of bureaucracy but hold no one specific agency in foreign aid accountable for any one specific task. Planners in foreign aid use the old failed models of the past—the “Financing Gap”, the “poverty trap”, the government-to-government aid model; and the “expenditures = outcomes” mentality. Searchers in foreign aid would imitate the feedback and accountability of markets and democracy to provide goods and services to individuals until homegrown markets and democracy end poverty in the society as a whole. An example of the more promising “searchers” approach in foreign aid is 2006 Nobel Peace Laureate Mohammad Yunus and Grameen Bank.

PLA Notes 32: Participation, Literacy and Empowerment - PLA (Participatory Learning and Action Notes, 1998)

Physically distanced adaptive management - Proud, Emma (LearnAdapt, 2020)
Opportunities and challenges for local leadership

PFM Reform Through PDIA: What Works and When it Works - Harris, Jamelia; Lawson, Andrew (Public Financial Management Blog - IMF, 2022)

Perverse Payment by Results: frogs in a pot and straitjackets for obstacle courses - Chambers, Robert (IDS - Participation, Power and Social Change, 2014)
Posts about Robert Chambers written by idsppsc

Perspectives on Participation in Design - Sanders, Elizabeth (Wer gestaltet die Gestaltung? Praxis, Theorie und Geschichte des partizipatorischen Designs, 2013)
Keywords: Design

Performance Story - Dart, J.; Mayne, J. (Encyclopedia of Evaluation, 2005)
This is a concise description of ‘Performance Story’ written by Jess Dart and John Mayne for the Sage ‘Encyclopaedia of Evaluation’.

Peer reviews - Guidance for facilitators and participants - Wadley, Ian (Mediation Practice Series, 2021)
Essential points for practitioners and donors • Mediation offers a cost-effective and proven method for resolving armed conflict. Between 1985 and 2015, 75 per cent of armed conflicts in the world were resolved through agreement rather than by force. In most cases these processes will have involved third party facilitation or support. • Professional mediators understand the high stakes involved in their work to prevent, mitigate and resolve armed conflict. In addition, they and their financial supporters are increasingly required to demonstrate ‘value-for-money’ to ensure continued funding. • However, traditional monitoring and evaluation (M&E) methods are not well suited to this task, typically imposing artificially linear project models on a dynamic conflict situation, as well as compliance reporting that moves attention away from real value. • Traditional M&E methods tend to focus on documenting the past and generating vast amounts of data, rather than enabling timely adaptation of the project in the present. • Traditional M&E approaches rely heavily on external evaluation consultants. Even in the best of cases this may interfere with the mediation process and impose a heavy time burden on the project team, leading to low acceptance of traditional M&E approaches by mediation practitioners. • In contrast, an ideal M&E approach for mediation should deliver useful insights in even the most dynamic and sensitive mediation environments, impose a light reporting burden, and be readily accepted by mediation teams. It should protect discretion and trust, enable rapid adaptation, and also provide some assurance that donor funds are being well spent.

Peacebuilding design, monitoring, and evaluation: A Training Package for participants and trainers at intermediate to advanced levels - Ernstorfer, Anita; Barnard-Webster, Kiely (Carnegie Corporation, 2019)
This training package includes 7 Training Modules and a set of Annexes (Annexes A-O). The Training Modules build on each other and should ideally be used in a sequenced way in a training setting. However, for groups with specific training needs around particular areas, modules can also be used individually, but need to be tailored by the trainers and facilitators to meet the needs of specific audiences. The annexes provide worksheets and hand-outs that can be used as resources during the training for specific modules and exercises.

Peacebuilding and Statebuilding Strategy for SDC’s work in fragile and conflict contexts - SDC (Swiss Agency for Development and Cooperation SDC, 2015)

PDIA Toolkit - A DIY Approach to Solving Complex Problems - Samji, Salimah; Andrews, Matt; Pritchett, Lant; Woolcock, MIchael (Center for International Development at Harvard University, 2018)
The PDIAtoolkit is designed to guide you through the process of solving complex problems which requires working in teams. We call it a Do-it-Yourself (DIY) kit, where the ‘you’ is a committed team of 4-6 people mobilized to work together to solve a complex problem that cannot be solved by one person.

PDIA Notes 2: Learning to Learn - Samji, Salimah (Building State Capability, 2016)
written by Peter Harrington After over two years of working with the government of Albania, and as we embark on a new project to work with the government of Sri Lanka, we at the Building State Capa…

PDIA in action - Andrews, Matt; Samji, Salimah (Center for International Development, Harvard University, 2021)
Learning from our experience in 2020, we asked the alumni of our HKS Implementing Public Policy (IPP) Executive Education program, if they wanted to work with our students on their nominated problems. Eight IPP alumni, William Keith Young, Adaeze Oreh, Milzy Carrasco, Kevin Schilling, Artem Shaipov, George Imbenzi, David Wuyep, and Raphael Kenigsberg, who had been trained on PDIA and implementation, signed up to work with our students. Thirty-seven students signed up to take the course beginning January 26th, 2021. The students worked across eight teams and adopted a problem driven approach to foster learning that could help their authorizers develop an action learning strategy to their nominated challenge. This book highlights the students’ work drawing from their blogs as well as the event series. There are 8 sections, one for each of the teams and the problems they worked on during the course. We hope you enjoy reading their stories! Scan the QR Code at the end of each section to learn more.

PDIA Course: Alumni are already practicing what they learned - Samji, Salimah (Building State Capability, 2017)
written by Salimah Samji We offered 4 free PDIA online courses between November 2015 and June 2016. They were well received and 365 people, living in 56 countries, successfully completed the course…

Payment by Results in Development Aid: All That Glitters Is Not Gold - Clist, Paul (The World Bank Research Observer, 2016)
Payment by Results (PbR), where aid is disbursed conditional upon progress against a pre-agreed measure, is becoming increasingly important for various donors. There are great hopes that this innovative instrument will focus attention on ultimate outcomes and lead to greater aid effectiveness by passing the delivery risk on to recipients. However, there is very little related empirical evidence, and previous attempts to place it on a sure conceptual footing are rare and incomplete. This article collates and synthesises relevant insights from a wide range of subfields in economics, providing a rich framework with which to analyze Payment by Results. I argue that the domain in which it dominates more traditional forms is relatively small and if it is used too broadly, many of the results it claims are likely to be misleading. The likelihood of illusory gains stems from the difficulty of using a single indicator to simultaneously measure and reward performance: ‘once a measure becomes a target, it ceases to be a good measure.’ This does not mean PbR should not be used (indeed it will be optimal in some settings), but it does mean that claims of success should be treated with caution.
Keywords: Conditionality; Contract theory; F35; Foreign Aid; Multitask model; performance

Passive and active adaptive management: Approaches and an example - Williams, Byron (Journal of Environmental Management, 2010)
Adaptive management is a framework for resource conservation that promotes iterative learning-based decision making. Yet there remains considerable confusion about what adaptive management entails, and how to actually make resource decisions adaptively. A key but somewhat ambiguous distinction in adaptive management is between active and passive forms of adaptive decision making. The objective of this paper is to illustrate some approaches to active and passive adaptive management with a simple example involving the drawdown of water impoundments on a wildlife refuge. The approaches are illustrated for the drawdown example, and contrasted in terms of objectives, costs, and potential learning rates. Some key challenges to the actual practice of AM are discussed, and tradeoffs between implementation costs and long-term benefits are highlighted.

Partnering with communities to co-design humanitarian health strategies: A SeeChange CommunityFirst Framework for implementation in MSF projects - Chapela Trillo, Violeta; Farber, Jessica (MSF, 2034)
The CommunityFirst Framework is intended to be implemented by field teams at MSF. The theoretical aspects and evidence presented on the importance of community engagement are intended for all MSF staff seeking to learn more about why and how to shift the way we work with communities as humanitarians. We believe this guideline, and other tools like it (including OCA’s Person-Centred Approach Guidance07, and MSF Vienna Evaluation Unit’s Guidance for Involving Communities08), to be an important contribution to the growing movement of communities and humanitarian actors who are pushing for changes in the humanitarian system that translate to dignity, health, justice, equity and self-determination for communities around the world. Specifically, the CommunityFirst Framework is intended to guide MSF teams to co-design health strategies with communities, throughout all stages of the project cycle, for exploratory missions, projects that are just opening, projects that have been running for some time, or those that are closing. At the time of publication, the CommunityFirst Framework has been tested in pilot projects in: (1) Madre de Dios, Peru (MSF OCP, August 2022), (2) Tonkolili, Sierra Leone (MSF OCA, November 2022) and (3) Anzoátegui, Venezuela (MSF OCB, February 2023) The experiences from these pilots (feedback from teams, implementation results, adaptations to each context, etc.) have informed the adaptation of the Framework. CommunityFirst builds on existing community engagement work inside MSF and contributes a practical framework for co-designing health initiatives with communities. To avoid duplicating efforts and resources around community engagement inside MSF, the appendices in this guideline largely refer to already existing MSF resources.09 This guideline is meant to be a living document that can evolve and be adapted given the experience of MSF staff and community members and diverse community contexts. This guide can be used by anyone in MSF who is interested in partnering with communities to improve the responsiveness and impact of their humanitarian programs. This is the first iteration of the document. Subsequent iterations will be published based on additional testing during future phases of the CommunityFirst TIC project.

Participatory Systemic Inquiry - Burns, Danny (2012)
This article explores Participatory Systemic Inquiry processes through two examples of practise. The first is about embedding public engagement in UK higher education, the second is about water infrastructure development and local capacity development in small towns situated around Lake Victoria. These examples illustrate why it is necessary to understand the wider systemic dynamics within which issues are situated, and how this helps to identify workable and sustainable solutions to problems. It describes the learning architectures which were constructed to hold the local and thematic inquiries and then to extend them. It also demonstrates the methods which operationalised these processes and explores some of the methodological differences between this approach and other approaches to qualitative and participatory research.

Participatory Rural Appraisal - Better Evaluation (Better Evaluation)
"Participatory Rural Appraisal (PRA) recently renamed Participatory Learning for Action (PLA), is a methodological approach that is used to enable farmers to analyse their own situation and to develop a common perspective on natural resource management and agriculture at village level. PRA is an assessment and learning process that empowers farmers to create the information base they need for participatory planning and action. Outsiders contribute facilitation skills and external information and opinions. Many different tools have been developed for use in PRA. There are four main classes: tools used in group and team dynamics; tools for sampling; options for interviews and dialogue; and options for visualisation and preparing diagrams. Most countries have had some experience with PRA and local publications are available. IIED regularly reports on new developments in its PLA notes (Pretty et al 1995)." (Bie, 1998)

Participatory Monitoring & Evaluation: Learning from change - Guijt, Irene; Gaventa, John (IDS Policy Briefing, 1998)
Development organisations need to know how effective their efforts have been. But who should make these judgements, and on what basis? Usually it is outside experts who take charge. Participatory monitoring and evaluation (PM&E) is a different approach which involves local people, development agencies, and policy makers deciding together how progress should be measured, and results acted upon. It can reveal valuable lessons and improve accountability. However, it is a challenging process for all concerned since it encourages people to examine their assumptions about what constitutes progress, and to face up to the contradictions and conflicts that can emerge.

Participatory evaluation. What is it? Why do it? What are the challenges? - Zukoski, A.; Luluquisen, Mia (Community-based public health policy & practice, 2002)
Yogi Berra was right when he remarked, “If you don’t know where you’re going, you’ll wind up somewhere else.” When we collectively apply our hopes and energies to improving our communities, how do we know if we’re making the right choices along the way? How will we notice when we are spinning our wheels and what changes to make? How will we know that we are making a difference? And how will we know what success looks like to everyone in the group? By taking a community-based public health approach to our work, we create an opportunity to engage in a particular type of evaluation—participatory evaluation – that can help answer those questions. For those groups that are interested in this approach, this policy brief discusses the key concepts of participatory evaluation and some tips for applying it. We also present some real-life examples from two evaluators who work with grantees of the Partnership for the Public’s Health Initiative.

Participatory evaluation design and sense-making - making meaning through co-constructed lenses - King, Julian; Elliott, Kerry; Hollingsworth, Hilary (Australian Evaluation Society International Evaluation Conference, 2023)
Based on a presentation at Australian Evaluation Society International Evaluation Conference, Brisbane, 28 September 2023. The importance of participatory approaches to evaluation is well-understood. Involving the right people and balancing voices at the stakeholder table can help to make evaluations more valid, credible, useful, and actually used. Not all evaluations, however, are done in participatory ways. This presentation shares the experience of three evaluators in two recent projects with clients and stakeholders who were unaccustomed to participatory evaluation. In both cases, new ways of working were introduced, requiring participants to come into the evaluation tent, collaborate and learn with us. Engagement processes deliberately focused on incrementally building deep stakeholder input into evaluation design - including key evaluation questions, context analysis, program theory, rubrics, methods of data gathering, and reporting formats. The resulting evaluation frameworks represent an agreed, shared set of lenses for making sense of the evidence. Through this process, relationships and trust were cemented, stakeholders understood the evaluation process and endorsed the resulting frameworks. The client and sector have become more interested, more engaged stakeholders and more savvy evaluation participants and consumers. As the evaluations progress, the evaluation design builds in ongoing stakeholder involvement in reviewing emergent footprints of system change and making evaluative judgements. Participatory approaches are many, varied, and flexible. We will share our reflections and generalisable learning from the participatory approaches we applied in these evaluations. We will invite the audience to reflect on their own practices and experiences, and provoke them to consider whether participatory evaluation is 'just' a methodological option or a foundational element of good evaluation.

Participatory Evaluation - Better Evaluation (Better Wvaluation)
Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study. A participatory approach can be taken with any impact evaluation design, and with quantitative and qualitative data. However, the type and level of stakeholder involvement will necessarily vary between different types, for example between a local level impact evaluation and an evaluation of policy changes (Gujit 2014, p.1). It is important to consider the purpose of involving stakeholders, and which stakeholders should be involved how, in order to maximise the effectiveness of the approach.

Participatory design with marginalized people in developing countries: Challenges and opportunities experienced in a field study in Cambodia - Hussain, Sofia; Sanders, Elizabeth B.-N.; Steinert, Martin (International Journal of Design, 2012)

Participatory Budgeting in Scotland: An overview of strategic design choices and principles for effective delivery - Escobar, Oliver; Harkins, Chris (2015)
Description

Participatory Budgeting as if Emancipation Mattered - Baiocchi, Gianpaolo; Ganuza, Ernesto (Politics & Society, 2014)
Participatory Budgeting has by now been widely discussed, often celebrated, and is now instituted in at least 1,500 cities worldwide. Some of its central features—its structure of open meetings, its yearly cycle, and its combination of deliberation and representation—are by now well known. In this article, however, we critically reflect on its global travel and argue for more careful consideration of some of its less well-known features, namely the coupling of the budgeting meetings with the exercise of power. We disaggregate PB into its communicative and empowerment dimensions and argue that its empowerment dimensions have usually not been part of its global expansion—and this is cause for concern from the point of view of emancipation. We thus discuss the specific institutional reforms associated with empowerment in the original version as well as its analytic dimensions. We also address some of the specific dangers of a communication-only version of PB as well as some suggestions for reintroducing empowerment.

Participatory approaches: a facilitator's guide - VSO (VSO, 2004)
This book provides a set of guidelines for people who will be involved in participatory processes and projects with specific design focus on VSO (Voluntary Services Overseas) volunteers. It looks at appropriate levels of participation; pitfalls of participatory approaches (PA); best practice in facilitation; and tools for participation. The guide is organised into three parts (I) Principles, (II) Methods and (III) Toolkit. Part I gives a background to PA with a comparative analysis of PA in relation to top-down approaches, and within the range of PAs; looks at the role PA in VSO; discusses how to facilitate participatory processes with multiple stakeholders; presents a framework for PA on different levels of participation; and examines key facilitation skills needed to support participatory activities. Part II collates a range of participatory methods that have been used successfully in the field by VSO volunteers. Methods are categorised according to this suitability for use at different stages of a project process. Examples are also given of methods that can be used for specific purposes, such as participatory organisational appraisal and gender/diversity analysis. Part III gives tips on how to choose the most appropriate tool and how to organise participatory workshops and small group activities. It also systematically records a range of tools used by development workers all over the world with reference to what tool is appropriate in what situation. A profile of each tool includes guidelines on its purpose, potential applications and variations, as well as possible pitfalls. Illustrative case studies taken form real experiences of development workers in the field are also included.

Participatory Approaches - Guijt, Irene (Methodological Briefs: Impact Evaluation, 2014)
This guide, written by Irene Guijt for UNICEF, looks at the use of participatory approaches in impact evaluation. Using participatory approaches means involving stakeholders, particularly those affected by intervention, in the evaluation process. This includes involvement in the design, data collection, analysis, reporting, and management of the study. Excerpt "By asking the question, ‘Who should be involved, why and how?’ for each step of an impact evaluation, an appropriate and context-specific participatory approach can be developed. Managers of UNICEF evaluations must recognize that being clear about the purpose of participatory approaches in an impact evaluation is an essential first step towards managing expectations and guiding implementation. Is the purpose to ensure that the voices of those whose lives should have been improved by the programme or policy are central to the findings? Is it to ensure a relevant evaluation focus? Is it to hear people’s own versions of change rather than obtain an external evaluator’s set of indicators? Is it to build ownership of the UNICEF programme? These, and other considerations, would lead to different forms of participation by different combinations of stakeholders in the impact evaluation."

Participatory adaptive management leads to environmental learning outcomes extending beyond the sphere of science - Fujitani, Marie; McFall, Andrew; Randler, Christoph; Arlinghaus, Robert (Science Advances, 2017)
Resolving uncertainties in managed social-ecological systems requires adaptive experimentation at whole-ecosystem levels. However, whether participatory adaptive management fosters ecological understanding among stakeholders beyond the sphere of science is unknown. We experimentally involved members of German angling clubs (n = 181 in workshops, n = 2483 in total) engaged in self-governance of freshwater fisheries resources in a large-scale ecological experiment of active adaptive management of fish stocking, which constitutes a controversial management practice for biodiversity and ecosystem functioning when conducted inappropriately. The collaborative ecological experiments spanned several years and manipulated fish densities in 24 lakes with two species. In parallel, we experimentally compared changes in ecological knowledge and antecedents of proenvironmental behavior in stakeholders and managers who were members of a participatory adaptive management treatment group, with those receiving only a standard lecture, relative to placebo controls. Using a within-subjects pretest-posttest control design, changes in ecological knowledge, environmental beliefs, attitudes, norms, and behavioral intentions were evaluated. Participants in adaptive management retained more knowledge of ecological topics after a period of 8 months compared to those receiving a standard lecture, both relative to controls. Involvement in adaptive management was also the only treatment that altered personal norms and beliefs related to stocking. Critically, only the stakeholders who participated in adaptive management reduced their behavioral intentions to engage in fish stocking in the future. Adaptive management is essential for robust ecological knowledge, and we show that involving stakeholders in adaptive management experiments is a powerful tool to enhance ecological literacy and build environmental capacity to move toward sustainability. Participatory active adaptive management leads to far-reaching environmental learning outcomes in stakeholders. Participatory active adaptive management leads to far-reaching environmental learning outcomes in stakeholders.

Participatory action research - Cornish, Flora; Breton, Nancy; Moreno-Tabarez, Ulises; Delgado, Jenna; Rua, Mohi; de-Graft Aikins, Ama; Hodgetts, Darrin (Nature Reviews Methods Primers, 2023)
Participatory action research (PAR) is an approach to research that prioritizes the value of experiential knowledge for tackling problems caused by unequal and harmful social systems, and for envisioning and implementing alternatives. PAR involves the participation and leadership of those people experiencing issues, who take action to produce emancipatory social change, through conducting systematic research to generate new knowledge. This Primer sets out key considerations for the design of a PAR project. The core of the Primer introduces six building blocks for PAR project design: building relationships; establishing working practices; establishing a common understanding of the issue; observing, gathering and generating materials; collaborative analysis; and planning and taking action. We discuss key challenges faced by PAR projects, namely, mismatches with institutional research infrastructure; risks of co-option; power inequalities; and the decentralizing of control. To counter such challenges, PAR researchers may build PAR-friendly networks of people and infrastructures; cultivate a critical community to hold them to account; use critical reflexivity; redistribute powers; and learn to trust the process. PAR’s societal contribution and methodological development, we argue, can best be advanced by engaging with contemporary social movements that demand the redressingl of inequities and the recognition of situated expertise.

Participation Schemas: a tool to characterize collaborative participation - Prieto-Martín, Pedro (PDD2014, Contemporary difficulties and future prospects for Participatory and Deliberative Democracy, NewCastle, 2014)

Participation Makes A Difference: But Not Always How And Where We Might Expect - Gaventa, John (Development Outreach, 2011)

Participation in ICT Development Interventions: Who and How? - Thapa, Devinder; Sæbø, Øystein (The Electronic Journal of Information Systems in Developing Countries, 2016)
The aim of participatory development (PD) in the context of using Information and Communication Technologies (ICT) for development (ICT4D) is to empower underprivileged communities and disadvantaged segments of the stakeholders. The literature on ICT4D is replete with empirical evidence showing that ICT interventions often fail since they are often externally initiated, with very limited involvement from the affected (Heeks, 2002). Clearly, the principles and concepts of PD are relevant to ICT4D. However, we should not consider PD a panacea but need to understand the caveats and processes by which PD happens. Questions to ask include: What are the various challenges in PD? Who are the relevant stakeholders? Why and how do actors enrol in the project? How do we create sustainable ICT4D projects through PD? To understand these research questions, we present a case analysis of a project in Nepal called the Nepal Wireless Networking Project (NWNP). Investigating the specific initiatives that they enabled, telemedicine, education and jobs, we propose that the key participants in the NWNP were activist actors and the affected and that activists drew upon existing Social Capital to enrol the affected through a process explained by Actor Network Theory (ANT). In the process, they built other forms of Social Capital, which in turn extended the benefits of PD to several mountain villages.

Participation for Humanitarian Innovation - Toolkit - Smith, Amy; Thompson, Martha; Saida Benhayoune; Crespo Cardona, Omar (Elrha, 2023)
A resource designed to help organisations, teams and individuals manage innovation journeys responsibly and successfully. We have partnered with MIT D-Lab to develop a new resource to drive greater diversity and inclusion within project design and implementation. The Participation for Humanitarian Innovation (PfHI) toolkit sets out a robust approach to setting expectations for and monitoring the degree of participation within research and innovation projects for, with, and by people affected by crisis. The PfHI toolkit is composed of five tools: Opportunity Adviser: Identify and prioritise the desired benefits of participation while weighing the potential barriers. Participation Matrix: Agree on the precise degree of participation to target at a given project stage. Resource Navigator: Select tools and processes to address the needs of stakeholders, the project objectives and context. Quality Guidance: Ensure engagements are delivered to the highest standard. Assessment Matrix: Collectively evaluate the degree of participation achieved during an activity or project phase, learn and adapt. By applying these tools before, during and after a research and/or innovation project, implementors can ensure that stakeholders/end-users are included and participating at the highest possible degree. We encourage users of the PfHI toolkit to consider how else to integrate the tools into existing practices. For instance, the Evaluation Matrix could be used to supplement existing MEAL activities to: Establish baselines to measure changes in participation over time. Track the degree of participation/engagement of stakeholders. Regularly assess the degree of participation. Seek feedback from participants about the degree of their participation. Monitor the progress of activities related to stakeholder engagement. Our Participation for Humanitarian Innovation toolkit represents our ongoing commitment to responsible research and innovation across our portfolio of grants and for the humanitarian sector more broadly. We hope you will consider downloading and using the toolkit on your next project.

Participation for Humanitarian Innovation - Background Paper - Smith, Amy; Thompson, Martha; Saida Benhayoune; Crespo Cardona, Omar (Elrha, 2023)
A resource designed to help organisations, teams and individuals manage innovation journeys responsibly and successfully. We have partnered with MIT D-Lab to develop a new resource to drive greater diversity and inclusion within project design and implementation. The Participation for Humanitarian Innovation (PfHI) toolkit sets out a robust approach to setting expectations for and monitoring the degree of participation within research and innovation projects for, with, and by people affected by crisis. The PfHI toolkit is composed of five tools: Opportunity Adviser: Identify and prioritise the desired benefits of participation while weighing the potential barriers. Participation Matrix: Agree on the precise degree of participation to target at a given project stage. Resource Navigator: Select tools and processes to address the needs of stakeholders, the project objectives and context. Quality Guidance: Ensure engagements are delivered to the highest standard. Assessment Matrix: Collectively evaluate the degree of participation achieved during an activity or project phase, learn and adapt. By applying these tools before, during and after a research and/or innovation project, implementors can ensure that stakeholders/end-users are included and participating at the highest possible degree. We encourage users of the PfHI toolkit to consider how else to integrate the tools into existing practices. For instance, the Evaluation Matrix could be used to supplement existing MEAL activities to: Establish baselines to measure changes in participation over time. Track the degree of participation/engagement of stakeholders. Regularly assess the degree of participation. Seek feedback from participants about the degree of their participation. Monitor the progress of activities related to stakeholder engagement. Our Participation for Humanitarian Innovation toolkit represents our ongoing commitment to responsible research and innovation across our portfolio of grants and for the humanitarian sector more broadly. We hope you will consider downloading and using the toolkit on your next project.

Participation by the poor in Luang Prabang tourism economy: current earnings and opportunities for expansion - Ashley, Caroline (ODI, 2006)

Participation by All - The keys to sustainability of a CRS food security project in Niger - Cekan, Jindra; Kagendo, Rutere; Towns, Alexandra (Valuing Voices, 2016)
This project that ran from 2006-2012 in Niger and was implemented by three NGOs: CRS, Cooperative for Assistance and Relief Everywhere (CARE), and Helen Keller International (HKI) under the direction of United States Agency for International Development (USAID) Office of Food for Peace (FFP) as a multi-year assistance program (MYAP) to support food security activities in the Dosso, Tahoua, and Zinder regions. PROSAN focused on increasing agricultural production and agro-enterprise, improving household health and nutrition status, reinforcing the capacities of health agents, and enhancing community resiliency. Here are the highlights from the report which itself is an excerpt from a longer analysis we did. Also please note one Annex highlights the similarlties/ differences we found to USAID/ FFP’s 4 elements of sustainability: AIM, METHODS, AND RESEARCH QUESTIONS The aim of this sustainability evaluation was to explore perceptions of sustainability from Nigeriens involved in PROSAN, former CRS staff and donors. It focused on evaluating participants’ adherence to project outcomes and their creation of new innovations. It also evaluated partners’ involvement in sustaining project outcomes. This evaluation used qualitative and quantitative methods including community mapping, focus group discussions, beneficiary interviews, and key stakeholder interviews. The evaluation was carried out in six communities in the Dosso region, with more than 500 interviewees, focusing on the following research questions: Sustainability of activities and groups: Are the communities sustaining the activities three to five years after the end of the project? What can we learn from the communities and their post-project implementation partners? Spread and unexpected outcomes: If the project was considered a success in the eyes of the community, how well did it spread? Fostering Sustainability: What are the long-term prospects for continued sustainability? FINDINGS Three years after PROSAN’s conclusion, the project was considered a success by community members, national partners, the implementer (CRS), and donor (USAID) staff. The main findings include: 1. SUSTAINABILITY OF ACTIVITIES AND GROUPS Eighty percent (80%)[*] of all activities were reported to have become self-sustained and community innovations have emerged: On average, households reported moving from being food secure for 3-6 months per year during PROSAN to 8-12 months at the time of this evaluation, which is a remarkable impact [1]. CRS_Niger_PROSAN_Sustainability_Evaluation_pdf [1] Women reported greater income through the increase in sales of food that was produced and processed due to the grain mills [1]. Respondents also reported improved household health, hygiene, and nutrition, with 91% of survey respondents indicating that their health and sense of well being had improved, especially through the efforts of the health posts and clinics that CRS helped build and the government of Niger’s efforts in sustaining them with resources and staff [1]. Community groups/committees have continued and are well-supported by NGO partners: 81% of the committees set up by PROSAN were functioning at the time of this evaluation, with many participants discussing ways to sustain best practices within their communities, and members still receiving regular trainings or updates [1]. Several new and refresher trainings come through national partners, NGOs, and new channels such as radio programs [1]. Some new NGOs and international organizations have built upon PROSAN’s success, for instance, by using land previously managed by PROSAN for a new vegetable gardening training program, building hygiene programs on past health awareness efforts, or extending agricultural credit for further inputs [1]. Twenty percent (20%) of implemented activities were not sustained or have stagnated: While hygiene practices were sustained by households and there was widespread latrine construction, sanitation was poor in the villages, and most latrines had fallen into disrepair [1]. Fewer than 50% of women reported practicing exclusive breastfeeding for children less than six months of age [1]. While almost half of all health committees no longer exist, new health clinics staff have replaced some of the work of the committees with health and agricultural promotion messages now being sent via radio, television, and cell phones [1]. Literacy training and theater groups have completely ceased [1]. With the exception of the Système Communautaire d’Alerte Précoce-Réponses aux Urgences’ (SCAP-RU) SCAP-RU early warning system which has expanded, other resilience activities such as roadwork and caring for the environment are a lesser priority due in part to the lack of food and cash-incentives to continue doing them [1]. 2. SPREAD AND UNEXPECTED OUTCOMES New innovations and ceased activities reflected the project’s legacy: Community innovations have emerged such as collective funds paying for cleaners of the new health center, community-imposed sanctions for births occurring outside of the health centers, and the monitoring of savings from well water sales. National partners have praised the project, with many lamenting its withdrawal. One non-PROSAN village told an Agriculture Ministry staff and potential NGO partner that “No one should bring a program here unless it is like PROSAN.” PROSAN-trained masons, well repair technicians, and village youth have learned land recuperation techniques (zai holes, bunds and demi-lunes) that helped generate income beyond project communities. Project activities that received free inputs have largely stopped being implemented once the incentives were withdrawn such as Food for Training (FFT), Food for Work (FFW), or Cash for Work (CFW) (e.g. literacy, seedlings, latrines, theater etc.); nonetheless the inputs were highly valued and have continued to support agriculture and health (carts, bicycles). 3. FOSTERING SUSTAINABILITY The following areas were identified as potential barriers to sustainability that could be systematically explored in other projects: Although most committees are still functioning, there are no processes in place to engage and train youth and new inhabitants of the villages. While village communities have been maintained, there is an increasing lack of ministry resources (e.g., staff, transportation, and communications) to take the place of NGOs like CRS after a program ends. There is little management of knowledge around project data, which is further exacerbated by staff changes in NGOs, government ministries, and donors. Project data (proposal content, monitoring data, evaluation results, participant lists, partner names, and exit agreements) must be managed ethically, locally and be held online, accessible for future projects to use and for villages to conduct self-evaluations.

Participation as assemblage: Introducing assemblage as a framework for analysing participatory processes and outcomes - Stage, Carsten; Ingerslev, Karen (Conjunctions. Transdisciplinary Journal of Cultural Participation, 2016)
The article presents a yet unexplored framework for analysing the multidimensionality and dis/connections of participatory processes and their outcomes by using the concept of the ‘assemblage’ (DeLanda, 2006). The case is an eight-month collaboration between a task force initiated by Central Denmark Region, the socio-economic company Sager der Samler, and citizens. The collaboration is aimed at bringing together and working across various institutional and user perspectives to act on a societal challenge. The analysis is theoretically based on a review of existing theories of participation and typologies for analysing and evaluating participation. In particu- lar, the analysis focuses on the assemblage approach as a way of acknowledging the institutional, affective, mate- rial and power-related complexity of participatory processes. The assemblage approach helps to analytically stress that the process under investigation should be evaluated both with a more traditional focus on decision-making or power allocation, as well as taking into account the social, personal-affective and material benefits produced, and the potential for change in the relationship between public administration and citizens.

Participation and Power in Climate Change Adaptation Policies: Vulnerability in Food Security Programs in Nepal - Nagoda, Sigrid; Nightingale, Andrea J. (World Development, 2017)
The article explores the moments wherein participatory approaches in climate change adaptation (CCA) policies contribute to reinforcing, rather than transforming, the underlying causes of vulnerability. Using the case of food insecure households in the district of Humla in northwestern Nepal, the study demonstrates that the same social and power relations that are driving local vulnerability dynamics, such as caste, gender, and access to social and political networks, also play important roles in shaping the impact of CCA policies. By tracing Nepal’s CCA programs, starting with the local level, through district to international-national level dynamics, the study adds insights into the barriers to exclusion that embed power relations all the way through the chain of policy development. The purpose is to better understand how CCA can perpetuate rather than alleviate the conditions that create differential vulnerability patterns at village level. It raises questions about how whether CCA programs are an adequate response to increasing vulnerability for some of the world’s most marginalized people.

Participation and accountability in development management - Brett, E. A. (The Journal of Development Studies, 2003)
This article critically reviews the role of participatory theory in managing development projects and programmes in poor countries. Participation has emerged in response to global demands for greater individual and social control over the activities of state and private agencies, and especially to the manifest failures of traditional ‘top-down’ management systems in LDCs. Claims made on behalf of these participatory methodologies are critically reviewed and a distinction is drawn between strong and weak versions of the theory. Empirical evidence is then considered to evaluate the effectiveness of these methodologies, using long-standing insights of social science theory to show that participation can succeed for specific kinds of projects and programmes in favourable circumstances, but is unsuitable for many others. It commonly fails in contexts where local conditions make co-operative and collective action very difficult, or where it is manipulated by implementing agencies to justify their own actions or poor performance.

Participación Digital y mejora democrática - un camino sin regreso - Palumbo, Gabriel (Luminate, 2020)
Tal vez como nunca en la historia, la democracia está siendo desafiada. A los viejos problemas, muchos no resueltos, se les agregan otros, surgidos de la incorporación de tecnología, de la emergencia de nuevos actores sociales y de una conformación de la subjetividad política cada vez más compleja e impredecible. Esta investigación pretende reactualizar algunos de los debates clásicos sobre la democracia y, al mismo tiempo, sentar una base empírica que permita pensar de qué modo la tecnología puede ayudar a mejorar la calidad democrática. Según nuestro estudio, la participación digital está promoviendo modificaciones importantes en la relación entre la ciudadanía y los decisores políticos, ampliando los espacios de proximidad. Esto sucede, fundamentalmente, porque baja las barreras de acceso y de incentivos haciendo más fácil, más efectiva y más mensurable la participación ciudadana. La participación digital aumenta el períodos electorales y fomenta el asociacionismo cívico como posibilidad cierta y concreta de presionar sobre los decisores políticos. Lo digital habilita, además, participación offline y genera espacios de pedagogía cívica. La capacidad de implantación de temas en el debate público que se puede generar a través de ella mejora la conversación pública, impactando a su vez la calidad de la democracia. Desde Luminate entendemos que favorecer los espacios de rendición de cuentas desde la ciudadanía hacia el poder político es una apuesta al robustecimiento de la experiencia democrática y que las formas digitales están abriendo posibilidades que otras opciones no pudieron lograr. Esperamos que esta investigación sea un aporte para poder seguir reflexionando y avanzando en esta dirección.

Parliament and the people - Rumbul, Rebecca; Moulder, Gemma; Parsons, Alex (2018)
How digital technologies are shaping democratic information flow in Sub-Saharan Africa

ParEvo - Web-assisted participatory evolution of scenarios - Davies, Rick (2020)
ParEvo is a method of exploring alternative futures or histories, using a participatory evolutionary process (hence ParEvo). The process is designed to be used by multiple people, to produce a branching structure of storylines about what did, or could, happen. Participants are anonymous and can choose the extent to which they collaborate with others. Participants are also able to tag, comment on, and evaluate the storylines that are generated. Completed exercises can be analysed using downloaded data describing the content of storylines and the structure of people’s participation. Outcomes can be both cognitive (e.g. how we think about the future) and behavioural (e.g. how we respond to the future).

Paradoxes and partnerships: a study of knowledge exploration and exploitation in international development programmes - Lannon, John; Walsh, John N. (Journal of Knowledge Management, 2019)
Purpose – This paper aims to look at how organisational partnerships balance knowledge exploration and exploitation in contexts that are rife with paradoxes. It draws on paradox theory to examine the partnership’s response to the explore-exploit relationship.

Paradigms, Poverty and Adaptive Pluralism - Chambers, Robert (IDS, 2010)
This paper explores participatory methodologies (PMs) associated with a paradigm of people, contrasted with a dominant paradigm associated with things.

Pando - Root Change (Root Change, 2020)
Pando is a platform that gives organizations a way to visualize, learn from, and engage with the social systems in which they work. We have designed Pando to help users build trust, strengthen relationships, and work together to achieve greater social impact. Grounded in social network analysis, Pando allows users to collect and visualize organizations and their relationships in real-time on simple and easy-to-use relationship maps. Relationship maps are managed by map administrators. These are actors interested in learning about a particular system and making the tool available to those working in the system. Pando is integrated with Keystone Accountability’s Feedback Commons, an online tool that allows map administrators to collect and analyze qualitative feedback about levels of trust and relationship quality among map participants. Key features of Pando include relationship mapping, feedback surveys, and dashboards.

Overview for Social Sector Leaders: Assessing Innovation Potential for Social Impact - Global Knowledge Initiative (2016)
We aim to provide decision makers with greater insight and confidence into the process of assessing innovation impact potential. Rather than considering the role innovation plays after an investment is made, or based on historic evaluations of how innovation has or has not delivered solutions to a problem, this approach is forward-looking. This customizable toolset assesses the future impact that innovation can deliver in a system to tackle particularly complex problems

Overcoming Premature Evaluation - Roche, Chris (2016)
Guest post from Chris Roche on practical ways of introducing adaptive management, learning from failure etc in aid programmes

OutNav - Evaluate outcomes and impact - Matter of Focus (Matter of Focus, 2021)
Make better use of your data and information to learn, improve and tell an evidenced story of the difference your organisation is making.

Outcome Measurement in Local Governance Programmes: a Power Dimension - McGee, Rosemary; Pettit, Jethro (WIP Paper, 2013)

Outcome mapping: learning brief - Buell, Stephanie; Malallah, Haneen; Mason, Paige (Briefing Paper, 2020)
Adaptive programmes recognise that certain changes, particularly in behaviours, are complex, non-linear and difficult to measure. This briefing note explores the use of outcome mapping (OM) as a monitoring, evaluation and learning (MEL) approach to track behavioural change and inform adaptation for two programmes: the Pathways to Resilience in Semi-arid Economies (PRISE) research consortium and the Accountability in Tanzania programme (AcT I and AcT II). It discusses the implementation of OM, the ways in which it has enabled adaptation and enabling contexts in order to identify key considerations for MEL specialists and programme managers as they determine whether OM may be the right fit, and how best to use the approach. Key messages OM has a number of different benefits as a MEL approach, including unpacking different uses of information at different levels of programme implementation; helping to develop a common language around progress markers; and going beyond monitoring to inform adaptation throughout implementation. These benefits are important aspects of monitoring, evaluation and learning for adaptive management (MEL4AM), as they provide richer evidence for decision-making at a frequency that could mean real-time learning and change. OM works best when it is embedded throughout the organisation, and as part of programme and organisational culture, rather than tasked to a MEL unit or individual.

Outcome mapping for health system integration - Tsasis, Peter; Evans, Jenna; Forrest, David; Jones, Richard Keith (Journal of Multidisciplinary Healthcare, 2013)
Health systems around the world are implementing integrated care strategies to improve quality, reduce or maintain costs, and improve the patient experience. Yet few practical tools exist to aid leaders and managers in building the prerequisites to integrated care, namely a shared vision, clear roles and responsibilities, and a common understanding of how the vision will be realized. Outcome mapping may facilitate stakeholder alignment on the vision, roles, and processes of integrated care delivery via participative and focused dialogue among diverse stakeholders on desired outcomes and enabling actions. In this paper, we describe an outcome-mapping exercise we conducted at a Local Health Integration Network in Ontario, Canada, using consensus development conferences. Our preliminary findings suggest that outcome mapping may help stakeholders make sense of a complex system and foster collaborative capital, a resource that can support information sharing, trust, and coordinated change toward integration across organizational and professional boundaries. Drawing from the theoretical perspectives of complex adaptive systems and collaborative capital, we also outline recommendations for future outcome-mapping exercises. In particular, we emphasize the potential for outcome mapping to be used as a tool not only for identifying and linking strategic outcomes and actions, but also for studying the boundaries, gaps, and ties that characterize social networks across the continuum of care.

Outcome Mapping FAQs - Outcome Mapping Learning Community (Outcome Mapping, 2018)
FAQ resource produced by IDRC's Evaluation Unit, with contributions from Kaia Ambrose, Sarah, Earl, Jan Van Ongevalle and Julius Nyangaga. We've added it here to answer some of your OM related questions.

Outcome Mapping: Building Learning and Reflection into Development Programs - Earl, Sarah; Carden, Fred; Smutylo, Terry; Patten, Michael Quinn (IDRC Books, 2001)
"Outcome Mapping provides not only a guide to essential evaluation map-making, but also a guide to learning and increased effectiveness, and affirmation that being attentive along the journey is as important as, and critical to, arriving at a destination." - Michael Quinn PattonMore and more, development organizations are under pressure to demonstrate that their programs result in significant and lasting changes in the well-being of their intended beneficiaries. However, such "impacts" are often the product of a confluence of events for which no single agency or group of agencies can realistically claim full credit. As a result, assessing development impacts is problematic, yet many organizations continue to struggle to measure results far beyond the reach of their programs. "Outcome Mapping" recognizes that development is essentially about people relating to each other and their environment. The originality of this approach lies in its shift away from assessing the products of a program to focus on changes in behaviour, relationships, actions, and activities in the people, groups, and organizations it works with directly. In doing so, "Outcome Mapping" debunks many of the myths about measuring impact. It will help a program be specific about the actors it targets, the changes it expects to see, and the strategies it employs and, as a result, be more effective in terms of the results it achieves. This publication explains the various steps in the outcome mapping approach and provides detailed information on workshop design and facilitation. It includes numerous worksheets and examples.

Outcome mapping: A method for tracking behavioural changes in development programs - Smutylo, Terry (ILAC Brief, 2005)
This guide published by the Institutional Learning and Change (ILAC) Initiative provides a detailed overview of using outcome mapping as an evaluation tool. Contents Expressing results as changes in behaviour Outcome mapping terms The three stages of outcome mapping Figure 1. The three stages and twelve steps of outcome mapping Stage 1. Intentional design Figure 2. The four basic questions of the intentional design stage Stage 2. Outcome and performance monitoring Stage 3. Evaluation planning Using outcome mapping The Ceja Andina Project The Agro-industry and Market Development project for Arracacha

Outcome Mapping - Better Evaluation (Better Evaluation, 2013)
Outcome mapping (OM) is a methodology for planning, monitoring and evaluating development initiatives in order to bring about sustainable social change. As the name suggests, its niche is understanding outcomes; the so-called ‘missing-middle’ or ‘black box’ of results that emerge downstream from the initiative’s activities but upstream from longer-term economic, environmental, political or demographic changes. At the planning stage, the process of outcome mapping helps a project team or program be specific about the actors it intends to target, the changes it hopes to see and the strategies appropriate to achieve these. For ongoing monitoring, OM provides a set of tools to design and gather information on the results of the change process, measured in terms of the changes in behaviour, actions or relationships that can be influenced by the team or program.

Outcome Harvesting. Understanding social change results in complex circumstances - outcomeharvesting.net (2018)
This website is a source of applications, events and resources to support the development of a community of practitioners of Outcome Harvesting. BetterEvaluation is one of three sister sites of OutcomeHarvesting.net

Outcome Harvesting - Wilson-Grau, R. (Better Evaluation, 2015)
Outcome Harvesting collects (“harvests”) evidence of what has changed (“outcomes”) and, then, working backwards, determines whether and how an intervention has contributed to these changes. Outcome Harvesting has proven to be especially useful in complex situations when it is not possible to define concretely most of what an intervention aims to achieve, or even, what specific actions will be taken over a multi-year period.

Outcome Harvesting - Wilson-Grau, Ricardo; Britt, Heather (Ford Foundation, 2012)
Outcome Harvesting was developed by Ricardo Wilson-Grau with colleagues Barbara Klugman, Claudia Fontes, David Wilson-Sánchez, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz, and Wolfgang Richert. Over the past 8 years, Outcome Harvesting and has been used to monitor and evaluate the achievements of hundreds of networks, non-governmental organizations, research centers, think tanks, and communitybased organizations around the world. This brief is intended to introduce the concepts and approach used in Outcome Harvesting to grant makers, managers, and evaluators, with the hope that it may inspire them to learn more about the method and apply it to appropriate contexts. Thus, it is not a comprehensive guide to or explanation of the method, but an introduction to allow evaluators and decision makers to determine if the method is appropriate for their evaluation needs. Where possible, we have included examples to illustrate how Outcome Harvesting is applied to real situations. For each case story, organizations were asked to provide a description of the outcome and a summary of the role played by the organization. Sometimes they added other information such as the outcome’s significance. Some details and identifiers were redacted for confidentiality purposes. A draft of this brief was graciously commented on by Bob Williams, Fred Carden, Sarah Earl, Richard Hummelbrunner and Terry Smutylo. The final text is, of course, the sole responsibility of the authors and editor.

Our priorities in international assistance - GAC (GAC, 2018)
For Global Affairs Canada, development innovation includes new or improved business models, policy practices, approaches, technologies, behavioral insights or ways of delivering products and services that benefit and empower the poorest and most vulnerable people in developing countries.

Our Data, Our Process: Re-imagining the Community-Led Monitoring and Evaluation Discussion Samahan para sa Pagsulong at Pag -unlad ng Capaculan (SPPC), Nancy Bobadilla, Gunjan Veda - SPPC, Samahan para sa Pagsulong at Pag -unlad ng Capaculan; Bobadilla, Nancy; Veda, Gunjan (Asia Pacific Journal of Evaluation, 2024)

Organizational Performance Index (OPI) Handbook: A practical guide to the OPI tool for practitioners and development professionals - Pact (Pact, 2015)
This handbook is intended to help development practitioners understand how to use and what can be achieved by Pact’s Organizational Performance Index (OPI) tool. This document will provide practitioners and managers with information on how the Pact OPI works and how best to incorporate it in their country strategy or program.

Organizational Learning: A Theory of Action Perspective - Argyris, Chris; Schon, Donald A. (Addison Wesley, 1978)