Monitoring and Evaluation

From microLINKS Wiki

Contents 1 Introduction 2 The Role of the Causal Model in Performance Monitoring and Impact Assessment 3 What Constitutes Good Performance Monitoring? 4 What Constitutes A Rigorous Impact Assessment? 5 What Are the Steps in Implementing an Impact Assessment? 6 Where Can I Find Additional Resources? 7 Navigation

Introduction

As its name implies, monitoring and evaluation (M&E) consists of two basic components: performance monitoring and evaluation (see table below). Monitoring and evaluation serve distinct purposes.

Performance monitoring involves the tracking of project outputs and outcomes as indicators of project effectiveness, or the extent to which the project achieves its stated objectives. Performance monitoring is primarily a management tool. It is akin to a roadmap that provides information indicating whether the project is on the right road to its intended destination (or intended impact) and, if not, how to get back on to the right road. Performance monitoring does not (indeed cannot) tell the project whether it has arrived at its final destination. For this, a statistically valid control group is required, and few performance monitoring systems include a valid control group. Performance monitoring tends to focus on outputs and outcomes (the left-hand side of the causal chain) and on those indicators that are easier to capture and measure on an on-going basis (see figure below).

Monitoring	Evaluation
Performance Monitoring: The tracking of project outputs and outcomes as indicators of project effectiveness.	Project Evaluation: The evaluation of project implementation as measured against the project's scope of work, deliverables, personnel requirements, etc. Not intended to attribute impact to project operations. Impact Assessment: Analysis of whether project objectives (outcomes and impacts) were achieved and can be attributed to project operations. Requires a counterfactual.

Evaluation in turn consists of project evaluation and impact assessment. Project evaluation refers to an evaluation of project implementation, as measured against the project’s scope of work, deliverables, personnel requirements, and so forth. Project evaluations may include a baseline and follow-up but typically include a single observation point, often at or near the end of the project.

Project evaluations may also include an assessment of project effectiveness, but like performance monitoring, do not include a valid control group and thus cannot be used to attribute outcomes or impacts to project operations. Project evaluations, nonetheless, can produce useful findings on project operations, and if designed and timed appropriately, can be used to guide management decision-making. End (or near end)-of-project evaluations, however, tend to be useful primarily for external stakeholders.

Like performance monitoring, project evaluation tends to focus on the left-hand side of the casual chain, although often with a heavier emphasis on project activities and the organizational structures, policies, systems, and day-to-day operations that drive project results. A specific type of project evaluation is the so-called "process evaluation," which assesses whether the project was implemented--including the procedures undertaken, the decisions made, and the services delivered--as intended. By documenting the project's development and operation, the process evaluation uncovers reasons for successful or unsuccessful performance, and provides information for potential replication.

Impact assessment is an evaluation whose purpose is specifically to attribute outcomes and impacts to project operations.

The Role of the Causal Model in Performance Monitoring and Impact Assessment

All value chain development projects are based on a causal model showing the causal (or logical) links between project activities and expected outputs, outcomes and impacts. Underlying the links in the causal model is a set of theorized causal relationships that project designers believe to be true. The importance of the causal model for performance monitoring and impact assessment is that it forces project administrators and evaluators to articulate the critical causal relationships underlying project design and evaluate the degree to which they make sense and/or are justified.

What Constitutes Good Performance Monitoring?

Performance monitoring consists of a number of related tasks. Chief among them is the selection of "key performance indicators" that allow the management to monitor project performance over time.

Monitoring key performance indicators does not by itself provide sufficient information to evaluate and assess project performance. It typically needs to be supplemented with other quantitative and qualitative data collection methods so as to understand the background drivers for the trends and results revealed by the key performance indicators. Useful data collection methods include key informant interviews, focus group discussions, small-scale targeted surveys, market scanning, secondary research, or rapid assessments. By utilizing a "tool box" of quantitative and qualitative data gathering methodologies that complement and mutually reinforce each other, projects can "triangulate" to a greater understanding of their effectiveness.

Performance monitoring, however, entails more than data collection. Data collection needs to be embedded within a "system." Implied by the word "system" is a process for transforming data into useful information. Included in this process are a number of tasks that must be performed if the system is to operate efficiently and effectively; tasks that include, among other things, the reporting, management, analysis, dissemination, verification, and use of data. A breakdown in any of these tasks will compromise the validity and usefulness of the performance monitoring system.

In developing a performance monitoring system, value chain projects can follow a set of widely validated best practices that includes matching system design to resources and technical capacities, training, participation, pilot testing, and oversight and monitoring.

Once the performance monitoring system has been finalized, the details should be captured in a series of Performance Indicator Reference Sheets (PIRS) for each key performance indicator. The PIRS is a summary resource that describes how the performance monitoring system is operationalized.

What Constitutes A Rigorous Impact Assessment?

Impact assessment rigor is determined by the following four criteria: internal validity, external validity, construct validity, and statistical conclusion validity. For more on this topic, please click here.

1. Internal validity is the extent to which the impact assessment establishes a credible counterfactual. Internal validity can be suspect when certain types of biases in the design or conduct of the impact assessment could have affected observed results, thereby obscuring the true direction, magnitude, or certainty of the treatment effect. Selection bias is a primary source of bias. Selection bias occurs when there are systematic differences in observable (e.g., gender, education, climate, market access)and unobservable (e.g., ambition, risk orientation, entrepreneurial spirit) characteristics between the treatment and control groups.
2. External validity is the extent to which the impact assessment findings are generalizable to other value chain projects.
3. Construct validity is the extent to which the impact assessment design and data collection instruments accurately measure the project's causal model.
4. Statistical conclusion validity means that the researchers have correctly applied statistical methods and identified the statistical strength/certainty of the results.

Impact assessment rigor further depends on a variety of other factors that need to be incorporated into the assessment design, implementation, and analysis, including triangulation, methodological transparency, sound data collection methods, and methodological appropriateness.

For more on impact assessment methodologies, see the Impact Assessment Primer Series article #2, “Methodological Issues in Conducting Assessments of Private Sector Development Programs”and Primer Series article #3 “Collecting and Using Data for Impact Assessment.”

What Are the Steps in Implementing an Impact Assessment?

Conducting a good impact assessment of a value chain project involves the following steps (the steps assume two research rounds--a baseline and follow-up):

1. Select the Project(s) to be Assessed.
2. Conduct an Evaluability Assessment.
3. Prepare a Research Plan.
4. Contract and Staff the Impact Assessment.
5. Carry out the Field Research and Analyze its Results.
6. Disseminate the Impact Assessment Findings.

The Private Sector Development Impact Assessment Initiative (PSD-IAI) team is conducting a series of impact assessments on four USAID private sector development projects to demonstrate and refine this approach to impact assessment:

• Brazil: Micro and Small Enterprise Trade-Led Growth Program
• India:Growth-oriented Micro Enterprise Development (GMED) Program
• Kenya: Kenya BDS and Kenya Horticultural Development Programs
• Zambia: PROFIT Program

Another impact assessment of the USAID-funded Development of a BDS Market in Rural Himalayas project was conducted in 2007.

Where Can I Find Additional Resources?

The PSD-IAI is developing, testing, and publishing guidelines for credible impact assessments. To further this agenda, PSD-IAI is publishing an Impact Assessment Primer Series that explains many of the key concepts and operational steps summarized above:

1. IA Primer Number 1: Assessing the Impact of New Generation Private Sector Development Programs
2. IA Primer Number 2: Methodological Issues in Conducting Impact Assessments of Private Sector Development Programs
3. IA Primer Number 3: Collecting and Using Data for Impact Assessment
4. IA Primer Number 4: Developing a Causal Model for Private Sector Development Programs
5. IA Primer Number 5: Causal Models as a Useful Program Management Tool: Case Study of PROFIT Zambia
6. IA Primer Number 6: Planning for Cost Effective Evaluation with Evaluability Assessment
7. IA Primer Number 7: Common Problems in Impact Assessment Research

The PSDIAI has recently published two additional papers:

• Assessing the Effectiveness of Economic Growth Programs
• Time to Learn: An Evaluation Strategy for Revitalized Foreign Assistance

Navigation

Go to Value Chain Framework

Go to Value Chain Analysis

Go to Value Chain Development main page

Go back to the top