Standard 18: Learning from and leveraging results during close-out.
Engage participants, partners, donors, host or local governments and other stakeholders in project close-out evaluation and reflection activities with the intent of learning from and leveraging the project.
Conduct a final evaluation or after-action review which engages stakeholders in interpreting results and generating learning.
-
Why
End of project final evaluationA final evaluation systematically assesses a project’s relevance, efficiency, effectiveness, impact and sustainability on a defined population and draws from data collected via the monitoring system, as well as any other more detailed data. A final evaluation usually includes an endline survey to allow for comparison with baseline data, if available, of relevant project indicators. and after-action reviewAn after-action review is a reflection on the successes and failures of a project (or activities within a project). An end of project after-action review brings together a team to discuss a recently completed project, in an open and honest fashion, and does not require the collection of additional data from project participants (After-action review definition adapted from Better Evaluation). processes focus on engaging key stakeholders, including community members, CRS, and partners, in understanding overall project, and generating and documenting learning, to help CRS and partners design better projects in future. An effective project final evaluation or after-action review process:
- Answers key questions about project relevance, effectiveness, efficiency, impact and sustainability.Note that these are standard project evaluation criteria. Not all will be applicable to project after-action reviews.
- Brings in a range of stakeholder perspectives to challenge bias and interpret findings.
- Creates a space for CRS and partners to jointly identify and document lessons learned including regarding the project theory of change.
- Generates useful information that benefits future projects.
- Facilitates internalization of project learning, thereby increasing stakeholder ability to leverage and use that learning.
-
Who
- Primary responsible: Project manager or chief of party (PM/CoP) and the MEAL lead for the evaluation (with external evaluator if applicable)
- For evaluations, the MEAL leadThe MEAL lead for the evaluation may be the country program MEAL manager, project MEAL coordinator, or other CRS MEAL staff. For internal evaluations, this individual typically leads the CRS evaluation team. For external evaluations especially, this individual often takes on the role of “evaluation manager”, serving as the main point of contact between the project team and generally ensuring the evaluation runs smoothly and is completed in a timely manner. individual designated as the "evaluation manager"The “evaluation manager” serves as the main point of contact between the project team and an external evaluator and focuses on ensuring the evaluation runs smoothly and is completed in a timely manner. For more on the evaluation manager role, see the Managing and Implementing an Evaluation resource. oversees the evaluation process, with support from the PM/CoP. The MEAL lead/evaluation manager (or external evaluation consultant if applicable) also prepares evaluation data for review during the evaluation reflection event.
- The evaluator typically prepares, facilitates, and documents the evaluation reflection event.
- The PM/CoP leads the after-action review or works with a CRS facilitator external to the project team to plan and implement the after-action review.
- Others involved: CRS and partner project team members (including sector specialists, MEAL staff, and operations staff); country program senior management team (SMT); regional technical advisors; country program and/or regional business development (BD) staff; members of the project governance structure; IDEA staff as applicable
- CRS and partner project team members (sector and MEAL staff, as well as operations staffOperations staff participation will vary by project. For example, for projects with significant supply chain management or ICT4D components, staff from these areas will be closely involved in the evaluation. ) participate in the after-action review or final evaluation process.
- Members of the SMT and the project governance structureThe project governance structure is a group that makes decisions about a project that are beyond the project manager’s sole authority. Establishing a project governance structure ensures that the right decisions are taken at the right time and with the right degree of stakeholder participation. The size and composition of project governance structures vary by project, ranging from a very small body comprised of the PM and a project sponsor, to a project board with representation from a variety of stakeholders. may participate in the final evaluation process including the evaluation reflection event.
- Regional technical advisors (MEAL and sectoral) support the project team to interpret final evaluation findings.
- Country program and/or regional BD staffIf there is a country program BD staff, he or she may participate. Regional BD staff may participate where feasible for particularly strategic projects. may participate in the final evaluation reflection event; IDEA staff may also participate in such events for strategic, centrally-funded projects.
-
When
- In the final 6 months of the project, unless otherwise requiredFor example, in some multi-year projects, donor requirements may necessitate conducting the final evaluation at the end of the penultimate year/beginning of the final project year, to align with the timing of project baseline data collection.
- Schedule the final evaluation and reflection event or after-action review in the project close-out plan. Be sure to hold the final evaluation reflection event before finalization of the evaluation report (keep in mind any deadline for report submission).
-
How
This key action is implemented in accordance with CRS’ MEAL Policies and Procedures, specifically Procedures 3.6 and 3.7.Procedure 3.6: Conduct final evaluation to measure and document project progress and contribute to larger agency learning.
Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.Note: While there are many common steps in the process of conducting a final project after-action review and a final project evaluation, the final evaluation process is lengthier and more complex. Given that final evaluations are more common in CRS projects, the guidance below is written from the perspective of a final evaluation. Key adaptations to that guidance and other differences in after-action review processes are highlighted in the after-action review section below.
Follow these steps to ensure an effective, participatory final evaluation or after-action review process:
Final evaluation (including the evaluation reflection event)
- The MEAL lead/evaluation manager (as applicable) and PM/CoP use the Managing and Implementing an Evaluation guidance to support the internal or external evaluator to:
- Finalize all evaluation tools and train data collectors on their use.
- Collect quality data.
- Conduct preliminary data analysis.
For an internal evaluation, a CRS MEAL staff typically leads an internal evaluation team.
- The evaluator (internal or external) works with the PM/CoP and MEAL staff to prepare an agenda, discussion materials, and a facilitation guide for the reflection event to review and interpret evaluating findings and identify learning from the evaluation.
- Use CRS’ Guidance on Monitoring and Evaluation and other resources from MEAL Procedure 3.7.MEAL Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.
- TIP: Identify an individual who can document key discussion points during the event, along with lessons learned, final conclusions and recommendations. While the evaluator is responsible for preparing the report of the reflection event, given the evaluator’s facilitation responsibilities, note-taking support can help with the completeness and accuracy of the reflection report.
- The evaluator facilitates the participatory evaluation reflection event with CRS and partner project staff, in accordance with MEAL Procedure 3.7.MEAL Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.
- The PM/CoP and evaluation manager (as applicable) support the evaluator to finalize the reflection event report.
- Keep the report succinct (ideally three pages or less) and focused on key discussion points related to findings, recommendations, and lessons learned.
- Disseminate the meeting notes or final event report to participants within a week.
- The evaluator prepares a draft of the full evaluation report and any other deliverablesFor example, drafts of communications pieces or other documents CRS can use in sharing evaluation findings with stakeholders (per MEAL Procedure 7.2). While the evaluation scope of work should have detailed CRS’ expectations and preferred formats for these deliverables, review these expectations as the evaluator begins to work on the final report and related deliverables. from the scope of work, shares with the project team for review and feedback, and addresses feedback.
- Ensure the final evaluation report follows the structure agreed in the final evaluation terms of reference (including agreed annexes) and includes a summary of the evaluation methodology and any limitations of the same.
- Ensure the report clearly responds to the evaluation questions, referencing evidence gathered through the evaluation process, highlighting stakeholder inputs, and incorporating learning from the evaluation reflection event.
- Coordinate internal review and feedback on the draft report within a timeframe that allows for timely finalization of the evaluation report. Evaluation report reviewers typically include the PM/CoP, evaluation manager, sector leads/regional technical advisors and the regional MEAL advisor as applicable, the head of programming, and other key CRS and partner project staff.
- If the final evaluation report will be submitted to a donor, time permitting, request a review by an IDEA staff or another CRS staff member with strong understanding of the donor’s requirements and interests.
- If the evaluator does not incorporate CRS’ feedback in the final version of the report, ensure s/he provides justification.
- The PM/CoP shares the final evaluation report with the CRS and partner project team and members of the project governance structure, and integrates end-of-project achievements, key findings and learning into the final project past performance reference (PPR) per MEAL Procedure 8.1Procedure 8.1: Complete end-of-project past performance references (PPRs) to document project learning and post to Gateway. (see Standard 18, key action 5 for guidance on saving the PPR and evaluation and reflection reports to GatewayPer the audit trail requirements for MEAL procedure 3.6, projects and emergency responses must upload evaluation or after-action review reports to Gateway within 90 days of the end of data collection activities. See procedure 3.6 for detailed requirements ).
- The PM/CoP and project team bring the learning from the final evaluation process into end-of-project sharing and dissemination events with project stakeholders (see Standard 18, key action 3 for detailed guidance).
- Click here for Guidance if a final after-action review is required.
While after-action reviews do not involve collection of primary data, steps 2-4 and 6-7 above are still applicable, with the following adaptations and tips:
- The PM and MEAL lead use the customized review questions from the after-action review TOR (see Standard 18, key action 1) to finalize the agenda and develop a facilitation guide and materials for the after-action review event.
- Use the good practices from the MEAL Procedures Narrative (procedures 3.6 and 3.7).
- Do not skip the step of developing a facilitation guide and materials for the after-action review! Too often, project teams take a “check the box” approach to the after-action review and miss the opportunity to deepen project team learning. Adapting the standard after-action review questions to include probing questions or prompts customized to the specifics of the projects is one way to avoid this. Another key step is to develop a facilitation guide detailing what project data to prepare for the review, and how to present project data and structure discussion around the review questions in a manner that encourages deeper reflection and generates meaningful project learning.
- Consider incorporating materials and activities that help teams look at project achievements, feedback, and other information in new ways and from different perspectives. These might include include “data placemat/data headlines” or “quotables” activitiesA “data placemat” is a tool for presenting qualitative and quantitative project data in thematic groupings designed to encourage stakeholder interaction with and interpretation of the data. In a “data headlines” activity, participants capture key points that they see emerging from project data (e.g. data provided in a data placemat). A data headlines activity can include instructions to develop different kinds of headlines (positive, critical, etc.) to challenge participants to look at data from different perspectives.
In a “quotables” activity, participants receive a sheet with quotes representing the perspectives of different project stakeholder. After reviewing the quotes, participants try to match the quote to the stakeholder who said it. – see the “Data Internalization Guidance” under tools.
- If an individual external to the project teamAfter-action review facilitators from outside the project team might include the head of programming or regional MEAL advisor. has agreed to facilitate the after-action review event, the PM briefs him or her prior to the event on any sensitivities to be aware of.
- If per the terms of reference for the after-action review, the process includes a pre-review survey to collect CRS and partner staff perspectives on the project experience and achievements, send reminders to those invited to complete the survey. Prior to the after-action review event, the MEAL lead or external facilitator compiles and analyzes the survey responses to identify and prepares to share key themes and feedback in the face-to-face after-action review event.
- If the after-action review will include a review of final project achievement against relevant project indicators, the MEAL lead prepares final indicator information (e.g. indicator tracking table) and any supporting visuals.
- In some cases, especially with dispersed project teams (for example, in a multi-country project), it may be appropriate to organize multiple after-action review events.
- The tip to identify a note-taker is equally if not more important for an after-action review, as thorough documentation is commonly overlooked in after-action review processes.
- Even if the after-action review doesn’t produce endline project data, it’s still important to integrate the summarized learning from the review into the final project past performance reference (PPR) (see Standard 18, key action 5 for guidance).
- The PM and MEAL lead use the customized review questions from the after-action review TOR (see Standard 18, key action 1) to finalize the agenda and develop a facilitation guide and materials for the after-action review event.
Reflect on the close-out process itself: Identification of learning from the project close-outKeep in mind CRS’ broad definition of project close-out as a process that includes programming and operations activities both before and after the project end date. For projects funded by institutional donors, the term “project close-out” also encompasses the specialized activities of award close-out. process itself is also important for continuous improvement in project management. Project evaluations typically don’t include this type of review, given their timing within the close-out process. If the reflection event to review and interpret final evaluation findings is held close to the end of the project close-out process, consider building into the event an internal review of successes and challenges in the project close-out process. Otherwise, organize a short after-action review by CRS and partner staff focused on the close-out process. Use the Close-out Process After Action Review tool.
-
Partnership
- Engage partners fully in the after-action review or final evaluation process, including the evaluation reflection event to analyze and interpret findings and identify project learning.
- In projects where one or more partners is only involved in discrete components of the project, tailor reflection event participation appropriately, and ensure there are opportunities for partner staff to hear about overall project learning (e.g. participation in wider project close-out events).
- In the case of large partner project teams, work with partners to identify the staff best-placed to contribute to the evaluation reflection event. Support partners as needed to share learning from the reflection with project staff who did not participate in the reflection event, along with any other staff in their organizations who might benefit from project learning (see MEAL Procedure 3.7.MEAL Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning. ).
When CRS is a sub-recipient- The prime is responsible for the final project evaluation and related learning events.
- CRS may organize its own learning events with partners for the final evaluation if the event organized by the prime is not sufficiently inclusive or is not likely to draw out the kind of learning more useful for CRS and partners. Follow the process outlined in steps 2-4 above for any CRS-organized events.
Emergency projects- Follow the same process for emergency projects, telescoping the steps as needed.
Key resources
Tools and templates
-
Additional tools and templates for evaluations, after-action reviews, and participatory reflection events (see CRS' MEAL Procedures 3.6 and 3.7)
-
Getting Started Guide: Practical Learning at CRS (exercises)
-
Guidance on Planning and Conducting an Evaluation (from CRS' Guidance on Monitoring and Evaluation)
-
M&E Short Cuts series: Communicating and Reporting on an Evaluation
-
M&E Short Cuts series: Managing and Implementing an Evaluation
-
Past Performance Review (PPR) templates (from CRS' MEAL Procedure 8.1)
Other resources
-
Institutional Strengthening: Building Strong Management Processes (ISG): Chapter 10, Monitoring and Evaluation
- In the final 6 months of the project, unless otherwise requiredFor example, in some multi-year projects, donor requirements may necessitate conducting the final evaluation at the end of the penultimate year/beginning of the final project year, to align with the timing of project baseline data collection.
- Primary responsible: Project manager or chief of party (PM/CoP) and the MEAL lead for the evaluation (with external evaluator if applicable)