Standard 18: Learning from and leveraging results during close-out.

• Primary responsible: Project manager or chief of party (PM/CoP) and the MEAL lead for the evaluation (with external evaluator if applicable)
  • For evaluations, the MEAL leadThe MEAL lead for the evaluation may be the country program MEAL manager, project MEAL coordinator, or other CRS MEAL staff. For internal evaluations, this individual typically leads the CRS evaluation team. For external evaluations especially, this individual often takes on the role of “evaluation manager”, serving as the main point of contact between the project team and generally ensuring the evaluation runs smoothly and is completed in a timely manner.  individual designated as the "evaluation manager"The “evaluation manager” serves as the main point of contact between the project team and an external evaluator and focuses on ensuring the evaluation runs smoothly and is completed in a timely manner. For more on the evaluation manager role, see the Managing and Implementing an Evaluation resource.  oversees the evaluation process, with support from the PM/CoP. The MEAL lead/evaluation manager (or external evaluation consultant if applicable) also prepares evaluation data for review during the evaluation reflection event.
  • The evaluator typically prepares, facilitates, and documents the evaluation reflection event.
  • The PM/CoP leads the after-action review or works with a CRS facilitator external to the project team to plan and implement the after-action review.

• Others involved: CRS and partner project team members (including sector specialists, MEAL staff, and operations staff); country program senior management team (SMT); regional technical advisors; country program and/or regional business development (BD) staff; members of the project governance structure; IDEA staff as applicable
  • CRS and partner project team members (sector and MEAL staff, as well as operations staffOperations staff participation will vary by project. For example, for projects with significant supply chain management or ICT4D components, staff from these areas will be closely involved in the evaluation. ) participate in the after-action review or final evaluation process.
  • Members of the SMT and the project governance structureThe project governance structure is a group that makes decisions about a project that are beyond the project manager’s sole authority. Establishing a project governance structure ensures that the right decisions are taken at the right time and with the right degree of stakeholder participation. The size and composition of project governance structures vary by project, ranging from a very small body comprised of the PM and a project sponsor, to a project board with representation from a variety of stakeholders.  may participate in the final evaluation process including the evaluation reflection event.
  • Regional technical advisors (MEAL and sectoral) support the project team to interpret final evaluation findings.
  • Country program and/or regional BD staffIf there is a country program BD staff, he or she may participate. Regional BD staff may participate where feasible for particularly strategic projects. may participate in the final evaluation reflection event; IDEA staff may also participate in such events for strategic, centrally-funded projects.

• In the final 6 months of the project, unless otherwise requiredFor example, in some multi-year projects, donor requirements may necessitate conducting the final evaluation at the end of the penultimate year/beginning of the final project year, to align with the timing of project baseline data collection.
  • Schedule the final evaluation and reflection event or after-action review in the project close-out plan. Be sure to hold the final evaluation reflection event before finalization of the evaluation report (keep in mind any deadline for report submission).

This key action is implemented in accordance with CRS’ MEAL Policies and Procedures, specifically Procedures 3.6 and 3.7.Procedure 3.6: Conduct final evaluation to measure and document project progress and contribute to larger agency learning.
Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.

Note: While there are many common steps in the process of conducting a final project after-action review and a final project evaluation, the final evaluation process is lengthier and more complex. Given that final evaluations are more common in CRS projects, the guidance below is written from the perspective of a final evaluation. Key adaptations to that guidance and other differences in after-action review processes are highlighted in the after-action review section below.

Follow these steps to ensure an effective, participatory final evaluation or after-action review process:  

Final evaluation (including the evaluation reflection event)

1. The MEAL lead/evaluation manager (as applicable) and PM/CoP use the Managing and Implementing an Evaluation guidance to support the internal or external evaluator to:

• Finalize all evaluation tools and train data collectors on their use.
• Collect quality data.
• Conduct preliminary data analysis.

2. The evaluator (internal or external) works with the PM/CoP and MEAL staff to prepare an agenda, discussion materials, and a facilitation guide for the reflection event to review and interpret evaluating findings and identify learning from the evaluation.

• Use CRS’ Guidance on Monitoring and Evaluation and other resources from MEAL Procedure 3.7.MEAL Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.
• TIP: Identify an individual who can document key discussion points during the event, along with lessons learned, final conclusions and recommendations. While the evaluator is responsible for preparing the report of the reflection event, given the evaluator’s facilitation responsibilities, note-taking support can help with the completeness and accuracy of the reflection report.

3. The evaluator facilitates the participatory evaluation reflection event with CRS and partner project staff, in accordance with MEAL Procedure 3.7.MEAL Procedure 3.7: Reflect on evaluation findings with partners and other stakeholders to generate appropriate recommendations and inform agency learning.
4. The PM/CoP and evaluation manager (as applicable) support the evaluator to finalize the reflection event report.

• Keep the report succinct (ideally three pages or less) and focused on key discussion points related to findings, recommendations, and lessons learned. 
• Disseminate the meeting notes or final event report to participants within a week.

5. The evaluator prepares a draft of the full evaluation report and any other deliverablesFor example, drafts of communications pieces or other documents CRS can use in sharing evaluation findings with stakeholders (per MEAL Procedure 7.2). While the evaluation scope of work should have detailed CRS’ expectations and preferred formats for these deliverables, review these expectations as the evaluator begins to work on the final report and related deliverables. from the scope of work, shares with the project team for review and feedback, and addresses feedback.

• Ensure the final evaluation report follows the structure agreed in the final evaluation terms of reference (including agreed annexes) and includes a summary of the evaluation methodology and any limitations of the same.
• Ensure the report clearly responds to the evaluation questions, referencing evidence gathered through the evaluation process, highlighting stakeholder inputs, and incorporating learning from the evaluation reflection event.
• Coordinate internal review and feedback on the draft report within a timeframe that allows for timely finalization of the evaluation report. Evaluation report reviewers typically include the PM/CoP, evaluation manager, sector leads/regional technical advisors and the regional MEAL advisor as applicable, the head of programming, and other key CRS and partner project staff.
• If the final evaluation report will be submitted to a donor, time permitting, request a review by an IDEA staff or another CRS staff member with strong understanding of the donor’s requirements and interests.
• If the evaluator does not incorporate CRS’ feedback in the final version of the report, ensure s/he provides justification.

6. The PM/CoP shares the final evaluation report with the CRS and partner project team and members of the project governance structure, and integrates end-of-project achievements, key findings and learning into the final project past performance reference (PPR) per MEAL Procedure 8.1Procedure 8.1: Complete end-of-project past performance references (PPRs) to document project learning and post to Gateway. (see Standard 18, key action 5 for guidance on saving the PPR and evaluation and reflection reports to GatewayPer the audit trail requirements for MEAL procedure 3.6, projects and emergency responses must upload evaluation or after-action review reports to Gateway within 90 days of the end of data collection activities. See procedure 3.6 for detailed requirements ).
7. The PM/CoP and project team bring the learning from the final evaluation process into end-of-project sharing and dissemination events with project stakeholders (see Standard 18, key action 3 for detailed guidance).

• Click here for Guidance if a final after-action review is required.

  While after-action reviews do not involve collection of primary data, steps 2-4 and 6-7 above are still applicable, with the following adaptations and tips:

  • The PM and MEAL lead use the customized review questions from the after-action review TOR (see Standard 18, key action 1) to finalize the agenda and develop a facilitation guide and materials for the after-action review event.
    • Use the good practices from the MEAL Procedures Narrative (procedures 3.6 and 3.7).
    • Do not skip the step of developing a facilitation guide and materials for the after-action review! Too often, project teams take a “check the box” approach to the after-action review and miss the opportunity to deepen project team learning. Adapting the standard after-action review questions to include probing questions or prompts customized to the specifics of the projects is one way to avoid this. Another key step is to develop a facilitation guide detailing what project data to prepare for the review, and how to present project data and structure discussion around the review questions in a manner that encourages deeper reflection and generates meaningful project learning.
    • Consider incorporating materials and activities that help teams look at project achievements, feedback, and other information in new ways and from different perspectives. These might include include “data placemat/data headlines” or “quotables” activitiesA “data placemat” is a tool for presenting qualitative and quantitative project data in thematic groupings designed to encourage stakeholder interaction with and interpretation of the data. In a “data headlines” activity, participants capture key points that they see emerging from project data (e.g. data provided in a data placemat). A data headlines activity can include instructions to develop different kinds of headlines (positive, critical, etc.) to challenge participants to look at data from different perspectives.
       
      In a “quotables” activity, participants receive a sheet with quotes representing the perspectives of different project stakeholder. After reviewing the quotes, participants try to match the quote to the stakeholder who said it. – see the “Data Internalization Guidance” under tools.
  • If an individual external to the project teamAfter-action review facilitators from outside the project team might include the head of programming or regional MEAL advisor.  has agreed to facilitate the after-action review event, the PM briefs him or her prior to the event on any sensitivities to be aware of.
  • If per the terms of reference for the after-action review, the process includes a pre-review survey to collect CRS and partner staff perspectives on the project experience and achievements, send reminders to those invited to complete the survey. Prior to the after-action review event, the MEAL lead or external facilitator compiles and analyzes the survey responses to identify and prepares to share key themes and feedback in the face-to-face after-action review event. 
  • If the after-action review will include a review of final project achievement against relevant project indicators, the MEAL lead prepares final indicator information (e.g. indicator tracking table) and any supporting visuals.
  • In some cases, especially with dispersed project teams (for example, in a multi-country project), it may be appropriate to organize multiple after-action review events.
  • The tip to identify a note-taker is equally if not more important for an after-action review, as thorough documentation is commonly overlooked in after-action review processes.
  • Even if the after-action review doesn’t produce endline project data, it’s still important to integrate the summarized learning from the review into the final project past performance reference (PPR) (see Standard 18, key action 5 for guidance).

Reflect on the close-out process itself: Identification of learning from the project close-outKeep in mind CRS’ broad definition of project close-out as a process that includes programming and operations activities both before and after the project end date. For projects funded by institutional donors, the term “project close-out” also encompasses the specialized activities of award close-out. process itself is also important for continuous improvement in project management. Project evaluations typically don’t include this type of review, given their timing within the close-out process. If the reflection event to review and interpret final evaluation findings is held close to the end of the project close-out process, consider building into the event an internal review of successes and challenges in the project close-out process. Otherwise, organize a short after-action review by CRS and partner staff focused on the close-out process. Use the Close-out Process After Action Review tool.