Standard 18: Learning from and leveraging results during close-out.

• Primary responsible: Project manager or chief of party (PM/CoP) with MEAL leadThe MEAL lead for the evaluation may be the country program MEAL manager, project MEAL coordinator, or other CRS MEAL staff. For internal evaluations, this individual typically leads the CRS evaluation team. For external evaluations especially, this individual often takes on the role of “evaluation manager”, serving as the main point of contact between the project team and the external evaluator(s) and generally ensuring the evaluation runs smoothly and is completed in a timely manner. For more on the evaluation manager role, see M&E Short Cuts: Planning an Evaluation.  or individual designated as the “evaluation manager”
  • The PM/CoP ensures that the final evaluation (whether internal or external) and/or after-action review is planned and implemented in a timely manner.
  • The MEAL lead works with the PM/CoP to develop the evaluation terms of reference (required for all evaluations - see CRS MPP 3.2Procedure 3.2: Develop terms of reference (ToR) for baseline and evaluation events ), scope of work, and detailed evaluation plan; for external evaluations, the PM/CoP and evaluation manager work with the evaluator to develop a detailed evaluation plan.

• Others involved: CRS and partner MEAL staff; CRS and partner project staff, including any sector leads; head of programming (HoP); country representative (as applicable); regional MEAL advisor
  • CRS and partner staff (MEAL and non-MEAL), including any sector leads, contribute to the development of evaluation and/or review objectives, questions and methods as well as the initial evaluation dissemination plans.
  • The HoP and, as applicable (e.g. for strategic projects) the country representative, contribute to the development of evaluation or after-action review objectives and questions and review the draft terms of reference (TOR).
  • The regional MEAL advisor reviews the draft TOR as required by CRS MEAL policies and proceduresProcedure 3.3: Regional MEAL advisors review baseline and evaluation ToRs to improve evaluation quality and ensure appropriateness to information needs.  to determine quality and appropriateness to project scope and information needs.

Donor-commissioned final evaluations: In some complex projects, the donor may take responsibility for the final evaluation process, including contracting the external evaluator(s). In such cases, CRS’ role will be to liaise with the donor and the external evaluator throughout the evaluation planning process, to ensure timely and appropriate CRS contributions and effective coordination with the project team and other stakeholders.

• Initiate planning of the final evaluation or after-action review so that it can be conducted in the last six months of the project or per donor requirements (see below).
  • For a final project evaluation, begin preparations early to allow sufficient time for development of the terms of referenceMPP 3.2: Develop terms of reference (ToR) for baseline and evaluation events. (TOR), scope(s) of work (SOW), selection of an evaluation team, and finalization of evaluation methodology, tools, and work plan.
  • Schedule the evaluation or after-action review process keeping in mind the timing of final project activities and project staffing transitions.Avoid planning a final evaluation too early, when significant project activities are still taking place and consuming the time and attention of the project team. At the same time, it is important not to plan the final evaluation or after-action review too late in the close-out period, when key staff may have already left the project team.
     
  • Factor into evaluation planning any seasonal considerations (e.g. academic or agricultural calendars; weather-related access considerations; etc.) that may affect data collection and stakeholders' ability to contribute to the evaluation process.

Evaluation planning considerations for complex and multi-year projects:

• For complex, multi-year projects, the project team may need to start preparing for the evaluation much sooner than six months before the project end date.  It may take up to a year to develop the TOR and scopes of work, contract an evaluation consultant, finalize evaluation methodology and tools, conduct the evaluation, hold the reflection event, finalize the evaluation report, and disseminate results.
• In some multi-year projects, donor requirements may necessitate conducting the final evaluation at the end of the penultimate year/beginning of the final project year, to align with the timing of project baseline data collection. In such cases, start evaluation preparations as part of annual planning for the project's penultimate year. Plan for a final project after-action review as part of project close-out. 

Follow these steps to plan and implement an effective final evaluation or after-action review.

What’s the difference between a final evaluation and an after-action reviewPer MPP 3.6, a final evaluation or after-action review is required only when CRS is the prime recipient, with the following parameters based on total project or response value below:
a. ≥$5 million: external evaluation
b. Between $1M and $5M: internal or external evaluation
c. ≤$1million: after action review or internal or external evaluation ?

A final evaluation is conducted at the end of a project and typically includes an endline survey, or other data collection methods as appropriate, to allow for comparison with baseline data (if available) for key project indicators. A final evaluation systematically assesses a project’s relevance, efficiency, effectiveness, impact and sustainability on a defined population and draws upon monitoring and evaluation data to answer evaluation questions, meet stakeholder information needs, and generate learning. An after-action review is a reflection on the successes and failures of a project (or activities within a project). It brings together a team to discuss a soon-to-be or recently completed project, in an open and honest fashion, and does not require the collection of additional data from project participants. The focus of an after-action review is ensuring that learning takes place to support continuous improvement. (After-action review definition adapted from Better Evaluation)

1. The MEAL lead and PM/CoP review the plans for the final evaluation (or after-action review) included in the approved project proposal and budgetDuring project design, CRS will have developed a general schedule and budget for the final evaluation or after-action review based on CRS’ MEAL Policies and Procedures (see Procedure 3.6) and, as applicable, available information about any donor requirements. . For externally-funded projects, they also revisit any donor requirements and additional considerationsFor example, a project mid-term evaluation or review may have generated points for follow-up or areas for focus in the final evaluation. CRS and donor learning needs and interests may also have evolved during project implementation. for the evaluation or review (based on internal discussions or discussions with a project donor).
2. The MEAL lead and PM/CoP work with the project team to identify a preliminary set of key questions for the evaluation or review and to determine the best timing for the process.

• Where possible, incorporate this discussion into the final annual or quarterly project review and planning meeting as the project approaches its final year (for multi-year projects) or quarter (for projects < 12 months).
• For final evaluations of externally-funded projects, engage the HoP, country representative (as appropriate), and other staff with strong knowledge of the donor in brainstorming the evaluation objectives and key questions.
• Ensure that the timing and sequencing of the evaluation or review is well coordinated with other activitiesFor example, for a final evaluation, ensure that major evaluation activities are not likely to overlap with periods of intensive project implementation or other major project events. For an after-action review or the project reflection event that follows final evaluation data collection and preliminary analysis (per CRS MEAL policies and procedure), time the event early enough to allow for the participation of key CRS and partner project staff, keeping in mind that some positions may phase out before the project end date. and event s in the project close-out plan, including the timing of project staff transitions.

One or more review or evaluation reflection events? The decision to plan for one or multiple after-action review or final evaluation reflection events will depend on several factors, including the scale and geographic scope of the project and project partnerships, the number and profile of stakeholders who should be directly engaged in the process, the timing, and the budget available. For complex projects, it may be appropriate to organize a series of events, with outputs from one event informing the next. For multi-country projects, it may be appropriate to organize an event with key project stakeholders in each country, followed by a central-level discussion/final event to further analyze, interpret, and reflect at the overall project level.

3. With inputs from the CRS and partner project team and others consulted in step 2, the MEAL lead and PM/CoP develop a TOR for the evaluation or after-action review.

• Refer to MEAL policies and procedures (MPP) resources for MPP 3.2Procedure 3.2: Develop terms of reference (ToR) for baseline and evaluation events to identify good examples relevant to the project context.
• Review the project stakeholder analysis and any project communications plan to identify stakeholder information needs to address in the evaluation or after-action review.
• Reflect on the key discussion points from quarterly and annual project meetings to identify key successes and challenges or outstanding questions appropriate for further examination through the evaluation or review process.
• Customize “standard” questions to focus on the most relevant project-related issues.
  • If the project included any programming or operations approaches that were new for the project team or stakeholders in the target area, new partnerships, or other notable new elements (e.g., use of ICT4D), include questions related to these new project elements.
  • Consider programming and operations-related project management issues, e.g. factors that influenced the effective management of project time, scope, budget, and stakeholder relationships, etc.
  • See CRS’ Guidance on Monitoring and Evaluation, chapter on Planning and Conducting an Evaluation, for additional tips.
• Note: Sensitive issues may surface during the review of project management issues. Include in the TOR an approach to review these issues during the evaluation or after-action review process, as well as the most appropriate way to document and disseminate related findings. Keep in mind that final evaluation and after-action review reports may be shared outside the project team.

Question customization is important for after-action reviews, too! While after-action reviews present a rich opportunity for learning, too often they are approached as a “check the box” exercise with only a cursory review of stock questions. The TOR for a good after-action review process includes customized, probing questions that help the project team deepen its reflection or approach a question from a new perspective. For example, under the standard question, “What could have gone better?”, develop a set of sub-questions or prompts to generate discussion around specific components of the project (e.g. the distribution process, participant selection, consortium coordination, exit/transition planning, etc.). Consider building appropriate data internalization approachesFor example, an activity like “data placemat/data headlines” or “quotables” – see the USAID LEARN Data Internalization Guidance under “Other Resources”. into the TOR include methodologies and activities to facilitate deeper reflection on the after-action review questions.

4. After finalizing the TOR, the PM/COP and MEAL lead develop the necessary scope(s) of work (SOW).

• For an after-action review, a SOW is needed if the facilitator will be from outside the project team. Ensure the SOW clarifies the preparation and follow-up responsibilities of both the facilitator and designated project team members (e.g. the PM and MEAL lead).
• For complex projects, a single evaluator may be insufficient. In such cases, CRS may need to develop an SOW for a consultant agency with diverse evaluation capacities, or separate SOWs for each profile needed on the evaluation team.
• For external evaluations, the CRS project team may have the option of contracting an evaluator external to CRS, or a CRS staff member with evaluation experience who is external to the project team.

5. For final evaluations, the MEAL lead/evaluation manager and PM/CoP ensure the evaluator/evaluation team designs an appropriate evaluation methodology, data collection tools, and detailed evaluation work plan in line with the TOR and SOW.

• In the case of internal evaluations, the CRS MEAL lead often serves as the evaluation team lead and takes responsibility for developing the evaluation methodology, data collection tools, and work plan.
• Seek assistance from MEAL technical staff as needed.
• See the M&E Short Cuts series: Preparing for an Evaluation and Managing and Implementing an Evaluation for guidance on final preparations.