National Consultancy: EYE Endline Evaluation; Team Leader: Public Health or Sexual Reproductive Health Expert & Gender/Adolescent and Youth Expert, Kampala-Uganda

Please specify the position you are applying for in your application 

Purpose of the Consultancy:

To undertake the endline evaluation for the project, “My Body, My Life, My World - Empowering Young girls and women to decide over their own bodies and Ensure universal access to comprehensive adolescent and youth friendly SRHR information and services (EYE universal SRHR)” implemented by UNFPA in partnership with CARE and Marie Stopes Uganda (MSU) in Kamuli and Mayuge districts with support from Norwegian Government. 

How you can make a difference:

UNFPA is the lead UN agency for delivering a world where every pregnancy is wanted, every childbirth is safe and every young person's potential is fulfilled.  UNFPA’s strategic plan (2022-2025), reaffirms the relevance of the current strategic direction of UNFPA and focuses on three transformative results: to end preventable maternal deaths; end unmet need for family planning; and end gender-based violence and harmful practices. These results capture our strategic commitments on accelerating progress towards realizing the ICPD and SDGs in the Decade of Action leading up to 2030. Our strategic plan calls upon UN Member States, organizations and individuals to “build forward better”, while addressing the negative impacts of the Covid-19 pandemic on women’s and girls’ access to sexual and reproductive health and reproductive rights, recover lost gains and realize our goals.

In a world where fundamental human rights are at risk, we need principled and ethical staff, who embody these international norms and standards, and who will defend them courageously and with full conviction.

UNFPA is seeking candidates that transform, inspire and deliver high impact and sustained results; we need staff who are transparent, exceptional in how they manage the resources entrusted to them and who commit to deliver excellence in programme results.

Scope of Work

BACKGROUND

The United Nations Population Fund works to deliver a world where every pregnancy is wanted, every childbirth is safe and every young person’s potential is fulfilled. We are also the leading United Nations agency that supports the Government of Uganda to generate population data for planning and decision making. We ensure the empowerment of women, adolescents and young people to live their dreams and fulfill their potential.  UNFPA is active in 56 districts across Uganda.

With support from the Norwegian Government, UNFPA in partnership with CARE and MSU the EYE project is aimed  at  increasing  the utilization of integrated SRHR services by adolescents and youth aged 10-24 in Uganda with a focus on the two high-SRHR burdened districts of Eastern Uganda (Kamuli and Mayuge).  The two districts have a population of about 1.1M people, where  poverty levels are 45% and 49% for Kamuli and Mayuge respectively; and Teenage pregnancy rate is at 23.8% and 20.9% for  Kamuli and Mayuge respectively. 

The EYE project responds to identified problems of high maternal mortality rate; high levels of teenage pregnancies, low use of contraceptives; and weak programming for GBV prevention and response. The project’s goal is to ensure adolescents and young people (10-24), especially those most vulnerable and in hard-to-reach areas, can exercise their sexual and reproductive health and rights (SRHR) by empowering them to make informed choices, accessing high-quality integrated SRHR and GBV services through strengthened health facilities, and fostering a supportive enabling environment at the sub-national level.

MAIN PROJECT RESULTS

The main project results include the following:

 PURPOSE AND SCOPE OF THE STUDY 

UNFPA is seeking qualified Ugandans to conduct an endline study of the achievement of the EYE universal SRHR project during the three years of implementation (2023-2025). 

The purpose of this evaluation is to assess to what extent the EYE programme outcomes and outputs were achieved including the overall results, effects/changes realized, good practices, challenges faced, and lessons learned. The endline evaluation will assess the relevance, effectiveness, efficiency, impact, coherence, and prospects for sustainability of the EYE programme based on the OECD/DAC criteria and UNFPA evaluation guidelines. These will also be assessed to determine the contribution of the EYE programme towards 9th Government of Uganda/UNFPA Country Programme outcomes. 

The evaluation will apply appropriate methodology for assessing principle cross-cutting elements of equity and vulnerability, gender equality and human rights. It will be based on the guiding principles, norms and standards for evaluations adopted at UNFPA. 

Objectives

Scope 

In relation to the above, the endline evaluation will cover the 2 programme districts of Kamuli and Mayuge.  Contribution analysis will be employed as the overall analytical framework of this evaluation to identify the contribution that the EYE programme has made to the observed set of changes (i.e. positive/negative, intended/unintended effects) among targeted beneficiaries and communities. Mixed methods will be employed to generate and triangulate qualitative and quantitative data across multiple sources to answer the evaluation questions.

The evaluation will follow a collaborative and participatory approach ensuring close engagement with programme beneficiaries, implementing partners, district local government leadership, MDAs, and other key stakeholders as will be informed by an initial rapid desk review and inception meetings with key stakeholders which will support the refinement and finalisation of the evaluation methodology and analytical framework to be included into the inception report. 

All information will be analysed and triangulated using standardised methods (e.g. structured coding and content analysis). Results from the endline evaluation will be validated through a (virtual or physical) workshop with stakeholders from national and district levels and disseminated through a regional dissemination workshop (physical). 

Specific Tasks

Time horizon

The EYE universal SRHR project is a three-year project, which is in its final year of implementation. Therefore, within the framework of the above evaluation objectives, this endline evaluation will cover the period from January 2023-December 2025.

Geographic scope

At the national level, the main sectors to focus on will be the Ministry of Health, Ministry of Gender Labour and Social Development and Ministry of Education. Otherwise much of the data collection will be from the 2 districts of focus; Kamuli and Mayuge.

EVALUATION CRITERIA AND PRELIMINARY EVALUATION QUESTIONS

Evaluation Criteria

In accordance with the OECD/DAC evaluation criteria and UNFPA global evaluation guidelines, the endline evaluation will examine the following evaluation criteria: relevance, effectiveness, efficiency, impact and prospect for sustainability. It will also use the evaluation criterion of coherence to assess the extent to which the UNFPA Uganda CO harmonized interventions under the NORAD programme with other actors, promoted synergy and avoided duplication amongst SRHR and Health Development Partners (HDP) and  the framework of the UNCT Delivering as one. Relevance The extent to which the project design was relevant for its different stakeholder groups (including primary beneficiaries) and to national policies of the GOU, and Country Programme. How effectively the programme adapted its strategies and interventions to contextual changes to achieve results. Effectiveness The extent to which the EYE universal SRHR project results and targets have been achieved, including both intended and unintended effects, and the extent to which these outputs have contributed to the achievement of the Country programme outcomes.  Efficiency The extent to which the project outputs and outcomes have been achieved in the most economic and timely way, including an assessment of how funds, expertise, personnel, and implementation modalities contributed to, or hindered the achievement of results.

Coherence The internal and external coherence of the EYE universal SRHR project, including the coherence between specific project components, other UNFPA programmes, as well as coherence with other partners’ SRHR and GBV interventions in the Kamuli and Mayuge districts Sustainability The continuation of benefits from the EYE universal SRHR project after its termination, linked to their continued resilience to risks. Impact The extent to which the EYE universal SRHR project has generated or is expected to generate significant positive or negative, intended or unintended, impact for its beneficiaries

                Indicative Evaluation Questions

The evaluation questions presented below are indicative and preliminary. Based on these examples, at the inception phase, the evaluators in consultation with the Evaluation Reference Group (ERG), are expected to develop a final set of evaluation questions. The examples here serve the purpose of illustrating to the evaluator’s key questions of interest to the ERG.

Relevance

Effectiveness

Efficiency

Sustainability

Coordination

Coherence

Impact

The evaluation will assess the extent to which the project had an impact (positive or negative, intended or unintended) on the targeted beneficiaries. 

The final evaluation questions and the evaluation matrix will be finalized by the evaluation team in the design report.

APPROACH AND METHODOLOGY

Evaluation Approach/ Design

The evaluation will use a cross-sectional mixed-methods design, which involves both quantitative and qualitative approaches of data collection. The Evaluation design will build on the baseline assessment methodology to ensure consistent measurements of the pre and post project intervention impact. 

Theory-based approach

The evaluation team will be required to verify the theory of change underpinning the EYE universal SRHR project and use this theory of change to determine whether changes at output and outcome levels occurred (or not) and whether assumptions about change hold true. The analysis of the theory of change will serve as the basis for the evaluators to assess how relevant, effective, efficient and sustainable the support provided by the EYE universal SRHR project was during the implementation period.

Contribution analysis 

As part of the theory-based approach, contribution analysis will be employed as the overall analytical framework of this evaluation to identify the contribution that the EYE universal SRHR project has made to the observed set of changes (i.e. positive/negative, intended/unintended effects) among targeted beneficiaries and communities. Specifically, the evaluators will use a contribution analysis to explore whether evidence to support key assumptions exists, examine if evidence on observed results confirms the chain of expected results in the theory of change, and seek out evidence on the influence that other factors may have had in achieving desired results. 

Accordingly, the evaluation team will reconstruct the logic behind the EYE universal SRHR project interventions (theory of change) for the period under evaluation. The theory of change (ToC) reflects the conceptual and programmatic approach taken by UNFPA over the period under evaluation including the most important implicit assumptions underlying the change pathway. The ToC will include the types of intervention strategies or modes of engagement used in program delivery, the principles/guiding interventions, the elements of the intervention logic, the type and level of expected changes and the external factors and influence and determine the causal links depicted in the theory of change diagram. The ToC will be tested during the field and data collection phase.

Evaluators will analyse and interpret the logical consistency of the chain of effects: linking programme activities and outputs with changes in higher-level outcome areas, based on observations and data collected along the chain. This analysis should serve as the basis of judgment by the evaluators on how well the programme has contributed to the achievement of the intended results (outputs and outcomes) in the EYE universal SRHR programme document.

The indicative and preliminary evaluation questions listed above must be complemented by questions on sets of assumptions that capture key aspects of the intervention logic. The data collection for each of the assumptions will be guided by clearly formulated quantitative and qualitative indicators which must also be included in the evaluation matrix.

Methodology and tasks

The consultants will detail their methodological approach in their Inception report, as per the TOR, including the tools they will use. The methodology and approach must incorporate human rights and gender equality perspectives.   

The evaluation team shall develop and conduct the evaluation in accordance with the UNEG Norms and Standards for Evaluation, Ethical Guidelines for Evaluation, Code of Conduct for Evaluation in the UN System, and Guidance on Integrating Human Rights and Gender Equality in Evaluations. The evaluators will be requested to sign the UNEG Code of Conduct  prior to starting their work.

The methodological design of the evaluation shall include in particular: (i) a theory of change; (ii) a strategy for collecting and analyzing data; (iii) specifically designed tools for data collection and analysis; (iv) an evaluation matrix; and (v) a detailed evaluation work plan and agenda for the field phase. 

Evaluation Process

The evaluation process can be broken down into three different phases that include different stages and lead to different deliverables: inception phase; field phase; reporting phase; and phase of dissemination and facilitation of use. The evaluation manager and the evaluation team leader must undertake quality assurance of each deliverable at each phase and step of the process, with a view to ensuring the production of a credible, useful and timely evaluation.

Inception Phase

In the inception phase, the evaluation manager will lay the foundation for communications around the evaluation. All other activities will be carried out by the evaluation team, in close consultation with the evaluation manager and the ERG. This phase includes:

 At the end of the inception phase, the evaluation team will develop an inception report that presents a robust, practical and feasible evaluation approach, detailed methodology and work plan. The evaluation team will develop the inception report in consultation with the evaluation manager who will submit it to the ERG for review. The ERG members will provide feedback and comments to the inception report during the 1st ERG meeting.

 Field Phase

The evaluation team will collect the data and information required to EYE universal SRHR project the evaluation questions in the field phase. Towards the end of the field phase, the evaluation team will conduct a preliminary analysis of the qualitative data to identify emerging findings that will be presented to the CO and the ERG.[1] The field phase should allow the evaluators sufficient time to collect valid and reliable data to cover the thematic scope of the EYE universal SRHR project  evaluation and be able to compare baseline and endline data. A period of up to 3 weeks for data collection (HFA, KII, FGD, site visits) is planned for this evaluation. However, the evaluation manager will determine the optimal duration of data collection, in consultation with the evaluation team during the inception phase.

 The field phase includes:

●        Meeting with the UNFPA Uganda CO staff to launch the data collection.

●        Meeting of the evaluation team with relevant programme officers at the UNFPA Uganda CO.

●        Data collection at national and sub-national levels.

 At the end of the field phase, the evaluation team will hold a debriefing meeting with the CO and the ERG to present the emerging findings from the data collection. In addition to the ERG members, selected stakeholders who participated in the data collection, including representatives of DLG and IP field teams will be invited to participate in the meeting virtually. The meeting will serve as a mechanism for the validation of collected data and information and the exchange of views between the evaluators and important stakeholders. It will enable the evaluation team to refine the findings, which is necessary so they can then formulate their conclusions and develop credible and relevant recommendations.

 Analysis and Reporting Phase

In the analysis and reporting phase, the evaluation team will continue the analytical work initiated during the field phase. The quantitative data collected through the HFAT, HMIS and NGBVD will be cleaned, entered and analyzed using appropriate software to be determined by the evaluators. Qualitative data will be entered and coded using Nvivo, Dedoose or similar software to facilitate structured content analysis. The evaluation team will jointly conduct a virtual or onsite analysis workshop to triangulate draft findings and conclusions by evaluation question based on the various sub-components of the endline evaluation (HFA, HMIS/NGBVD, qualitative etc.).

The evaluation team shall develop a draft evaluation report that summarizes the context, methodology, findings, conclusions and recommendations). The draft report will take into account the comments and feedback provided by the CO and the ERG at the debriefing meeting at the end of the field phase. Additionally, the evaluators should write a stand-alone report on the HEALTH Facility Assessment, which will be annexed to the draft evaluation report. The final report will summarize and triangulate the key findings from the HFA as part of the overall contribution analysis.

Prior to the submission of the draft report to the evaluation manager, the evaluation team will perform an internal quality control against the criteria outlined in the Evaluation Quality Assessment (EQA) grid. The evaluation manager will subsequently review the draft evaluation report, using the same criteria (defined in the EQA grid). If the quality of the report is satisfactory (in form and substance), the draft report will be circulated to the ERG members for review. In the event that the quality of the draft report is unsatisfactory, the evaluation team will be required to revise the report and produce a second draft.

The evaluation manager will perform his/her review of the draft final report against the completed evaluation matrix (to ensure that the analysis - responses to the evaluation questions - rests on credible data and information and is, in fact, evidence based). S/he will also collect and consolidate the written comments and feedback provided by the members of the ERG. Based on the comments, the evaluation team should make appropriate amendments, prepare the final evaluation report and submit it to the evaluation manager. The final report should clearly account for the strength of evidence on which findings rest to support the reliability and validity of the evaluation. Conclusions and recommendations need to clearly build on the findings of the evaluation. Each conclusion shall make reference to the evaluation question(s) upon which it is based, while each recommendation shall indicate the conclusion(s) from which it logically stems.

UNFPA may decide to conduct a physical or virtual co-creation workshop to draft the recommendations jointly with UNFPA and IP programme staff, representatives of the Embassy, or the wider ERG. If this option is chosen, the evaluation team is still responsible for formulating the final recommendations based on the co-creation workshop deliberations.

 The evaluation report is considered final once it is formally cleared by the ERG.

Dissemination and Facilitation of Use Phase

In the dissemination and facilitation of use phase, the evaluation team will develop a PowerPoint presentation of the evaluation results that summarizes the key findings, conclusions and recommendations of the evaluation in an easily understandable and user-friendly way.

The evaluation matrix

The evaluation matrix is center-piece to the methodological design of the evaluation. The matrix contains the core elements of the evaluation. It outlines (i) what will be evaluated: evaluation questions for all evaluation criteria and key assumptions to be examined; and (ii) how it will be evaluated: data collection methods and tools and sources of information for each evaluation question and associated key assumptions. By linking each evaluation question (and associated assumptions) with the specific data sources and data collection methods required to answer the question, the evaluation matrix plays a crucial role before, during and after data collection.

 As the evaluation matrix plays a crucial role at all stages of the evaluation process, it will require particular attention from both the evaluation team and the evaluation manager. The evaluation matrix will be drafted in the inception phase and must be included in the inception report. The evaluation matrix will also be included in the annexes of the final evaluation report, to enable the evaluation report’s users to access the supporting evidence for the answers to the evaluation questions.

Data collection methods

To be able to make meaningful assessment of trends, the methods used at baseline and MTR will be adopted at endline.

Field visits – Site visits will be conducted on sample basis during which group interviews and focus group discussions (FGD) will be conducted with direct and indirect beneficiaries, including in-school and out of school young people [2], men and women, health care workers, teachers, parents, religious, cultural and political leaders. The evaluators will be required to take into account ethical considerations when collecting information. The proposed field visit sites, stakeholders to be engaged and interview protocols will be outlined in the inception report to be submitted by the evaluation team. When choosing sites to visit, the evaluation team should make explicit the reasons for selection. The choice of the locations to visit at sub-national level needs to take into consideration the implementation of the project components in those areas and done in consultation with the UNFPA project technical team and the evaluation manager.

The evaluation team is expected to dedicate up to three weeks for data collection in the field. The data collection tools that the evaluation team will develop, shall be presented in the inception report.

 Data validation

The evaluators will check the validity of the collected data and information and verify the robustness of findings at each stage of the evaluation, so they can determine whether they should further pursue specific hypotheses (related to the evaluation questions) or disregard them when there are indications that these are weak (e.g. contradictory findings or lack of evidence).

The evaluators will use a variety of mechanisms to ensure the validity of the information and data collected[3] including but not limited to:

 The specific validation mechanisms will be further elaborated in the inception report.

 Sampling strategy

The evaluation will use different sampling strategies suitable for the variables and data sources identified in the evaluation matrix. For the qualitative primary data collection, purposive sampling will be used to sample districts and key informants for the KIIs, group interviews, and FGDs, and sites to be visited. UNFPA will provide an initial overview of the interventions supported by the EYE universal SRHR project, the locations where these interventions have taken place, and the stakeholders involved in these interventions. 

 Building on the initial stakeholder map and based on information gathered through document review and discussions with UNFPA staff, the evaluators will develop the final stakeholder map. From this final stakeholder map, the evaluation team will select a sample of stakeholders at national and sub-national levels who will be consulted through interviews and/or group discussions during the data collection phase. These stakeholders must be selected through clearly defined criteria and the sampling approach outlined in the inception report. In the inception report, the evaluators should also make explicit what groups of stakeholders were not included and why. The evaluators should aim to select a sample of stakeholders that is as representative as possible, recognizing that it will not be possible to obtain a statistically representative sample.

 The evaluation team shall also select a sample of sites that will be visited for data collection for the KIIs, FGD, group interviews and direct observations, and provide the rationale for the selection of the sites in the inception report. UNFPA will provide the evaluators with necessary information to access the selected locations, including logistical requirements and security risks, if applicable. The sample of sites selected for visits should reflect the variety of interventions supported by the EYE universal SRHR project, both in terms of thematic focus and context.

The evaluation team should consider how to include non-beneficiaries in the sample, with a focus on using qualitative methods to assess the effectiveness of the programme strategies in reaching beneficiaries with SRHR/GBV information and services, creating demand and changing health seeking behaviours. With regards to programme reach and coverage, the evaluation team could consider including questions to measure dose-response relationship at programme sites for different interventions - and then explore reasons for non or under exposure (recall bias notwithstanding). 

 The final sample of stakeholders and sites will be determined in consultation with the evaluation manager, based on the review of the inception report.

 Ethical clearance

The Evaluators will seek Local Institutional Review Board (IRB) approval of the inception report and protocols using a fast-tracked process. The evaluation team shall finalise the inception report based on feedback from IRB/UNSCT.
 

Deliverables

Monitoring and Reporting

The consultants will provide periodic updates (in writing) on progress of the assignment, at a frequency to be agreed upon by the M&E and technical teams from UNFPA. The consultant will present all key milestone products to UNFPA M&E and technical teams, who will in turn share them with the MOH Research and Evaluation Committee.
 

Management of the study

The consultant will work under the direct supervision of the M&E Specialist, with technical support from the ISRHR and GEWE teams. The consultant will present all key milestone products to the Evaluation Reference Group. The Research and Evaluation committee of the Ministry of Health will guide the consultancy and vet all the consulting firm’s work to ascertain scientific rigor. Approval by the MOH Research and Evaluation committee is a pre-condition for the assignment to be considered finalized and that it has met the expected quality standards. The consultant will work in accordance with the procedures and activities agreed upon, and expected to deliver outputs as per agreed schedule

Expected travel
The consultants will be expected to travel widely, to the 2 target districts 

    Evaluation Quality Assessment Checklist 

 1. Structure and Clarity of the Report

To ensure the report is user-friendly, comprehensive, logically structured and drafted in accordance with international standards.

2. Executive Summary  

To provide an overview of the evaluation, written as a stand-alone section including key elements of the evaluation, such as objectives, methodology and conclusions and recommendations.

3. Design and Methodology

To provide a clear explanation of the methods and tools used including the rationale for the methodological choice justified. To ensure constraints and limitations are made explicit (including limitations applying to interpretations and extrapolations; robustness of data sources, etc.)

4. Reliability of Data

To ensure sources of data are clearly stated for both primary and secondary data. To provide explanation on the credibility of primary (e.g. interviews and focus groups) and secondary (e.g. reports) data established and limitations made explicit.

5. Findings and Analysis

To ensure sound analysis and credible evidence-based findings. To ensure interpretations are based on carefully described assumptions; contextual factors are identified; cause and effect links between an intervention and its end results (including unintended results) are explained.