Senior Expert for the design and implementation of an AI-Powered evaluation management system

Result of ServiceUNRWA is seeking an individual consultant to design and implement an automated online system for managing evaluations, including tracking recommendations and analysing evaluation reports. The system should meet the requirements outlined below, recognising that these may be adjusted—expanded or streamlined—based on a detailed needs assessment to be conducted at the start of the consultancy. Automated recommendation tracking: • The platform must automatically monitor recommendations arising from evaluation reports. The aim is to reduce the need for manual data entry and enable the timely provision of updates on the status of implementation. Evaluation Report Metadata Management • The system should be able to record and display key information about each evaluation report, including details such as report type (e.g., mid-term, final, decentralised, or centralised), year, subject, and other relevant metadata. It should also enable users to search and filter reports based on these fields. AI-powered document analysis and coding The platform should include artificial intelligence features such as, but not limited to, the following: • The system should be able to automatically create short summaries of each report’s findings for different criteria. For example, it could highlight the main achievements and challenges related to “effectiveness,” as well as generate summaries for areas such as “relevance,” “gender equality,” or “sustainability.” It should also apply structured coding to the reports to support both detailed and high-level analysis. • Extract and present evidence on various topics and thematic areas from several evaluation reports. For instance, the system should be able to automatically pull out and display all evidence related to “gender equality” or “capacity development” from a set of evaluation reports, showing the relevant excerpts. • Convert existing manually coded data into the new system and integrate these historical records into the automated framework. Secure, cloud-based platform The solution must be delivered as a secure, cloud-based platform. It should: • Comply with data protection standards, ideally integrating with UNRWA’s Microsoft Copilot environment or equivalent secure infrastructure. • Offer a private workspace for internal users while providing the option to publish approved reports and summaries on a public website. Work LocationWork will include home-based assignments, and potential travel. Expected durationThe level of effort will be around 80 days over the course of the assignment, which is scheduled to take place between September to December 2025. Duties and ResponsibilitiesThe phases of this consultancy are: Phase 1: Assessment & Design • Conduct comprehensive needs assessment with the Evaluation Division to understand current workflows and business requirements. • Map existing evaluation processes to identify automation opportunities and AI integration points. • Audit historical evaluation reports to assess data volume, formats, and migration needs. • Develop technical architecture for Microsoft ecosystem integration and design database. Phase 2: AI Development & Platform Building • Develop models for document analysis, summarization, and evaluation-specific coding. • Create automated recommendation extraction and tracking. • Build core cloud-based platform with secure authentication and role-based access. • Implement search/filtering capabilities and automated workflow systems. • Identify and assess potential solutions to implement the new platforms /automated processes. Phase 3: Testing, Deployment & Training • Establish iterative testing process with the Evaluation Division to validate AI accuracy. • Conduct pilot testing with historical reports and perform security compliance testing. • Deploy platform in UNRWA's secure cloud environment. • Provide user training and create documentation for ongoing system maintenance. The consultancy will ensure seamless integration with existing workflows while delivering automated processes that reduce manual effort and strengthen evidence-based decision-making. Qualifications/special skillsMINIMUM QUALIFICATIONS AND EXPERIENCE • Advanced degree in Computer Science, Information Technology, Information Systems, Data Science, or related technical field. • Minimum 10 years in system development, with experience in AI/machine learning applications and cloud-based platform development. • Demonstrated experience in document analysis, and automated text classification systems. • Experience with secure cloud deployments, preferably Microsoft environments, and data protection compliance. • Proven track record of integrating systems with existing organizational infrastructure and databases. • Experience leading technical consultancies from needs assessment through deployment and training. • Experience with UN agencies. DESIRABLE QUALIFICATIONS • Experience with evaluation/audit frameworks, UN system knowledge, or similar organizational contexts. • Professional work experience in the MENA region. • Familiarity with UNRWA and its programmes. • Understanding of the Palestinian refugee context would be advantageous. LanguagesFluency in written and spoken English is required. Additional InformationCOMPETENCIES • Management skills including work planning. • Strong ability to work with non-technical stakeholders and translate complex technical concepts into practical solutions. • Excellent analytical, communication and writing skills in English. CONDITIONS OF SERVICE • Remuneration for this consultancy will be P4 equivalent and will depend on the qualifications and experience of the candidate. The contract will be deliverable-based; • Work will include home-based assignments, and potential travel for approximately 80 days. No FeeTHE UNITED NATIONS DOES NOT CHARGE A FEE AT ANY STAGE OF THE RECRUITMENT PROCESS (APPLICATION, INTERVIEW MEETING, PROCESSING, OR TRAINING). THE UNITED NATIONS DOES NOT CONCERN ITSELF WITH INFORMATION ON APPLICANTS’ BANK ACCOUNTS.