Background: With the support of UNFPA Mongolia, the Facebook-based Mandukhai chatbot was launched in 2021, to deliver accurate and confidential information on sexual and reproductive health (SRH) to adolescents. This AI-powered bot was designed to increase access to SRH information and services and empower its users with the correct knowledge and informed decision-making. Since its installation at the National Center for Maternal and Child Health (NCMCH) and the appointment of a live agent for troubleshooting, Mandukhai demonstrated promising results, achieving an accuracy level of 62% in June 2021, and through ongoing enhancements to its functionality, the chatbot’s accuracy improved to 73% by the end of 2022.
Since its launch, there has been limited information available regarding its performance and level of accuracy in the response of the Mandukhai chatbot including user experience. Additionally, no formal evaluation has been conducted on the bot’s compliance with the data security requirements as per the Mongolia ICT Sector Medium-Term Development Policy (2022-2027), and health-related ethical considerations. Recent advancements in AI technology have the potential to significantly enhance the performance of AI-powered chatbots, even in complex interactions involving sensitive information. Given this potential, the evaluation will also focus on the aspects that the chatbot has “learned” since it was launched.
Therefore, UNFPA Mongolia is seeking to hire a national individual consultant (IC) to perform a comprehensive technological assessment of the Mandukhai chatbot. The evaluation will utilize specific criteria designed for chatbot performance metrics, focusing on accuracy in response rates, overall user experience, and responsiveness to user needs. To capture a broad spectrum of users’ experiences, the assessment will include a diverse group of users, particularly adolescents from remote areas, ensuring the dataset is representative and inclusive.

Key outputs and activities expected to respond to the objective of the consultancy:

1. Develop the evaluation design: The consultant will be responsible for conducting a comprehensive desk review of existing documentation and researching best practices and standards for chatbot evaluation metrics. This includes studying use cases of advanced AI-powered chatbots in the medical field. By gathering background information and understanding the objectives, design, and any previous evaluations of the chatbot, the consultant will develop an evaluation matrix. This matrix will feature defined performance metrics derived from past conversation logs, including accuracy or effectiveness, engagement, user interaction, operationalities such as fallback and escalation, and error and context awareness metrics.

2. Usability Testing: Usability testing is essential for evaluating the chatbot, aimed at ensuring that it is intuitive, efficient, and satisfying for users. The consultant will conduct one-on-one sessions to observe participants as they interact with the chatbot, completing specific tasks. Immediate feedback will be collected through observation, post-session questionnaires, or brief interviews. It is important to recruit adolescents from varied demographics, including differences in age, gender, education level, and geographic location, to participate in the usability testing to ensure comprehensive and relevant results.

3. Scalability and performance testing are crucial for evaluating how well the chatbot manages numerous queries or simultaneous conversations, providing insights into its scalability. Performance testing should encompass the chatbot’s depth of understanding of the Mongolian language, AI algorithms, and conversation flow and ability to handle tones, proficiency in handling context, multi-turn conversations, and variations in user input, memory-related functions such as recalling the previous conversation and generating of dashboard and statistics. These tests will help determine the chatbot’s strength and readiness to function effectively under varied and demanding usage scenarios.

4. Organize stakeholders’ consultation meetings to validate the findings of the chatbot evaluation and improvement plan with recommended actions for further enhancement of the bot level of AI functions.

5. Provide a final consultancy report: The consultant will provide the final consultancy report of the evaluation for further improvements and required updates to the chatbot based on user interaction and scalability testing results.

Duration and working schedule:

The duration of this consultancy is expected to be 33 non-consecutive days within July 15 –Sep 10 in Ulaanbaatar, Mongolia. The consultant’s work will be based on deliverables, and it is not tied to the working hours.

Place where services are to be delivered:

The consultant will work from her/his workplace.

Delivery dates and how work will be delivered (e.g. electronic, hard copy etc.):

The selected consultant is required to provide the following deliverables to ensure that the chatbot meets the desired standards for functionality, user satisfaction, and overall effectiveness.

1. Desk review report along with the bot evaluation matrix, by July 26, 2024.

2. A draft findings of the bot usability and performance testing analysis, by August 20, 2024.

3. National validation workshop report with presentations (findings and Scalability Plan) with recommendations, by September 1, 2024

4. A draft consultancy report in both English and Mongolian for stakeholders’ feedback including UNFPA Mongolia, by September 5, 2024.

5. Final reports for the UNFPA approval, by September 10, 2024

All deliverables will be submitted in English and Mongolian and shall follow the structure and detailed outlines agreed with the UNFPA Country Office.

Monitoring and progress control, including reporting requirements, periodicity format and deadline:

The consultant should work in close consultation with the UNFPA SRHR team and provide progress updates through weekly email as per the schedule jointly confirmed upon the start of the consultancy.

The reporting requirements, formats, and deadlines will be refined during the first briefing session.

 

Requirements: 

Education:

• Postgraduate degree in computer science, data science, mathematics, or statistics

Knowledge and Experience:

• At least 3-5 years of professional experience as a program or software developer including an understanding of natural language processing (NLP) and machine learning (ML) algorithms and Experience with bot development platforms.
• Proficiency in analyzing user interactions and feedback to improve bot functionality and user satisfaction.
• Proven experience with analytics tools to monitor bot performance user engagement and survey design.
• Proven experience in troubleshooting and resolving issues.
• Awareness of the ethical implications of bot interactions, especially in sensitive areas such as health.
• Experience in working with the United Nations and engaging with government and ministries, as well as CSOs, is desirable.
• A proven track record of producing high-quality reports and policy recommendations.

Languages:

• Mongolian native speaker
• Technical level of English (both in reading and writing), including the ability to review professional materials.