Human-centered design boosts digital data collection in Rwanda

Photo: CIAT/StephanieMalyon

Usability testing of mobile phone dietary survey in Rwanda raised response rate from 58% to 70% (75% among women) and halved the number of low-quality responses.

The widespread adoption of mobile phones around the world has raised hopes of reaching farmers in low- to middle-income countries, both to provide agricultural advice and to collect data. Yet the unique behavioral challenges posed by farmers have largely dissipated the impact of a wave of digital services and projects that has sprung up around the world.

Researchers working under the CGIAR Initiative on Digital Innovation faced this challenge trying to collect data on diet quality in Rwanda. They had developed a system that uses mobile phones to collect high-frequency and disaggregated data on the quality of people’s diets across the entire country. This represents information that did not exist before, and a vital resource for decision-makers to detect when market conditions affect food security, and who is most affected.

The system adapts an existing survey, the Diet Quality Questionnaire (DQQ), so that it can be sent to users of VIAMO, a social enterprise that provides mobile phone-based information services to almost a tenth of the Rwandan population, in return for a small amount of airtime for completion. DQQ is used to monitor diets globally by the Global Alliance for Improved Nutrition (GAIN), which can now include data from Rwanda thanks to the mobile phone survey.

A team of researchers working under the CGIAR Initiative on Digital Innovation faced this challenge trying to collect data on diet quality in Rwanda. They had developed a system that uses mobile phones to collect high-frequency and disaggregated data on the quality of people’s diets across the entire country. This represents information that did not exist before, and a vital resource for decision-makers to detect when market conditions affect food security, and who is most affected.

It consists of a simple survey, the Diet Quality Questionaire (DQQ), sent to users of VIAMO, a social enterprise that provides mobile phone-based information services to almost a tenth of the Rwandan population in return for a small amount of air time.

Video: Learn more about the Live Food System Monitoring project in our webinar

Although the team was successful in creating a system capable of generating a large dataset at a fraction of the cost of traditional methods and with similar levels of accuracy, they were concerned that many people, especially older generations, had difficulty completing the survey. Before starting a second round of the survey, they turned to colleagues in the Inclusive Design and User Research team at the Alliance of Bioversity International and CIAT to investigate if they could improve participation.

If behavioral constraints prevent digital innovations from benefiting farmers and other groups, human-centered design starts by identifying and working with these constraints rather than expecting people to adapt. It provides of a practical set of research methods that can be applied at any point in the project cycle. In this case, the team combined three approaches: expert assessment of the survey usability, in-person tests with eight users, and a comparison of usage metrics before and after the usability recommendations were implemented.

“The usability test demonstrated how human-centered design can significantly improve system performance even mid-development,” said Daniel Jimenez, senior scientist at the Alliance. “This publication provides clear evidence of the impact of human centered design: improving survey completion rates, enhancing data quality, and reducing costs.”

Expert assessment

Usability experts assessed the survey according to ten principles for interactive design. This highlighted several improvements that could be made, such as a status indicator of how many questions remained or improvements for users to redo sections or complete the survey in more than one session.

The 10 Nielson principles for heuristic design. Credit: Interaction Design Foundation, [CC BY-SA 4.0] (https://creativecommons.org/licenses/by-sa/4.0/).

Usability testing

Eight testers were recruited from the Rwandan population, four female and four male. For a small payment, they agreed to take the survey in a room set up with cameras to observe their interactions with the survey and their body language. After 10 minutes spent taking the survey, they were interrupted and asked about their experiences filling out the survey. Afterwards, all these observations were compiled into a “user journey” that identifies the different steps users go through and highlights points where challenges were faced and how frequently they occurred.

Usability test set up: (Left) Participants were filmed by the computer placed in front of them recording their activities; (Right) Interactions with the system were recorded by the hanging phone and streamed to two observers.

“This testing approach allowed the team to step into the users’ shoes and see the innovation from their perspective,” said Charlotte Schumann, an advisor on user research and human-centered design for the project. “It’s remarkable how small adjustments to service design can have a big impact on the user experience.”

Both the expert and usability testing identified similar opportunities for improvements, such as changing words that might be confusing, avoiding error messages in English, and shortening questions to fit on small screens. All these issues were addressed for the second version of the survey.

Results

To understand whether the changes had an effect, the team looked at two metrics. The first was the proportion of people that completed the survey having started it. The second was a measure of quality, detecting if it was likely that people skipped through the survey to receive their reward by answering ‘yes’ to the questions. Responses that deviated from the median for ‘yes’ responses by a certain threshold were flagged as low-quality.

The second version of the survey, which collected almost 30,000 responses between September 2023 and March 2024, showed improvements in completion rate and quality across both sexes and all age groups. The completion rate improved to 70%, up from 58% in the first version, while the proportion of low-quality responses dropped to 11.7% down from 22.6%. The improvements had the greatest impact in completion rate for female respondents, and for the quality of responses from older respondents, demonstrating how human-centered design is necessary to build inclusive systems.

The proven benefits of human-centered design

The goal of the crowdsourced diet quality monitoring project was to create a new high-frequency, population-level dataset at a much lower cost than traditional methods. Improving the system with human-centered design lowered the cost still further: as fewer responses had to be discarded as lower quality, this saved US $8,000 - or 10% of the cost per respondent. A further US $1,500 was saved by increasing the completion rate.

The usability testing took place over just two days, costing US $20,000. If it had been incorporated from the first phase of the survey, it would have already paid back through cost savings and will continue to do so as data continues to be collected. In addition, by increasing completed responses by women and the quality of responses from older generations, it gives a better representation across different sectors of the Rwandan population.

"Human-centered design can have a relatively high upfront cost, but we have shown this is more than outweighed by the long-term benefits in greater inclusivity, quality of outputs and cost savings, even with commonly used technologies such as mobile phone surveys," said Anna Muller, inclusive design and user research team lead at the Alliance. "Even though usability testing is worthwhile for solutions that are already deployed, it is even more beneficial to implement them at an early stage of product development".

"We hope this work encourages those developing digital innovations based on digitalized data collection to incorporate human-centered design early on to maximize its benefits,” added Jimenez.