Across the world, responding to the COVID-19 pandemic has required immense levels of trust between citizens and governments to protect public health and adhere to safety measures. The Centre for Economic Research in Pakistan (CERP), a leading independent policy institution focused on improving decision-making through rigorous quantitative research, set out to measure the extent that citizen compliance on state directives to reduce the spread of COVID-19 depends on whether citizens trust the state and believe it to be acting in their own interests. This article presents how they did so via computer-assisted telephone interviewing (CATI) in major urban centers in Punjab, Pakistan using SurveyCTO.
How was the survey conducted?
The Trust and Compliance Survey project consists of one baseline survey round that takes 15-20 minutes to conduct over the phone and an endline survey that is soon to be scheduled. The questions focus on citizen attitudes towards government and state institutions (e.g., trust, perceived capacity, benevolence), self-reported behavior regarding social distancing and reporting symptoms, and perceived norms. Out of a sample size of 15,000, the CERP Survey team, the data collection unit at CERP, conducted 5,700 surveys, with a response rate of 38%. The sample came from an ongoing CERP project, The Social Compact project, which seeks to strengthen the link between the provision of local services and local property tax collection in urban Pakistan.
The CERP Survey team used SurveyCTO’s CATI starter kit to design and implement this phone survey project. Using case management, a data collection workflow centered on cases that can represent any unit of evaluation in the project (e.g., households, patients, water pumps), they were able to assign cases to enumerators and track the progress of contacting respondents.
Enumerators used their personal mobile phones to conduct the survey. They called respondents using SurveyCTO’s “phone-call-dialer” field plug-in that launched the dialer application on their devices and they captured responses in SurveyCTO Collect, the mobile app. (SurveyCTO has since developed the “phone-call” field plug-in that launches calls using phone numbers stored in SurveyCTO forms and also displays masked and labeled phone numbers for respondent privacy protection.) The survey form was programmed to capture three contact attempts and discontinue attempts if the respondent was unresponsive after the third call. Additionally, the times to call back respondents were scheduled and displayed on the “Manage cases” tab on enumerator phones.
The CERP Survey team also modified the sample dataset definition included in the SurveyCTO CATI starter kit to help enumerators better track calls. If respondents missed enumerators’ first call attempts and returned the calls, enumerators would have to quickly find the case of that particular respondent in their “Manage cases” tab, which only displays respondent names. But if enumerators didn’t have the name beforehand, it was challenging to identify the case. To make it easier, CERP made a simple change to the file so that it would display respondent phone numbers as well.
How were enumerators recruited and trained?
The CERP Survey team assembled a team of 25 enumerators, some of whom already had experience conducting in-person surveys for previous CERP projects. For new hires, CERP was able to recruit from traditional channels and conduct interviews over the phone. CERP also administered a hiring test via SurveyCTO to assess knowledge of survey best practices and adeptness with using SurveyCTO.
The enumerator team was separated into three groups. Project managers conducted four virtual training sessions over Google Meet for each group that included the following components:
- Screen sharing to demonstrate how to use SurveyCTO Collect and the case management workflow.
- PowerPoint presentations that covered the SurveyCTO platform components and the survey questionnaire.
- Virtual mock interviews within each group to model approaches for asking sensitive questions and engaging with uncooperative respondents.
- Remote assessments to test enumerators’ understanding of the survey protocol and data collection tool.
They held additional training sessions midway through the project for enumerators who were experiencing challenges. Each group also had its own WhatsApp group to discuss experiences and support each other.
How was data quality monitored?
The CERP Survey team used audio audits, a SurveyCTO feature that enables recordings of some or all survey administration for a greater understanding of how the surveys are actually being conducted. Audits were applied randomly and also to the portion of the interview when respondents were asked to provide consent and answer more sensitive questions.
They also monitored how surveys were being conducted with device sensor readings that measure sound and conversation in real-time. Project leads first conducted surveys to capture normal levels of sound and conversation and used them as benchmarks to identify outliers that they investigated through back-checks.
Lastly, the team added another layer of quality control by applying SurveyCTO’s text audits to more sensitive questions to capture the amount of time spent on each one. If the results of any of these monitoring approaches indicated consistent issues with enumerator performance, the enumerator was retrained.
Learn more
More information about CERP’s broader efforts to track the impacts of COVID-19 and develop evidence-based responses can be found here. For any questions about the Trust and Compliance Survey and other projects, reach out to survey@cerp.org.pk.
To learn more about how to use SurveyCTO for phone surveys, explore the CATI starter kit and watch this webinar recording for a helpful overview. If you’re a SurveyCTO user and need help, our support team is available 24×7. Go to the Support Center and click on the link to “Submit a support request.” No question is too small.
