How Will AI Reshape Data Collection? With Lawrence Li and Dr. Christopher Robert of SurveyCTO
- Premiere Date: October 26, 2026
Show notes & key takeaways
In this episode of Survey & Beyond: The Data Collection Podcast, host Marta Costa sits down with Dr. Christopher Robert, Founder Emeritus of Dobility, and Lawrence Li, President and CEO of Dobility, the company that powers SurveyCTO. Together, they examine how AI is changing the data collection lifecycle, from instrument design and fieldwork to data quality, data security, and the credibility of empirical research, and what organizations should do now to prepare.
What You’ll Learn:
- Where AI is already changing the data collection lifecycle
- How AI makes full-scale data quality checks economically viable
- What organizations overestimate and underestimate about AI in survey research
- How AI affects data privacy risks, and how it can help mitigate them
- Why AI should be managed like a junior colleague
- How AI adoption will differ across industries and low-connectivity settings
- And more!
Lawrence Li is the President and CEO of Dobility, the company behind the SurveyCTO data collection platform. He brings a background in consulting and investing, and leads Dobility’s strategy at the intersection of technology and social impact, with an emphasis on data quality, data security, and responsible AI adoption.
Dr. Christopher Robert is the Founder Emeritus of Dobility. A technologist by training, he began collecting data for social impact in Cambodia in 2003 and later relied on large-scale field data for his doctoral research in public policy. Persistent data quality problems with paper-based methods led him to build the tools that became SurveyCTO.
If you enjoyed this episode, make sure to subscribe, rate, and review on Apple Podcasts, Spotify, and YouTube. Instructions on how to do this are here.
Episode transcript
Lawrence Li & Christopher Robert Transcript
Christopher Robert [00:00:00]: I’m an AI optimist, as I’m sure every listener will have figured out by now already. One of my big hopes is that AI will help us get more from the data we collect in at least sort of two different ways. One is it reduces the cost of making better and more use of the data we collect because we can get more out of qualitative data, which I mentioned earlier. It can also help us to draw from prior data we’ve collected and combine data in novel ways.
Marta Costa [00:00:31]: Today on Survey and Beyond, we are joined by Lawrence Li, President and CEO of Dobility and Dr. Christopher Robert, Dobility’s Founder. Lawrence offers his experience as an investor and operator, while Chris brings his background in development, economics, and technology. Together, they are here to explore what meaningful access to better data collection looks like today and how emerging artificial intelligence technology could reshape the way organizations collect, protect, and use data. Lawrence and Chris, welcome to Survey and Beyond.
Lawrence Li [00:01:09]: Thank you. Excited to be here.
Christopher Robert [00:01:010]: Yeah, same here. Thanks for having us.
Marta Costa [00:01:12]: Thank you. It’s an honor. And we usually start the podcast just getting to know a little bit more about our guests. And today is particularly exciting because we have the founder of Dobility, and we have the current president and CEO. And you two have very distinct backgrounds. So, Lawrence, for example, you have a lot of experience in consulting and investing. Chris, you started with technology and then public policy and development economics. So, can you just let our listeners know what drew you to these fields and maybe, Lawrence, we can start with you.
Lawrence Li [00:01:48]i: I’ve always been interested in how technology transforms the world around us in general, but especially over the course of my career in terms of new business models, how technology transforms how organizations work, what new services and new functions are available, and really, over time, learning more about it. I was excited to be an operator and get into a technology company. And it was only after kind of looking for that and meeting Chris and Dobility that my eyes were really opened to the intersection of technology and social impact, and as I got to know Dobility and SurveyCTO, really, understanding the interesting technical and user challenges around data collection, something I certainly took for granted and I think very commonly taken for granted. So, it’s been a little bit of an unorthodox path here, but I’ve been really enjoying it.
Christopher Robert [00:02:34]: Yeah. And my path has been a little bit roundabout as well. Like you mentioned, I sort of started out as a technologist, and then I really had my first real experience collecting data for social impact back in 2003 when I was living in Cambodia. And then in 2009, it became really important to my dissertation research when I was getting my PhD in public policy. Back then we were using paper and pencil methods. We were trying to catch and resolve quality issues after the fact in the office. We were even building custom software for double entry to try to improve the quality. And we still kept finding that however much time and money and energy we put in, data quality was still not quite good enough. And so, this was really a threat to the analysis that I was doing personally, but I thought it was a threat more broadly to public policy and social impact and the evidence we’re using to guide our work. And so I became really interested in finding a better solution. And that’s how I just kind of ended up at this intersection of data collection and technology in the social impact space.
Marta Costa [00:03:44]: So, today I’m really eager to hear your thoughts about AI and specifically in this field of data collection. And I think I want to start with maybe a high-level question around what do you think is fundamentally changing about the way organizations are collecting and using data?
Christopher Robert [00:04:05]: So, I think in general, change has been pretty gradual, more gradual than I had expected, to be honest. And that’s particularly true among the users and use cases we work with a lot at SurveyCTO. Our users tend to be pretty cautious and deliberative in how they adopt new technologies. And so, I don’t think they’ve been particularly quick to embrace change. I do think that by now everybody’s starting to see that there are specific opportunities. So, for example, leveraging qualitative data. We used to collect a lot of qualitative data almost incidentally with our data collection efforts, but for the most part it might be useful when you happen to be looking at a specific submission, a specific interview, but we didn’t have really great ways to leverage and analyze that data effectively at scale. And so I think there’s growing openness to the ways that AI allows that to happen. And that’s, I think, slowly sort of becoming a little bit transformative within data collection, even for our users. I think people can also do more analysis faster than before, and they can build instruments better and faster than before, particularly if they use an AI agent and they use an appropriate skill, like what we’ve tried to do with the SurveyCTO agent skill, where you have an AI agent that’s able to really put a lot of effort into doing things right, but doing them quickly. I still think that these changes are mostly at the margins of the work, though. So, I wouldn’t say that’s fundamentally different than it was, but I certainly think that it will be as time goes on.
Lawrence Li [00:05:44]: I’m with Chris on this, and I think one really interesting framing here is maybe we’re seeing AI tools sort of change the capacity for certain tasks. And when you think about that at a workflow level, there’s an immediate impact, but then there’s a meta impact. The immediate impact may be, hey, we used to only listen to one out of every thousand audio recordings and audit the submission, and today it’s possible to use transcription to review all of them and compare all of them, and that changes one bottleneck. Another one that we’re familiar with is that powerful platforms like SurveyCTO are often harder to learn. There’s a steeper learning curve, and often people are drawn to easier-to-use tools or even just do-it-for-me, getting consultants and external resources to use the tools. And that’s another bottleneck that I think AI is already moving the needle on in terms of change. We’re seeing a lot of use of our AI assistant, our AI form tools, to really accelerate people’s use of our platform. I think today we’re seeing those individual tasks change, but it’s easy to look ahead and say, as organizations digest that change, the fact that that capacity at an individual step of the workflow changes should actually lead to the overall workflow changing, whether it’s how you approach instrument design, whether it’s how you approach quality check, process design, knowing now that you can have every single audio audit reviewed, or knowing that you have this kind of AI processing capacity, or even how you decide on building versus buying a powerful platform versus a simpler platform, services versus tools and capacity in-house, all of that, I think, will be affected by the individual changes that AI is driving that Chris talked about. So, I think there’s a clear path to the marginal and incremental changes becoming more transformative as organizations kind of digest the implications.
Marta Costa [00:07:31]: And if we zoom in a little bit and we take a look at the data collection life cycle. So, usually when we see a data collection project, we think about stages. There is the design stage, the collection stage where we collect the data, the monitoring stage, and then exporting and the analysis of the data to eventually use it to make important decisions. So, if we take a look at all these stages within a project, where do you think you see the biggest transformation from AI?
Lawrence Li [00:08:05]: When you think about the full data lifecycle, I think the one we’re seeing, or I’m seeing a lot, and I’m really excited about is organizations leveraging AI to think about more customization in terms of the tools they use. I think in our space, in particular, but in a lot of industries, you’ve seen this sort of dynamic where software is built for maybe the modal user or the biggest user, and then the rest of the organizations or adjacent similar use cases have to twist themselves to use that software or a specific stack and kind of really almost organize their people and organization around the software rather than the other way around. I think AI has made it possible, certainly, to build things from scratch. But what I see a lot of is building, knitting, integrating, customizing in a way that makes the overall stack serve the organization’s specific purpose more effectively. And so, certainly, I think that’s a change that I’m really excited about because I think it’s really positive for a lot of user organizations and it’s great, frankly, for a platform like SurveyCTO that brings a lot of power and customizability, but hasn’t always benefited from that. So, definitely that’s been one of the biggest things we’ve seen both from existing users and prospective ones.
Christopher Robert [00:09:14]: Yeah. And like I mentioned earlier, I think there’s been early contributions sort of at the very beginning and toward the end of the overall life cycle. So, when it comes to design and getting assistance with design and building instruments on one side and then getting help with cleaning and analysis on the other, I think those are, in some ways, I think about researchers’ use of AI as sort of starting on those ends and sort of moving inward. But if you think about what Lawrence was just saying about being able to customize tools, one concrete example of this is that we released field plugin capabilities. I think it was like early pandemic, so spring 2020, and we thought this would open up tremendous customization opportunities. But the reality is that it was a little too technical and too complicated for the vast majority of users to really leverage. So, they couldn’t customize the actual data collection experience, whether it was a web form or something on the device offline. Whereas now, if they use, for example, the agent skill, like, so many more people can build and test field plugins to get exactly the collection experience they want. That’s just one example. And I think there’s just going to be more and more and more examples of that kind of thing where people are able to do this level of customization that before was just out of reach for everyone but the most well-resourced teams.
Marta Costa [00:10:39]: And do you think like in the future there’ll be a different part of the data life cycle that AI will have a big impact on?
Christopher Robert [00:10:48]: I have lots of hopes. I’m an AI optimist, as I’m sure every listener will have figured out by now already. One of my big hopes is that AI will help us get more from the data we collect in at least sort of two different ways. One is that it reduces the cost of making better and more use of the data we collect because we can get more out of qualitative data, which I mentioned earlier, without a whole team of qualitative coding assistance and so on, and it can help us also analyze quantitative data in new ways. I think it can also help us to draw from prior data we’ve collected and combine data in novel ways. That was just a little too difficult before. And then basically an even bigger hope is that it helps us to start better leveraging other people’s data so that we’re not all reinventing the wheel; we’re not all collecting the same data again and again and again, sort of enormous cost for everyone. Despite Herculean efforts, it’s just never really worked out very well using other people’s data, often because it’s sort of difficult to share in the first place. And then if people share it, it’s sort of difficult to fully understand, like the collection process and the methods used in the field to collect the data. Nobody really knows whether they can trust it. And so basically that’s just been too hard. And I hope that AI will help us get over that in a few key ways. And between all of that, I sort of hope that we end up having to collect less data, which will speed research, reduce costs, and hopefully increase quality. So, that’s sort of my big picture hope.
Lawrence Li [00:12:30]: I definitely see that future that Chris is describing. And I think for our listeners, collecting less data can mean really different things. So, one example might be that today, we’re used to collecting in a fairly enumerated, intermediated form, structured data context. And with the power of AI processing, it now makes much more sense to leverage things like long-form audio, qualitative responses, video, and photos. So, are you collecting less data if you don’t fill out a 300-question form, but instead have a 30-minute conversation and take 5 pictures, and those 5 pictures can be analyzed for myriad fields? Maybe. Maybe not. That’s sort of the kind of question in data collection workflows that I think AI will imminently introduce. I also hope for one that AI drives adoption of features and functionalities and practices that we sort of all know are important, but are often underutilized because of time or cost. I think certainly we see a lot of this around data quality and data security: users who tell us, “We know this is best practice; we haven’t figured out how to set this up”, or “We know this is best practice; we could do it on SurveyCTO, but we can’t always afford that.” And we just know that’s important. But it hasn’t quite risen to the threshold of something we can tackle. And hopefully the efficiency introduced by AI tools, the fact that it speeds up the learning curve, individual work, whether it’s at the form design level, setting up encryption, whether it’s just making post-processing a lot more convenient, that sort of swings into having organizations leverage the existing quality and security features that we’ve always invested so much time in and really believe is central to making it safe and responsible to collect this level of data.
Marta Costa [00:14:08]: I’m curious, what do you think people could be underestimating or overestimating about AI’s impact in data collection?
Christopher Robert [00:14:19]: So, I think that’s a good one, and I’m sure opinions differ pretty radically on this. I think I saw very early on with the new generative AI options that were out there that people immediately became really eager to use AI to actually do the data collection, collect data from people. It’s a little like what happened with like crowdsourcing data collection when there was a little wave of attempts at that; basically, it’s expensive, and it’s messy to have humans interviewing other humans. And so I feel like everybody’s been super eager to, like, get out of that any which way that’s possible. But I think people are overestimating how ready AI is for that and how ready human respondents are for that. In terms of underestimating, I think a lot of people are still underestimating the value that AI can add during instrument design or analysis, sort of two ends of the data collection process. I think a lot of people think really highly of their own training and judgment, which, fair enough, we all have egos, or they have quality and privacy concerns, and they’re not quite sure how to resolve those. And so, I think that many people, I think, are still underestimating the value that’s there and available today. But that said, lots of people, and I’ve been guilty of this myself, overestimate AI’s ability to work with minimal supervision. I think right now there’s some sort of balance required. AI offers incredible promise and can really speed up and improve our work, but it really needs those of us who have expertise and judgment to manage that AI closely, kind of, like, you would with junior colleagues. You wouldn’t say, “Oh, I’m not going to work on a team, and I’m not going to have junior colleagues, and I’m not going to let them help with my work because I don’t trust them to do everything perfectly.” Of course you want to integrate junior colleagues, but you also need to use your own expertise and judgment in guiding them and ensuring that outputs are high quality and the work is high quality. And I think that’s kind of how I think about AI today as well.
Lawrence Li [00:16:20]: I have a very similar take. I think if I had to kind of nutshell it, I would say I still think people underestimate the power of tools that AI enables and the power that AI brings to tasks, but they overestimate the seamlessness or ease of adopting it. And I think often the two get conflated. And so people will come in, find something that feels a little bit technical, feels a little bit hard to set up, or see a hallucination early and not think about the ability to set up layers of quality checks, etc., and walk away from the tool. On the other hand, especially with how fast the capabilities are progressing, even folks who are staying pretty close to the leading edge sometimes will demonstrate something to them, and it will blow their minds. I think the underlying power is really, really impressive. But there is a layer of kind of whether it’s thought, like Chris said, parallel managing junior hires. But even like I was saying earlier, thinking about how the removal of a specific constraint can change the rest of the workflow. And a lot of the benefits will come from this meta-level judgment, this organizational and process kind of adoption and judgment. I think that ends up getting underestimated, and sort of the two end up creating this tension that leads maybe to underadoption, underutilization or misutilization. But it’s really kind of two sides of the same coin in a lot of ways. It’s easy to conflate, slightly harder to set up or requiring a little bit of human judgment to unlock, to less powerful than I think. And then you also see the individual things it can do and the speed with which it can do it, and it blows your mind, and it’s more powerful than you think. And really that’s because it’s not one axis. I think it’s sort of orthogonal problems, and they need to be thought about that way.
Marta Costa [00:17:54]: I was thinking about looking ahead, what AI could enable or what could become possible that now is not practical or not as scalable. I know you already mentioned a few things, even how people can now use and collect qualitative data as before, but I don’t know. Do you have any other things that come to your mind in terms of what can become possible that was not before?
Lawrence Li [00:18:20]: I mean, certainly there are a lot of these, and we could even go category by category. But the one big one I’m really excited about, just because I know it’s top of mind for a lot of our users and industries that do data collection at this scale, is, I think AI enables a massive step up in data quality. We at Dobility have spent a lot of time thinking about data quality, what features are necessary, and we’ve released these features, but the leverage of AI’s ability to speed up and lower the cost of tasks – just the example I gave at the top – organizations that review one in a thousand audio audits can now review every single one. And the audio audit might previously have been reviewed for major glaring discrepancies. Now you can have it transcribed and check for internal consistency against every field in a submission. That’s the kind of thing that just the sheer mechanical efficiency unlocks something we all knew was important, but was just totally cost-prohibitive.
Christopher Robert [00:19:14]: Yeah, and I think, if you don’t mind me sort of tapping back for a second and sort of more big picture on that. I think, as a researcher, the credibility crisis in empirical social science has been getting worse all the time. It’s kind of a big deal. There’s, like, reproducibility issues, credibility issues more broadly. And the question is, like, can people trust empirical research? And I think that AI might hurt at first because I think it’ll massively worsen data mining. And when you have data mining, it sort of undermines all of the assumptions that go into p-values and our sort of ideas of statistical significance. But I’m really, really hoping and expecting that it’ll help in one big area, which is, I think the fundamental statistical foundations on which we base most of our analyses are just not up to the task. And I think this is fundamentally why the credibility crisis is what it is. I think that frequentist statistics, statistical methods, are simple; they’re easy to teach, they’re easy to implement, but they’re an incredibly blunt instrument, and they have these flaws that are just absolutely inherent. And I think that we’ve known, I think since World War II, that Bayesian statistics are much more powerful and much more able to confront a sort of messy real world. But Bayesian methods are a lot harder to teach, and they’re even harder to implement in practice. And so, only if you have a ton of time, money and expertise to put into higher-quality prediction or decision-making like you do on Wall Street, those are basically the only places where you see Bayesian statistics being used today. I think that AI will shift to calculus so that basically Bayesian statistics can become much more the norm more broadly in social science research. And I think that will have dramatic long-term effects. I think smaller picture. I do think that we will see AI interviewing that’s sort of increasingly capable. That starts with high connectivity, developed world applications like job interviews; you’ve been seeing that for ages now, to eventually getting to lower connectivity, more developing world settings. But I think Lawrence has mentioned a couple of times now: photos, video, audio recordings – all of these can become first-class methods of data collection in a way that they just weren’t feasible or cost-effective before. So, one small example is like some of my old surveys, like we’d have survey questions about the material of the home’s roof or its floor, or what assets are in the living room, how many televisions do they have, things like this, and you can imagine now just running around and taking a quick video, and then AI extracts all the structured data you need from that video. And in fact, you might be able to extract more data than you thought you wanted to begin with. Maybe it’s not just televisions. Maybe you realize that there’s something else that’s really important and visible that distinguishes sort of higher-income households from lower-income households in different settings. And those are the kinds of things that I think AI is going to enable.
Marta Costa [00:22:25]: I will go back to the data quality theme later on, which is quite important to us. But now I feel like we have already discussed so many different things that AI already does that usually, you know, a data collector would do before. Like, for example, we mentioned how much is possible now in terms of designing forms, in terms of customizing forms, in terms of translations, transcriptions; there is already so much that AI can do. So, how do you see this role of the researcher or researcher in academia, but sometimes the form designer, the data collector, whatever the role is, how do you see this role changing now with AI?
Christopher Robert [00:23:12]: Well, I think for researchers this is a tough one, and it’s either extremely exciting or extremely terrifying depending on your perspective. I think right now there’s still incredible value for technical expertise in order to manage and sort of quality-check AI work, like I mentioned, but over time I expect that to actually change dramatically, and it’s really changing day-by-day. Technical expertise is becoming more and more of a commodity, and I think we’re on track for it to become basically free for everybody, including the expertise needed to quality-check and manage AI agents, which we can talk about later. But I think at that point we come back to a place where generalists armed with really deep interests, sharp judgment and really local knowledge might come to dominate research in social sciences again, the way they did at the beginning of social science when it was philosophers really thinking broadly. And before we developed super technical methods and hyper specialization in academia and in research in general, you had broadly thinking generalists who were able to do really incredible things. And I think we might see that again. Obviously, that will cause us to have to rethink who does what, how organizations organize themselves, basically everything about how research work is done. And in education, we have to figure out how to train people to live happily and productively in this new world. And unfortunately, I think that transition is going to be kind of a challenging one for all of us.
Lawrence Li [00:24:52]: And I’ll add one here related to Chris’s point, which is that what we’re describing is a massive change on a tiny timescale. And so, of course, one of the ways a researcher’s job will transform is to become a shepherd of that massive change. If we’re looking at the level of change in role that Chris is describing, senior folks in organisations – researchers – will have to help those organizations navigate this change, navigate the shift away from deep technical expertise being the gating factor and what people mostly provide, and shifting towards making sure human judgment is correctly identified, allocated, implemented with lower-cost, AI-empowered, maybe 10x as many studies, because now each study takes less time, and so the whole approval process should change and how senior you have to be before you can pilot in a kind of research topic might change. There’s a lot here that organizations, people, behaviour, and processes will have to change along with that. And it will certainly be a big part of the role of anyone close to data collection to be a part of that change and be driving it and be thoughtful about it in order to get there.
Marta Costa [00:26:01]: So, the million dollar question I guess is will there be any jobs left? Lawrence, are we concerned about the disruption to our users and their work?
Lawrence Li [00:26:12]: Certainly, I’m not smart or arrogant enough to say no. There’s no worries that you can imagine all sorts of levels of continued advancement in this technology that really do make even the sharp generalist judgment that Chris was describing irrelevant. I think for the purposes of my own thinking, I tend to categorize away from that and just say in that version of the world, maybe we have superabundance, maybe something unforeseeable happens and I can’t really claim for that, but I think if we narrow the scope and kind of look at the changes happening now in a historical context, greater demand for data collection as we have more value from the analysis, that would directly parallel historical technological revolutions. If the Industrial Revolution didn’t lead to the same quantity of goods being created with 110 the labor, instead we saw probably more than 10xing of the quantity and variety of goods that were being produced. And I think for those of us close to research and close to data collection, probably everyone we know has the next 10 things they’d like to look for or the amount they’d like to increase their sample size. The interesting and pertinent question that is not quite as high a priority as the one they’re working on. So, I do think there’s a lot of signals to really believe that AI unlocking capacity will inherently remove the need for the next tier of work and the next tier of work, and instead will become kind of a source of labor leverage, as we’ve seen in the past. I think in our space specifically, often human in the loop is required because SurveyCTO is used in a lot of logistically challenging situations, whether that’s low connectivity, whether that’s low accessibility, whatever the case may be, used in environments where people are absolutely necessary to even get in the proximity of the data you want to collect, irrespective of whether it’s qualitative or audio or video or photos. I think people are just sort of necessary there. We’re not within range of flock cameras or anything like that. So, that certainly helps in our spaces in particular. I think the other one I’d flag is that more and more SurveyCTO has been used in contexts where data collection isn’t what people think of as the activity they’re doing. It’s tangential or parallel to the thing they’re doing, and it happens to take up some of their time. So, we’ve talked about cases where SurveyCTO is used as an EMR in a clinic with low connectivity and a tight budget. In that case, the doctor, the nurse, the front desk check-in – none of them identifies as data collectors – and if you make data collection more seamless for them, that has done nothing to change the actual value-added task they’re doing. Similarly, if you’re talking about inspections or audits where someone with technical training is going to look at a building or a warehouse or validate agricultural practices, in a lot of these cases, the collection of data is necessarily a part of what they’re doing. But they’re not identifying as I’m primarily a data collector. In fact, you can imagine directly freeing up their time to apply their expertise to more photos of farms that are collected or whatever the leverage creating enablement of AI is. So, I do think, especially in our space, I feel pretty comfortable that what we’re seeing already and what we can project probably doesn’t involve the kind of displacement that certainly some doomerism views have mentioned. That’s not to say that that couldn’t happen with changes far beyond my ability to anticipate, but that kind of gives me a little more optimism.
Christopher Robert [00:29:29]: Yeah. And I think I would just emphasize something Lawrence mentioned in the beginning about the sort of quantity of what’s being produced. If we look back at the Industrial Revolution and sort of physical goods production, right, it wasn’t that the same number and variety of goods were being produced just with far less labor, and therefore everybody’s sort of out of work, like, that’s not what we saw at all. Rather, it enabled a massive increase in the production of these goods. You may have different opinions about walking into a modern Walmart, but like the fact that all of these goods are available at the kinds of cost they are was a function of what was a very disruptive change that played out over 100-plus years. I think what’s difficult about the current moment is that AI is coming to knowledge work much more rapidly. So, I think it’s exciting that we can see potentially a massive scale-up in knowledge work and all the benefits that come from that, including research, including policy analysis, including lots of other things, impact assessments; rather than rigorously measuring the impact of 1 out of 10 programs, you’re able to do it for 10 out of 10 programs. Like, I think that will be transformative and positive, but obviously still massively disruptive as we all sort of adjust to this, particularly on a very short sort of time horizon.
Marta Costa [00:30:54]: I want to elaborate a little bit on what Lawrence mentioned. Lawrence, you mentioned a lot of different industries, a lot of different types of companies and organizations that are collecting data, even people that might be collecting data, but they don’t see themselves as data collectors. I think in our podcast we even try to have a lot of different guests from different industries to show people that there are people around the world in so many different industries collecting data in different ways. And you both have experience working in these different industries. We can have international development. We also have market research, as you mentioned, warrants, inspections, audits, global health – so many different industries. Do you think that AI will impact these different industries in the same way or in different ways?
Christopher Robert [00:31:44]: I think there are a lot of axes on which there’ll be a lot of differences. So, for example, the mode of data collection, I think, matters a lot here. So, self-response web surveying, I think, will be transformed much faster than phone interviewing, for example. And I think that phone interviewing will be transformed much faster than in-person interviewing. And so there’s a lot of difference across the world and across sectors in the mode of data collection they’re using. And I think there’ll be different paces, connectivity, language, and different levels of comfort with technology will slow or speed the transformation in different places. I also do think different industries face different pressures. So, for example, we’re seeing that market pressure is really speeding AI adoption in a lot of private sector settings. To many people’s disappointment, many business leaders are fearful that if they don’t adopt AI, their businesses will be left behind. And by left behind, I don’t mean like in a FOMO kind of way, but business leaders are worried that if they don’t adapt quickly enough, their businesses will just fail because they were too slow to act and they failed to remain competitive. Social impact organizations, I think, particularly nonprofits, can operate pretty differently. I think they can move as quickly or as slowly as their funders allow. And frankly, from what I can tell, many funders are either straight up distrustful of AI, or at least not confident enough to press their grantees to adopt it before they’re ready. And so I think a lot of nonprofits we see moving more slowly and more cautiously. That said, in areas where funding is competitive, I think you’ll start to see the same kind of pressures that private sector orgs have been facing already, which is that if those orgs that are new or adopt AI more aggressively, if they’re able to deliver more for less, you might find organizations facing a lot of pressure in the sort of competition for donor dollars. So, they’ll start losing bids because they’re not cost-competitive the way they used to be. In academia, I think academia is another area that tends to move at its own pace, tends to be a very, very slow pace, but at least in the U.S and I’m not sure how much this is true globally, the traditional model of academia is kind of under siege, and its affordability is a pretty meaningful part of what’s fueled the current crisis. And so I think we might see academia start moving more quickly if colleges and universities believe that AI can help them reduce costs and become more affordable again. Because if they don’t, they’re kind of in big trouble.
Lawrence Li [00:34:29]: I think that all makes a lot of sense. I really like how Chris is exploring this from the perspective of kind of the stakeholders and the incentives they create. I think one other lens I might add here is that I think Chris went through different organizations, their types of stakeholders, and how they might therefore be incentivized to react to the general progress of AI. I think another cut here is that different organizations in different industries have different pre-existing pain points that are suited to being addressed by AI. So, for example, I might expect adoption of AI-powered scale data quality tools to vary widely depending on how much data quality was already an issue for the organization. In research, replicability is a big deal. In all of the carbon credits we’ve gone to, the credibility of the carbon credits themselves is a huge focus. These are areas where I’d expect that any time progress in AI tools kind of allows stepwise unlocks in these preexisting pain points, really, really fast adoption and kind of an eagerness to solve the problem, not because it’s AI, not because it’s technology, but because this problem’s already been plaguing them. And so, I do think there’s another lens to look at this, which is for a platform like SurveyCTO serving myriad users in a bunch of different industries. Certainly, one way to think about it is how their stakeholders think about AI in general. The other way that I find really relevant is thinking about the specific pain point within that industry that could be addressed by AI, maybe more narrowly or more immediately. I think you’re going to get a lot of attention and engagement on those dimensions.
Marta Costa [00:35:58]: What about different environments? So, we serve a lot of users in places where they have low Internet connectivity, sometimes less technical capacity. So, what do you think? What does AI impact look like in these settings?
Lawrence Li [00:36:16]: I think certainly you would expect the impact to phase out, roll out more slowly, as we’ve seen with virtually every technology. That being said, this is a topic that’s near and dear to my heart and near and dear to Dobility’s kind of mission as an organization. What we’ve seen is a platform that started maybe as a data collection research platform, but has really evolved into something that I feel bridges the gap created by connectivity and technical expertise, and really brings the power of a lot of modern technology into the hands of more people, more users, at a price point that’s far more accessible. When I talk about an EMR system built in SurveyCTO, it’s far cheaper than any EMR system they would be able to buy. And it works offline because SurveyCTO intrinsically works offline. So, yes, they need to return to connectivity; they need to sync up. But when you’re in a clinic with no connectivity, and you have to bring the devices back somewhere else, or you’re in a clinic with unreliable connectivity, we unlock all of the benefits that modern software has brought to healthcare in a way that would not be available otherwise. And then the same is true for supply chain audits; the same is true for agriculture. So, I think we would love to continue to play that same role in terms of AI adoption, which is really to facilitate and help be that last mile, help users cross the bridge, help a broad swath of technologies cross that air gap, so to speak, and do so responsibly, do so in a way that has data security in mind, has privacy in mind, make sure that users, the subjects or customers or patients of our users are well served and well protected while we help accelerate that rollout. So, certainly I wouldn’t expect it to happen everywhere all at once. That said, I think we are a big part of kind of making that a more democratized, more inclusive technological impact than maybe historically we’ve seen.
Christopher Robert [00:38:04]: I totally agree. I mean, I think in general, we’ll see things be slower and sort of more bespoke in these more challenging environments, but I do think it’ll still happen. And obviously organizations like Dobility hope to responsibly facilitate this transition and kind of be able to assist people through this. And I think a number of platforms and existing organizations who sort of dive in on AI early and figure out how to use it responsibly will also help the overall sector move despite all of the sort of technical and other challenges.
Marta Costa [00:38:38]: From a lot we have discussed today, it feels like AI can make technology more accessible. So, I wonder, do you think AI can help at least minimize the technology gap that exists today?
Christopher Robert [00:38:51]: Definitely. I mean, obviously, I need an AI optimist hat that I could just wear. But I think AI is bringing really rapid progress in the accessibility of just complex software and complex methods more generally. And I think we’re actually really close to having the kind of really simple voice interface with dynamic visuals that you see in science fiction, in Star Trek. It’s like people just talk to the computer; they say what they need, visuals come up to show them things that they need to see in order to then make better decisions. Honestly, I think we’re mostly there with the technology, but what’s holding us back right now is we’re really used to typing and clicking, and our home and office environments aren’t really well suited to all of us talking and listening versus clicking and reading. And how do we handle privacy, for example, if we’re all talking to our devices and we can overhear each other interacting with our devices? It seems like kind of a silly problem, given all the technical challenges involved in getting, like, a really fluid and effective voice interface, but it’s a real problem, and I actually don’t know how we’re going to solve that. We all just sit around whispering into our little microphones all the time? Like, I mean, it just seems ridiculous. So, I think there’s stuff still to work out, but I’m really hopeful that AI will sort of bridge that interface gap between sort of complex technology and methods and sort of the vast majority of people.
Marta Costa [00:40:21]: And now I want to go back to data quality that we already mentioned before. And the way that I’m thinking is we already mentioned that with AI we can work faster, we can handle more information, more data, but it feels like simultaneously that means that we can scale mistakes more easily too, which feels scary. So, what kind of data quality problems do you think AI brings, and what kind of measures or practices can people put in place just to at least minimize those?
Lawrence Li [00:41:00]: Yeah, I mean, for those of us who live and breathe data collection, see it implemented daily, I think I sort of frame AI data quality as a subset of overall data quality, which is obviously a topic we spend a lot of time and energy thinking about. And on that front, we’ve built a platform that has a lot of features for minimizing, preventing, identifying and correcting human data collection errors. And in a lot of places where we might see AI scaling volume, the errors will look similar and might be kind of pursuable through similar avenues. Certainly, there are other underlying areas that we can get into, but part of this is just kind of taking existing data quality tools and turning the scale up to 11, so to speak. So, that covers, I think, actually a decent set of them. I think another part of this is that the organizations that care most about data quality typically aren’t using an off-the-shelf feature, just at the individual user level. They themselves have built workflows and processes around audit and back checks. And in some ways AI can scale individual steps of that up. And in some ways AI might change that process a little bit, and each case will be a little bit different and thoughtful. But I think my view is a lot of the existing quality tools can be rethought and not necessarily need to be rebuilt or expanded, but rethought and used in ways that capture a lot of these AI quality issues and the scaling of the mistakes risks as you flow.
Christopher Robert [00:42:23]: I couldn’t agree more. And I think just kind of, again, stepping back to consider kind of the big picture of cue, you know, how to assure quality when AI gets involved. I honestly think we face fundamentally similar kinds of quality problems that any larger organization with the means to scale their work has always faced. I’m not a big organization guy, but basically, from what I’ve observed, right, large organization that have resources to hire lots of people and do lots of things, they scale their work, but they have to figure out a way to do that and maintain quality. I think that the problem with those of us who are starting to use AI and AI agents much more is that all of a sudden it’s kind of like all of us have entire teams of people doing work for us. I don’t know that OpenAI just announced Dots, and there was the OpenClaw thing earlier this year, and then Meta has Muse, and basically we’re seeing a world where all of us can have teams of people just working on our behalf. And we’re not used to that. I mean, most of us aren’t. I think if, maybe you’re a big company manager, like, you know just how to do all of this, and it’s just totally fine. But for most of us, this is kind of abnormal, and we’re not used to how to deal with that and how to manage quality when we’re scaling in that way. I think it sounds crazy, but at least in engineering work, I’ve seen how effective different AI agents can be at quality-checking other AI agents, especially when you give them sufficient guidance about what matters to you, like what are the problems you’re looking for and what are the problems you care most about. It’s kind of crazy to think, well, to solve AI quality problems, we just have AI check AI. It sounds nuts, but I’ve seen that work. It’s working more and more all the time and in more and more domains. So, I think that that’s a piece of it. But obviously, each of us still needs to fundamentally own and be responsible for the work that we do, the work that we vouch for, everything that’s produced with the help of AI. And it’s like, again, managers at an organization who take responsibility for the work done by their subordinates. I just think again, a lot of us aren’t super used to playing that role, but all of us have to learn how to play that role now.
Marta Costa [00:44:44]: And as a follow-up, Chris, you are mentioning that we need to be responsible for the work that we do. You also mentioned before about the trust in empirical research. You also mentioned how we should treat AI as a junior colleague and manage AI very closely. So, now that people have been adopting AI in their processes, how important do you think transparency is and how important it is for people to maybe create quality assurance processes in their work and to communicate those when they are reporting their data?
Christopher Robert [00:45:19]: I think it’s super important, and luckily it’s way, way easier than it used to be. So, if you go out to, like, the Harvard MIT Dataverse and you look at what’s shared about survey data that’s been shared, and you try to figure out how that data was collected and what their process looked like and how they dealt with quality and how did they clean the data, like, it’s hopeless. And partly it’s hopeless because, like, nobody has time to document all that, and even if they did, there’d be a 100-page PDF, and nobody’d have time to read it. So, basically, this has been a really big problem about how do we communicate and then process information about process. And again, leaning on engineering, where in software development AI methods have been adopted maybe faster than in any other area. We’ve seen, and this is true in the Dobility engineering team, as it is in pretty much teams all over the world now. Teams are dramatically better at documenting the work they’re doing because it’s part of how they use the AI. They have the AI document everything really thoroughly. They cross-check that. Every time they make a change, the AI goes and looks at the document and says, “Okay, well, what’s no longer current here? What’s no longer correct because of this change we made?” And it updates the documentation. Like, go into any organization’s SharePoint or Google Drive, open a document that documents a process or pretty much anything and just see how outrageously out of date and unhelpful it is. Like, AI is helping us to become much better about both documenting the things and then processing those; that said, people are still involved, and people can still lie and mislead. So, ultimately, I think trust is still going to be driven a lot by personal and organizational reputation. You’re going to have organizations and people that you trust are doing their work to make their work high quality, and then you’re going to rely on that trust, I think, because, basically, otherwise you just have to take all that documentation about process at face value, and maybe that’s not valid. So, I think there’s still going to be this very human element of reputation and trust, but at the same time, I think there’ll be this ability to make much more information available and consider, so for example, if I wanted to do a meta analysis, but I only wanted to consider data that was collected in a very specific way and dealt with a source of bias that I personally care a lot about and think has undermined lots of the research in the area, I can have AI agents go out and if the studies have documented themselves thoroughly enough, my AI agent can go out and find the studies that fit the criteria that I have and then help me do that meta analysis. And that’s just the kind of thing we haven’t been able to do traditionally.
Lawrence Li [00:48:16]: As we talk about this, I think about that meme of if both people in an email exchange adopt AI tools, person one is writing bullet points, their AI tool is drafting into a long, florid, beautiful prose email. Person B is receiving that email and having their AI tool turn it back into bullet points. And you could really just directly talk in bullet points. But I think the beauty of what Chris is describing is that that meme assumes that the prose is really removing information density, whereas I think Chris is talking about a class of problems where, if we could, we would provide a lot of information. If we could document our methodology, our processes, what would result is a long, long report. And for every individual reading that report, maybe what they need is an AI to generate 5 bullet points. And those 5 bullet points would be different depending on the meta-analysis or depending on what they were trying to get out of it. But the report itself is actually really information-dense. And historically, I think we’ve just had a really hard time navigating situations like that where if I need to create a 300-page report, even if there’s a good reason for that, I will probably do a poor job of it. And if you have to read a 300-page report to extract 10 bullet points, you will probably do a poor job of it. But it turns out AI tools allow us to create pretty darn good 300-page reports and then let you read it just for the things you care about. And all of a sudden, the sheer amount of information that can be transmitted can hopefully be trust-enhancing. Now, of course, underpinning this, like Chris said, is still personal and organizational trust, but I think really from my perspective, making a previously unscalable level of stakeholder engagement possible is really exciting. I think whether you’re thinking about corporate boards, donors and funders, or any sort of ecosystem where you’re trying to operate at scale through other people. We’ve had a certain level of opacity that is really limited by the human ability to process information and deliver information. And to the extent that AI can kind of scale that up in a meaningful way, where you’re not just taking 10 bullet points and turning them into 300 pages, but actually turning 300 pages of information into 300 pages, that ends up being a really, really big unlock. We experience this every year. We go through our compliance audits for SOC 2 and HIPAA, and we put a ton of time and energy into it. And we know that oftentimes our users who request the report either don’t really open it or open it to check two things, depending on their organization’s guidelines. They need to confirm one or two things. And we create this report. It’s not low information density. We don’t start with 10 bullet points and flesh it up. We go through a lot of time and energy to do this. You could imagine this becoming far more feasible to the study level, to the data level, to the AI tool usage level, at the data quality assurance level, whatever organizations are willing to let their stakeholders and their audience and other data-collecting or data analysis organizations know can now be transmitted far, far more easily.
Marta Costa [00:51:01]: The other thing that I wanted to bring to the table is privacy, because these are the two main, most important pillars for us. Data quality, but also data security. And so what do you think are the risks to privacy that AI imposes?
Lawrence Li [00:51:19]: It’s a great question. I think there’s a lot. Obviously, one thing. Chris’s example of the people whispering into their microphones is just as more and more of our daily life is observed, as data collection moves below this conscious level, into a level of convenience and abstraction, it’s easy to lose track of what’s being collected, what’s available, etc. And that introduces maybe not new risks per se, but maybe widens a class of existing risks that might just be leakage, carelessness, kind of personal information that shouldn’t make its way out there. I think the good news is data collection organizations are naturally at the forefront of this problem. They’ve had to think about it in their work before AI scaled up their work. So, they’re familiar with the shape of the problem. And we, serving them with tools, have been brought into that; we’ve been asked for specific features, whether that’s end-to-end public-key encryption, whether that’s making sure that data that we store on behalf of our users is secure. All of that is going to become more important as the surface area for these same types of privacy issues expands, and as the convenience and therefore the volume of data collected goes up, I think it’s more of a scaled version of an existing problem. And I think the good news is it’s a problem that has been thought through a lot by individual users, by researchers who want to use data responsibly, by entire industries like healthcare with HIPAA. It’s certainly not something we’re unaware of. We just need to make sure we do the work of scaling the solutions to the new power of our tools and the new scale of the field work we’re hopefully doing.
Christopher Robert [00:52:49]: And kind of as crazy as it sounds and as sort of nakedly optimistic as it sounds, I think keeping data safe is a hard problem. It’s always been a hard problem. If we move faster and we collect more data more quickly and it’s all flying around and we’re throwing at AI agents all over the place, obviously, that increases the risk of exposure. But I think that AI can help us here too. I mean, I think it’s a hard problem, but I think it’s one that we can get AI’s help to figure out the right ways to change our processes, to keep up and to scale, and also to build the better tools we need to be able to help keep that data protected. So, for example, one thing that I’ve been advocating for a while now is the idea of leveraging AI to much more safely identify how to turn a data set with lots of PII (personally identifiable information) into a safe, non-PII data set that can be shared. That’s traditionally been a kind of difficult thing. At first it was like, oh, you’d remove the obvious things. Then you realize, oh, well, there’s only one family with 6 daughters in this village. And so, therefore, if you know the village and you know there are 6 daughters, you can identify the household. And then it became a tough human problem. And then we started developing machine learning tools to try to help, but those were kind of like, not that great. I think that AI can actually help us now do much better at efficiently producing and filtering our data down to a safe subset that can be stored and shared more safely. And so again, here I think that AI might introduce or worsen some problems, but I think it also helps with the solutions.
Marta Costa [00:54:39]: So, to wrap up, we usually ask our guests to share some advice with our listeners. And so imagine that you need to advise organizations that design and implement data collection processes; what would be your advice? How can they best prepare for what’s coming concerning AI?
Lawrence Li [00:55:01]: I think at the most meta level, I’ve mentioned a few times this idea that organizations will need to be thoughtful about how their work changes. And I think having someone take point and ownership of that, with access to external advisors, etc., is just absolutely crucial because what you’ll see is individual AI tools removing bottlenecks, changing pain points. But what you’ll really need is someone who can look at that holistically and say, I see what we already have, I see what might be coming on the horizon, and this is what should change about us organizationally. These are the new shapes of problems. These are the new problems scaling up now that we’ve solved these two choke points; this is the third constraint. And so our organizational energy needs to be focused on this. I think you just need to have someone thinking about that and owning that, especially with technology that changes as quickly as AI. I will also echo a point that Chris has made a few times, which is that for organizations that do data collection, being creative about how to leverage data that’s already been collected will be a huge opportunity. AI certainly allows a lot of scalability in data quality tools, maybe, like with AI interviewing; if you’re interviewing a connected population, if you’re doing research on a connected population, it can scale your work. But I think there’s a lot of value lost or maybe trapped in data that’s already been collected, whether because the underlying methodology is not understood or even organizations use it in narrow ways. But you can go back and realize, oh, we went to this village, or we went to this set of villages, and part of what we collected was audio and pictures, and we can now conduct a whole new study with barely any new data collection, just leveraging those pictures. There’s just going to be tons of opportunity in really being thoughtful about what can be done with what’s already there and just the dimensions where AI blows out kind of the scale constraints of what we’re used to thinking about. So, those are the two things I’d really think about. It’s just keeping an eye on that organizationally and having someone to own that and then really reflecting on, for some of our users, decades of data that has already been collected and what really could be done with that.
Christopher Robert [00:57:05]: Yeah, and I think piggybacking on that, I think I would definitely just more enthusiastically collect more qualitative data today, knowing that A, there are great ways to use that data today, but B, 6 months from now there’ll be even better ways. And so basically, I would collect a lot more audio, video, photos, and obviously, I’d work really hard to secure all of that because, coming back to the day of privacy concern, but basically, that really, really rich data is becoming more valuable all the time. And so I would just lean in and collect more of that, particularly given the timeline of many of our projects. I mean, most of the projects I’ve worked on are 6 months minimum, usually more, 2 to 5 years. And so if you think about what’s going to be possible with data analysis and processing in 6 months, let alone a year or two, I think you really want to set yourself up to be able to take advantage of that.
Marta Costa [00:58:09]: Well, thank you so much, Chris and Lawrence, for your time, for your great thoughts and insights.
Lawrence Li [00:58:14]: Thank you, Marta. Appreciate you having us.
Marta Costa [00:58:16]: That wraps up today’s conversation with Lawrence and Chris. Today we discussed how AI can fundamentally change the way data is designed, collected, analyzed and turned into decisions. We also discussed what AI could mean across different industries and challenging field environments. And finally, the new questions AI creates around data quality, privacy, security, and what it will take to trust research in an increasingly automated world. So, check the episode notes to learn more. And don’t forget to subscribe.
Outro [00:58:49]: Thanks for listening to Survey and Beyond the Data Collection podcast by SurveyCTO. If you want to learn more about how SurveyCTO helps organizations collect reliable, secure and scalable data anywhere in the world, visit www.surveycto.com. And if you are a fan of Survey and Beyond, consider leaving us a rating or review on your favorite podcast app. Your feedback helps more listeners discover these conversations and stay connected to the latest thinking in data collection. Don’t forget to follow us on Apple Podcasts, Spotify or wherever you get your podcasts so you never miss an episode. On behalf of the entire SurveyCTO team, thanks again for joining us and we will see you next time.