[00:00:00] Mario:  I hate when people say human in the loop. No. No. No. Researcher in the loop. The same prompt will give you different answers. I just finished teaching an 8-week class on AI-assisted survey to the students at the University of Maryland in the survey program. The other thing we need to deal with is anxiety, because you need to realize that you will never keep up.

[00:00:21] Marta:  Today, on Survey and Beyond, we are joined by Mario Callegaro, founder of Callegaro Research. From pioneering online point of methodology to leading research at Google and now advising companies on AI-assisted research, Mario is here to show us why data quality matters more than ever and how technology can either strengthen or weaken it depending on how you use it. 

[00:00:44] Marta: Mario, welcome to Survey and Beyond.

[00:00:47] Mario:  Thank you. Yeah. I’m very excited.

[00:00:49] Marta:  So, to kick off our discussion today, could you maybe walk us through your career journey? I know you started with survey research. You transitioned to user experience in market research. So, just tell us a bit more about how this played out in your life.

[00:01:07] Mario:  Definitely. I started my career. I was a sociology student in Italy at the University of Trento. And after I took my first research methods class, I got very excited about surveys and especially questions on design. And so to the point that I actually even did the same exam twice, which is something that, actually, I checked, you can still do in Italy. You can do an exam twice, where the second time you just work on a specific subtopic. You agree with the professor on the topic, and you generate, like, a small paper or something.

[00:01:42] Marta:  By option.

[00:01:43] Mario:  It’s optional. Yes. Not mandatory. And so the second time I did detection on question on design because I was really excited about writing questionnaires. You know, how do you write the questions? You know? So, that’s where I got very excited. And then I was lucky enough that the University of Trento had a collaboration with the University of Berkeley in California, and we were using their own service software to do telephone interviews. And so I was recruited to run basically a small CATILAB, and with the help of a professor who came to teach us Compiegse, who came to teach us how to use the software and everything, we did a few studies like telephone service studies. So, I was training the interviewers, who were mostly students, and managing this 20-workstation kind of telephone interview center. That’s where I got into really listening to lots of surveys and asking questions and understanding how sometimes we think our questionnaire is great, and then you start talking to people, and they get confused on a specific question, for example. So, that was really an eye-opener. Then I started working at the university. I was in a department called sociology and social research for a few years until, eventually, I applied for a scholarship at the University of Nebraska because they had a master’s program in survey research and methodology. Still now, there are many. And the scholarship was from the Gallup organization, which I guess everybody knows. And luckily, I got the scholarship. I was able to move to Nebraska, and I did the master’s, and then my professor eventually convinced me to also do a PhD. So, I spent seven years there, give and take. And then after that, my first job was actually as a service scientist for KnowledgePanel, which is a probability-based panel of the U.S population. The company doesn’t exist anymore, but KnowledgePanel is still around, and the company has been acquired by Ipsos. So, you can still see, you know, Ipsos KnowledgePanel. And that was great. I was there for a couple of years, and I was really working on, you know, at the time, I’m talking about 2007, 2008, 2009. Web services were really taking off, and so we had all the issues of how do you design a question, or how do you show questions on a screen. We didn’t have mobile phone issues then. It was mostly desktop, basically, and understanding. And, obviously, the Internet speed, which was very slow. So, there was customers who wanted to show, like, a video, which now, you know, we don’t even think about it, but there were some technical challenges. So, I learned a lot, had an excellent team, then three years later, I got a ping from LinkedIn, a recruiter from LinkedIn saying, “Hey, we are looking for a survey scientist at Google.” I was like, what? I mean, Google surveys. I mean, funny enough, I was already living in Mountain View, which is the headquarter. But I only drove around Google, and I saw people going around by bicycle, you know, the funny colorful bicycle. But then, as much as I didn’t know anything else. Well, Google Search, you know? And then I did a bit of research, so maybe they do some survey. Then maybe they talk to their customers. Okay? And that’s exactly what the job was for a team, which is called quantitative marketing. And my first project was running a survey of our advertisers. And so that’s how I basically started my career at Google.

[00:05:04] Marta:  And now you are running your own company. Right?

[00:05:07] Mario:  Yeah. Exactly. Yeah. I left almost two years ago, running my own company, consulting on survey research, obviously, market research, because I was in marketing first, and then user experience. So, I also did a transition from, let’s say, market research to user experience research later on in Google. So, I spent about eight years in market research and seven in user experience, which I thought was great. So, I had a lot of fun doing that. And see now research methods change, you know, and also getting an idea of the difference between B2C and B2B. It’s very different. So, consumer research is one story. When you go to B2B, it’s a very different ballgame, and you need to understand how to do it properly because there are, I think, just much more difficult to do in general, and if we have time, we can even discuss that.

[00:05:58] Marta:  Now, I would just love to hear your perspective about data collection because that’s our main topic. And your career covered many areas, as you mentioned. It’s incredible. And I wonder, you have seen data collection happen in all these different areas, how different was data collection from one area to the other? How did this shape the way you think about it?

[00:06:21] Mario:  So, if we talk about the difference between market research and user experience research in terms of data collection, you have something in common, which are you’re talking pretty much to the same users, but from different angles and at different stages. So, when you are in market research, generally, you talk to your users after the product has already been launched. So, the product should be out there, and you talk to the users or customers, you know, depending on your definition. Generally, the difference between users and customers in some companies, like, somebody is a user if the product is free, and if they pay their customers. But from a research point of view, they are still the people who use your product with different levels of expectation, obviously. So, in market research, there is this stage difference. So, you start talking to the customer after everything’s been launched. And then you ask them different questions. It’s mostly about their experience. Obviously, there are lots of questions related to the marketing that you need to do about the product, and everything else. While in user experience, you generally start talking to users as soon as you have some kind of demo or prototype. So, you’re really much closer to the product. You are closer to the engineers. You are closer to the designers because, well, you’re also in the UX team. While generally from a market, it’s just a bit more distant from the designers, especially in a huge company. If the company is smaller, it’s much easier to make these connections. You know? So, in big companies, you know, there’s the marketing team and the UX team, and everything is different. Also, the chain of command is different. There’s no overlap. A few exceptions; there are some companies that are more like an insight team or something with different languages, where the UX and marketing are in the same team, which makes a lot of sense because you’re talking to the same users. And many times, one of the risks of having these two teams separate is that you might duplicate research without even knowing it because there are some overlap. So, I’ll give an example. The market research team might study churn and try to understand why some people stop using our product. Even the user experience team can do the same study because it’s still relevant to them. And so sometimes, yeah, I was actually in a situation where I was in UX, and someone in marketing was doing a churn study. I was like, maybe I should know that. Also, it’s important, even if you don’t know it, to have a consistent way to report and maybe a common repository; there’s also lots of software out there that you can use, where every report from marketing and UX should go there. So, you avoid duplication, everybody’s learning, and you don’t need to start from scratch, especially if the company is huge. If the company is much easier to know who’s doing what, but when the company is big like Google, there were cases in which there were some duplication, unfortunately, you know, or in any case, overlap, I would say. But, yeah, the main difference is the stage in which you talk to your customers and the kind of questions. So, in user experience, you do a lot of usability studies. You do lots of qualitative interviews. You try to understand how people use your software. If you are in B2B, then I had some colleagues who actually went into a company that was in cloud, and they just observed how they were, like, setting up some cloud products. Just in a very, let’s say, anthropology kind of style, where just to understand, did they look at the official help center, or did they just search online how to do something? How did they start using it? Or did they just do it? Did they talk to somebody? So, it’s fascinating to see these reports. So, that’s another reason why I moved to UX. It was like, I really wanna be closer to the product, while in marketing, you’re generally a bit more distant. And so it was great working with designers and making some changes based on your research, like we found this. And so you see the change actually in the product, and that’s this kind of satisfaction. I said, “Okay, you know, maybe hopefully, the product is gonna get better because of my research.”

[00:10:26] Marta:  This is great because this is also something that I wanted to explore with you. And I think you already mentioned a lot, which is when a lot of people, myself included, think about market research, and I think maybe more market research than user experience research is we think about surveys a lot. We think about web surveys. That’s how people are collecting data. But as you mentioned, there is so much more than that. And the data that you need is so much more than the data that you collect from the users. You need to understand how people are interacting with a specific product, their behavior. So, can you just dive into that a little bit more and just describe how do you collect data in this space?

[00:11:10] Mario:  You’re right. And there are also industry reports showing that the market research services still use data collection method for the good and the bad. So, we know it. And in fact, when I was at Google, I was doing survey office hours, so people could just book, like, half an hour with me. And I spent a lot of time basically telling people not to do a survey, which was funny. You know? I was a survey person telling people not to do it. And the reason is because it was not the most appropriate or efficient data collection method. So, many times it was not what the tool is able to do, especially if you want really in-depth knowledge, granular knowledge in a survey, you’re not gonna get it, so you should do qualitative studies. Or if you really want even more detailed, you look at something that, depending on the company, you call them logs or behavioral data or telemetry, depending on the company you work with, which is basically the interactions that are captured by the software that people are using, and we can talk about that later. But you can also look at those. That will give you what people do. You don’t know the why, but you do the what. So, let’s say if you have this brand new UI that your designer is very excited about, then you see that nobody’s clicking on that specific button. You see that from the logs, you don’t know why. Is it out of sight? They don’t see there’s something that is happening. And so then, when you need to do qualitative, when you do maybe a usability study where you see how people move their mouse, how they’re interacting with the tool. So, I would say in user experience surveys, they are very, very, very popular, but less popular than in market research. But even market research is just too many. At best, in market research, you might do a bit of focus groups, and maybe some interviews, but the interviews, in-depth interviews and usability studies are obviously very, very commonly used in user experience because it makes sense. And that’s especially when something’s very new, you have a prototype you can show the users so they can click around, they can see how they do it, they can talk out loud, and telling them what they are trying to do. And they fail, you can probe. So, you need to do it live, obviously, and trying to understand what was the friction, and then you can show this video to your designers and guess what? You know, we can change this. We can improve that. So, those are very commonly used methods. And there are gazillions of methods in user experience research. Now, as soon as I moved, I bought a couple of books, and I remember there’s one book. They have 100 methods, you know, that you can use. Maybe they’re very tiny, like card sorting. Okay. It’s one method, but it’s still so many. So, in a way, when you are in user experience, it should be like a Swiss knife. You need to know lots of methods because, going back to your initial question and what my approach is, I need to know your research question. And so it’s my job as a researcher to ask you questions about your questions. Because sometimes you just come to me like, “Oh, I wanna know this.” But then it’s not really what you want. And then when I ask a few questions, like, actually, you wanted that. And sometimes the answer is already there, so you don’t need to do any research. So, one research method is look what we did in the past internally. Oh, we already know it. Great. So, there’s no need to do research. Which is the best answer? Immediately. You know? Like, you can say, “Well, okay. Now that I know that, I wanna know more.” But then you don’t start from zero. You already have this data point or maybe data points, and you say, okay. Great. Now I know more about the topic, but then you may come back to me in a week and say, okay. Now that we have all this knowledge, we wanna know this specific thing because we are planning this new feature or something around that nature and wanted to know more. So, it’s part of the difference. So, know exactly what you wanna measure. And then the second question I ask, I mean, “Who are we gonna talk to?” So, users are a bit vague. Do you want to talk to a random group of people who are using it, or do you want to talk to the people who started using the product? So, they are novices, which, obviously, they have less experience, and they have a different experience, and different set of expectations. Or you wanna talk to the people who are already using it for, let’s say, a year, for example. Because, obviously, there’s also self-selection. And if they’re using it for a year, the people, for some reason, didn’t see that that was useful for they had already gone. So, they churned. You don’t see them. Or you can also follow them. So, you can see follow this cohort of people. So, let’s say you get a new Pixel phone, and you can talk to them during the first two weeks that somebody got the phone. How is this experience? Blah blah blah. And then you can talk to them after three months or after a year. And then you can also talk to the people who stop using it, asking them, “Why did you stop using this specific phone? What happened?” You know? Did you switch to another company or brand, and why? So, that’s how you follow up depending on different such questions. And that question can be market research, definitely. It can also be user experience research. Maybe, you know, it’s just something that didn’t work well in the phone, and they couldn’t do something and so they switched to something else.

[00:16:30] Marta:  Yeah. And how do you handle data quality in this context? Because you are gathering data from so many sources in so many ways, quantitative, qualitative. So, it sounds like it can be quite complex, the way you control quality in your data.

[00:16:48] Mario:  That’s a big topic. The main distinction is if you have your own list of people to talk to. So, you have a list of customers or users that you can reach out in different ways. And that’s one story, and that’s where generally the quality is very high because they’re using your product for sure, because you know it based on your internal data. In many companies, you can actually reach out to them. There is some kind of opt-in mechanism. So, you are opted in when you use the product into something called, let’s say, market research. So, not everybody, so we need to remember that the sample is a bit skewed in a way. But you have permission to contact them. And so you can reach out and say, “Hey. We are doing a study on this product. Do you wanna do an interview with me?” Or you can even send them a survey if you need to do a survey or other methodologies. That’s one story. And then the quality for these people is very high because you know they’re real. There’s no fraudster there. You can talk to them and everything. When you go off your list because you don’t have a list or you wanna reach potential customers, then that’s where everything gets very complicated very quickly because you say, well, we wanna talk to potential customers, for example. So, what do you do? You choose some kind of online, generally, sample or online panel or, like companies that provide users or respondents or whatever language. That’s where everything gets very tricky because, unfortunately, as we know, especially with recent debates, there’s a lot of fraud in the survey world, but even in other places where people are trying to pretend to be what they are not. And so if it’s qualitative, it’s a bit easier because you actually can see that it’s the real person, and you can do some extra stuff. But when it’s completely unmoderated, then it’s more difficult to know if that’s exactly the person you needed to talk to. And so you need to pay a lot of attention. And in the survey world, there are companies specialized now in, let’s say, cleaning out fraudsters. So basically, blocking them from entering your study before they actually even start with lots of different, very sophisticated technologies in a way. So, the industry is developing, but it’s kind of sad that we need to have this industry developing, and to clean up your sample instead of just having a good sample to start with, which is obviously a huge topic that the market research industry is debating at the moment. But, yeah, that’s what I always think about quality. And the other key concept I was also discussing with my colleagues was that when you choose a company to work with, many times you need to rely on a third-party company. I saw folks getting very excited about this company because it was cheaper and faster than other companies. But my question is, well, maybe there’s a reason, you know, to have cheaper and faster. So, that’s where it gets very complicated very quickly. But if you’re a good researcher and you know your topic and you did all the internal desk research, you talked to the expert, you are able to understand if the study results are completely off, or they don’t make any sense, or they’re completely different from, let’s say, internal data. Because many times you have internal data. Oh, you should. And so that’s another quality criterion. If a study is, like, complete, it comes out with very different outcomes than all the data collected so far, is it a one-off? Is it just a fluke? Or, I don’t know, you got very unlucky, or is it real? And many times, it’s not real.

[00:20:26] Marta:  So, you can do quality checks at least to know.

[00:20:29] Mario:  Correct. Exactly. And so the other skill that a researcher should have is not only in the methods, but also you need to know your product very well. So, let’s call it substantive knowledge. So, you need to read if you find them, third-party research, you know, industry data. So, you have the big picture. Because your competitors aren’t doing research as you. And many times they publish that, so you should read them. So, you have an idea of your space. And when you have an idea of your space, it’s much easier to see if your results are completely not credible, or if they are, actually, you can rely on them. And that goes back to triangulation. So, you have some data points. You did a study with your own customers. You get some number. You did a study with potential customers that cannot be completely different. Or you do a study. You do these competitive studies where you compare scores by your product and the competitors, and so you have some kind of baseline. So, for example, you can ask questions about quality of the support that you get from these products, so you can see how the perception of support is for your product versus your 2, 3 competitors and trying to work on that.

[00:21:47] Marta:  Something that I would also love to dig in is I know that now you are teaching, and maybe you are also working on some projects related to AI-assisted surveys. So, from your perspective, and this is a hot topic today. Everyone is talking about it. Everyone is using it. So, from your perspective, how do you think AI can improve? We are going to talk about how people are maybe overestimating a few things, but I would like to take a look at the positives first, maybe.

[00:22:23] Mario:  So, my mental model on AI helping is, first, we need to think about who’s helping. So, generally, we talk about helping the researcher, but that’s just one act, or let’s say, in this equation. AI can help the respondent if he’s in a survey, for example. And I have some examples there. AI can even help the people collecting the data. Let’s say the interviewer or the supervisor, if that is the case. So, this is just to start them on a mental model. Then looking at the positive, and as you know, everything is changing very quickly, I just finished teaching an 8-week class on AI-assisted survey to the students at University of Maryland in the survey program, and we went through 150 papers on anything you can imagine about the different stages of a survey project from even starting doing a literature review, which you should always do, to actually write in the final report. So, the whole cycle. And it’s very similar in research in general. If it’s not a survey, it can be qualitative research. It can be something else. And you have the general or generic tools, like the one we all know, and then you have all the custom tools, which is another huge team, because there are so many that you would never keep up with. So, there are some places where I think AI and LLM is working. It’s pretty good, you know, there is, I call it, I hate when people say human in the loop. No. No. No. Researcher in the loop. The human doesn’t know. If you don’t know a topic, AI writes in a way that is so confident you can’t believe anything. But if you are an expert or expert in the loop, then you understand the limitation, and sometimes you might disregard or you might reprompt and see what’s going on here. So, with that in mind, think about doing a literature review or desk research. How difficult it is, and especially when you are new to a project, you’d start by reading lots of reports, which you should read anyway. But having, like, a nice summary, it’s something that’s not perfect, but it’s getting better and better. And there are generic tools like the deep research function in most common LLM tools that you can imagine. They’re all called the deep research anyway. So, from Gemini to ChatGPT to Copilot is all called the deep research. And so you have basically a summary across, I don’t know, 120 different websites, which probably you would not be able to do manually. So, that’s something. But you can think about the same thing about your internal database. So, if you have an internal database of research, it gives you, like, a nice summary. So, you start getting, like, let’s say you move teams, you go to a new product. What? How do you get up to speed? Are you gonna read all the results reports? No. You don’t even know what they are. But now if the company did a good job in archiving them, and now with an LLM on top to summarize them, then link them to the original reports. K? So, you can check, you can learn more. Then you have, like, an initial starting point, which will save you a lot of time to get up to speed. And, also, you become a substantive knowledge expert, which you should be anyway. So, that’s where I think it’s a great way to help. Then we have all the tools that are doing, I would say, a good job in summarizing text, you know, like interviews and everything, with different results, and they’re all very different. So, you use three different tools, and you get two different results. So, that’s something we need to deal with. Let’s just say coding open-ended answers is another topic where lots of people are excited, and it’s getting better and better. You know, there are some good studies out there. And then all the code, writing code. So, you know, if you need to write some statistical analysis, code with your preferred coding tool can be Python, R, or anything. In this case, there’s a lot of help, you know, and it’s like having an assistant there. So, that’s definitely, I think a lot of people are, they would agree with me on this, let’s say, assistant and help. As long as you have in mind that there are always hallucinations even now, they’re not going away anytime soon. An example I did with my students was I found a request for proposal online. Very common. You know? You get a request for a proposal, which was for a survey, obviously. It was a survey in different counties. It was a, let’s say, 40-page RFP. And I asked them to try two tools to see how they summarize it. If you are working in a survey shop or in a company, in a market research company, you’re gonna get the RFPs. So, how do you process them? Do you read it all, or you have the LLM doing a quick read and give you the basic info? Like, number one, when is the deadline to answer? Because if you miss the deadline, then you’re not gonna get that job. So, maybe that’s number one. You know, very important and other things. And in many cases, even when I did my own test, there were some hallucinations. You know? I remember I had an RFP with a scoring system, which is very common. You know? We score each company based on this scoring system, and the sum was, at different points, the sum was, I don’t know, let’s say up to 60. So, 10 points from this, 10 points from this. And then this LLM tool completely changed the scale to 1000. And when I re-prompt, obviously, the LLM is gonna ask for forgiveness, but that was it. So, if I wouldn’t have read the original report, that probably was gonna a mistake because that’s key. And the students were shocked, you know, like, what? You know, this is 2026. And we were using the latest model; we were not using something else. So, that’s something just to keep in mind. I mean, the mental model is like a research assistant who’s very eager to please you, but because he’s an assistant and he’s, let’s say, young to speak, this entity doesn’t know much. And so doesn’t know how to check for something that, in that case, like, if the scale is 60, you cannot just convert everything to 1000. So, that’s something that just to keep in mind.

[00:28:27] Marta:  So, do you think people are overestimating AI a little bit?

[00:28:31] Mario:  Yeah. For sure. Also, because, you know, the marketing material is very hyped up, and so we just need, but it makes sense. You know? There’s a lot of pressure from the investors to get the money back, and many companies are not making any money, by the way. So, I’m not surprised that there is a bit of hype. And I understand it’s very exciting. So, it’s not that it’s all negative. And you want to be excited. You want to be an early adopter, but you just need to keep in mind that, you know, everything that is seen is not always perfect. Also, we need to deal with something that we are not used to doing, which is uncertainty. So, the same prompt will give you different answers. And for a researcher’s mind, it’s very hard to, you know, we don’t do well with uncertainty. You know? We want well. And so, because the model is probabilistic. And then the tools give you different answers every time. So, what I did with my students is to use, there is a tool where you can see the same prompt coming out from two different LLMs, left and right. So, it’s like on a table. Very nice, and you start getting the message that, even with the same prompt, you have very different answers. And so that’s why it’s very disconcerting, like, which one should I believe or which one is closer to what I really meant, or you know, which one is, and some outputs are just shorter, some outputs are longer, you know, depending on still with the same problem. That’s what we need. We need to be able to deal with uncertainty. And the other thing we need to deal with is anxiety, because you need to realize that you will never keep up. And if you don’t accept that, you’re gonna be stressed forever. You know?

[00:30:05] Marta:  More and more. Yeah. No. But clearly, AI is posing new data quality risks. So, you now need to pay attention to different things. And, certainly, it’s a tool. It can help in a lot of things, but it can also be dangerous if you don’t pay attention and you’re not careful with it.

[00:30:24] Mario:  Not careful, and you’re not an expert, especially. So, because it sounds really good. But it’s not only a hallucination, it’s also something might be missing. Like, something very important might be missing. And because you don’t know, you don’t know what you don’t know, it seems okay. So, doing that and a bit of more, like traditional and also reading, learning more from the original. So, it’s good to find the original sources, but then read the original sources because that’s how you learn, and you build your internal knowledge.

[00:30:54] Marta:  So, looking ahead and taking into account all this new technology that is rising, how do you think the research teams will operate in the future?

[00:31:06] Mario:  Always hard to talk about the future. My best guess is, well, ideally, as I said before, the market research team should be closer to the user experience team for more like an organizational point of view. And if AI can help, for example, to transfer knowledge, having this repository, for example, of research done by different teams and make you aware that somebody else in the other team is doing similar research than you, or they did it in the past. Well, that’s definitely very useful. I probably guess that companies will have their own internal LLM tool trained on the internal data, which makes a lot of sense. So, you get more than a general tool which is trained on, let’s say, the Internet, which still, we don’t know what it is. We just don’t know where the data is coming from. That’s another problem of LLMs that are difficult to reproduce, almost never. And also they are non-transparent. So, you don’t know which data sources were used to train and which were not, for example, and it’s very hard to know. And so, if you have an internal tool, you kind of know where the data is coming from. And also, you can always double-check because that’s in it. So, I would envision then. There are some folks in some conference saying that probably the difference between qual and quant is gonna become more and more blurred with AI, because now if you know less on the quant side, I can help, for example, on that. So, I don’t have a strong opinion yet, but I can see that coming. I don’t know if it’s positive or negative, but definitely, this very strong distinction, I guess, is becoming more and more blurred. Also, when you see job postings, they’re looking for, like, mixed method, more and more folks who can handle a bit of data and also a bit of qualitative, which makes a lot of sense. So, probably we would need more. The new researcher would need to be more equipped in mixed-method research than just say, “Oh, I only do stats, or I only do qual.” Because, actually, either you work together with the qual and quant or you just do everything yourself, because the quant will never give you all the answers, and the qual, same story. So, generally, it’s when you have them together that you get the highest amount of insights and knowledge. So, that’s probably what’s gonna happen, hopefully.

[00:33:32] Marta:  And taking that into account, we might have a lot of young researchers listening to our podcast. Do you have any specific advice for them? What should they focus on? What kind of skills would be important for them to have?

[00:33:47] Mario:  Definitely. So, we just discussed how everything is very daunting and stressful because there’s something new coming out every day, a new research, a new study. You go on LinkedIn, and you can never keep up. So, I would focus on one or two topics that you are really passionate about so you can follow those. Otherwise, if you want to follow everything, you might do that as full time job, and that was it, basically. Skills definitely need to learn about how LLMs work. And it’s technical, but you need to read a bit. Prompt engineering is still important. When I read a paper that is coming out, I don’t even read the paper. I go to the prompt first and see what was done there. You can say which tool, but then the tool, by the time the paper is out, is already updated. But at least you have an idea about the prompt, and you understand what was done, what kind of input was given to the machine, in a way. So, a different way of reading research. Then you read everything else. If the prompt is not there, it’s like, well, why? I wanna know. You know? It needs to be more transparent. So, that’s something. And then now more than ever, there are lots of webinars, so it’s much easier to keep up with webinars; many are free, you know. My general suggestion is to join one or two or even more professional associations. They organize lots of webinars and training so you keep up. Because the moment you leave your education and you start working, you don’t devote your entire time to learn, so you need to keep up. And so that’s generally the way to do it. And so it might be easier. And then the community, you know, creating, discussing with your colleagues and exchanging. Because if, let’s say, you have a group of people and everybody’s passionate about different topics, then let’s say you meet once a week. You do, like, lunch and learn, and anybody can say, well, you know, I learned about these new things, or this new study came out. Now, I’m gonna try to redo it with my own data. So, that’s another way to share knowledge and avoid the anxiety of keeping up with everything, which is basically humanly impossible.

[00:35:49] Marta:  And with that in mind, do you have any resources you want to share with our listeners?

[00:35:55] Mario:  Just one. By the time the podcast is out, it should be out. But in case it’s not out, I’ll announce it. So, I’m gonna co-edit a special issue of journal, which is called Survey Practice. So, Survey Practice is online, open access, so anybody can read the journal from the American Association of Public Opinion Research, which is really targeting practitioners. So, the articles are not like journal articles in terms of length. They’re short and are very practical. And with my colleague, Sara Ball from LMU Munich, we are editing the special issue on AI-assisted surveys. And we are looking for practical experiments or practical examples of how AI was helping the researcher, or maybe the respondent, or maybe the interviewer, or the supervisor in the survey world. So, it’s open to anybody, and private companies and academic is the same. And we are taking advantage of the tool, which is all online, so you can show images, you know, you can show a gift or even a small video, how you, I don’t know, let’s say you did some coding of open-ended answers, or you developed a tool to convert a questionnaire from Google Docs to this specific survey tool. That’s just a couple of examples, so anybody can, and then once it’s out there, they can talk to you, so create a bit of a discussion. So, that call for papers is gonna come out soon and hopefully, by the end of the year. We are not waiting until the issue is full. As soon as a paper is ready, we’re gonna put it out there so you can start reading it right away. So, we wanted to make it very, very practical and also for students, like, if you are teaching any kind of AI assist or something, which exercises do you give to your students? And how did they do it? What did they learn? That’s another topic that we would love to hear from. Because a lot of people are now tasked by teaching students how to use AI. Well, if you need to teach, obviously, you have the theory, you read, research, but then you’ll only learn, especially with AI, if you start prompting and see what happens. So, which kind of exercise will you give to your students, and how do they do them? Are they excited? You know? What works? What doesn’t work? That’s something that I would love to get for the special issue.

[00:38:11] Marta:  Wonderful. Thank you so much, Mario. It was a pleasure having you here.

[00:38:15] Mario:  Thank you. Yes. It’s a great discussion.

[00:38:17] Marta:  That wraps up another episode. Today, Mario and I discussed market research versus user experience research, the structural differences between the two fields, and how to avoid duplicate research silos inside large organizations. We also discussed ensuring data quality, why researchers must triangulate results with internal baseline data and maintain deep substantive knowledge of their products. And finally, the reality of AI in research, treating LLMs as eager assistance while staying vigilant against hallucinations, missing data, and model uncertainty. Check out the show notes for a link to Mario’s upcoming special issue on AI-assisted surveys, and don’t forget to subscribe. 

[00:39:05] Marta: Thanks for listening to Survey and Beyond, the data collection podcast by SurveyCTO. If you want to learn more about how SurveyCTO helps organizations collect reliable, secure, and scalable data anywhere in the world, visit www.surveycto.com. And if you are a fan of Survey and Beyond, consider leaving us a rating or review on your favorite podcast app. Your feedback helps more listeners discover these conversations and stay connected to the latest thinking in data collection. Don’t forget to follow us on Apple Podcasts, Spotify, or wherever you get your podcasts so you never miss an episode. On behalf of the entire SurveyCTO team, thanks again for joining us, and we will see you next time.