Hosted by Turo
Huge thanks to the Design, Research, and ReOps teams at Turo for hosting this event 🌟 Ethnio organizes events like these to celebrate our community and, of course, for any excuse to eat pizza.
Panelists
Was there a live stream or recording?
Not this time. We went old school—an analog event that lived and died on a random Thursday in April in downtown SF. But good news: we’ve got a full transcript available down below.
How dare you have a panel that isn't about AI. Are you trying to anger the robots?
There has been lots of work on measuring the impact of UXR, and lots of events lately around AI. There have also been more events around ReOps in the last few years, which is awesome. We haven't seen as much discussion about the impact and ROI of ReOps, especially from people who work inside and outside of program management and ReOps. We were eager to hear thoughts about how the ROI of ReOps had been measured and perceived. Thanks to everyone who came to hang out!
UX Research Ops ROI – Opening Question
Nate
I have my very first question to our esteemed panel.
And that question is basically about the anchor to UX research. So, we're in ResearchOps, let's say, and we're interested in our ROI. How tied are we to the perception of value that research in general has?
Are we exclusively tied to it, or does Ops have its own ROI? That's my first question to you guys.
Denise
Starting with an easy one. Go ahead.
Nate
And I can kind of elaborate on that a little bit, just to say that, you know, because in Ops, we're not actually providing, like, the insights technically, but we're presenting maybe the findings. We're more operational, obviously. Do we get a chance to prove our own ROI somehow, or is it really just UXR that proves that, and we're kind of just logistics in the back?
Ops as Strategy and Value Partner
Megan
I can kick us off here. So, I think oftentimes the two functions are intertwined. You can't have research operations without a research team.
But, you know, I think the perception is always that operations means just tactical work and just scheduling of research participants. But there's so much more that operations can deliver to our stakeholders...
...Can we work with you?
And so, you know, we are very intertwined with research, but we are able to prove our value outside of, you know, our specific work by partnering with those across the company who have the same goal of getting input directly from our customers.
Denise
I think here at Turo, maybe we have a slightly different sort of organizational setup. So we're a small but mighty research team, and we have one, you know, sort of king of research operations, Arturo over here...
Like, is there value to research without research ops?
Because I think for us, there's so much that we wouldn't be able to do or wouldn't be able to do at the same quality and the same scale and the same speed without research operations.
Erika
And I'll give the designer's answer. It depends...
So I think the ops people can be the best advocates, right? Because you see what's going on, and you see what makes research go efficiently.
And a lot of designers... are afraid to talk about, like, doing things quickly.
But it's, like, if you, the whole point of all this work is to learn things that help you do your business better.
Nate
Totally, totally. I mean, it's so fascinating because to the extent that we're not 100% tied to research impact... is there a way to do that?
The Danger of Measurement as Target
Erika
So I'll start here. We have to be mindful, right, of a phrase you've probably heard of...
And the tricky part is when you're defining a measurement or quantifying something to see how you're doing...
But first, right, you need to describe before you quantify, and I think that's what people miss in this.
I really just think it's a critical thinking exercise, right, and it's doing research...
It's people and it's interactions, and you're trying to change behavior...
Denise
98.6? I think it was 98.6, but then I saw something on the Internet.
Erika
See? Yeah, so you know that's a good indicator, but you do that for reasons...
And then in chasing that number, you're not destroying what yielded that number.
Denise
Do you feel like every company or every team needs to come up with their own bespoke framework?
...Or is there something out there you think that could be used?
Erika
It really doesn't. It's like I think the most important thing is not to be dogmatic.
Like everybody's messed this up with Apple, right?
Everything is just a tool.
Nate
But that's such a fascinating perspective because I feel like... Is the business empowering researchers?
Is the business empowering product teams?
SLAs and Feedback Loops
Megan
Yeah, I mean, if I've learned anything by working at Instacart for the last four years is that things move incredibly fast...
But I think, like, everyone cares about numbers, obviously...
It doesn't have to be a blaming mechanism...
But we need to support one another...
Nate
Yeah. And does it feel like we're heading towards a score?
Megan
I think at Instacart, we, what we're focused on...
We do measure things like, did we meet our SLAs?
But another thing we measure is no-show rates...
Nate
Yeah. And I would just guess from design and research side, probably not a huge, you know, goal...
Denise
I don't think that's a conversation that we're having here.
I'm sort of curious if other people are experiencing that.
Nate
So the people thing here is what I want to highlight...
Megan
I don't know if there's a way to put a score or have an ROI measure of collaboration...
It's hard. We have about six designing research program managers...
It's just the human element that we bring to the role...
Creative Metrics and Brand Implications
Nate
That was such a good point. I think the other thing is, are there creative numbers?
Erika
I mean, ultimately what ops does is... make it more effective and make it more focused.
I don't know how to say like, this wasn't annoying.
Denise
It's making me think of... agile scrum workflows...
Megan
Yeah, well thought, because it does, I mean, I know you said you don't use the word efficiency...
Nate
Yeah. And, I mean, I'm going to go way out on a limb here...
Erika
I'm just a huge advocate for having the ops in-house...
Megan
And having a research operations team in-house allows the team to build that context over time...
Erika
Like, there are so many brand implications about how easy it is to recruit people...
Operational Roadmaps and Organizational Maturity
Hollis @ Turo
I didn't really have a question. It was more about just the way that I kind of think about...
So just to share that perspective to let you know...
But, you know, the way that I think about and approach research ops is...
Nate
That's awesome. Great perspective...
External Perspectives
Audience: Viji Hari @ Verily
This is more of a comment than a response to your question...
Then re-ops is like now in a very thick, are they going to be the blockers or are they going to be our partners?
Nate
Oh, I love that. Are they going to be the blockers or are they going to be our partners?
Audience: Academic
So first of all, I'm an academic...
Audience: Kevin Simmons @ Volvo
All right. So I work at a company with a pretty small team...
So I was curious to pose the question...
Tooling and Automation Skepticism
Audience: Anna Hall @ Wells Fargo
I'm sorry, that's a question for the panel...
But there is a prevalence of this view is that automation out there is going to replace our jobs...
Erika
Can I respond to this one? I'm a veteran of the knowledge management wars...
Fundamentally, the most efficient thing is for people to have a conversation...
Megan
I think an interesting tactic to just challenge that assumption is...
Erika
Oh, I just thought of a metric...
Denise
I feel like research is the cure for the loneliness epidemic...
Celebrating Customers and Empathy in Research
Megan
Yeah, I can speak to a way that this worked really well...
We put together a program called Customer Love Month...
Erika
There's a kind of consulting project that I love...
Measuring Quality and Confidence
Audience: Quality measuring woman
So I'm going back and thinking about goals and, like, when we measure...
Erika
First, you have to describe quality, right? It, like, goes back to that...
You have to start with an agreed-upon description...
Megan
Yeah, that resonates. We have to define what quality means...
Erika
It makes people really itchy. Like, this is the problem...
But we're never going to have certainty. We're going to have varying levels of confidence...
Megan
Do you ever do, like, certainty sandwiches...
Erika
No, it really isn't. Like, it's all sales...
So you have to be like, hey, man, how do you feel? You can't change a feeling with a fact, right?
If you've ever been to therapy, you've heard that one. I love you.
