When is a research panel right for your organization?

Are you grappling with the challenges of consistently recruiting participants for your user research studies? You're not alone. Many of our clients face the daunting task of assembling a diverse, engaged, and qualified group of individuals for each new project. 

A user research panel—a group of people you have selected, and who have agreed, to take part in your research on an ongoing basis—might be at the top of your list of solutions to look into. However, for every benefit touted by a user research panel, there is a lot of time, effort, and money required that might not be worth the investment for your team.

So how do you know if investing in a research panel is the almighty solution to all your problems and the right path forward? You analyze the decision through two methods: assessing the shadow sides of the touted benefits and quantifying the ROI of building a research panel.

In this article we’ll cover:

• The benefits and shadow sides of building a research panel: time, engagement, cost, and quality of insights.
• How to measure the ROI of a panel by benchmarking your current state and predicting whether a panel will improve these metrics. 
• And how to move forward once you’ve decided whether or not a panel is right for you.

The shadow sides

Whether it's addressing the need for speed, increasing engagement, achieving cost-effectiveness, or uncovering deeper insights, building a research panel presents a range of benefits that may significantly improve your user research process. However, it’s important to take a more critical look at the pros to unveil the cons: 

Time

Pro:
‍Recruiting participants for a study is much quicker once a panel has been established.

Cons:
‍Building the panel is an ongoing task.

Managing & maintaining the panel takes time, specifically keeping updated information like demographics and engagement metrics.

Engagement

Pro:‍
A higher percentage of panelists apply for studies when they are engaged in your panel, making it quicker and easier to recruit for user research and general product feedback sessions. 

Con:
‍Creating engagement with your panelists takes time and requires you to steward them through many emails to build a connection with them. Research panel best practices say that you should set an expectation with your panelists about how often you will contact them, and stick to it. Even in months when you don’t have any studies to invite them to, you’ll need to engage them in other ways: a panel newsletter, for example.

Cost

Pro:
‍Recruiting new participants for every study is a costly and time-consuming endeavor. Having a panel to pull from for each study will reduce the total cost of running the study by reducing the amount of time that is spent and circumventing recruitment fees. 

Con:
‍Your panel may require you to invest in a new management tool, which could set you back thousands per year. Or, you may look into hiring a new team member just to manage the panel—again, thousands per year. The cost savings vs. new investment may not be worth it when you look into it, especially if you’re not running many studies in a year.

Quality of Insights

Pro:
‍Having access to the same participants over multiple projects can lead to deeper insights because of shared trust and transparency.

Cons:
‍The potential for bias in your research increases two-fold when recruiting from your research panel, especially if your panel is relatively small and you’re talking to the same panelists often.

Your panelists might be biased toward providing primarily positive feedback because they want to stay in your good favor and continue to be tapped for their feedback. Or, they may feel pressured to answer in a particular way because they have gotten to know you well. 

Assessing the ROI

So what does an assessment of these benefits vs. drawbacks look like? For us, it’s benchmarking your current situation and then estimating (or even running an experiment) to determine whether a research panel improves upon your decided metrics.

Take this example as inspiration: 

Recently, we worked with a client to determine whether or not a research panel was an appropriate solution to some of their team’s recruitment challenges. Due to the nature of their organization, recruitment for studies often took weeks to get off the ground. They have a huge base of customers to pull from already and wanted to engage the ones interested in participating in research more frequently through a panel to save them both time and effort overall. 

Their goal was to increase the speed of recruitment and increase the level of engagement they had with users to reduce the overall effort of recruitment for user research.

We started by benchmarking these three metrics: 

1. Speed: We measured the time between when they requested a user list from their analytics team and when the team was able to launch the recruitment effort of their 5 most recent projects and found that, on average, it takes 9.6 days to get a recruitment effort off the ground.
   ‍
2. Engagement: We measured the number of people invited to a study vs. the number who applied to get an average conversion rate of 21%.
   ‍
3. Variability of Audience: We looked at their 10 most recent studies to see if we could compile a list of standard user characteristics that could be used as recruitment criteria for all their studies going forward. A panel is only useful if you’re able to use it often.

From here, we could look at these metrics and estimate—or in some cases measure— the ROI of the panel. 

Take “speed” for example. Implementing a research panel would allow the team to circumvent an internal process that extended their timelines—a process that takes 9.6 days on average right now, which we estimated could be cut down to just 4.5 days from start to finish.

However, the time needed to build the panel well could be several months’ investment for a team already stretched thin. Considering the time needed to manage the panel every week, we questioned whether the investment was worth it.

When you look at improving engagement (% of users invited that apply to participate) we can hypothesize that it will improve with a research panel, but many factors play into that and we don’t know for sure that it will work. How many weeks or months of trust-building will it take to see the type of engagement that would make a meaningful difference for the team?

Because the client already had a large database of customers, we could run a small experiment over several months with recurring studies to understand, in this small population, if engagement increased and how much effort truly went into making that happen.

The value of building a research panel comes down to how often you are going to be able to use it. A specific pitfall of our client was that their screening criteria were often hyper-specific and were not things that would be easy to indicate in a panel. So, if 80% of your current studies can’t be satisfied by the 10-20 screening criteria you collect from your panelists then it’s likely not the right path forward.

For this client, the go/no-go criteria were clear: if we cannot prove that a research panel will make their processes faster and improve their engagement with users, then a research panel is not the right solution—at least right now. 

Though these are standard, universal benefits, a research panel may have other potential benefits for your team, like team morale or UX maturity improvement. These can also be benchmarked through analysis of metrics, sentiment surveys, etc., and we encourage you to think about what metrics would be meaningful for your organization.

Building a research panel is not always the solution to your recruitment problems as it can be a huge investment of time, effort, and money that might not fix anything. But you’re still left with a recruitment issue that’s making your team slow to insights.

So what do you do? 

There are options.

• You can build a research cohort, which is basically a research panel, but much smaller. If you have several studies that require the same type of users, recruit ~20-30 users to participate over several months. Remember to rotate these users out occasionally to avoid panelist fatigue and bias in your work.
  ‍
• You can hire someone to manage research operations, taking the burden of recruitment off your team’s shoulders, and putting it on those of an expert.
  ‍
• Or, you can take a look at other parts of your research process and see if those problems can be resolved. In the case of our client example above, working with their analytics team to learn the specific skills needed (writing an SQL to pull their list) would have solved the problem and given the researchers the control they needed over recruitment.

Assessing and benchmarking your current state and estimating, predicting, or measuring the future state will help you confidently move forward when deciding what’s best for your recruitment process. If you’re interested in how to best build a research panel, check out our step-by-step guide!

What's next?

Have a project you’re working on and need some support? Reach out to us.
‍
Do you just want to chat about product, UX, research, process, and methodologies? We’re down for that too. Let's chat.‍

Do you want to avoid talking to another human being right now? We get it. Sign up for our Curious Communications newsletter to stay up to date on all things UX and other curiosities. We’ll hit your inbox every few weeks.