Lessons from Mentoring on AI Bias and User Research

Students conducted a series of tests, prompting Adobe Firefly to generate images based on prompts such as “a teacher in a classroom,” “people exercising in a gym,” “a happy couple,” etc. After which, they documented their observations, synthesized their findings, and created a final poster that commented on the biases they saw in the generated images. You can see some of their final designs here.

While I was there to observe and advise, this was also a great learning opportunity for me! This experience helped reinforce the value of the critical components of user research—asking the right questions, setting clear goals, and never leaving an insight without a “so what?”

Over the course of the week, I learned:

• You can apply a research mindset to any mentoring conversation: There’s questions that get surface-level responses and there’s questions that encourage the other person to open up about how they’re thinking and feeling. When advising student groups on their projects, I found that asking questions like “Tell me more about that,” “Why is that important?” “What are you trying to achieve?” and “What’s getting in the way?” helps get to the heart of the conversation more quickly.

• The importance of goal-setting: The more intentional you are when using a new digital tool, the more satisfied you’ll be with your output. This was true for the groups of students using AI for image generation. When their prompts were less specific and thoughtful, the AI filled in gaps on its own—often leaning on biases and producing unsatisfactory outcomes. Goal setting is also important when I’m running a research study: If I didn’t have clear goals before collecting data, I may not have enough data to tell the full story. However, in this case, I would choose to conduct secondary research rather than relying on assumptions to fill in those gaps.

• ‍Seeing an insight all the way through: Every step of creating an insight is important—from observation and analysis to reporting. These stages lay the groundwork for the most critical aspect: determining what the insight means and what should be done about it. Student groups that crafted clear, actionable "so what" messages in their final work stood out to me, because that demonstrated the value of the work. As creative problem-solvers, researchers must go beyond reporting just the facts to provide a perspective that drives meaningful action. This ability to connect findings to solutions is what transforms insights into impact.

• AI exists in the context of all in which you live and what came before you: You think you just fell from a coconut tree? (meme explanation here) The bias students observed in AI-generated imagery speaks to the bias of the stock photos that exist today. So this bias problem isn’t new. Since the visuals that are being fed into the AI machine are stereotypical, noninclusive, and eerily shiny, we can expect the output to have the same problems.

Mentoring these students was a reminder that even though AI is coming for our jobs, the uniquely human skills of critical thinking, creativity, and ethical decision-making will always set us apart and remain essential in shaping meaningful design solutions.