Start with the decision, not the data

For years early in my career I collected the data first and worked out what it was for later.

When I started out in UX, I was quick to reach for interviews and focus groups.

I loved the research part. Hand me a problem and I’d be off, booking sessions and gathering all the insight I could.

What I didn’t do was stop and ask what the data would actually teach me, or which decision it was meant to help me make.

I just wanted the data.

When I moved into the people space, the same habit showed up with dashboards. We’d share reports on the standard metrics, engagement, turnover, adoption of our processes, without being clear on what decision any of it informed.

Mark and I discussed how Crewmojo was seeing a version of this where every time a customer asked for a new type of report…

Inevitably they would always come back with more questions where they needed an iteration on the report to answer.

But that is actually the point - data should help you answer a question and make a decision, and a good answer will always surface the next decision worth making.

In this newsletter I share a simple framework for how to work out what data you should be gathering and we’ll share a little treat coming soon from Crewmojo…

Dashboards are only as useful as the questions they answer

The trouble with how our tools approached reporting before is that they were fairly fixed and focused on what data was available to show, rather than what questions should be answered to help us make better decisions.

So we adapted.

We stop asking the questions we couldn’t get answered quickly… and we start asking the ones we could.

That’s how you end up with a dashboard full of data that isn’t really useful.

Data showing you a box has been ticked isn’t the same as the thing working to solve the problem you set out to solve.

Crewmojo has rebuilt their reporting feature around exactly this challenge. Instead of picking from fixed or pre-defined reports, you ask it what you want to see and iterate from there.

But how do you know what questions to ask?

Start with the decision, not the data

Years ago at a UX conference I heard Behzod Sirjani, then a senior researcher at Slack, name a mistake I found myself repeating: doing exactly what a stakeholder wanted and collecting exactly the data they asked for.

He shared a simple framework that I’ve used ever since for deciding what data I need to gather and how: decision, evidence, data, approach.

• 🎯 Start with the decision. What are you actually trying to decide, and can your team even make that call? If there are several decisions tangled together, separate them.
• 🧭 That tells you the evidence you’ll need. What has persuaded your leadership before? Learn how they think, because the most rigorous data won’t move a decision if it isn’t the kind of evidence they trust.
• 📊 The evidence tells you the data. What type do you need, does it already exist somewhere, and is it reasonable and ethical to collect?
• 🛠️ Only then, choose the approach. What do you want to learn, from whom, how, and how will you make sense of it.

When it comes to your reports then, for any metric you track, you should be able to answer two things:

• What are you trying to learn?
• What will you change as a result?

With AI changing the way we access data, it’s about to become even more vital that we flip the order from thinking about what data we have, to what decision our data can help us make.

Try this before your next reporting cycle

Take one report or dashboard you currently send.

Write down the decision it’s meant to inform, and what you would actually change based on what it shows.

If you can’t answer either, then go back to what decisions you need to make and what data will help you make it before reconfiguring the reports that will help you move forward.