The 10 Levels of Data Mastery as a PM (Most get stuck at L4)

+ Common mistakes Product Managers commit at each level

Here are 10 levels to judge how deep a Product Manager works with data.

Sadly, many PMs get stuck at Level 4 & self-declare themselves to be "data-driven".

LEVEL 1: Identifying metrics

The PM knows what metrics need to be tracked to measure product or feature success. Ex: conversions, retention rate, MRR, NPS, activation rate etc.

🔸 Common mistakes: listing out too many metrics than what is really required. This is the stage where vanity metrics (that have little bearing on the larger goal) start getting tracked and reported on.

LEVEL 2: Tracking data

The PM then works with ops to track the data consistently using the best medium possible e.g. setting up instruments like Mixpanel, Segment, Pendo, Heap or Google Analytics.

🔸Common mistakes: Sometimes it’s better to track metrics through native database queries or a Google Data Studio hooking into a data model vs. a Google Analytics instance with inaccuracy issues.

LEVEL 3: Treating data

The PM understands the biases in the data & sanitizes/addresses them before commencing analysis.

🔸 Ex: A PM at this level would know how averages with datasets containing several outliers get skewed. They may use medians if the data set works better with that aggregate.

LEVEL 4: Reporting data

The PM periodically summarizes & shares the numbers with the team & leadership to create alignment.

🔸 The sad fact is that most of these reports are devoid of commentary. They are just a heap of numbers dumped in a Slack channel or email thread which, over time, starts getting ignored.

<-- Most PMs stop here -->

LEVEL 5: Contextualizing data

The PM helps the reader understand what the significance or weight of the data is through supporting information i.e. adds context.

🔸 "Context" comes in different forms but is often expressed via:
(a) Rate of change: how the metrics has changed over time
(b) Benchmarks: how data fares against industry standards or comparable products.
(c) Ratios: what the numbers mean in relation to other metrics. Ex: "1 out of 4 users who watch a demo purchase an item."

LEVEL 6: Visualization

The PM understands how to surface data visually such that it instantly drives a point home & influences the reader.

🔸 Common mistakes: trying to choose "pretty" visualizations rather than "fit" ones. Ex: choosing a pie chart to depict a breakup of a data point with > 30 values.

LEVEL 7: Insights

The PM synthesizes "past" data with other evidence parameters (e.g. customer feedback) & aptly communicates & documents what this means for the product.

🔸 Ex: stating a cogent argument backed with data points why a step in the onboarding process needs to be revamped or eliminated completely.

LEVEL 8: Foresights

The PM catches on a trend to predict "future" behavior & uses it to formulate a strategy to capitalize on it.

🔸 Ex: an ecommerce PM notices a promising rise in no-result search patterns for a particular niche & proactively plans to build that category up.

LEVEL 9: Translation

The PM is able to translate the insights & foresights into an action plan that leverages the data appropriately.

🔸 Common mistakes: attempting to build something that may optimize one metric but compromise another. Ex: shortening a sign up process but giving up on user profile richness.

Level X: Optimize

The PM is able to gain maximum mileage on a metric through a series of meaningful experiments. They consolidate this learning but most importantly, recognize when it's time to move onto other priorities.

🔸Common mistakes: attempting to over-optimize a metric when the upside doesn't justify the effort.

Conclusion

There seems to be this unchecked romanticism in Product Management circles about being data-informed. However, while all the excitement is encouraging, very few can describe what it means to use data as a tool to build better products.

The reality is that most PMs fall somewhere in this first half of this 10-level spectrum. Very few are able to utilize metrics to make meaningful, impactful decisions.

At the same time, it’s important to note that every initiative or feature does not require all 10 levels of data skills. Some are smaller bets where basic applications suffice. However, for the bigger bets or core value propositions, PMs need to strive to traverse most of these levels to unravel big wins.

It’s important for Group Product Managers and CPOs to understand the differences between these levels and guide their PMs accordingly.