The challenges with survey-based research

PMs need to interpret surveys with a grain of salt.

Here's a little story.

In my early days as a manager, I had to conduct a performance appraisal.

Each question on the appraisal form had a 6-part rating scale.
1 = Below Expectations.
...
5 = Exceeds Expectations.
6 = Above & Beyond Call of Duty

To my surprise, when I received appraisal ratings of my reports, most of them had rated themselves a 6 on more than 90% of the questions.

During my 1:1s, I asked them to justify their scores.

Some reasoned that they did that because they had put in extra hours at work during that time. Others gave such scores because they "did whatever they were asked to do".

I had to explain that a value like "Communicate openly & honestly" had nothing to do with putting in more time.

Thus, the self-appraisal scores varied with each employee's nature, self-awareness & alignment with the definition of what each notch on the scale meant.

--

Here's the fallacy we fall into:
Just because a survey has a point-scale doesn't mean it suddenly becomes "objective" or "empirical". In fact, it still carries subjectivity, only now it's conveyed numerically.

The same phenomenon occurs with customer feedback.

Net Promoter Score is an example of a 10-point scale that allows you to understand how likely a customer will refer your product to their network.

Scores will be fragmented based on each person's mental model:
🚩 Generous customers will be willing to rate a decently satisfactory experience a 9.
🚩 Grumpy customers would rate a 5-star Apple-like experience a 7.
🚩 Unforgiving customers will remember that one negative incident in an ocean of stellar experiences and rifle down the provider's reputation with say, a 3.

And so on.

This isn't to say NPS is useless. With enough data points & segmentation, it can still be a handy "trend" indicator. But for reasons mentioned above, it shouldn't be the sole performance metric.

The solution?

COMBINE survey data with the following:

1) Actual usage
How frequently do they accomplish key goals on the product? Measure how your value metric moves. (ex: for Uber, it might be rides booked/month).

2) Review retention events - do they keep renewing? This indicates how keen they are to retain the value you offer.

+ Most importantly

3) Get qualitative feedback by talking to users BUT also asking the right questions:

For example,
✔️ When was the last time you used the product? What for?
✔️ If were to close down tomorrow, how would that affect you?
✔️ What should we start, stop or keep doing?
✔️ What problems have we solved for you since you started using us?
✔️ Who, in your network, are you willing to refer us to in the next X days?

Rating scales don't tell the entire story. Augment them with a good old conversation.

Thank you for reading Behind Product Lines. This post is public so feel free to share it.

Share

Tips for PMs jumping from B2C to B2B

Someone asked me the following question on LinkedIn:

Q: “I’m a Product Manager moving from a B2C product to a B2B one. Is there a difference I should expect?”

I thought I'd share my response with the wider community.

The biggest adjustments that one has to make when moving from B2C:

B2B products can involve a matrix of stakeholders as opposed to individuals.

In B2C, you’re dealing with a single individual and their needs. (Of course, that doesn’t necessarily mean it’s simpler.)

In B2B, there’s usually 1-2 people researching solutions who will first try your product or request a demo. However, as you get closer to purchase, it’s normal for an entire committee of people to get involved. B2B is where you'll see the classic divide between buyer and user.
The B2B end-to-end buying journey is quite different.

If you're not operating with a PLG model, then the sales cycles are longer & complex with considerations on non-functional parameters like compliance, security, data ownership & integrations.
B2B is not a user volume game.

You have to think in terms of company accounts, how you can retain & expand them over time.

Image Credit: Product Intuition

The Buying Roles

If you're moving into the enterprise SaaS space, you’ll see different buying roles within this group of people.

End-User

The person who will make use of your product. An end-user alone may not be able to influence decisions. However, if enough end-users bandy together, it can factor in a buying decision.

Decision Maker

The person who will ultimately write the cheque. Typically, a C-level executive.

Champion

The person who is enthusiastic about your product & actively puts in a good word to others in the committee.

Blocker

A person on the committee that can raise a deal-breaking objection and bring the sale to a halt. Legal can be a blocker if they find contractual issues. IT can be a blocker if they don’t find your security protocols sufficient.

So, why should we care as a PM? Isn’t all this relevant for sales?

Product Managers need to care because the committee’s objections can be deeply linked with the product.

For example, an end user want a product that ease their woes and make them look good to their supervisors.

The decision maker may want detailed reporting to prove the worth of the product in business terms.

The champion may call for a product demos or collateral to be able to influence decision makers.

Most importantly, blockers can throw a lot of challenges.

Some examples:

Finance & Procurement can raise concerns about product pricing but also how the product accepts payments, records audits, processes billing & issues invoices.
Legal & Compliance may want assurance on GDPR compliance, ownership of data, migration tools to import and export data, permission control to regulate data access.
IT & Infrastructure can throw curveballs around security, infrastructure, backup policies, redundancy, single-sign on support, integrations, APIs & more.

All of these considerations are important when you’re planning your roadmap.

Because you need to know what the product needs to offer to be able to successfully sell it to a committee & put it in their hands.

Some practical tips that came in handy for me when transitioning to B2B SaaS:

Go through a few buying cycles with sales rep - from lead acquisition to deal closure. Understand what it takes to close one account.
Deliver demos to a few of the top accounts & listen to objections, needs, priorities. Get a feel of the jargon those stakeholders use.
Document problem statements from each committee member. Explore how product can assist in overcoming their challenges.
Attempt to “interview” the end user and decision maker separately to isolate needs.

Behind Product Lines

Why PMs need to be careful with NPS Surveys