Notion AI: 5 Lessons for Product Managers
Discover more from Behind Product Lines
Notion AI: 5 Lessons for Product Managers
+ Must-have Resource: 23 Book Summaries for Product Managers
Hello product champs,
It’s been another whirlwind of a week for product companies in the AI arena.
Some announcements that caught my eye:
Spotify announced it’s AI-powered music recommendation engine: DJ,
Meta rolled out it’s own large language model for governments & researchers and
Based on my conversations with the PM world, I’ve realized that a number of product companies are feeling intense FOMO and are reacting to this hype by rolling out their own AI integrations based on Open AI’s APIs. I know several PMs are going to feel the pressure for quick time-to-market on this and thus, in this newsletter, I wanted to caution the product tribe on a few things to look out for as they embark on this journey.
In this edition of Behind Product Lines, we’ll cover the following topics:
23 Book Summaries for Product Managers
5 PM Lessons from Notion AI
23 Book Summaries for Product Managers
Reading PM books is easy. Retaining them is hard.
I've often struggled with trying to remember an anecdote or lesson I got from a specific book.
So, I thought I’d solve this problem myself using a little AI magic.
I created a Notion repo with short summaries & takeaways from 23 product/startup books that I've preferred over time.
No, these aren’t the only books I recommend and they aren’t rank ordered in any way.
I leveraged ChatGPT to help me build this which saved a lot of time.
You can find the Notion repository here.
This repo might be helpful for 2 categories of people:
1. Those professionals that have read the books and need to jog their memories every once in a while.
2. Aspiring product people that have yet to read these books & would like to get a preview into what they would potentially learn.
Disclaimer
These summaries are by no means a replacement for reading or listening to the actual book. One should still make an effort to grab a copy if they really want to grasp the author's viewpoint.
Hope it helps!
5 PM Lessons from Notion AI
It seems everyone’s throwing in their towel in the AI ring these days.
In all fairness, I’m part of that bandwagon. vFairs announced their AI suite too.
At the same time, it’s only when we started integrating with Open AI that I realized the limitations that I had to grapple with.
If you’re using the vanilla APIs without training the model on the kind of content you want to optimize it for, your customers will see sub-standard results.
AI is a building block and it needs to be treated as such. Unless you wrap it with a solid user experience where the user can swiftly understand how they can get a job done faster, you won’t get far in terms of traction.
Open AI results aren’t perfect. In fact, Sam Altman himself says that ChatGPT is incredibly limited but “good enough at some things to create a misleading impression of greatness.” He further adds, “it's a mistake to use ChatGPT for important tasks.”
We noticed that we would have no visibility to what kind of results our users are getting if we didn’t plan for it early. Therefore, training the model and establishing a feedback loop are active conversations in the development team.
Now, the latest notable player in the AI ring is Notion. It is also based on Open AI’s APIs (GPT-3).
Of course, comparing ChatGPT with Notion isn’t really an apples-to-apples comparison. One is a prototype to prove the value of AI models. The other is a productivity suite add-on. I don’t quite rate ChatGPT as a MVP either as it doesn’t position itself in any particular product category. However, Notion AI’s execution does show how one can take a promising technology and construct a compelling user experience around it.
A few lessons for myself that might be helpful for Product Managers planning to embrace AI in their respective areas:
1. Educate the user on what’s possible.
ChatGPT was a revolutionary breakthrough but what fueled it’s growth was widespread information on the kind of magical things you could do with it.
From December 2022 onwards, there was a flurry of ChatGPT screenshots on social media that clarified how people - from content writers to marketers to sales people to HR - could leverage it to get their “jobs” done faster.
People were curious and thirsty for use cases they could try ChatGPT with. That’s partly why the ChatGPT e-book for PMs took off.
Notion AI didn’t rely on ChatGPT’s tailwinds. It doesn’t leave the onus of discovering use cases to the user at all.
As you’re typing out anything on it’s editor, you can hit space bar to get a list of things you can do right now. That instantly inspires users to adopt the tool for various jobs they may have in mind.
Lesson #1:
Audiences want to know exactly what your AI enhancement can do. Surface the possibilities clearly and demonstrate the value.
2. Educate on limitations.
As I mentioned above, no AI model is perfect. It’s very easy for users to buy into the technology and become blind to it’s limitations.
It’s not just that it might come up with poor results. It might actually produce outputs that are morally, logically or ethically wrong and can have, at times, have adverse consequences. As a Product Manager, you need to bake in some guardrails.
For example, Notion AI shows a disclaimer right below the text bar to help user’s understand what they’re getting into. It’s not afraid to use strong words like “inaccurate” and “misleading”.
Here’s what the disclaimer links out to:
Rather than tooting the AI horn indefinitely, it’s important to be transparent from the start to preempt inbound grievances (or worse, legal trouble).
Moreover, many product people are forgetting that privacy is a critical matter.
You cannot simply assume that training your model on user-generated data moving forward is a given. This will require explicit permission and approval or, in Notion’s case, clear declaration that user data will not be leveraged in any way.
Lesson #2
Let your users know about the limitations upfront. Don’t assume AI models are perfect. They will issue less than ideal responses that you don’t want to be liable for.
3. Weave it into the fabric of the product.
As the Open AI integrations pour in, I’m seeing some products are creating dedicated AI modules separate from the core user experience to simplify development. This, in the long run, will be a mistake and will lead to lower adoption.
The beauty of Notion AI is that as I’m selecting text or editing, the AI assistant is right there. The AI layer needs literally no learning curve as it lives in the environment that I’m already familiar with.
Moreover, unlike ChatGPT that focuses more on generating/regenerating content, Notion AI has a lot more focus on improving content that you’ve already written by offering operators like “Improve writing”, “Make shorter”, “Change tone of voice” etc.
After a simple permission to enroll me in a trial and a tip on the keyboard short cut (space bar), I’m good to go. Moreover, new users to Notion won’t find it hard to discover the feature either.
Lesson #3
Don’t isolate your AI capabilities. Use it as a foundational building block to lift your current product experience.
4. Capture instant feedback.
This is something that most Product Managers will miss as they hurry towards shipping out an AI-enabled product increment: collecting feedback on responses.
I highly encourage Product Managers to first learn a bit about how ChatGPT works behind the scenes. Here’s an article I recommend written by Marco Ramponi.
An important excerpt from the article:
Reinforcement Learning from Human Feedback
The method overall consists of three distinct steps:
Supervised fine-tuning step: a pre-trained language model is fine-tuned on a relatively small amount of demonstration data curated by labelers, to learn a supervised policy (the SFT model) that generates outputs from a selected list of prompts. This represents the baseline model.
“Mimic human preferences” step: labelers are asked to vote on a relatively large number of the SFT model outputs, this way creating a new dataset consisting of comparison data. A new model is trained on this dataset. This is referred to as the reward model (RM).
Proximal Policy Optimization (PPO) step: the reward model is used to further fine-tune and improve the SFT model. The outcome of this step is the so-called policy model.
Step 1 takes place only once, while steps 2 and 3 can be iterated continuously: more comparison data is collected on the current best policy model, which is used to train a new reward model and then a new policy.
Thus, the AI model is based on reinforcement learning and that feedback loop is what helps it to improve over time. That’s why Notion AI is allowing users to rate their responses (plus give them permission to review the original prompt), so that they can understand where the model is failing and remedy it with targeted training.
Lesson #4:
Create feedback capture mechanisms to figure out early where the AI model is serving sub-optimal results.
5. Prove value before you make your ask.
Another mistake products will make is that they will assume they don’t have to prove any value (because people have tried out ChatGPT) and will force users to “pay before they play”.
In Notion AI’s case, I was able to use the AI features for a generous amount of time. I was in “research mode”, so naturally my usage was on the higher side. By the time, I got the upgrade prompt, I had already charted out some primary use cases that Notion AI could help with in my daily routine.
Notion AI is priced at $10 per member per month at the moment including free plans.
Note on pricing
Based on the rest of the market, the fee isn’t a radical amount but for people like me who use Plus, it doubles our subscription amount instantly. Time will tell if this pricing strategy will pay off.
Lesson #5:
Adopt a reverse-trial model. Let people play around with your AI facility & truly acknowledge it’s value before you put up a paywall.
These are exciting times. However, let’s not get ahead of ourselves.
ChatGPT has opened a world of opportunities. But it’s a world with latent laws and limitations. Without knowing what those are, the magical fad of AI will come to a grinding halt as soon as the euphoria wears out.
Learn before you leap.
Notion AI: 5 Lessons for Product Managers
Solid book summary approach.