Google has 14 products with more than a billion users, and yet the company’s head of product for frontier models says the team still has “more work we need to do” on the most obvious consumer opportunity sitting in front of it. That gap — enormous distribution on one side, an unfinished assistant over Gmail, Calendar, and Docs on the other — is the tension running through a wide-ranging conversation with Tulsee Doshi, who leads product for Gemini and Google’s other frontier models, on Lenny’s Podcast.
Doshi rejects the premise that Google has stepped back from the frontier. She points to Gemini 4, codenamed “Argon,” which she says was imminent at the time of recording, and says plainly that Google wants to win the frontier conversation. But she immediately reframes the frontier as one point on a broader curve. Google builds full-stack — from hardware through models to end products — and its stated goal is a range of models across the cost-to-quality curve, serving a search user seeking information, an engineer building something new, and a cyber defender protecting infrastructure, not just whoever has the smartest chatbot.
The strategic claim underneath is that Google does not have to win on a single axis. It can also win by making existing products better, through AI Mode and AI Overviews, and by owning the efficient end of the curve. The 14-billion-user figure, which she flags as one she should double-check, is the distribution half of that argument.
What she does not resolve is why Google has not shipped a strong assistant that works across its own data, while third-party products connect to that data and do. Doshi says Google wants its data to power magical experiences, that the team is working on “some amazing things,” and that “there is more work we need to do here on this front.” The host notes that Josh Woodward, in an earlier interview, gave a similar watch-this-space answer. It is the one moment where the breadth strategy sounds less like a choice than a gap.
A model is not a normal product, and the PM has to manage the magic
The episode’s most distinctive argument is that a model does not behave like a conventional product with predictable outputs. Doshi reaches for Snape’s potions class: you put in all the ingredients and are thoughtful about what you expect, but “there will be a little bit of magic,” and that magic will be surprising and unique.
“The model isn’t predictable. It’s kind of like Snape’s potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic.”
She is careful not to overdo the mystery. Much of the process is predictable: researchers monitor training runs, track metrics, and intervene when needed. What remains is the model’s “vibes” — how a capability shows up downstream, how it feels to communicate with the model, its behavior and persona. Those qualities are partly trained in and partly discovered through evaluation and testing, which is why the PM’s job is twofold: shape what goes in, then look at what comes out, find where the model unexpectedly shines, and turn that into product.
This connects to how Google specifies model behavior. A dedicated model-behavior team works with PMs to define what “good” looks like for Gemini specifically, translates that into a written spec, then into rubrics used for evaluation, then into training data and reward modeling. Doshi frames the model’s personality as a kind of UX in its own right: how transparent the model is, how well it communicates, what questions it asks back.
Evals and vibes are two halves of the same skill, and both start small
Doshi treats structured evaluation and intuitive testing as complementary, not rival camps. On evals, she concedes Google is still figuring out how to upskill its own PMs. Her advice for a first eval is deliberately modest: start with five prompts, build to around 20 that stress-test a set of capabilities and show signal between two models. That is enough to be useful without solving every methodological question at once. Start by judging outputs yourself, she says, because even articulating what “good” means for a long multi-turn trajectory is genuinely hard, and the exercise produces clarity before it produces scale.
On vibes, her advice is to test models on work you actually care about, because that is where you will see the magic. People on social media gravitate to different probes — 3D animation and visual aesthetics, creative writing and synthesis, hard financial analysis — and the right task is one where current models show “signs of life” but are not yet perfect by your own definition, leaving room to see how different models handle it differently.
She warns that evals must be thought through holistically. The judge model’s quality matters as much as the model being judged, and whether you evaluate on your production stack with all mitigations and filters, and how the model is served, can change measured performance. Audio makes this vivid: the content of what a model says is only part of quality; naturalness, tenor, and inflection are also affected by serving, so an end-to-end evaluation setup matters.
Her most concrete example of eval design came from her six years leading product for responsible AI at Google. Early skin-tone evaluation used the Fitzpatrick scale — six tones built for healthcare purposes, skewed extremely light, and inadequate for a global population.
“Six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose.”
Working with Harvard professor Monk, whose research had produced a 10-tone scale from anthropological and sociological work, Google replaced the six-tone scale in its evaluations and used the 10-tone version to improve representation in efforts including the Pixel camera and image generation.
| Eval scale | Tones | Origin | Consequence |
|---|---|---|---|
| Fitzpatrick scale | 6 | Built for healthcare | Skewed light, could not represent a global population |
| Monk skin tone scale | 10 | Anthropological and sociological research | Adopted by Google for evaluations including the Pixel camera and image generation |
The 60/20/20 rule: ship what works now, build for what comes next
Doshi’s framework for how far ahead of model capability to build splits a product into three parts.
“You want to actually build features that aren’t necessarily always working today. Build such that 60% of what you’ve built works now and your prototype is at least functional.”
About 60% should work today, so the prototype is functional. About 20% is not perfect but works well enough, and she can see how models will get there soon. The remaining 20% is where she sees only glimmers of light but believes cracking it would produce a magical experience. She says the exact numbers are fungible but the thesis holds.
The contrast is with standard startup advice to build an MVP that works. Her version keeps the product useful today while positioning it to improve as models improve. She acknowledges the difficulty: even as someone who works on the models, she cannot say precisely where they will be in three months, so she prefers concrete proxies — where is my product now, where do I see glimmers — over forecasting model capability.
She extends the point to product architecture. Customers sometimes build rigid systems around where a model was, then struggle when the model improves.
“Consistently retest, because you might be even hampering what you’re getting out of a new model because you’re trying to force fit it into your existing structure.”
The fix is to keep retesting and to be willing to remove scaffolding. The host cites Boris Cherny’s advice to periodically strip out accumulated skills and see what the model does unaided. Doshi defines a harness simply: the wrapper around the model — system instructions, tools, skills, and context — that gives the model an environment to operate in.
Safety confidence comes from monitoring, and from AI hardening systems
Doshi’s position on AI safety is that optimism is only justified through vigilance. She does not claim the problem is solved; she says the path to optimism runs through being careful, thoughtful, and intentional, and that the critical requirement is having enough people worried and interrogating how models are built.
The specific mitigation she highlights is monitoring the model’s chain of thought to detect whether it is misaligned with its directed actions, then intervening. She says this both raises confidence in deployment and improves understanding of where misalignment appears, feeding back into model improvement.
She also argues the safety debate underweights AI’s defensive role. With Gemini 4, Google is partnering with cyber defenders, including Wiz — which she describes as now a Google company — on identifying and patching vulnerabilities. She cites Wiz’s “scan for good” program working with a hospital system to find and patch a vulnerability. The host notes a recent All-In discussion predicting heavy investment in AI-driven cyber defense as offensive AI improves, and observes that he has not read of Google’s models hacking anyone. Doshi answers that Google prefers stories about fixing vulnerabilities.
Google is hiring PMs for taste, model intuition, and tolerance for constant change
Doshi endorses the idea of a PM resurgence, with a specific mechanism: when anyone can build anything easily, building something great becomes the differentiator, and taste becomes the scarce input. The PM’s job is to separate signal from noise and find the insight that unlocks a good experience. At a company building many products, she adds a second requirement — someone who can drive consensus and turn chaos into order, because speed without that produces chaos.
Her hiring criteria are concrete. She wants PMs who understand the vibes of Google’s models or others, who have an opinion, who play with the models daily, who have their own use cases for judging quality, who stress-test capabilities themselves, and who will get scrappy with data — dive in, build an eval, pull the pieces together. She also wants adaptability, because the space changes every two weeks and a roadmap written two months ago will need rewriting.
On developing the existing team, she says the PMs are teaching her. Some have leaned in and built real skill with internal tools; the team is building a shared repository of reusable “skills” (technical, not human) so one PM’s workflow benefits others, and running workshops so more PMs gain confidence reading data and long traces. Her biggest lever is time and space: sprinting constantly leaves no room to learn the tools, but that investment pays back roughly tenfold. She describes a planned two-day November workshop pulling all PMs out of their day jobs to build and play. She does not know whether Google still runs the 20% time program, and says her PMs are already working “150%.”
AI adds heft to requirements — it does not replace the PM
Doshi’s read on how AI has changed product management: prototyping is the obvious shift, and data science is the second — PMs now have data at their fingertips for data-driven decisions. She pushes back on the social-media framing that “prototypes are the new PRD” or “evals are the new PRD.” What those slogans actually show, she argues, is that the PM has moved from writing requirements and assuming someone else — a data scientist, engineer, researcher, or designer — will supply the substance, to being responsible for going one step further and putting heft behind the requirements in whatever format that takes.
On AI-generated documents, she recommends separating data from insight: the model should supply the data — pulling dashboards, metrics, and Twitter and Reddit analysis for a two-weeks-post-launch usage update — and the PM should supply the insight. Google is investing in building the skills that produce the right data so these artifacts are well structured, which in turn frees time for creative thinking.
On burnout, she says everyone on her team is putting in all the hours and that she is actively asking how to make the environment sustainable. Her diagnosis is that burnout comes less from hours than from two other sources: churn and chaos, and work you do not enjoy — grunt work or work that feels wasted. Her hope is that AI tools remove the grunt work and free time for thinking and creative work.
For builders and investors, the implications cluster in three places. First, architecture: any product wrapped rigidly around one model version is a depreciating asset, which is why Doshi’s insistence on retesting and stripping scaffolding matters more than any single benchmark. Second, talent: if taste and model intuition are the scarce inputs, the moat shifts from engineering headcount toward judgment, and companies that let PMs spend time learning internal tools expect roughly a tenfold return on that time. Third, strategy: Google’s stated answer to the frontier race is breadth across the cost-quality curve, so no single model release decides the outcome — and its distribution advantage means the unfinished assistant over Gmail and Calendar is the launch worth watching. The unresolved question is operational. Doshi does not yet have a sustainable-workload answer for a team she says is working at 150%, with a two-day offsite as the current intervention — the cost of the frontier race she says Google intends to win.