Many, many more case studies available on request.
AI Coach
Defining the voice of an AI health coach and building the eval framework to keep it honest
As MyFitnessPal introduced AI Coach to ~700k Premium users, we faced a question the product had never had to answer before: what does "good" mean for an AI agent, giving advice in a category where language about food, weight, and calories can easily tip into judgment or harm?
I led the definition of the coach's core voice, establishing how it should balance empathy with efficiency, accuracy with personalization, and claimed helpfulness with actual agent ability.
Because our classic testing doesn’t solve for non-deterministic outputs, I partnered with our AI engineer to design our llm-as-judge eval system: a rubric scoring every response on safety, voice, and correct tool use (flow execution) across a set of key scenarios. This gives us objective signal on whether prompt changes are moving the coach in the right direction.
That framework is becoming the standard the team runs against every coach update before it ships, which shifted my role from "writes the prompt" to owning the quality bar for how the product speaks and behaves.
Streaks
Dissecting user motivation
Streaks began with a hypothesis about milestones: if we celebrated key days (3, 7, 10), users would be more likely to push toward the next one, driving our core metric of week-over-week food logging. For the first test, I worked with our business intelligence team to source social proof claims like "You're in the top 50% of users!". Users came back to log 3% more food and 1% more days in week 2.
Frequency | Learning: More celebrations = Good
That result opened a bigger question: how far could frequency go? We kept testing more celebration days until we were celebrating every single day from day 2–10. That experiment reached stat-sig +1.5% average app open days and +2% average food logging days compared to control.
Timing | Learning: Drive the Action → Reward cycle
Alongside frequency, we tested timing: celebrating a user the instant they logged food and incremented their streak, versus waiting until they returned to the Today screen. The in-the-moment celebration won, even though it interrupts the user mid-task. The suppressed celebration was stat-sig negative on average app open days. The principle this surfaced: celebrations are welcome distractions as long as they feel meaningful, and reinforce a powerful Action → Reward cycle.
Copy | Learning: Lean on social proof and users’ identity
With frequency and timing solved, I turned to copy. My hypothesis was that novel, varied copy each day would outperform repetition. But, it didn't. leaderboard-style social proof still won with a small neutral-to-negative result on the variant. The takeaway reframed how I thought about streak copy going forward: users respond less to novelty than to feeling part of something larger than themselves.
That insight opened a new question: What about Day 1, before leaderboards become relevant? I tested a day-1 streak for returning users leaning on identity: "Mary is back!" It won with a stat-sig +2% average app open days, confirming that identity-based framing can be as powerful as social proof.
Streaks Day 2-10 Copy Experiment: Leaderboard vs. Novelty
Streaks Day 1 Existing Users Copy Experiments
Streaks Day 1 New Users Copy Experiments