The Marketing Formula that Propelled Duolingo to 500M Users

01:40:30
https://www.youtube.com/watch?v=RLndfgfjVMM

Summary

TLDRThe episode of the Logan Bartlett Show features an extensive conversation with Luis von Ahn, the co-founder and CEO of the language learning platform Duolingo. Duolingo, valued at $9 billion, differentiates itself through a strong culture of product iteration and user engagement. The conversation delves into how this culture is built around continuous improvement, A/B testing, and creative risk-taking. Luis discusses Duolingo's early decision to focus solely on improving user retention rather than monetization, eventually leading to significant market growth, especially in the U.S., where 80% of users were not previously learning a language before using Duolingo. Luis also narrates how Duolingo's mascot, Duo, became a viral meme, contributing to the company’s marketing strategy. The company's revenue model, which relies on a freemium approach with ads and subscriptions, evolved from a firm belief in providing free education. Duolingo also avoids relying heavily on performance marketing, instead relying on organic and creative engagement strategies. Through these approaches, the company has been able to significantly grow its user base and improve educational outcomes, pushing boundaries without becoming overly reliant on traditional marketing avenues.

Takeaways

  • 📈 Duolingo's success is built on a culture of continuous product iteration and user engagement.
  • 🦉 The company's mascot, Duo, has become a viral meme, boosting their marketing efforts.
  • 💡 A/B testing is crucial for refining product features and understanding user behavior.
  • 📊 User feedback is important, but data-driven decisions are prioritized to enhance the learning experience.
  • 📱 The platform capitalized on mobile growth, bringing language learning to new audiences.
  • 💸 Duolingo utilizes a freemium model, offering a blend of free access and paid features to sustain revenue.
  • 🚫 By avoiding heavy reliance on performance marketing, Duolingo focuses on creative and organic growth strategies.
  • 🎯 The team maintains a focus on high user retention before monetization to grow the market.
  • 👥 Luis von Ahn highlights the importance of hiring exceptional and kind talent for company culture and success.
  • 🔄 Duolingo's expansion beyond languages, to areas like math and music, broadens its educational impact.

Timeline

  • 00:00:00 - 00:05:00

    The episode features a conversation with Luis von Ahn, co-founder and CEO of Duolingo, discussing the company's culture of product iteration, AB testing, and the viral success of their mascot, Duo. Luis also reflects on Duolingo's unexpected path to becoming a billion-dollar business in language learning.

  • 00:05:00 - 00:10:00

    Luis explains that the key to Duolingo's success was a focus on user retention for the first five years, even without monetization. The company prioritized making the product engaging to grow the language learning market, particularly in countries where such tools were scarce.

  • 00:10:00 - 00:15:00

    Duolingo started testing changes to their product only after realizing they couldn't objectively measure their impact otherwise. A set of tests revealed they weren't good at predicting success, which led them to implement a rule where every change had to be tested.

  • 00:15:00 - 00:20:00

    Luis recounts how initially they bet on which product change would work best but found that data often contradicted their instincts. This convinced them of the necessity of an AB testing infrastructure, which now runs over 500 tests per quarter, encouraging a culture where failing tests is acceptable.

  • 00:20:00 - 00:25:00

    The discussion highlights how effective AB testing at Duolingo involves putting a team on optimizing significant metrics, such as 'time spent learning.' If issues arise, they are addressed to avoid issues like those experienced by Groupon due to over-testing.

  • 00:25:00 - 00:30:00

    They discuss the balance between using data and listening to user feedback. Duolingo employs user research but relies heavily on AB testing to identify real user struggles, supported by a knowledgeable internal team that's adept at understanding what works.

  • 00:30:00 - 00:35:00

    Luis talks about marketing strategies, emphasizing the importance of understanding user needs rather than just feature suggestions. Duolingo engaged social media users organically by leaning into existing memes about their brand, using this to drive effective and cost-efficient marketing.

  • 00:35:00 - 00:40:00

    Initially having no marketing budget, Duolingo relied on PR and the quirky appeal of their mascot to gain traction. The first significant marketing changes came with the arrival of their CMO, who steered them towards leveraging viral social media content effectively.

  • 00:40:00 - 00:45:00

    Duolingo capitalized on memes portraying their owl mascot humorously, which went viral. They explored creative and engaging marketing angles instead of traditional methods, emphasizing humor in their campaigns to boost engagement without heavy reliance on ROI metrics from social content.

  • 00:45:00 - 00:50:00

    Luis discusses the challenges of setting metrics in a way that doesn’t exhaust resources or frustrate users, like what happened with Groupon. They run rigorous reviews to ensure their product changes lead to genuine improvements without negative consequences.

  • 00:50:00 - 00:55:00

    Luis emphasizes Duolingo’s reliance on iterative design and rapid user feedback rather than traditional market studies, which enabled them to scale effectively. The strategy involves building a culture where it’s okay to fail fast and learn from it.

  • 00:55:00 - 01:00:00

    Marketing at Duolingo evolved under the new CMO, shifting from PR-focused efforts to building a brand through viral content. Luis highlights how being data-driven and responsive to user-created content allowed them to optimize their strategies.

  • 01:00:00 - 01:05:00

    The integration of humor and timing in their marketing proved successful, particularly leaning into user-driven content about Duolingo's owl mascot. Their marketing’s evolution shows a shift from simply gaining attention to engaging deeply with potential users.

  • 01:05:00 - 01:10:00

    Luis illustrates Duolingo’s agility by discussing how they leverage temporary ad space and adaptable marketing strategies to remain cost-effective while maximizing reach. They use this flexibility to focus on organic growth and user content.

  • 01:10:00 - 01:15:00

    He talks about Duolingo's early attempts at innovating language learning without prior knowledge in the field and how initial misconceptions were overcome through user feedback and AB testing, refining their approach to teaching languages effectively.

  • 01:15:00 - 01:20:00

    Luis confirms the success of Duolingo largely thrived on understanding the market was different than initially anticipated. Their strategy included focusing heavily on the user experience and integrating learning improvements based on extensive tests.

  • 01:20:00 - 01:25:00

    The conversation turns to how Duolingo scaled its operations to become a public company, bringing in both commercial success and social impact. The approach includes balancing between free services and premium offerings to support the business model.

  • 01:25:00 - 01:30:00

    Discussing the company culture, Luis highlights the importance of maintaining high hiring standards and incentivizing employees. They use team reviews to hold the line on quality, ensuring the right people are in place to maintain company values.

  • 01:30:00 - 01:35:00

    The discussion includes insights into maintaining innovation while managing growth, emphasizing metrics without stifling creativity. Luis stresses that understanding deeper user issues comes from staying close to user needs and experiences.

  • 01:35:00 - 01:40:30

    Closing the episode, Luis reflects on personal work habits and learning approaches that have contributed to his success, along with insights on maintaining discipline and work-life balance. His focus remains on impactful, mission-driven work.

Show more

Mind Map

Video Q&A

  • Who is Luis von Ahn?

    Luis von Ahn is the co-founder and CEO of Duolingo and previously developed reCAPTCHA, which he sold to Google.

  • What is Duolingo?

    Duolingo is a language learning platform known for its engaging, iterative approaches to education and is publicly valued at $9 billion.

  • How does Duolingo maintain high user engagement?

    Duolingo focuses on incremental product improvements, such as higher user retention through engaging features and A/B testing.

  • What is the significance of A/B testing at Duolingo?

    A/B testing is crucial for Duolingo's product development, allowing them to determine the actual effectiveness of features in improving user engagement.

  • What kind of culture does Duolingo promote?

    Duolingo promotes a culture of continuous improvement, creativity, and data-driven decision making, encouraging risk-taking and learning from unsuccessful A/B tests.

  • How did Duolingo's mascot become viral?

    The mascot, Duo, became a viral meme through strategic marketing and user engagement, leaning into internet culture and humor.

  • What is a major revenue strategy for Duolingo?

    Duolingo employs a freemium model with optional ads and subscriptions for enhanced features to generate revenue.

  • Why did Duolingo initially avoid performance marketing?

    Duolingo avoided performance marketing initially due to its expense and the fact that the company wasn't generating revenue at the start.

  • What is the philosophy behind Duolingo's product iteration?

    The philosophy is to improve the product continuously, making it sticky and engaging through user-focused experiments and data utilization.

  • How does Duolingo balance user feedback with data?

    While user feedback is considered, Duolingo prioritizes data-driven decisions to determine what truly enhances user experience.

View more video summaries

Get instant access to free YouTube video summaries powered by AI!
Subtitles
en
Auto Scroll:
  • 00:00:00
    welcome to Logan Bartlett show on this
  • 00:00:01
    episode what you're going to hear is a
  • 00:00:02
    conversation I have with Louise Von on
  • 00:00:05
    now Louise is the co-founder and CEO of
  • 00:00:07
    dualingo a $9 billion business in the
  • 00:00:09
    public markets that builds an
  • 00:00:11
    application for language learning L and
  • 00:00:13
    I talked about a number of different
  • 00:00:15
    things including how they built a
  • 00:00:16
    culture of iterating on product and AB
  • 00:00:19
    testing different features as well as
  • 00:00:22
    how they got their mascot Duo to become
  • 00:00:24
    a meme and go viral on different social
  • 00:00:27
    channels really fun conversation with
  • 00:00:29
    someone that is very very bright
  • 00:00:31
    including having a PhD in computer
  • 00:00:34
    science but applied that to building
  • 00:00:36
    both duol lingo as well as the prior
  • 00:00:38
    company that he sold to Google called
  • 00:00:40
    recapture which you might be familiar
  • 00:00:42
    with you'll hear that conversation with
  • 00:00:44
    Louise here
  • 00:00:53
    now I want to talk about like building a
  • 00:00:55
    culture of iteration AB testing
  • 00:00:57
    continuous Improvement all that stuff I
  • 00:00:59
    kind of have a opinion that in some
  • 00:01:01
    companies or some markets uh there's an
  • 00:01:04
    opportunity that opens up and it's just
  • 00:01:06
    a debate of which compan is going to run
  • 00:01:08
    through uh and build the equity value
  • 00:01:11
    that comes out of that I'm not sure if
  • 00:01:13
    you agree with this but my sense is that
  • 00:01:16
    you guys were in the right place right
  • 00:01:17
    time with regard to mobile and you were
  • 00:01:20
    right about your thesis around language
  • 00:01:23
    but it was actually the culture of
  • 00:01:25
    incremental Improvement and just
  • 00:01:28
    continuously getting better with the
  • 00:01:30
    product that has led to the distance how
  • 00:01:34
    big of a business you've been able to
  • 00:01:35
    build I don't know if you agree with
  • 00:01:37
    that that thesis that it wasn't
  • 00:01:38
    necessarily inevitable that there would
  • 00:01:40
    be an8 billion public company built
  • 00:01:43
    around language
  • 00:01:45
    learning yeah I I don't think it's
  • 00:01:48
    necessarily the case that a that an8
  • 00:01:50
    billion doll company needed to exist in
  • 00:01:53
    language learning um it's true that we
  • 00:01:55
    were at the at the right time with
  • 00:01:57
    mobile for sure and we capitalized for
  • 00:01:59
    that uh capitalize on that but I really
  • 00:02:02
    think we were able to create a product
  • 00:02:05
    that
  • 00:02:07
    um got people very engaged and and such
  • 00:02:11
    a thing didn't exist for language
  • 00:02:12
    learning and and I'll tell you the
  • 00:02:14
    biggest stat that that supports that 80%
  • 00:02:17
    of our users in the United States were
  • 00:02:19
    not learning a language before dual
  • 00:02:21
    lingual just we basically massively
  • 00:02:24
    grown the market in certain countries
  • 00:02:26
    not in every country but in certain
  • 00:02:27
    countries we've massively grown the
  • 00:02:29
    market and it's I think it's just
  • 00:02:31
    because I I mean the main reason for it
  • 00:02:33
    is because for the first five years of
  • 00:02:36
    the
  • 00:02:37
    company the only thing we did was work
  • 00:02:40
    on making dualingo have higher user
  • 00:02:43
    retention and that's not paid user
  • 00:02:45
    retention that's just free we we were
  • 00:02:47
    not making any money for the first five
  • 00:02:48
    years of dualingo we make zero dollars
  • 00:02:50
    like we were not making any money and
  • 00:02:54
    because we were not making any money we
  • 00:02:55
    also weren't spending any money on
  • 00:02:56
    marketing so there was kind of nothing
  • 00:02:59
    to do I mean we we weren't spending any
  • 00:03:01
    money on marketing so there was not much
  • 00:03:02
    to do there we weren't making any money
  • 00:03:04
    so there was not much to do in terms of
  • 00:03:06
    like you know working on the
  • 00:03:07
    monetization of dueling or or anything
  • 00:03:09
    like that the only things we could work
  • 00:03:10
    on are teach better and keep people
  • 00:03:14
    engaged and so that's all we did for
  • 00:03:17
    five years we simply made the product
  • 00:03:20
    stickier and stickier and it really made
  • 00:03:23
    a big difference how do you build a
  • 00:03:24
    culture of like AB testing and iteration
  • 00:03:27
    and people willing to take chance is
  • 00:03:30
    knowing that you might make mistakes and
  • 00:03:33
    that you can undo them if so it's not
  • 00:03:35
    like when we started we thought we're
  • 00:03:38
    going to build a company around AB
  • 00:03:39
    testing this is not you know the way it
  • 00:03:41
    started is we started we started with
  • 00:03:44
    the product and we we would make some
  • 00:03:45
    changes to it and
  • 00:03:49
    um we would make like whatever the
  • 00:03:51
    changes were like okay we're going to
  • 00:03:53
    add
  • 00:03:53
    a screen at the end of a lesson that has
  • 00:03:56
    fireworks or whatever random
  • 00:03:58
    changes and two things were happening
  • 00:04:01
    number
  • 00:04:03
    one we would add the string and we
  • 00:04:06
    couldn't tell if the thing was better or
  • 00:04:07
    not because you know we would we would
  • 00:04:10
    see that the metrics were like either
  • 00:04:12
    kind of flat but if they were flat we
  • 00:04:13
    always found an explanation like oh it's
  • 00:04:15
    because it's because this week is
  • 00:04:16
    Thanksgiving or whatever it is and
  • 00:04:18
    that's why normally we would have gone
  • 00:04:20
    down but because we uh you know added
  • 00:04:24
    the screen we didn't go down or
  • 00:04:25
    something like that like whatever
  • 00:04:27
    couldn't tell if things were actually
  • 00:04:28
    helping or not
  • 00:04:31
    and in addition to that we we couldn't
  • 00:04:35
    tell which idea was good and which
  • 00:04:37
    wasn't because we're like well if we
  • 00:04:38
    couldn't tell if that was a good thing
  • 00:04:40
    or not we you know I don't know was that
  • 00:04:42
    a good idea or not and so at some point
  • 00:04:45
    um it occurred to us you know what we
  • 00:04:46
    should actually run one ad test and
  • 00:04:49
    figure out if if that thing was actually
  • 00:04:52
    good um and we run I remember we ran a
  • 00:04:55
    test I don't remember exactly what the
  • 00:04:57
    feature was but I remember there were
  • 00:04:58
    like four options
  • 00:05:00
    and we each one of us bet on which the
  • 00:05:02
    option was going to be best and we
  • 00:05:04
    thought of ourselves as very good
  • 00:05:05
    product people and it turned out none of
  • 00:05:08
    us were right and then after that we
  • 00:05:11
    thought oh crap I guess we really had no
  • 00:05:13
    idea you know we ran the a test the
  • 00:05:15
    thing that one was not the thing any of
  • 00:05:16
    us thought was was going to win and then
  • 00:05:19
    for the next few weeks we ran a number
  • 00:05:21
    of other AB tests and we always would
  • 00:05:25
    say try to predict what was going to win
  • 00:05:28
    and we realized we actually quite crappy
  • 00:05:30
    at predicting what was going to win and
  • 00:05:32
    that's when we really convinced
  • 00:05:34
    ourselves well look we don't really know
  • 00:05:38
    what is better for a product I mean we
  • 00:05:41
    have some idea but it's not you know
  • 00:05:42
    it's better than 50% we know much better
  • 00:05:45
    than 50% so you know at that point we
  • 00:05:48
    kind of started this rule that every
  • 00:05:50
    change up du a linga is a Navy test and
  • 00:05:52
    so we built a pretty sophisticated
  • 00:05:54
    infrastructure which was way ahead of
  • 00:05:55
    its time by now I think you know this
  • 00:05:58
    was more than 10 years ago was way ahead
  • 00:05:59
    of its time in terms of how good it was
  • 00:06:01
    also for the size of company maybe
  • 00:06:03
    Google at the time has such a good
  • 00:06:04
    infrastructure much probably better than
  • 00:06:05
    ours but for a small company you know it
  • 00:06:08
    was kind of unheard of to have this
  • 00:06:10
    infrastructure but we built an
  • 00:06:11
    infrastructure that really allowed us to
  • 00:06:12
    make everything an test and I think that
  • 00:06:16
    that really made a made a big difference
  • 00:06:19
    the other thing that I think in terms of
  • 00:06:20
    a culture you we make we make it clear
  • 00:06:24
    that it's totally okay to run AB tests
  • 00:06:26
    that don't succeed and you know we have
  • 00:06:28
    the stats you know I don't know off the
  • 00:06:30
    top of my head anymore but I mean we run
  • 00:06:32
    something around 500 a test per quarter
  • 00:06:36
    and look something like 60% of them
  • 00:06:38
    succeed and the other 40% fail that
  • 00:06:40
    means hundreds of ab tests that we run
  • 00:06:43
    per quarter
  • 00:06:44
    fail and it's okay everybody's
  • 00:06:47
    everybody's done that and I also like to
  • 00:06:49
    tell the story that some of the best
  • 00:06:50
    features of dual
  • 00:06:51
    indle uh have been things that I've been
  • 00:06:54
    vly against at first most notably the
  • 00:06:57
    the streak I think was one that you said
  • 00:06:59
    you originally pushed back on is that
  • 00:07:01
    right no the streak actually was not the
  • 00:07:04
    streak I was you know you don't really
  • 00:07:06
    remember anymore but I'm pretty sure
  • 00:07:09
    came I'm pretty sure I was the one who
  • 00:07:11
    brought it up but there are there's more
  • 00:07:13
    than one person that will tell you that
  • 00:07:15
    they're pretty sure they're the one who
  • 00:07:16
    brought it up so success has many
  • 00:07:18
    fathers right or mothers like I'm not
  • 00:07:20
    the thing is I'm not even lying to
  • 00:07:21
    myself like I actually have a memory
  • 00:07:24
    remember that I remember that but there
  • 00:07:26
    are other people who claim that too so
  • 00:07:27
    and I believe them so we don't really
  • 00:07:29
    know who actually so I was not against
  • 00:07:31
    the streak but I was against our
  • 00:07:32
    leaderboards we have this league system
  • 00:07:34
    in our leaderboards I was against that
  • 00:07:36
    because we have tried leaderboards many
  • 00:07:38
    times of before and they had failed and
  • 00:07:40
    yeah this was like the fourth attempt
  • 00:07:42
    and somebody came and they said we're
  • 00:07:43
    going to do leaderboards I'm like no
  • 00:07:45
    don't do that you we've tried it we've
  • 00:07:47
    tried it many times you're just not
  • 00:07:49
    going to work but they were very um
  • 00:07:52
    committed to it and turns out it one of
  • 00:07:54
    the best things we ever done the the
  • 00:07:55
    league system at Theo how do you think
  • 00:07:58
    about like listening in to people's
  • 00:08:00
    feedback I've heard you use the example
  • 00:08:02
    before of like people thought it was too
  • 00:08:04
    bright the green was too bright and they
  • 00:08:06
    all complained about it and then you
  • 00:08:08
    said you changed it and they said okay
  • 00:08:10
    wait never mind it's not too bright even
  • 00:08:11
    though nothing actually happened how do
  • 00:08:13
    you think about like trusting the data
  • 00:08:15
    versus listening to the users and the
  • 00:08:17
    balance between that and product
  • 00:08:20
    design a couple of things to say the the
  • 00:08:23
    good news about dueling or what makes it
  • 00:08:24
    a little easier about dueling was that
  • 00:08:27
    we can be the users too there's some
  • 00:08:29
    that you designed where you're not the
  • 00:08:30
    user I mean for example for a while I
  • 00:08:33
    was you know we we we have a dualingo
  • 00:08:36
    ABC which is an app to teach you how to
  • 00:08:37
    read I'm not the user I know how to read
  • 00:08:40
    I just I'm just not the user I'm also
  • 00:08:42
    not like four years old um but for
  • 00:08:44
    dualingo I am actually the user and
  • 00:08:47
    so for that we we do take advantage of
  • 00:08:50
    that I mean everybody that is in the
  • 00:08:52
    product in a dual lle we ask them and
  • 00:08:55
    you kind of have to really be a big user
  • 00:08:56
    of it and that makes a big difference so
  • 00:08:59
    that's one thing you listen to yourself
  • 00:09:02
    um when we talk to users we have to take
  • 00:09:04
    everything they say with a grain of salt
  • 00:09:07
    um we just we have found that doesn't
  • 00:09:09
    mean don't listen to them but we have
  • 00:09:10
    found definitely do not build features
  • 00:09:13
    based on the things that you are ask I
  • 00:09:16
    mean usually there's a deeper reason for
  • 00:09:19
    why they're asking for a feature so what
  • 00:09:21
    what we're interested in is not what's
  • 00:09:23
    your idea for a feature we're not very
  • 00:09:25
    interested in that we're interested in
  • 00:09:26
    what you know where you're having where
  • 00:09:29
    you're confused or where you know where
  • 00:09:31
    you're having trouble or something like
  • 00:09:33
    that and and if you have an idea we want
  • 00:09:35
    you to tell us why you have it an idea
  • 00:09:38
    what why is that you're suggesting that
  • 00:09:39
    because that usually gets at what the
  • 00:09:41
    actual user problem is um so you know we
  • 00:09:45
    we do we do use of research we try to
  • 00:09:47
    all that but ultimately I think what
  • 00:09:50
    works best for us is we have a group of
  • 00:09:52
    people at dualle which fortunately has
  • 00:09:54
    grown over time but it's a group of
  • 00:09:57
    people that started with one or two
  • 00:09:58
    people who just because we ran so many
  • 00:10:01
    AB tests we had a really good idea of
  • 00:10:03
    what worked and what didn't and by now
  • 00:10:06
    we have a group of I don't know how many
  • 00:10:09
    call it 15 to 20 people in the company
  • 00:10:12
    who are just
  • 00:10:15
    excellently um ATT tuned to what works
  • 00:10:18
    and what doesn't with an IB cast it just
  • 00:10:21
    it's they're very good um and and I
  • 00:10:23
    think that's what what's worked the most
  • 00:10:24
    we just when you have under your belt
  • 00:10:26
    when you have hundreds of reps where
  • 00:10:28
    you're like I tried that
  • 00:10:29
    and you know did it work no it didn't
  • 00:10:31
    work I tried that and you know did it
  • 00:10:32
    work didn't work and you do that
  • 00:10:34
    hundreds of times you just get a really
  • 00:10:35
    really good nose and I think that's the
  • 00:10:38
    thing that has worked the most for us
  • 00:10:39
    having having this group of people I've
  • 00:10:41
    heard you say that when you find a
  • 00:10:42
    metric that you want to optimize you'll
  • 00:10:44
    actually form a team that focuses on the
  • 00:10:47
    optimization how does that is that a
  • 00:10:48
    cross functional team across product
  • 00:10:50
    engineering marketing like what does a
  • 00:10:52
    team look like maybe give an example of
  • 00:10:54
    a metric that you you thought about like
  • 00:10:56
    how it actually comes to be yeah I'll
  • 00:10:58
    give some examples I mean one for
  • 00:11:00
    example is
  • 00:11:02
    um a metric that we call time spent
  • 00:11:05
    learning
  • 00:11:06
    TSL a while ago a few years ago we
  • 00:11:09
    decided you know we should have people
  • 00:11:11
    spend more time in the app because the
  • 00:11:13
    more time they spend the more they learn
  • 00:11:15
    it's very simple uh so let's form a team
  • 00:11:19
    around it and what a team usually is is
  • 00:11:21
    we have um usually two team leads
  • 00:11:26
    sometimes three uh usually there's an
  • 00:11:29
    engineering team lead and then there's a
  • 00:11:31
    product manager team lead sometimes
  • 00:11:34
    there's a design team lead and sometimes
  • 00:11:36
    there's a um a learning scientist team
  • 00:11:39
    lead um but usually it's two and then we
  • 00:11:43
    put in that team we put designers we put
  • 00:11:46
    Engineers we may put one or two more
  • 00:11:48
    program product managers depending on
  • 00:11:50
    how how big the team is um and and then
  • 00:11:54
    they work and they can touch any place
  • 00:11:56
    in the app after a few months
  • 00:11:59
    they um they figure out what moves that
  • 00:12:02
    metric in the case of time spent
  • 00:12:04
    learning what moves the metric is the
  • 00:12:06
    leaderboards and so the time spent
  • 00:12:08
    learning team mainly touches the
  • 00:12:10
    leaderboards that's what they
  • 00:12:12
    do and as you think about the uh the
  • 00:12:16
    unintended consequences of setting these
  • 00:12:19
    metrics I think I've heard the example
  • 00:12:21
    of the streak uh people were just
  • 00:12:24
    signing into the app just to keep their
  • 00:12:26
    streak going maybe speak a little bit
  • 00:12:29
    about that or the Groupon example of
  • 00:12:32
    when a channel gets too saturated and
  • 00:12:36
    the balance of local optimization versus
  • 00:12:39
    long-term value creation yeah this is
  • 00:12:42
    the tyranny of metrics I mean that we
  • 00:12:44
    have to watch out for that a lot our
  • 00:12:47
    themes are so good at at optimizing
  • 00:12:52
    metrics that sometimes they lose sight
  • 00:12:56
    of what's actually important they're
  • 00:12:57
    just it's just a machine to optimize a
  • 00:12:59
    metric um and they're very good item by
  • 00:13:02
    the way the people that we hire you know
  • 00:13:04
    are usually these people that have
  • 00:13:05
    optimized their entire life I mean they
  • 00:13:07
    were probably the best student in their
  • 00:13:10
    preschool I just kind of optimized their
  • 00:13:13
    entire life so they're massively good at
  • 00:13:14
    optimizing and they'll optimize the hell
  • 00:13:16
    out of something and so we have to
  • 00:13:17
    really watch out and you know the way we
  • 00:13:19
    watch out for that is by we have this
  • 00:13:21
    process for product review where you
  • 00:13:23
    know every change of the app to the app
  • 00:13:26
    gets reviewed by a a senior group of
  • 00:13:28
    people who are kind of overseeing the
  • 00:13:31
    whole thing so they have to keep in mind
  • 00:13:34
    you know they're the ones who stop
  • 00:13:35
    spamming us and stuff like that but you
  • 00:13:38
    know when you just let teams just fully
  • 00:13:40
    optimize the metric this lead to familyy
  • 00:13:42
    behavior and you know I I this we have a
  • 00:13:44
    a story that we use inside the company a
  • 00:13:46
    lot um which is comes from group one
  • 00:13:48
    from the founder of group one who told
  • 00:13:50
    us this story um who said uh look
  • 00:13:54
    originally with group one we have this
  • 00:13:56
    rule that we could only send one email
  • 00:13:58
    per day per user because who wants to
  • 00:14:01
    receive more than one you know kind of
  • 00:14:02
    marketing email per day from a
  • 00:14:04
    company and uh at some point some
  • 00:14:08
    product manager said well can we try
  • 00:14:11
    sending two and you know the founder
  • 00:14:13
    said no we only send one and the prodog
  • 00:14:15
    manager reled this amazing argument
  • 00:14:17
    which is are you against knowledge we
  • 00:14:19
    just let's just let's just see let's
  • 00:14:22
    just see can we and of course that's a
  • 00:14:23
    very hard argument to argue against no
  • 00:14:27
    of course I'm not against so it's fine
  • 00:14:28
    you but there's no way in hell that
  • 00:14:30
    we're going to launch to like we're just
  • 00:14:32
    going to see what it does so they they
  • 00:14:34
    tried it they saw what it does it turned
  • 00:14:36
    out the matches were way too good not to
  • 00:14:39
    launch it it would have been
  • 00:14:40
    irresponsible not to launch it like my
  • 00:14:42
    God this been preached everything so
  • 00:14:44
    much and so you know of course they they
  • 00:14:48
    they launched it um and uh then you know
  • 00:14:52
    I was like well we tried two what about
  • 00:14:55
    three they tried three free was better
  • 00:14:57
    they launched it and and they had a
  • 00:14:59
    tested their way to a larger number I
  • 00:15:01
    don't remember the exact number but like
  • 00:15:02
    seven emails per day and then
  • 00:15:04
    immediately after that the whole Channel
  • 00:15:07
    died because everybody started marking
  • 00:15:10
    their emails as spam because it's too
  • 00:15:12
    much and then they just a tested their
  • 00:15:14
    weight to killing the channel and the
  • 00:15:16
    reason for that is because they didn't
  • 00:15:18
    have full information about what was
  • 00:15:19
    going on all the metries they could see
  • 00:15:21
    were good but they actually couldn't see
  • 00:15:22
    how many people are Marching their
  • 00:15:23
    emails a Spam and that's always true you
  • 00:15:25
    never have full information about what's
  • 00:15:27
    going on in the user you just have some
  • 00:15:29
    metrics and from what you can see it's
  • 00:15:32
    good but you're not seeing for example
  • 00:15:34
    uh that maybe you're you're getting
  • 00:15:36
    people to spend more time in the app but
  • 00:15:38
    you're actually frustrating them you're
  • 00:15:39
    not seeing their frustration you're just
  • 00:15:41
    seeing the time and you're like oh okay
  • 00:15:42
    well that's good and so you know we we
  • 00:15:45
    combat that by basically you know it is
  • 00:15:48
    okay at do there are so many test we
  • 00:15:50
    just don't run even if when people say
  • 00:15:53
    are you against knowledge I'm like yep I
  • 00:15:54
    actually am against knowledge I don't
  • 00:15:56
    want to know what happens when you send
  • 00:15:58
    30 no ifications in one minute I don't
  • 00:16:00
    want to know sorry and so because
  • 00:16:03
    because in general we do things and
  • 00:16:05
    that's why it's good that we're users of
  • 00:16:06
    the app it's like would you want to
  • 00:16:07
    receive seven emails per day uh that are
  • 00:16:11
    all marketing emails would you want to
  • 00:16:12
    do that no I wouldn't okay then don't do
  • 00:16:14
    that and that I think that has really
  • 00:16:16
    helped us um do the right thing here and
  • 00:16:19
    not drive our product to the ground I
  • 00:16:22
    think a lot of brothers particularly
  • 00:16:23
    games Dy themselves to the ground when
  • 00:16:27
    you know they're just like well we're
  • 00:16:28
    just going to monetize more and monetize
  • 00:16:30
    more and monetize more and it works for
  • 00:16:31
    a while until at some point it's just
  • 00:16:33
    like you know I by the way I wouldn't
  • 00:16:35
    know how it is with Facebook anymore but
  • 00:16:37
    at some point then Facebook their
  • 00:16:38
    Facebook app kind of just had so many
  • 00:16:41
    ads that I think they kind of ran it to
  • 00:16:43
    the ground too
  • 00:16:45
    um I haven't used it in a long time but
  • 00:16:47
    this is you know if they started with
  • 00:16:48
    like one ad per whatever 10 posts at
  • 00:16:51
    some point it was I don't know how many
  • 00:16:53
    ads you know do you set constraints on I
  • 00:16:56
    mean push notifications and emails is a
  • 00:16:58
    very obvious one but are there
  • 00:17:00
    constraints about things that you won't
  • 00:17:03
    change within the app and those are
  • 00:17:05
    non-starters or is is push notifications
  • 00:17:09
    really a uh a oneofone thing because it
  • 00:17:12
    can actually be marked as spam no no
  • 00:17:15
    it's not just um we have a lot of
  • 00:17:18
    channels in the app that our product
  • 00:17:21
    managers get very good at so for example
  • 00:17:23
    when you finish a lesson on dualingo
  • 00:17:25
    there is a sequence of screens that we
  • 00:17:28
    call session end so there course should
  • 00:17:31
    be one that says you finished and here
  • 00:17:33
    are your points but we start adding
  • 00:17:35
    other ones for example you increase your
  • 00:17:37
    streak good you increase your streak
  • 00:17:39
    that's another screen but we have other
  • 00:17:40
    ones too that are just like hey uh by
  • 00:17:43
    the way do you want to subscribe to uh
  • 00:17:44
    super dualle or we add another one that
  • 00:17:47
    oh by the way have you tried our podcast
  • 00:17:49
    it's basically an ad we just have
  • 00:17:52
    decided to start adding things in there
  • 00:17:54
    and it it works every time you add
  • 00:17:55
    another session on screen it does what
  • 00:17:57
    we wanted h but you can't have 30
  • 00:18:00
    session in screens after you finish one
  • 00:18:02
    lesson right so this is just like at
  • 00:18:04
    some point if you finish one lesson you
  • 00:18:06
    have 30 screens that you have to click
  • 00:18:07
    through the channel dies I mean it just
  • 00:18:10
    it just makes the whole thing so for
  • 00:18:11
    example we have limits you know we just
  • 00:18:14
    cannot have more than a certain number
  • 00:18:16
    of screens at the end of a lesson uh and
  • 00:18:19
    and then there's a whole pretty
  • 00:18:21
    sophisticated system that tries to it's
  • 00:18:22
    it's kind of like an auction it's not
  • 00:18:24
    exactly an auction but there's there's a
  • 00:18:26
    pretty sophisticated system that tries
  • 00:18:27
    to figure out okay can only show three
  • 00:18:29
    which three should I show um that's the
  • 00:18:32
    type of thing that we done and there's a
  • 00:18:33
    lot of channels like that in the app you
  • 00:18:35
    know I'm is red dots turns out if you
  • 00:18:37
    put a red dot somewhere in the screen
  • 00:18:39
    people are going to tap on it well after
  • 00:18:41
    a while when we when that was discovered
  • 00:18:43
    by a uh enterprising product managers
  • 00:18:46
    the app had like 30 red dots all over
  • 00:18:48
    the place and then the channel dies
  • 00:18:50
    because if there are 30 red dots you
  • 00:18:52
    don't click on any of them there's just
  • 00:18:53
    30 red dots um so that that's the type
  • 00:18:56
    of thing that that you know we use the
  • 00:18:58
    group on story unlock for and then how
  • 00:19:00
    do those things roll up to a decision
  • 00:19:02
    like is it does It ultimately land on
  • 00:19:04
    your desk or someone else's that like
  • 00:19:06
    hey we might be uh Pennywise pound
  • 00:19:09
    foolish on uh on these types of tests
  • 00:19:12
    and and this is a little hacky or
  • 00:19:14
    gimmicky to approach it this way every
  • 00:19:17
    Tuesday and Thursday for two hours we do
  • 00:19:20
    this thing called product review which
  • 00:19:21
    is I am there there's usually three
  • 00:19:24
    reviewers there's me there's somebody
  • 00:19:26
    from product manager and somebody from
  • 00:19:27
    the
  • 00:19:29
    and every change to the app gets 15
  • 00:19:32
    minutes locked and we decide whether
  • 00:19:34
    it's okay or not and after product
  • 00:19:36
    review uh we spend uh what we call
  • 00:19:39
    debrief for that day and we just go
  • 00:19:42
    through everything that we approv or
  • 00:19:44
    didn't approve and we make decisions
  • 00:19:45
    about like hey um this seems like a like
  • 00:19:49
    a more General thing let's let's add a
  • 00:19:51
    we We call we we internally have been
  • 00:19:53
    calling them FAS let's had like don't do
  • 00:19:56
    that anymore so that's the type of stuff
  • 00:19:58
    there's a lot of feedback in in this
  • 00:20:00
    where we we try to make ourselves a
  • 00:20:02
    better product team that way so we spend
  • 00:20:05
    yeah I mean every Tuesday and Thursday
  • 00:20:08
    multiple hours kind of
  • 00:20:10
    discussing what is okay and what is not
  • 00:20:12
    okay in terms of um features throughout
  • 00:20:15
    if you were uh an entrepreneur or uh an
  • 00:20:17
    executive at a company and were trying
  • 00:20:20
    to copy or mimic elements of just how
  • 00:20:24
    iterative of a culture you all have
  • 00:20:26
    built I sort of think like consumer apps
  • 00:20:28
    maybe gaming is the furthest extreme of
  • 00:20:31
    just what a knif edge it's on in which
  • 00:20:34
    you need to iterate and do as much as
  • 00:20:36
    you can to keep users and then
  • 00:20:38
    enterprise software is probably on the
  • 00:20:40
    the far other side once three years ago
  • 00:20:44
    yeah yeah exactly exactly if you were
  • 00:20:46
    trying to inject some more of this DNA
  • 00:20:49
    uh into a company is there any tactics
  • 00:20:53
    or anything that uh outside of what
  • 00:20:55
    we've spoken about here or philosophies
  • 00:20:57
    that you would as to a Founder that's
  • 00:20:59
    thinking about
  • 00:21:01
    this you know one thing you did say
  • 00:21:04
    right is I don't believe that this
  • 00:21:06
    iterative approach is what's best for
  • 00:21:08
    every single
  • 00:21:11
    product it happens that with
  • 00:21:13
    Duolingo a lot of things came together
  • 00:21:16
    that made it so that it works really
  • 00:21:18
    well so for example we don't have to pay
  • 00:21:21
    licensing fees for stuff we don't
  • 00:21:23
    license music that doesn't allow us to
  • 00:21:26
    play this song and that country or
  • 00:21:28
    whatever we don't have to worry about
  • 00:21:30
    that so we don't have to worry we don't
  • 00:21:31
    have to worry about that type of stuff
  • 00:21:34
    um we don't we don't have we also don't
  • 00:21:37
    run something where it's like life or
  • 00:21:40
    death where if something changes
  • 00:21:41
    somebody's going to you know they're
  • 00:21:43
    going to lose their job or whatever we
  • 00:21:45
    can easily just change the app at any
  • 00:21:46
    point in time um so there's just a lot
  • 00:21:48
    of things that that came together so
  • 00:21:51
    that it makes it so that we really can
  • 00:21:54
    just continually improve the product and
  • 00:21:57
    with the other thing is that we can
  • 00:21:58
    deliver the product you know we can
  • 00:22:00
    change the app often I mean we we change
  • 00:22:02
    the app once a week There's a new
  • 00:22:03
    version of the app once a week so we can
  • 00:22:04
    change the app often there's a lot of
  • 00:22:06
    this that that makes it so that we
  • 00:22:08
    really can
  • 00:22:09
    change uh very fast and and can iterate
  • 00:22:12
    and because we can then we do it um and
  • 00:22:16
    and if what's what if that's what you
  • 00:22:18
    want to do um I would say the most
  • 00:22:21
    important things are make it so
  • 00:22:24
    that uh there's no shame in failing uh
  • 00:22:27
    for the AB tests and that that's very
  • 00:22:30
    important make it so that you
  • 00:22:33
    can um iterate pretty quickly
  • 00:22:37
    so I mean that's those to me are
  • 00:22:41
    probably the biggest things now shifting
  • 00:22:43
    gears a little bit to marketing I'm not
  • 00:22:44
    sure I would have guessed uh that a
  • 00:22:47
    professor at Carnegie melon starting a
  • 00:22:50
    company would be the the one especially
  • 00:22:54
    computer science background right it
  • 00:22:55
    would be the one to build a world class
  • 00:22:57
    marketing organization and brand and
  • 00:23:00
    it's it was not intuitive uh to to me um
  • 00:23:05
    can we talk a little bit about how you
  • 00:23:06
    think about marketing and where that
  • 00:23:08
    fits into do a lingo strategy including
  • 00:23:13
    maybe brand and paid user acquisition
  • 00:23:16
    influencer organic all that stuff yeah
  • 00:23:19
    marketing is has had a long history of
  • 00:23:21
    dueling I think
  • 00:23:23
    um you know the first several years
  • 00:23:29
    we had a mar marketing budget of zero so
  • 00:23:33
    can't spend any money just
  • 00:23:36
    can't uh the reason for that was because
  • 00:23:38
    we were making no money and I thought it
  • 00:23:39
    was silly to spend money acquiring uses
  • 00:23:42
    and by the way you know when we were
  • 00:23:43
    launching everybody was that's kind of
  • 00:23:45
    when Performance Marketing was getting
  • 00:23:47
    like you know really growing you're like
  • 00:23:49
    oh my God you can just go to Google and
  • 00:23:51
    pay them five bucks and they give you a
  • 00:23:52
    user it's amazing just do that um I
  • 00:23:55
    thought it was silly to do that you know
  • 00:23:57
    pay Google five bucks or however many
  • 00:23:59
    dollars for a user when we were going to
  • 00:24:00
    make no money from a user so that just
  • 00:24:02
    seemed like throwing money away so this
  • 00:24:04
    is why we never got under the
  • 00:24:05
    Performance Marketing boat uh we just
  • 00:24:07
    didn't but what we did you know at first
  • 00:24:10
    marketing you for us meant
  • 00:24:13
    PR and it actually when you were small
  • 00:24:17
    first when you were small and be 10
  • 00:24:20
    years ago PR was actually quite a strong
  • 00:24:23
    Channel I think it's a much less strong
  • 00:24:25
    Channel now because you know it's kind
  • 00:24:27
    of social I'm taking over a lot of
  • 00:24:30
    generally news outlets don't have as
  • 00:24:32
    much traffic as they used to but that
  • 00:24:35
    was quite a big channel for us um and so
  • 00:24:38
    we had a good story for that and that
  • 00:24:40
    helped the other thing that helped us is
  • 00:24:42
    that we had this Green Owl that was kind
  • 00:24:44
    of weird um and I think people started
  • 00:24:46
    taking to it and so you would see like
  • 00:24:49
    posts on Reddit or posts that would go
  • 00:24:51
    viral but it would be you know kind of
  • 00:24:53
    once every few
  • 00:24:54
    months post on Twitter or post on Reddit
  • 00:24:57
    or one of these would go viral and it
  • 00:24:59
    had to do something with owl or
  • 00:25:00
    something like that
  • 00:25:03
    um we became significantly more
  • 00:25:07
    professional with marketing uh when we
  • 00:25:10
    hired um our first CMO who was Cammy
  • 00:25:15
    dway um Cammy had been the CMO for um
  • 00:25:18
    you know very fancy company she was in
  • 00:25:20
    the Heyday of Yahoo she was the CMO for
  • 00:25:22
    Yahoo then she was the head of all
  • 00:25:24
    marketing at Nintendo which is just
  • 00:25:27
    crazy a crazy marketing budget and then
  • 00:25:29
    she came to us and then you know I said
  • 00:25:33
    well we have no marketing budget and
  • 00:25:35
    then she said I'm see I can't operate
  • 00:25:37
    that way you got to give me something
  • 00:25:39
    and she convinced me and I don't know I
  • 00:25:41
    don't remember anymore the first year
  • 00:25:43
    she may have convinced me for something
  • 00:25:44
    like $30 million of marketing which to
  • 00:25:46
    me seemed like a my God what are you
  • 00:25:49
    doing and for her it must have been the
  • 00:25:50
    smallest marketing budget she ever had
  • 00:25:52
    in her entire life um but she was really
  • 00:25:55
    quite a trooper for it because she was
  • 00:25:57
    like okay well I'll operate with this
  • 00:25:59
    and she started trying a lot of stuff
  • 00:26:02
    and she hired a really good team um and
  • 00:26:05
    she started trying a lot of stuff the
  • 00:26:06
    majority of it didn't work we tried
  • 00:26:08
    stuff like oh you know what I'm going to
  • 00:26:10
    try a radio ad for Latinos in the US who
  • 00:26:13
    want to learn English stuff like
  • 00:26:16
    that kind of a lot of it wasn't working
  • 00:26:19
    but what we did start
  • 00:26:21
    noticing was that more and more there
  • 00:26:24
    were uh these memes kept popping up
  • 00:26:28
    relating to our owl and we weren't doing
  • 00:26:30
    much for that we were just you know we
  • 00:26:33
    were trying our own marketing stuff
  • 00:26:34
    whatever it is we were trying it's not
  • 00:26:36
    that it was working zero but it was not
  • 00:26:37
    home
  • 00:26:38
    runs but more and more we started
  • 00:26:40
    noticing that like oh my God there's
  • 00:26:42
    another another mean of some person uh
  • 00:26:47
    that decided to say that they're scared
  • 00:26:49
    of the dueling gu because it's going to
  • 00:26:51
    kidnap their
  • 00:26:52
    family and that's when you know Cammy
  • 00:26:55
    and a few people in the team and me and
  • 00:26:57
    all of us decided you know what why
  • 00:27:00
    don't we lean in on
  • 00:27:01
    this that works like a lot of the stuff
  • 00:27:05
    we working like we're doing yeah know
  • 00:27:07
    it's it's half ass working it's not not
  • 00:27:10
    working like that so we started leaning
  • 00:27:13
    in on it and we had a group of people
  • 00:27:15
    that she had hired that were just really
  • 00:27:17
    creative and really had their finger on
  • 00:27:18
    the pulse and then we started you know I
  • 00:27:21
    remember we did one video which was an
  • 00:27:23
    April Fool's campaign where we hired um
  • 00:27:27
    the idea for theol campaign was that we
  • 00:27:29
    were going to make a video of one of
  • 00:27:31
    these really cheap looking lawyers
  • 00:27:33
    lawyer videos where they're like you
  • 00:27:35
    know I will get you millions of dollars
  • 00:27:38
    uh and and we we s we thought okay let's
  • 00:27:41
    make a video just like that like the
  • 00:27:42
    crappy video saying uh for a lawyer that
  • 00:27:46
    is looking for people who had been
  • 00:27:47
    kidnapped by The dualingo
  • 00:27:50
    Owl and you there was so many good
  • 00:27:53
    things by the way a really interesting
  • 00:27:54
    thing is we actually went to try to get
  • 00:27:56
    one of the companies that make these
  • 00:27:58
    crappy videos and we talk to them and
  • 00:27:59
    we're like hey but we want it like like
  • 00:28:01
    crappy like the videos you make and
  • 00:28:02
    they're like what do you mean crappy
  • 00:28:04
    like we make excellent work okay and
  • 00:28:07
    they kind of didn't and in the end we
  • 00:28:08
    didn't hire any of them we actually
  • 00:28:09
    hired a very very fancy company and told
  • 00:28:13
    them to make it crappy and they actually
  • 00:28:14
    understood how to do that and it was
  • 00:28:16
    very good as opposed to
  • 00:28:17
    telling just do your normal thing just
  • 00:28:20
    yeah make that that wasn't working so
  • 00:28:22
    well we ended up hiring a very fancy
  • 00:28:24
    company um to to to make a very crappy
  • 00:28:27
    looking video what year is this that you
  • 00:28:29
    guys are doing 2019 maybe that was 2019
  • 00:28:32
    I think so there was some precedent like
  • 00:28:34
    I sort of think of um Dollar Shave Club
  • 00:28:38
    in their video back in whatever
  • 00:28:41
    20134 as kind of the first satirical
  • 00:28:44
    brandish video out there so you guys had
  • 00:28:47
    that was amazing precedent they did a
  • 00:28:49
    great job with it we we did but we
  • 00:28:50
    weren't thinking about that we I do know
  • 00:28:53
    about that video and everything we just
  • 00:28:54
    weren we were just like look this is
  • 00:28:55
    what this is what people are saying
  • 00:28:57
    anyways so we put out that video and it
  • 00:28:59
    went really quite viral and the video by
  • 00:29:03
    the way to be clear I mean it's a very
  • 00:29:06
    uh inside joke of okay so so the push
  • 00:29:10
    notifications you have which I want to
  • 00:29:12
    talk about in a second people uh made
  • 00:29:15
    memes that they are aggressive uh in
  • 00:29:18
    nature or passive aggressive in nature
  • 00:29:21
    and so that the owl's going to come find
  • 00:29:24
    you and then you made a derivative joke
  • 00:29:27
    about that which is if you're a lawyer
  • 00:29:30
    uh you're a lawyer looking for the
  • 00:29:32
    people that the owl actually stole so
  • 00:29:35
    it's like a very inside joke right
  • 00:29:38
    multiple we made multiple derivative
  • 00:29:40
    jokes for it I mean we made another one
  • 00:29:41
    where we also um as a joke product we
  • 00:29:45
    said you can you can actually pay us to
  • 00:29:46
    send the owl to like stand in front of
  • 00:29:49
    you until you do your lesson and stuff
  • 00:29:51
    like that um so yeah we lean in on that
  • 00:29:54
    and that started working really well and
  • 00:29:57
    and and that's by the way working really
  • 00:30:00
    well as a percentage of how did you even
  • 00:30:01
    think about the you guys are so data
  • 00:30:03
    driven how did you think about the ROI
  • 00:30:06
    or the could you just see have much we
  • 00:30:08
    didn't have much Roi at first all we
  • 00:30:10
    could tell was that the videos or the
  • 00:30:12
    things we were posting were being
  • 00:30:13
    watched a lot that's the only thing we
  • 00:30:16
    could tell eventually we ended up
  • 00:30:18
    getting data because um one of the
  • 00:30:20
    marketing people decided that they were
  • 00:30:22
    going to um that they were going to do a
  • 00:30:24
    a you know we added a screen at the
  • 00:30:26
    beginning when sign up to do a lingo how
  • 00:30:28
    did you hear about us and we did a lot
  • 00:30:30
    of um number crunching there and we
  • 00:30:33
    figured out there's really a very high
  • 00:30:34
    correlation people actually answer
  • 00:30:36
    pretty truthfully there it's not 100%
  • 00:30:38
    but it's pretty truthfully and we're
  • 00:30:39
    able we use that data and we're now able
  • 00:30:41
    to really figure out where things come
  • 00:30:42
    from and that really works um but at
  • 00:30:45
    first all we knew is that a lot of
  • 00:30:47
    people were watching our stuff um and we
  • 00:30:50
    weren't paying for it which which was
  • 00:30:52
    great I mean people were watching that
  • 00:30:54
    stuff and we weren't paying for it how
  • 00:30:55
    did you I mean this is something
  • 00:30:57
    innately a Founder can do I guess but
  • 00:31:00
    you're you're taking a chance by making
  • 00:31:02
    your brand and the icon of the owl uh
  • 00:31:08
    into a meme even more than just the
  • 00:31:11
    internet's already doing right and so
  • 00:31:14
    how did you think about like clearly you
  • 00:31:16
    must have a good sense of humor to find
  • 00:31:18
    this funny yourself but there's a risk
  • 00:31:20
    associated with that right you're a
  • 00:31:22
    learning app you don't want to be taken
  • 00:31:24
    as a joke necessarily we discussed it
  • 00:31:28
    there was a risk and we discussed it but
  • 00:31:30
    in the end you know it there was a risk
  • 00:31:33
    but it
  • 00:31:34
    wasn't it didn't seem like a existential
  • 00:31:37
    risk at first it's like what just
  • 00:31:38
    publish that video what's the problem
  • 00:31:41
    and you know we kept working and we kept
  • 00:31:43
    doing it more and more I think that was
  • 00:31:45
    we kind of boiled the Frog into it I
  • 00:31:46
    mean yeah it just didn't seem like that
  • 00:31:48
    big of a risk at first um and and I did
  • 00:31:51
    learn something pretty big here um we
  • 00:31:55
    hire um a a a person who uh was straight
  • 00:32:01
    out of college and Cammy hired uh this
  • 00:32:04
    person a woman named zarya who's amazing
  • 00:32:06
    um who um you know I remember I remember
  • 00:32:10
    her hiring meeting um you know Cammy was
  • 00:32:12
    trying to explain that we should hire
  • 00:32:13
    zarya and you know I saw her wrest me I
  • 00:32:15
    the whole thing and I'm
  • 00:32:17
    like why are we hiring this person I
  • 00:32:19
    don't understand Cam's like trust me
  • 00:32:21
    look you have to hire some young people
  • 00:32:23
    because they understand young people you
  • 00:32:25
    don't anymore Lise and um just trust me
  • 00:32:29
    on this one I'm like okay C me and zarya
  • 00:32:33
    came on and pretty quickly she said you
  • 00:32:35
    know what you know where young people
  • 00:32:37
    are on Tik Tok I'm like my God Tik Tok
  • 00:32:39
    [ __ ] people dancing I don't
  • 00:32:42
    understand
  • 00:32:43
    it and uh it's like look let let me make
  • 00:32:46
    this video they made a video they showed
  • 00:32:48
    it to me I'm like I I don't even
  • 00:32:50
    understand what this
  • 00:32:51
    is I have no idea what this is in fact
  • 00:32:54
    this not going to work because nobody's
  • 00:32:56
    going to understand this this what I
  • 00:32:58
    told but they said well will you let us
  • 00:33:01
    and sure carry on I don't think it's
  • 00:33:04
    going to work anyways they posted a
  • 00:33:07
    couple of videos and they went crazy
  • 00:33:09
    viral like I don't know how many any you
  • 00:33:12
    lost track 20 million views some crazy
  • 00:33:14
    viral
  • 00:33:15
    video and I still didn't understand why
  • 00:33:18
    why it was even funny I just did not get
  • 00:33:20
    it and one of the things that I
  • 00:33:21
    understood which I a major shift you
  • 00:33:23
    know most of my time at dualingo and
  • 00:33:26
    even before dualingo SP designing
  • 00:33:27
    products when you design a product it is
  • 00:33:30
    important that every single one of your
  • 00:33:31
    users understand what you're doing it
  • 00:33:34
    needs to be understandable and I was
  • 00:33:35
    applying that lens to our marketing I
  • 00:33:37
    thought well everything we put out has
  • 00:33:39
    to be at least understandable by the
  • 00:33:42
    world this thing you put out makes no
  • 00:33:44
    sense to me how in fact most people
  • 00:33:46
    won't understand it and what I learned
  • 00:33:49
    that with marketing that's okay
  • 00:33:50
    especially with things like social media
  • 00:33:53
    that Target it to the people you know
  • 00:33:55
    somehow Tik Tok targeted it to the
  • 00:33:56
    people that understood it and that was
  • 00:33:59
    fine and so you know that that worked
  • 00:34:02
    out and we started really leaning in on
  • 00:34:05
    Tik Tok quite a bit and then we started
  • 00:34:07
    getting worried that we were too reliant
  • 00:34:09
    on Tik Tok at least for our marketing um
  • 00:34:13
    uh so we started uh figuring out how to
  • 00:34:16
    how to get better at that um we started
  • 00:34:19
    this we tried the same thing on YouTube
  • 00:34:20
    shorts and it works on YouTube as well
  • 00:34:23
    so by now our Tik Tok and our YouTube
  • 00:34:26
    provide about the same number of of new
  • 00:34:28
    users to doal lingo which is awesome the
  • 00:34:30
    other thing is we started replicating
  • 00:34:32
    the same formula in multiple countries
  • 00:34:34
    so now we have accounts Tik Tok accounts
  • 00:34:36
    and and YouTube accounts on you know in
  • 00:34:39
    Germany in Japan in uh Brazil in Spanish
  • 00:34:44
    speaking world and in in a bunch of
  • 00:34:47
    different countries in Vietnam and
  • 00:34:48
    they've all succeeded uh and they all
  • 00:34:51
    use the same
  • 00:34:52
    formula and and so we're you know it's
  • 00:34:56
    it's worked out really well but but the
  • 00:34:57
    the formula you
  • 00:34:59
    know there's a number of things but I
  • 00:35:01
    think the biggest thing about the
  • 00:35:02
    formula is if you look at our Tik Tok
  • 00:35:05
    stuff and our our marketing stuff it
  • 00:35:07
    does it's not selling to you it's just
  • 00:35:10
    funny crap and like there's no button
  • 00:35:12
    that says download now or buy now or
  • 00:35:15
    anything like that it's just we now have
  • 00:35:17
    built the personality for the owl and um
  • 00:35:21
    people love it and I think that that
  • 00:35:23
    really gets a lot of people you know
  • 00:35:25
    it's basically free marketing we don't
  • 00:35:27
    we don't pay for that it's not true that
  • 00:35:28
    we don't pay for any marketing but that
  • 00:35:31
    that we don't pay for was there a
  • 00:35:32
    session in which you needed to come up
  • 00:35:34
    with what the personality of of the owl
  • 00:35:38
    was it's just it's iterative it's it's
  • 00:35:41
    evolved and um it really didn't mean I
  • 00:35:44
    would say the first glimmer of a
  • 00:35:46
    personality came this was many years ago
  • 00:35:49
    from the passive aggressive notification
  • 00:35:51
    and this one was one I added and it was
  • 00:35:53
    it was as follows and but I didn't think
  • 00:35:55
    it was going to be this this big right I
  • 00:35:56
    think it was um it was because you know
  • 00:36:00
    we've
  • 00:36:01
    always stopped sending notifications we
  • 00:36:04
    stopped using dual lingo so we you know
  • 00:36:05
    if you don't use dualingo we stop after
  • 00:36:07
    like it used to be about five days we're
  • 00:36:09
    like okay after five days you know
  • 00:36:11
    whatever you've given up we're not going
  • 00:36:12
    to spam you forever and it occurred to
  • 00:36:14
    me you know what if we're going to stop
  • 00:36:16
    we may as well tell people that we're
  • 00:36:18
    stopping why not so the last
  • 00:36:20
    notification on the fifth day that we
  • 00:36:21
    that we started sending was uh these
  • 00:36:24
    reminders don't seem to be working we're
  • 00:36:26
    going to stop sending them for now or
  • 00:36:27
    something to that effect and it turned
  • 00:36:29
    out this we didn't expect it turned out
  • 00:36:31
    that we really got people to come back
  • 00:36:33
    because they you know it's this passive
  • 00:36:35
    aggressive thing that they you know they
  • 00:36:37
    felt bad that Dingo was giving up on
  • 00:36:39
    them and that is that is the first
  • 00:36:41
    glimmer of a personality that was just
  • 00:36:43
    like oh this owl really wants me to be
  • 00:36:45
    there and he's GNA be passive aggressive
  • 00:36:47
    and that's I think that's kind of what
  • 00:36:48
    gave rise to a lot of the memes but I
  • 00:36:50
    mean you know we didn't and and so the
  • 00:36:53
    personality of the owl has really
  • 00:36:55
    evolved
  • 00:36:57
    over time and it's been kind of
  • 00:37:00
    co-created by a lot of different
  • 00:37:02
    dualingo employees but also by the
  • 00:37:04
    community because they've made a lot of
  • 00:37:06
    this stuff and
  • 00:37:07
    so there like there hasn't been now what
  • 00:37:10
    we have had meetings about especially
  • 00:37:12
    for some of the videos you know with a
  • 00:37:13
    lot of the videos we you know we are
  • 00:37:16
    learning brand after all and so we don't
  • 00:37:17
    want to do things that are lewd but with
  • 00:37:19
    a lot of the videos we are um we're
  • 00:37:23
    unhinged and so um we do have uh a
  • 00:37:28
    system in place for
  • 00:37:30
    not posting things that we're going to
  • 00:37:32
    regret later the system has failed a few
  • 00:37:35
    times um but we do have a whole approval
  • 00:37:38
    mechanism uh for not posting really
  • 00:37:40
    crazy crap what what do you do when you
  • 00:37:45
    uh when it feels it's crossed the line
  • 00:37:48
    and uh and something just offended
  • 00:37:51
    people in a way that you didn't expect
  • 00:37:53
    you just take it down and and move on we
  • 00:37:56
    usually take I mean we haven't that
  • 00:37:57
    hasn't happened too many times um we do
  • 00:38:00
    take it down by the way I don't know
  • 00:38:02
    good or bad usually the most offended
  • 00:38:04
    are the buing with
  • 00:38:06
    employees um so we the times we've taken
  • 00:38:09
    stuff down is more often than not is
  • 00:38:10
    because somebody in the company or a
  • 00:38:13
    group of people in the company are like
  • 00:38:14
    how could you do that um and so that's
  • 00:38:17
    that's what has happened but you know
  • 00:38:20
    but there was only one time that where
  • 00:38:23
    there was a major major backlash kind of
  • 00:38:25
    publicly which was when we
  • 00:38:28
    commented on um
  • 00:38:32
    the you know I couldn't have told you
  • 00:38:35
    this was going to be that bad there was
  • 00:38:37
    a our account our Tik Tok account left a
  • 00:38:39
    comment on a video about the Johnny Depp
  • 00:38:45
    Amber her thing and the comment said
  • 00:38:48
    something to the effect of like you guys
  • 00:38:50
    think Amber Heard is on Tik Tok or
  • 00:38:53
    something like I think that was the
  • 00:38:54
    comment and there was a huge huge
  • 00:38:56
    backlash because somebody posted saying
  • 00:38:59
    that dualingo does not care about
  • 00:39:03
    um uh like spousal violence or something
  • 00:39:08
    like that and there was a huge backlash
  • 00:39:10
    to the point where our um social media
  • 00:39:12
    team had to um make their their personal
  • 00:39:16
    accounts private because they were
  • 00:39:17
    getting death threats wow um so that's
  • 00:39:20
    when we learned to stay away from you
  • 00:39:23
    know there's certain things we just stay
  • 00:39:25
    away from even even if our comment even
  • 00:39:26
    if our comment is you know I don't think
  • 00:39:29
    that comment was
  • 00:39:30
    particularly incendiary but you know
  • 00:39:33
    there are some certain things we just
  • 00:39:34
    stay away from we had one one time we
  • 00:39:36
    have a a will joke on Tik Tok and
  • 00:39:39
    Instagram reals and YouTube shorts and
  • 00:39:42
    we had one one time that was um it was
  • 00:39:45
    when the market had corrected and uh it
  • 00:39:48
    was VC's six months ago versus VC's
  • 00:39:52
    today and the joke was VC's six months
  • 00:39:55
    ago were spend as much as you want like
  • 00:40:00
    are you are you really burning as much
  • 00:40:02
    cash as you can we can give you more
  • 00:40:04
    money blah blah blah then VC's today
  • 00:40:08
    were are you sure we need a marketing
  • 00:40:11
    team is that something and the joke was
  • 00:40:14
    that the VCS were flippant in their
  • 00:40:16
    advice right uh people took it as we
  • 00:40:20
    were making a joke about layoffs of a
  • 00:40:22
    marketing department right and it was
  • 00:40:26
    the joke was about us and VCS but I
  • 00:40:30
    understood where they were coming from
  • 00:40:32
    so we got a big that was a big outrage
  • 00:40:34
    and you know uh it was something that we
  • 00:40:37
    ended up I I think we apologized or
  • 00:40:39
    something but you just never if you're
  • 00:40:41
    going to take chances at these lines
  • 00:40:42
    inevitably you're going to touch one and
  • 00:40:44
    you're not sure I stand by that was just
  • 00:40:46
    a good joke and we were making fun of
  • 00:40:48
    EC's and not bounders it's impossible to
  • 00:40:51
    tell it's impossible to tell what
  • 00:40:53
    somebody's going to you know heard
  • 00:40:56
    mentality takes it you know they kind of
  • 00:40:58
    go against you um we haven't had a that
  • 00:41:01
    was the last this was maybe like a year
  • 00:41:03
    and a half ago or whatever we haven't
  • 00:41:04
    had a big problem since then um and and
  • 00:41:07
    I think it's important by the way if
  • 00:41:08
    you're running a marketing team like
  • 00:41:09
    ours which I think is is a major home
  • 00:41:12
    run machine um to be okay with it so you
  • 00:41:15
    know what I did when that happened and
  • 00:41:17
    they they thought I was going to fire a
  • 00:41:18
    few of them and I actually walked to the
  • 00:41:19
    desk of in fact the person that posted
  • 00:41:21
    and I said it's completely okay to stand
  • 00:41:24
    out is the risk being polarizing in some
  • 00:41:27
    way and so
  • 00:41:29
    inevitably there's going to be some
  • 00:41:31
    people that don't find it funny and
  • 00:41:34
    inevitably there'll be some percentage
  • 00:41:35
    of people that are offended and it's
  • 00:41:38
    hard to say where it's going to come
  • 00:41:39
    from so no that's that's great that you
  • 00:41:41
    went over it and said that that's yeah
  • 00:41:43
    the the account the checks and balances
  • 00:41:45
    are important and trying to get enough
  • 00:41:46
    diverse eyes on the things so that you
  • 00:41:49
    could understand where the totality of
  • 00:41:51
    your audience maybe is going to react to
  • 00:41:53
    it but it it's the cost of doing
  • 00:41:55
    business in some ways that inevitably
  • 00:41:57
    you're going to get hit with some of
  • 00:41:58
    these things and yeah we're going to
  • 00:41:59
    continue I mean and and by the way we've
  • 00:42:01
    had a number of things that we've had to
  • 00:42:03
    pull I mean the most recent one we had a
  • 00:42:05
    hilarious video that the marketing team
  • 00:42:06
    just made it was
  • 00:42:08
    hilarious um that we were going to post
  • 00:42:12
    close to Halloween we had gotten we had
  • 00:42:14
    gotten a famous person like in there it
  • 00:42:18
    it was it was
  • 00:42:19
    great but it had
  • 00:42:22
    kidnappings and uh the Israel Palestine
  • 00:42:26
    conflict erupted right around that time
  • 00:42:28
    and so we didn't post it we were so sad
  • 00:42:31
    that we couldn't I mean obviously there
  • 00:42:32
    was no way we should have posted that
  • 00:42:34
    but um in other times this would have
  • 00:42:36
    been an amazing video what about being
  • 00:42:38
    Scrappy on the brand marketing side I'd
  • 00:42:40
    heard a story about like you guys uh
  • 00:42:43
    just hanging around waiting for
  • 00:42:45
    billboard inventory to pop up and and
  • 00:42:48
    being able to take advantage of it can
  • 00:42:50
    you can you tell that story as well as
  • 00:42:51
    like other examples of just sort of
  • 00:42:53
    being uh Scrappy about opportunities
  • 00:42:56
    that present themselves you know because
  • 00:42:59
    historically our marketing budget has
  • 00:43:00
    been pretty small I mean it's still
  • 00:43:01
    pretty small our marketing budget now is
  • 00:43:03
    about 50 million a year which for a
  • 00:43:06
    company with a revenu this it's about
  • 00:43:09
    this about a fifth of what it should
  • 00:43:11
    be um you know we
  • 00:43:13
    don't we are very Scrappy I mean there's
  • 00:43:16
    this story with you know with the
  • 00:43:18
    Billboards we and the billboard was
  • 00:43:20
    going to be a funny billboard it was um
  • 00:43:21
    this was a billboard not necessarily so
  • 00:43:23
    much for marketing dual lingo but it was
  • 00:43:25
    for getting employees um so we just put
  • 00:43:29
    a billboard in the 101 in San Francisco
  • 00:43:32
    that said something or other like uh own
  • 00:43:36
    a home work in Tech move to Pittsburgh
  • 00:43:39
    and it just said dualingo or something
  • 00:43:41
    like that um yeah I
  • 00:43:44
    mean we only got that because we got a
  • 00:43:47
    really good deal we just hung around for
  • 00:43:49
    a long time until we got a good really
  • 00:43:50
    good deal um and but we Ed that we use
  • 00:43:53
    that same idea in a lot of ways by the
  • 00:43:55
    way that is exact
  • 00:43:56
    what we do with our Performance
  • 00:43:58
    Marketing so we do Performance Marketing
  • 00:43:59
    we don't do very much we
  • 00:44:01
    do you know I don't know on the order of
  • 00:44:05
    10 million bucks a year in Performance
  • 00:44:06
    Marketing a little more than that but
  • 00:44:07
    not much more than that and that's what
  • 00:44:08
    we do um and the what we do with
  • 00:44:13
    Performance Marketing is we we noticed
  • 00:44:15
    we were in a really good position which
  • 00:44:18
    is we don't particularly care which
  • 00:44:20
    country are users are coming from uh we
  • 00:44:23
    don't particularly care at what time we
  • 00:44:25
    acquire them we don't we don't need to
  • 00:44:26
    acquire them you know there are some
  • 00:44:28
    some companies that need to acquire them
  • 00:44:30
    you know for a specific date or for
  • 00:44:31
    something we just don't care and so one
  • 00:44:33
    of the things we do with our Performance
  • 00:44:35
    Marketing is we just have a system that
  • 00:44:36
    basically looks at the whole world and
  • 00:44:39
    so if for example right now ads are too
  • 00:44:42
    expensive in Italy we just don't do
  • 00:44:44
    anything in Italy we move all our budget
  • 00:44:46
    to France um or or or whatever if it's
  • 00:44:49
    too expensive in France we move it to
  • 00:44:51
    Vietnam or something so it just we're
  • 00:44:53
    always shifting budget a lot between
  • 00:44:55
    countries because we are we're in this
  • 00:44:57
    great position where we you know operate
  • 00:44:59
    in every single single country in the
  • 00:45:01
    world uh or have users in every single
  • 00:45:03
    country in the world and we are agnostic
  • 00:45:04
    as to where they come from um and so
  • 00:45:09
    that that's the type of stuff we do
  • 00:45:10
    we're we're we're very um very Scrappy
  • 00:45:12
    with that budget I heard when you were
  • 00:45:13
    starting to a lingo uh you had no
  • 00:45:16
    experience in teaching languages and so
  • 00:45:20
    you read a bunch of books and they all
  • 00:45:22
    contradicted each other or they they had
  • 00:45:24
    mixed messages on how to learn how to
  • 00:45:26
    teach I I don't know if that's true but
  • 00:45:29
    if so like how did you go about figuring
  • 00:45:31
    out what the core nucleus is of teaching
  • 00:45:35
    language and how to start we didn't I
  • 00:45:38
    didn't know how to teach languages I
  • 00:45:39
    mean I knew I knew Spanish because
  • 00:45:41
    that's my native language but I had
  • 00:45:43
    never taught Spanish um s my co-founder
  • 00:45:46
    knew German because that was his native
  • 00:45:48
    language and he never taught German and
  • 00:45:50
    so yeah we we read a bunch of books we
  • 00:45:53
    you know there were some commonality
  • 00:45:56
    about what all the books said how to
  • 00:45:57
    best teach a language and we kind of
  • 00:45:58
    took those unfortunately the books
  • 00:46:00
    didn't agree with each other they all
  • 00:46:01
    had their own method and they're all you
  • 00:46:03
    know um so we took the commonalities and
  • 00:46:06
    then we did what we thought made a lot
  • 00:46:08
    of sense by the way we made a bunch of
  • 00:46:09
    dumb mistakes that only computer
  • 00:46:10
    scientist would make so for example this
  • 00:46:13
    was a this was done um we thought hey we
  • 00:46:17
    this this this kind of approach to
  • 00:46:19
    deconstruct everything language is just
  • 00:46:22
    words so all we need to do is teach you
  • 00:46:24
    words and how to use them and if you
  • 00:46:26
    know all the words and how to use them
  • 00:46:27
    you would learn the language done and so
  • 00:46:30
    we're luy though so that's what we're
  • 00:46:32
    going to do and so the the first we one
  • 00:46:34
    of the first versions of dingo had just
  • 00:46:37
    we just were teaching you a list of
  • 00:46:38
    words ordered by frequency so you know
  • 00:46:41
    what the first word the most frequent
  • 00:46:42
    word is the teach you the first then the
  • 00:46:46
    second most common word whatever the
  • 00:46:47
    hell that is teach you that word Etc
  • 00:46:50
    that's a very bad way of teaching um and
  • 00:46:54
    uh you know it's also not true that
  • 00:46:57
    because you've learned enough words to
  • 00:46:59
    cover the first like 30 words or
  • 00:47:01
    whatever cover something like 20% of the
  • 00:47:02
    language if not true you know 20% of The
  • 00:47:05
    Language by knowing the first 30 words
  • 00:47:06
    that is just absolutely not true um but
  • 00:47:09
    that's kind of how we thought about it
  • 00:47:10
    so this is the type of stuff that we
  • 00:47:11
    were doing um over time we learned that
  • 00:47:16
    we could run AB tests to figure out how
  • 00:47:18
    to teach better and so that we started
  • 00:47:20
    getting better at that so we started
  • 00:47:21
    kind of figuring some stuff out
  • 00:47:23
    ourselves um and then but then we got
  • 00:47:26
    significantly more Pro um when we hired
  • 00:47:30
    our first person with a PhD in second
  • 00:47:32
    language acquisition um that was a we
  • 00:47:36
    should have done that earlier um yeah we
  • 00:47:39
    got a lot more Pro and and that that has
  • 00:47:42
    really helped and now what's interesting
  • 00:47:44
    is when we got these people with with ph
  • 00:47:46
    from second language AIS it's not like
  • 00:47:48
    they know how to teach with an app they
  • 00:47:49
    didn't because they know how to teach in
  • 00:47:51
    a classroom setting um but we knew how
  • 00:47:54
    to teach with an nap now we had our own
  • 00:47:56
    wonky ways of teaching with an app and
  • 00:47:58
    they had their own very professional
  • 00:48:01
    ways of teaching through classroom
  • 00:48:02
    settings um what has been really amazing
  • 00:48:05
    is kind of you know they've
  • 00:48:08
    learned how to take everything they know
  • 00:48:10
    and some of it applies to to an app some
  • 00:48:13
    of it doesn't they learn how to make
  • 00:48:14
    that kind of closer to the app and we
  • 00:48:17
    with the app and with AB testing and
  • 00:48:18
    everything have learned how to go more
  • 00:48:20
    closer to them and so at this point you
  • 00:48:23
    know I can I can say that compared to
  • 00:48:25
    years ago Dule teaches I know way better
  • 00:48:29
    um but we've understood that actually
  • 00:48:31
    there's just all these skills that you
  • 00:48:33
    need to learn and that they don't carry
  • 00:48:35
    to each other like for example you can
  • 00:48:36
    be amazing at reading a language and
  • 00:48:37
    have no idea how to speak
  • 00:48:39
    it and and we just didn't understand
  • 00:48:42
    that at first um so now you know we try
  • 00:48:44
    to teach you all the different skills
  • 00:48:45
    that that are required to to to learn a
  • 00:48:47
    language and you know we we understand
  • 00:48:49
    that the way to teach better is at first
  • 00:48:51
    you really need to teach things about
  • 00:48:53
    how to you know uh how to talk about
  • 00:48:55
    concretely about things and greetings
  • 00:48:57
    and stuff like that these are not the
  • 00:48:58
    most popular word or the most kind of
  • 00:49:00
    highfrequency words at all uh but they
  • 00:49:02
    are very useful to get started and so
  • 00:49:05
    that's um you we've learned that in the
  • 00:49:09
    in the early days when it was just you
  • 00:49:11
    and San uh you I I think we we talked
  • 00:49:14
    about this a little bit before we
  • 00:49:15
    started recording but neither of you
  • 00:49:17
    actually uh retained as users of of the
  • 00:49:21
    app necessarily and trying to learn new
  • 00:49:24
    languages was there was there a single
  • 00:49:27
    product unlock or an aha that you
  • 00:49:30
    realized about how to retain users or
  • 00:49:34
    teach people through it or was it just a
  • 00:49:35
    continuous process of these little tests
  • 00:49:37
    to get Inc incrementally better every
  • 00:49:40
    day so what happened with this is before
  • 00:49:42
    we launched
  • 00:49:44
    Duolingo um you know I made the first
  • 00:49:46
    version of the Spanish course and S made
  • 00:49:49
    the first version of the German course
  • 00:49:50
    and we were going to learn each other's
  • 00:49:52
    language I didn't know any German and he
  • 00:49:54
    didn't know any Spanish and we were
  • 00:49:55
    going to learn each other's language and
  • 00:49:56
    we ran into this problem that neither of
  • 00:49:59
    us wanted to do it I mean we would come
  • 00:50:00
    into the office and we would say hey did
  • 00:50:02
    you do your Spanish lesson you like no
  • 00:50:04
    man so so boring and I I thought the
  • 00:50:08
    same thing it was so boring to learn um
  • 00:50:11
    German um and that's when that's when we
  • 00:50:14
    realized that the hardest thing about
  • 00:50:15
    learning a language by yourself is stay
  • 00:50:17
    motivated and so that's when we started
  • 00:50:19
    like adding um you know adding all kinds
  • 00:50:23
    of game mechanics like progress bar at
  • 00:50:27
    the top at the end of a lesson we give
  • 00:50:29
    you points the lessons should be short
  • 00:50:31
    there all this stuff and this is before
  • 00:50:33
    we launched so we weren't a testing
  • 00:50:35
    anything but we added a bunch of stuff
  • 00:50:37
    so that at least we could swallow taking
  • 00:50:42
    a lesson I mean it's like okay well at
  • 00:50:44
    least it doesn't feel like a complete
  • 00:50:46
    chore feels yeah kind of fun and that's
  • 00:50:49
    you know we launched with that um and so
  • 00:50:52
    by the time we launched it was kind of
  • 00:50:53
    fun it wasn't as fun as it is today but
  • 00:50:55
    it was kind of of fun we had done no a
  • 00:50:56
    testing it was just at least with us we
  • 00:50:59
    had made it you know kind of engaging
  • 00:51:02
    and but but the the learning remained
  • 00:51:05
    which
  • 00:51:06
    was just um make it us engaging as
  • 00:51:11
    possible and so that that continued what
  • 00:51:14
    did you think this could be when you
  • 00:51:17
    were starting the company you had been
  • 00:51:19
    independently successful selling
  • 00:51:21
    recapture to Google and so you were able
  • 00:51:24
    to be a little bit more Mission driven
  • 00:51:26
    in your Pursuits uh I think which I my
  • 00:51:30
    understanding was that was an important
  • 00:51:31
    consideration in the early days did you
  • 00:51:32
    think that this would be a commercial
  • 00:51:35
    potential public company or was it hey
  • 00:51:37
    this is just a good service and maybe
  • 00:51:39
    it's a good business along the way I
  • 00:51:41
    never thought this would be a public
  • 00:51:42
    company um I thought you know I thought
  • 00:51:46
    we could do something that would help
  • 00:51:47
    teach people mainly English um I thought
  • 00:51:51
    that that that was going to be there I I
  • 00:51:53
    didn't know how big it was going to get
  • 00:51:56
    um when we were launching um Rosetta
  • 00:52:00
    Stone was like huge we thought at the
  • 00:52:03
    time was
  • 00:52:04
    huge um it's not even clear that we
  • 00:52:07
    thought we were going to get bigger than
  • 00:52:08
    them we just thought well we're free and
  • 00:52:10
    we're at least going to let people you
  • 00:52:13
    know learn that can't afford to pay and
  • 00:52:16
    andone was super expensive was like 500
  • 00:52:18
    bucks or something like okay well uh
  • 00:52:21
    we're going to let people learn that and
  • 00:52:24
    you know when they don't have to pay 500
  • 00:52:28
    bucks before we launched I thought if we
  • 00:52:30
    ever get to about 100,000 monthly active
  • 00:52:33
    users um that'll be success that's what
  • 00:52:37
    I thought and um yeah that's funny I
  • 00:52:39
    mean we get we get way more than that
  • 00:52:41
    new users per day now there there's a
  • 00:52:44
    great YouTube video I think it's around
  • 00:52:46
    the launch of uh of dualingo whatever uh
  • 00:52:50
    14 years ago or 12 years ago or whatever
  • 00:52:53
    it is but I believe it's on that video
  • 00:52:54
    you make make reference to this this
  • 00:52:57
    concept of the cap on human coordination
  • 00:53:01
    uh that historically had been around
  • 00:53:03
    100,000 people I think that was true of
  • 00:53:05
    the pyramids Panama Canal getting a man
  • 00:53:07
    on the moon that like that's been the
  • 00:53:10
    the when when when physical objects were
  • 00:53:13
    involved that was the most we could
  • 00:53:14
    scale to as as Humanity u c can you talk
  • 00:53:18
    a little bit about that and how you how
  • 00:53:20
    you thought that as an influence of
  • 00:53:22
    starting Duolingo so we always wanted
  • 00:53:25
    dealing with be free and it was one
  • 00:53:27
    potential way to monetize it we ended up
  • 00:53:30
    not monetizing this way but there was
  • 00:53:32
    one potential way of monetizing which
  • 00:53:33
    which was um look maybe we can get a lot
  • 00:53:37
    of people learning a language on
  • 00:53:38
    dualingo and at the same time while
  • 00:53:40
    they're learning they helping us
  • 00:53:41
    translate stuff because at the time
  • 00:53:43
    computers were just not as good today
  • 00:53:45
    they're essentially perfect but at the
  • 00:53:46
    time they weren't as good as they are
  • 00:53:48
    today translating stuff so we thought we
  • 00:53:50
    could rather than charging people uh to
  • 00:53:53
    learn a language or using ads we thought
  • 00:53:56
    we could use their work um to help like
  • 00:53:59
    translate stuff on the web and then we
  • 00:54:01
    could make money from that so for
  • 00:54:02
    example the idea was going to be
  • 00:54:03
    something like well if you write a news
  • 00:54:06
    piece in English like if you're CNN and
  • 00:54:08
    you write a news piece in English you
  • 00:54:09
    can send it to us and our users while
  • 00:54:12
    they're learning English they could help
  • 00:54:13
    translate it into Spanish because
  • 00:54:15
    they're native Spanish speakers and
  • 00:54:16
    that's part of their learning and then
  • 00:54:18
    once it gets translated by a users we
  • 00:54:19
    send it back to CNN and then we charge
  • 00:54:21
    them for the translation or something
  • 00:54:22
    like that we thought that was a
  • 00:54:24
    potentially good business model because
  • 00:54:26
    it you know no no money was involved uh
  • 00:54:30
    at least for the users turns out this is
  • 00:54:32
    a very bad business model so we ended up
  • 00:54:34
    not doing that but that was that was one
  • 00:54:36
    of the original ideas with dualing to
  • 00:54:37
    try to make money while keeping it free
  • 00:54:39
    it turns out a much better way to make
  • 00:54:41
    money while keeping it free is to use a
  • 00:54:43
    freemium model which is you can learn as
  • 00:54:45
    much as you want but you may have to see
  • 00:54:47
    some ads if you don't want to see ads
  • 00:54:49
    you can pay us turns out that works
  • 00:54:50
    really well that was not clear back then
  • 00:54:53
    that this would that this would work so
  • 00:54:54
    well yeah yeah so what year was it when
  • 00:54:57
    uh so so Leela sturdy from capg invested
  • 00:55:00
    and and told you hey you got to figure
  • 00:55:02
    out how to make some money here at some
  • 00:55:05
    point right what year was that she
  • 00:55:08
    invested in 2015 I believe we at that
  • 00:55:10
    point dualingo she invested a valuation
  • 00:55:12
    of about half a billion dollars with
  • 00:55:15
    with with roughly how many users zero
  • 00:55:19
    Revenue plus or minus zero Revenue we
  • 00:55:22
    must have had yeah I don't remember
  • 00:55:25
    anymore call
  • 00:55:27
    it I'm going to make up a number call it
  • 00:55:30
    single digit May one two million daily
  • 00:55:32
    active users one two million daily
  • 00:55:34
    actives and and if you looked at monthly
  • 00:55:35
    actives is
  • 00:55:37
    probably call it uh I don't know six
  • 00:55:41
    seven million monthly actives something
  • 00:55:43
    like that so so a rich valuation uh for
  • 00:55:47
    certainly for uh those times uh may
  • 00:55:51
    maybe yeah we were the good news is we
  • 00:55:52
    were growing a lot so we have just you
  • 00:55:54
    know we the previous year we had like
  • 00:55:56
    10x so it looks like we were drawing a
  • 00:55:58
    lot which was good and but you know she
  • 00:56:02
    said to me look um great product great
  • 00:56:07
    team um you got to make money and uh
  • 00:56:12
    yeah she she took me out to drinks after
  • 00:56:14
    after my third drink she said look at
  • 00:56:17
    you're not going to find uh she used
  • 00:56:19
    exactly this words you're not going to
  • 00:56:20
    find a bigger fool than Google like that
  • 00:56:23
    this is it like you know we invested we
  • 00:56:25
    like you this is great we believe in you
  • 00:56:27
    but you're not going to find a you know
  • 00:56:28
    you're not going to go get a valuation
  • 00:56:30
    hip from anybody else you're not going
  • 00:56:32
    to do that unless you make some money so
  • 00:56:35
    get it going and then um I got start
  • 00:56:38
    some yeah well and so now at this point
  • 00:56:41
    you have a very Mission driven company
  • 00:56:43
    right that are very uh aligned with your
  • 00:56:46
    democratizing access to to learning
  • 00:56:49
    languages and all that and so you come
  • 00:56:51
    back to them and say hey you're a little
  • 00:56:54
    hung over after a couple drinks with Lea
  • 00:56:56
    you start noodling it and you say hey we
  • 00:56:59
    we've got to figure this out I assume
  • 00:57:02
    not all of the company was uh was
  • 00:57:04
    excited to start showing ads or charging
  • 00:57:07
    subscriptions is that a fair fair
  • 00:57:08
    assumption yeah no they really were not
  • 00:57:11
    they were in fact um entirely against it
  • 00:57:14
    um most everybody that joined dualingo
  • 00:57:17
    joined because of the mission to really
  • 00:57:18
    deliver education to everyone um and so
  • 00:57:23
    you know I said to everyone we got to we
  • 00:57:24
    got to you know somehow pay for this
  • 00:57:27
    whole operation and what everybody you
  • 00:57:28
    know first question everybody said is
  • 00:57:30
    why well got to pay for your salary and
  • 00:57:34
    you've been paying our salary this whole
  • 00:57:35
    time what's the what's the problem you
  • 00:57:38
    know that's when I kind of told them
  • 00:57:39
    kind of what Lea had said you you can't
  • 00:57:41
    raise Venture Capital
  • 00:57:43
    forever um and
  • 00:57:47
    uh you know people finally were like
  • 00:57:49
    okay okay fine I this took like months
  • 00:57:52
    for people were like fine you got to
  • 00:57:54
    make money how are we by the way when
  • 00:57:56
    you have that I mean a lot of our
  • 00:57:57
    companies have dealt with different
  • 00:57:58
    versions of this in particular now
  • 00:58:00
    trying to get people back into office or
  • 00:58:03
    what whatever it is right different
  • 00:58:04
    social issues or the mission driven
  • 00:58:07
    nature of a of a company and trying to
  • 00:58:09
    Corral it in some way shape or form I
  • 00:58:12
    think as we're recording this uh there's
  • 00:58:14
    this there's this large business called
  • 00:58:16
    open AI That's maybe going through some
  • 00:58:18
    some elements of of this like how did
  • 00:58:20
    you actually go about building consensus
  • 00:58:24
    around
  • 00:58:25
    uh getting people on board with this it
  • 00:58:28
    sounds like you didn't do it by edict
  • 00:58:30
    and say Hey listen this is the way no no
  • 00:58:32
    it had to be consensus duing was
  • 00:58:34
    generally a consensus driven company it
  • 00:58:36
    had to I mean the way that works we were
  • 00:58:38
    I don't know how many employees we were
  • 00:58:39
    at that time call it 150 maybe keep or
  • 00:58:42
    take um you know there was clearly a
  • 00:58:46
    spectrum of how much people cared and
  • 00:58:48
    also spectrum of how influential they
  • 00:58:51
    were and um you know I spent time with
  • 00:58:54
    each with each individual who either
  • 00:58:57
    cared a lot or was very influential like
  • 00:58:59
    one-on-one and some of them were easy to
  • 00:59:02
    convince they were a few stragglers who
  • 00:59:03
    were just like really resisted the idea
  • 00:59:07
    um but you know it took it took a period
  • 00:59:10
    I don't know how long it was couple of
  • 00:59:11
    months
  • 00:59:13
    of you know people being you know
  • 00:59:16
    uh upset at at this and then when they
  • 00:59:20
    started seeing the mock of how they
  • 00:59:22
    became even more upset when they started
  • 00:59:23
    the first somebody saw a build of
  • 00:59:26
    dualing with ads in it people were like
  • 00:59:29
    what is this that's awful and it's
  • 00:59:33
    because yeah it's true I mean the ads
  • 00:59:34
    the truth is the Dual lingual product
  • 00:59:35
    experience is quite polished and then
  • 00:59:37
    you put an ad which is look like sorry
  • 00:59:39
    but most of these ads that you play on
  • 00:59:41
    apps look like crap yeah um so yeah it
  • 00:59:46
    took it took quite a bit to to figure
  • 00:59:49
    that out to to sorry for people to get
  • 00:59:50
    used to it um but you know the
  • 00:59:54
    explanation that really resonated most
  • 00:59:58
    with everybody is if we make money we
  • 01:00:01
    can use it to really speed up our own
  • 01:00:04
    mission and it was 100% true as soon as
  • 01:00:07
    we started making money we started
  • 01:00:08
    growing the employees a lot instead of
  • 01:00:10
    having like four separate product
  • 01:00:12
    pipelines now we have I don't know how
  • 01:00:14
    many product pipelines we have we have
  • 01:00:15
    like 50 separate you know kind of lanes
  • 01:00:19
    where people are are just improving the
  • 01:00:21
    product um and that's all because we you
  • 01:00:23
    know we make Revenue now how did you
  • 01:00:26
    think about the line between
  • 01:00:28
    subscription which my understanding is
  • 01:00:30
    that drives is a very small percentage
  • 01:00:32
    of the users but a high percentage of
  • 01:00:34
    Revenue and then advertising which is
  • 01:00:37
    the opposite when you were thinking
  • 01:00:39
    about this philosophically how did you
  • 01:00:41
    come up with where the lines of
  • 01:00:42
    demarcation were we didn't have too much
  • 01:00:44
    philosophical it was a lot AB testing
  • 01:00:46
    the only thing that we decided to do is
  • 01:00:49
    we're going to put an ad on the freeo
  • 01:00:51
    that we're going to do at the end of our
  • 01:00:53
    lesson we will put one
  • 01:00:55
    and that you know after 5 seconds is
  • 01:00:58
    skipable we're going to do that and we
  • 01:01:01
    decided we were going to do and then we
  • 01:01:03
    added an edict that said we're not going
  • 01:01:05
    to degrade the fre years experience
  • 01:01:06
    further than that that is it we're going
  • 01:01:09
    to degrade it by putting one ad at the
  • 01:01:11
    end of the lesson don't degrade it
  • 01:01:13
    more um we didn't know whether ads or
  • 01:01:17
    subscription was going to give us you
  • 01:01:20
    know that much more money in retrospect
  • 01:01:23
    I think I think subscript businesses are
  • 01:01:25
    a lot more well understood now than they
  • 01:01:27
    were at least digital subscriptions like
  • 01:01:29
    that for apps than they were a while ago
  • 01:01:31
    in retrospect this would have been
  • 01:01:32
    obvious but at the time it was not clear
  • 01:01:34
    to us that ads or subscriptions you know
  • 01:01:37
    which one was going to be the killer one
  • 01:01:39
    but after running it for about a you
  • 01:01:41
    know a few months it became clear that
  • 01:01:43
    subscription was so much of a Better
  • 01:01:45
    Business um and so yes today eight give
  • 01:01:49
    or take you know 8% of our um monthly
  • 01:01:52
    active users are paid subscribers
  • 01:01:55
    and 92% use it for free yet for a
  • 01:01:58
    revenue give or take 80% comes from the
  • 01:02:01
    subscription and you guys
  • 01:02:02
    philosophically will not charge people
  • 01:02:05
    for content right like that's uh you
  • 01:02:09
    don't want to become Rosetta Stone and
  • 01:02:10
    gatekeep content too much correct we
  • 01:02:14
    don't charge people for Content the only
  • 01:02:16
    things that are okay for us to charge
  • 01:02:18
    for are things that we have to pay
  • 01:02:20
    for so for example um we now have a here
  • 01:02:24
    and dual lingo which is dualingo Max
  • 01:02:26
    which has some features that have um
  • 01:02:29
    large language model usage we have to
  • 01:02:31
    pay for that because we don't have our
  • 01:02:32
    own large language model so we have to
  • 01:02:33
    pay for that and because we have a
  • 01:02:35
    variable cost on it we do put it behind
  • 01:02:38
    um you know you have to pay for it um
  • 01:02:41
    because otherwise that would be a major
  • 01:02:43
    money loser for us there's a rule I've
  • 01:02:45
    heard you guys have that if uh if five
  • 01:02:48
    people or less can build something as
  • 01:02:50
    well you won't charge for it is that is
  • 01:02:53
    that still true and how philosophically
  • 01:02:55
    do you think about that this is you know
  • 01:02:58
    it's one of those internally she called
  • 01:03:00
    the five Germans and a
  • 01:03:02
    dog five Germans and a dog can build it
  • 01:03:06
    uh why are why why why can we Char
  • 01:03:08
    because you know the reason we decided
  • 01:03:10
    not to do is because that you know
  • 01:03:11
    that's how that's how honestly that's
  • 01:03:14
    just how we beat Rosetta Stone I mean we
  • 01:03:16
    weren in German but we just made a thing
  • 01:03:19
    that was free and it was not made by 300
  • 01:03:22
    people it was made by very small number
  • 01:03:25
    of people and very quickly we took them
  • 01:03:27
    over in terms of usage not in terms of
  • 01:03:30
    Revenue because we're making any Revenue
  • 01:03:31
    but in terms of usage we took them over
  • 01:03:33
    very
  • 01:03:34
    quickly um and
  • 01:03:37
    so yeah we're going to get disrupted if
  • 01:03:39
    we build something and start charging
  • 01:03:41
    for it that can be built you know so
  • 01:03:43
    easily some company that has a small
  • 01:03:46
    amount of funding I mean for you know
  • 01:03:48
    five people you need five million bucks
  • 01:03:51
    of funding or less than that they can
  • 01:03:52
    build it and eat your business so that
  • 01:03:55
    that's been the rule um we're a little
  • 01:03:57
    less worried about that these days just
  • 01:04:00
    you know that was more of a worry a few
  • 01:04:01
    years ago when when our dominance was
  • 01:04:04
    not as big I mean at this point it's
  • 01:04:06
    just we have a lot of Moes and it's it's
  • 01:04:08
    become pretty hard to compete with us so
  • 01:04:10
    a little less worried about that but
  • 01:04:11
    that was a worry that was a pretty major
  • 01:04:12
    worry I'd say you know six years ago so
  • 01:04:17
    now you're actually branching outside of
  • 01:04:19
    languages uh how do you think about
  • 01:04:21
    doing that and then what's the hardest
  • 01:04:23
    part is it is it getting the brand is it
  • 01:04:26
    the content is it rebuilding elements of
  • 01:04:28
    the product what's the hardest part of
  • 01:04:30
    doing that it's creating the whole
  • 01:04:32
    learning experience um you know we're
  • 01:04:35
    branching out to math and music um
  • 01:04:37
    there's some differences in how you
  • 01:04:39
    teach math and music to languages and so
  • 01:04:41
    we're creating The Learning Experience
  • 01:04:42
    that's to me that's the hardest part
  • 01:04:44
    what's the most interesting difference
  • 01:04:45
    of how someone would learn a language
  • 01:04:47
    versus math or music well math needs
  • 01:04:50
    more
  • 01:04:51
    explanation than than languages
  • 01:04:54
    um it's also the exercises and there's a
  • 01:04:56
    lot more graphical elements to it it's
  • 01:04:58
    just a lot of math I mean some of it is
  • 01:05:00
    just numbers but a lot of it it just
  • 01:05:02
    especially you're going to teach
  • 01:05:03
    geometry and stuff like that it's it's a
  • 01:05:05
    lot more graphical stuff and and and you
  • 01:05:07
    have to start really you know you have
  • 01:05:09
    there a lot of things you have to kind
  • 01:05:11
    of figure out that we've now just
  • 01:05:13
    figured out with languages I mean we
  • 01:05:15
    figure out for example with languages
  • 01:05:16
    I'll tell you it took us many years to
  • 01:05:18
    figure this out and in retrospect it's
  • 01:05:20
    obvious ready look if you're going to
  • 01:05:21
    teach a language you you should align
  • 01:05:23
    yourself with this thing called the
  • 01:05:24
    common European framework of reference
  • 01:05:25
    it's called the CFR just do that don't
  • 01:05:28
    deviate um there's no such thing for
  • 01:05:31
    math for math there's something like
  • 01:05:32
    it's called like the common core but
  • 01:05:34
    it's not clear to us that that the
  • 01:05:36
    common core is as good as uh
  • 01:05:39
    CFR and so that's the type of stuff that
  • 01:05:42
    we just don't know and and also for math
  • 01:05:45
    um we're not entirely sure who our
  • 01:05:48
    audiences there's an audience we want
  • 01:05:51
    but there's an audience that we may end
  • 01:05:52
    up getting that is not what we want what
  • 01:05:54
    we want is everyone we want to do for
  • 01:05:57
    math what we did for languages which is
  • 01:06:00
    you know 80% of our users were not
  • 01:06:01
    learning a language before dual lingual
  • 01:06:03
    we want it to be the case that anybody
  • 01:06:05
    who's 25 years old or 30 years old or
  • 01:06:08
    however or they're eight years old they
  • 01:06:10
    just use dual lingle to learn math and
  • 01:06:12
    they get better at we want that to be
  • 01:06:14
    true it may end up being that the only
  • 01:06:16
    people we get are the people who are
  • 01:06:17
    forced to learn math because we just
  • 01:06:19
    can't make math fun enough uh but what
  • 01:06:22
    we would like is is to get to everyone
  • 01:06:24
    and with meth this is a real open
  • 01:06:26
    question can we get it so that your
  • 01:06:28
    average 35y old decides to spend five to
  • 01:06:33
    10 minutes a day getting better at math
  • 01:06:35
    can we do that I don't know that is what
  • 01:06:39
    the product team is T with and we have a
  • 01:06:41
    really good product team but I don't
  • 01:06:42
    know if they'll succeed because it's
  • 01:06:44
    interesting and and you alluded to this
  • 01:06:46
    of getting to spend five to 10 minutes a
  • 01:06:48
    day but uh you don't view your
  • 01:06:51
    competition as Rosetta Stone or a
  • 01:06:54
    private language tutor or or whoever
  • 01:06:57
    right how how do you think about your
  • 01:06:59
    competition and and what you're going up
  • 01:07:02
    against for that
  • 01:07:03
    individual competition is time um and
  • 01:07:07
    you know if you want to be more precise
  • 01:07:08
    it's time on the phone so a real you
  • 01:07:10
    know when we talk to our users and we
  • 01:07:12
    ask them you know the ones who like sto
  • 01:07:14
    using dualingo where you
  • 01:07:17
    stop I don't know we've done I'm sure
  • 01:07:19
    they exist but I I don't know if we've
  • 01:07:21
    done any user research where they told
  • 01:07:24
    us they've gone to a competitor like a
  • 01:07:26
    like a language learning competitor
  • 01:07:27
    that's not what people
  • 01:07:28
    say the most common thing that people
  • 01:07:30
    say well the first thing they says I
  • 01:07:32
    haven't stopped which they're just lying
  • 01:07:34
    about but once they once you convince
  • 01:07:37
    them that yes you know that they have
  • 01:07:38
    stopped they'll tell you what to be
  • 01:07:40
    honest I'm just spending more time on
  • 01:07:42
    Tik
  • 01:07:42
    Tok or Instagram or whatever the hell it
  • 01:07:45
    is so our real
  • 01:07:47
    competition is is you know the apps
  • 01:07:51
    games uh whatever it is the the things
  • 01:07:53
    that they take up your
  • 01:07:55
    time on your phone is there an
  • 01:07:58
    optimistic point about humans and their
  • 01:08:01
    desire to learn embedded in that that
  • 01:08:03
    they're willing to waste time on Tik Tok
  • 01:08:07
    or Instagram which presumably I think
  • 01:08:09
    most of that is uh or spend it learning
  • 01:08:13
    a language if the language experience is
  • 01:08:16
    comparable or enjoyable or easy like how
  • 01:08:19
    do you how do you think about the I mean
  • 01:08:21
    I believe that obviously we haven't
  • 01:08:23
    gotten there I mean obviously Tik Tok
  • 01:08:24
    has more uses than dualingo but I do
  • 01:08:27
    believe that
  • 01:08:28
    um as long as you're close to as fun as
  • 01:08:33
    tick Tok or as engaging or however you
  • 01:08:34
    want to call it addictive engaging
  • 01:08:36
    however you want to call it as long as
  • 01:08:37
    you're close I think we have a good good
  • 01:08:39
    shot because afterwards the good news
  • 01:08:41
    about dueling was afterwards you don't
  • 01:08:43
    feel like you wasted your time you at
  • 01:08:45
    the very least got somewhat better
  • 01:08:48
    Spanish um whereas if you spend 30
  • 01:08:51
    minutes you know scrolling on
  • 01:08:54
    Instagram you know you may learn a
  • 01:08:57
    couple of things but really mostly time
  • 01:09:00
    wastage um and so I we think that as
  • 01:09:03
    long as we're close as long as we're
  • 01:09:06
    close to how engaging that is we're in a
  • 01:09:08
    good spot and and again I think we've
  • 01:09:11
    gotten somewhat close but we're not
  • 01:09:14
    there uh you know we have I don't know
  • 01:09:17
    the latest numbers we released something
  • 01:09:19
    like 24 million daily active users we
  • 01:09:21
    don't have 500 million daily active
  • 01:09:22
    users so we got to get there yep there's
  • 01:09:25
    room to grow uh in terms of artificial
  • 01:09:28
    intelligence you all have been fairly
  • 01:09:30
    early in terms of using this can can you
  • 01:09:33
    talk about your original desire to maybe
  • 01:09:35
    build some of this stuff internally and
  • 01:09:38
    then uh where you are in the evolution
  • 01:09:41
    of of using tooling and what that's
  • 01:09:44
    going to mean from a user experience
  • 01:09:46
    standpoint yeah for artificial
  • 01:09:48
    intelligence um you know we've always
  • 01:09:51
    from the beginning for duelingo the idea
  • 01:09:52
    was we're not going to use humans to
  • 01:09:54
    teach
  • 01:09:55
    you there's a reason for that uh because
  • 01:09:59
    humans are expensive um you know if we
  • 01:10:02
    want to teach
  • 01:10:03
    literally whatever hundreds of millions
  • 01:10:05
    of people you got to find a lot of
  • 01:10:07
    teachers and they're expensive and you
  • 01:10:09
    got to pay for them and we're just not
  • 01:10:11
    going to use them basically we're going
  • 01:10:13
    to teach you with with a
  • 01:10:14
    computer um so that has always been the
  • 01:10:17
    idea and if you're going to do it with a
  • 01:10:18
    computer you know you got to make it so
  • 01:10:20
    the computer can teach us well as a
  • 01:10:21
    one-on-one human tutor and so we've been
  • 01:10:23
    working really hard to use AI to teach
  • 01:10:27
    as well as a one human Twitter um there
  • 01:10:31
    was a major shift of course in the last
  • 01:10:33
    year call it because of large language
  • 01:10:36
    models I and I don't think almost
  • 01:10:39
    anybody else could have told you five
  • 01:10:40
    years ago that large language models
  • 01:10:42
    were going to be this
  • 01:10:43
    good um turns out they're extremely good
  • 01:10:47
    so you know immediately when they came
  • 01:10:49
    out um we started you know applying some
  • 01:10:53
    of our AI efforts into that that's not
  • 01:10:55
    the only place where we've used AI we
  • 01:10:57
    use AI for a lot of different things um
  • 01:11:00
    but at least for conversational practice
  • 01:11:02
    this is the thing we were never able to
  • 01:11:04
    do uh without large language models we
  • 01:11:06
    just could not build a very good
  • 01:11:08
    conversational partner but today uh we
  • 01:11:12
    can build in a really good
  • 01:11:13
    conversational partner with a large
  • 01:11:14
    language model how has uh going public
  • 01:11:18
    made you all operate differently is it
  • 01:11:21
    been a good experience what would you
  • 01:11:23
    recommend for for people maybe thinking
  • 01:11:25
    about it for for us I think it's been an
  • 01:11:28
    excellent experience um for me has been
  • 01:11:30
    a really good experience I think there's
  • 01:11:31
    a couple of things I think we you know I
  • 01:11:33
    heard this from um Rich Barton uh who
  • 01:11:37
    started Zillow and a number of other you
  • 01:11:39
    know amazing companies he said look
  • 01:11:41
    going public is like joining uh the
  • 01:11:43
    major leagues like everything just
  • 01:11:45
    operates at a higher level and I believe
  • 01:11:47
    that I mean just everything started
  • 01:11:48
    operating on a higher level you just are
  • 01:11:50
    forced to do all these things that you
  • 01:11:53
    didn't have have to do before so you
  • 01:11:54
    don't do them it's like you know when
  • 01:11:55
    you get no visitors in your house you're
  • 01:11:57
    not really forced to clean but if you're
  • 01:11:59
    going to have visitors every day you
  • 01:12:00
    keep a pretty clean house um it's the
  • 01:12:03
    same thing like everything just we
  • 01:12:05
    operate a lot better I think for me in
  • 01:12:08
    particular I've been very lucky that we
  • 01:12:11
    have a really good CFO and finance team
  • 01:12:14
    so I haven't had my job hasn't changed
  • 01:12:17
    all that much because a lot of the
  • 01:12:18
    public company stuff gets done by our
  • 01:12:21
    finance team and I'm and other than just
  • 01:12:24
    regular doing regular earnings calls um
  • 01:12:27
    um it hasn't changed all that much for
  • 01:12:29
    me in particular but I think I've had a
  • 01:12:32
    fortunate experience for a number of
  • 01:12:34
    reasons but I think the probably the
  • 01:12:35
    biggest reasons reason why we've had a
  • 01:12:37
    fortunate experience is that
  • 01:12:40
    um
  • 01:12:42
    we got very good at
  • 01:12:45
    forecasting and we don't overpromise and
  • 01:12:49
    I think we've historically haven't we we
  • 01:12:52
    haven't just been one of these companies
  • 01:12:53
    that completely overpromises and so I
  • 01:12:55
    think that that has helped why do you
  • 01:12:58
    think there there haven't been more
  • 01:13:00
    successful consumer companies as we look
  • 01:13:03
    today you all are at your all-time high
  • 01:13:07
    despite having gone public you know
  • 01:13:10
    everyone else that sort of has gone
  • 01:13:11
    public in the 2021 vintage or whatever
  • 01:13:14
    uh are definitely below their all-time
  • 01:13:16
    high um I would say there there aren't
  • 01:13:18
    that many businesses that have been
  • 01:13:20
    created between five and
  • 01:13:23
    20 billion do$ 25 billion do in equity
  • 01:13:27
    value in within consumer do you think
  • 01:13:30
    there's there's some innate cap that you
  • 01:13:33
    either get super big like a Facebook or
  • 01:13:35
    a a Google or a Snapchat or it's it
  • 01:13:38
    feels like there's there's just not a
  • 01:13:40
    lot that exists at that level versus
  • 01:13:42
    enterprise software there's a ton do you
  • 01:13:44
    have a perspective on that yeah I mean
  • 01:13:47
    we asked ourselves this question um that
  • 01:13:49
    there does seem to be something to once
  • 01:13:52
    you get kind of big enough you just
  • 01:13:54
    immediately turned into like a
  • 01:13:56
    Facebook
  • 01:13:58
    um there do seem to be a little bit of
  • 01:14:00
    that uh I think another thing that has
  • 01:14:03
    happened certainly for a lot of kind of
  • 01:14:04
    the
  • 01:14:05
    smaller consumer things I I think I
  • 01:14:09
    think a lot of them were fooled by
  • 01:14:10
    Performance
  • 01:14:12
    Marketing I I think um we were very
  • 01:14:16
    fortunate not because we were so smart
  • 01:14:18
    it's not that we were so smart it's just
  • 01:14:19
    we just because we were not monetized we
  • 01:14:22
    didn't do performance Market
  • 01:14:24
    but Performance Marketing is such a drug
  • 01:14:26
    it's such an addiction and and once
  • 01:14:28
    you're in it you can't stop it and the
  • 01:14:30
    problem with it is that really cuts into
  • 01:14:32
    your margins to the point where you just
  • 01:14:35
    it's very hard to it's very hard to ever
  • 01:14:36
    become you know very profitable when
  • 01:14:38
    you're when you're relying on a lot of
  • 01:14:40
    these consumer apps that are not the
  • 01:14:44
    Instagrams of the world if you look at
  • 01:14:46
    where their users are coming from it's
  • 01:14:48
    overwhelmingly Performance Marketing
  • 01:14:50
    when you look at like 80% of your users
  • 01:14:52
    are acquired something like that you're
  • 01:14:54
    going to have a shitty margin that
  • 01:14:56
    that's that and so then then yeah
  • 01:14:58
    investors are going to think you're a
  • 01:14:59
    shitty business too because they have a
  • 01:15:00
    shitty margin fortunately for us we
  • 01:15:03
    never did rely on Performance Marketing
  • 01:15:05
    and so our margins are really good
  • 01:15:07
    because we don't do that um and I think
  • 01:15:10
    that's it I think I think in retrospect
  • 01:15:13
    I can understand that again it's not
  • 01:15:14
    because we were so smart back then but
  • 01:15:15
    in retrospect I can understand look the
  • 01:15:17
    truth of the matter with Performance
  • 01:15:18
    Marketing is that companies like
  • 01:15:20
    Facebook like meta or or Google um are
  • 01:15:24
    excellent at extracting all the value
  • 01:15:26
    for that like you know if there is more
  • 01:15:29
    value to extract if there's more margin
  • 01:15:31
    to extract they'll take it uh because
  • 01:15:34
    that's what they're good at and
  • 01:15:37
    so I I think that's another big reason I
  • 01:15:40
    think interesting so dualingo today has
  • 01:15:43
    approximately how many people uh
  • 01:15:45
    LinkedIn says 1700 does that sound about
  • 01:15:48
    right no that's a lot of um um I don't
  • 01:15:50
    know why that is there's some uh
  • 01:15:52
    contractors is there and who knows who
  • 01:15:54
    knows what else Randomness there is uh
  • 01:15:57
    we have about 750 employees 750 okay uh
  • 01:16:01
    I heard you or severon your co-founder
  • 01:16:04
    have been or were in every hiring
  • 01:16:07
    meeting is that still the case both of
  • 01:16:10
    us both of you all 750 people you've met
  • 01:16:15
    with well we've hired more than that
  • 01:16:16
    because we've also lost some people um
  • 01:16:18
    yeah I guess guess and by the way not
  • 01:16:20
    only that we've also turned down a lot
  • 01:16:22
    of people so yes we've been in hiring
  • 01:16:24
    meetings of thousands of people how do
  • 01:16:27
    how do you make that work TimeWise uh
  • 01:16:29
    and how is that a principle every week
  • 01:16:32
    we just do it every week There's a
  • 01:16:33
    meeting every week that lasts about half
  • 01:16:34
    an hour where they present us um you
  • 01:16:37
    know the hiring manager comes and
  • 01:16:38
    presents the case for the person that
  • 01:16:39
    they're uh going to look at for some
  • 01:16:41
    roles you know this go super fast
  • 01:16:44
    particularly for more kind of Junior
  • 01:16:47
    roles sometimes these meetings take a
  • 01:16:50
    long time and and you know we may go
  • 01:16:53
    through a lot of um back and forth but
  • 01:16:56
    it you know it works it's not like I
  • 01:16:58
    mean again it's not like the majority of
  • 01:17:01
    the people that apply for dual lingal
  • 01:17:02
    get rejected much earlier than that the
  • 01:17:04
    only ones that get to us are the ones
  • 01:17:06
    that where everything looks pretty good
  • 01:17:08
    and so they get to us so it's not that
  • 01:17:10
    many per week it's um got it so it's not
  • 01:17:12
    that you don't meet every employee
  • 01:17:15
    before they join but you sign off us not
  • 01:17:17
    we used to meet every employee that was
  • 01:17:19
    that that that was a while ago I mean
  • 01:17:21
    that probably stopped five years ago um
  • 01:17:25
    at this point it's it's really just the
  • 01:17:27
    case gets made to us yeah by the way
  • 01:17:29
    this is very important um and it's
  • 01:17:31
    because of this everybody else but us
  • 01:17:34
    and it's not because we're special
  • 01:17:35
    people I'll just tell you what it is
  • 01:17:37
    everybody else but us has a true
  • 01:17:38
    incentive to fill the role the the
  • 01:17:40
    person from the talent acquisition team
  • 01:17:42
    just wants to fill the role because
  • 01:17:43
    they're being paid to fill the role the
  • 01:17:45
    hiring manager has a hole in their
  • 01:17:46
    organization so they need to F the role
  • 01:17:48
    so everybody just has an incentive so
  • 01:17:50
    what happens is they're just like ah
  • 01:17:52
    this this price is good enough and so
  • 01:17:55
    they come in with a incentive to hire
  • 01:17:57
    whereas we have an incentive to hold the
  • 01:17:59
    line and by the way yeah again it's not
  • 01:18:01
    because we're such great individuals
  • 01:18:03
    with such great it's just a matter of
  • 01:18:05
    incend us because we do the same thing
  • 01:18:06
    because I've seen when s wants to hire
  • 01:18:08
    somebody he also has this problem and
  • 01:18:11
    when I want to hire somebody for my team
  • 01:18:12
    I also lower the bar and so we try to
  • 01:18:15
    you know it's it's simply it's not
  • 01:18:18
    because we are such great judges of
  • 01:18:21
    talent or whatever it's simply it's it's
  • 01:18:23
    a matter of incentives and that's why we
  • 01:18:25
    we we keep interesting so it's kind of
  • 01:18:26
    human nature if it's yours then you see
  • 01:18:29
    the pain of having to restart it and so
  • 01:18:31
    that's and that's exactly right I mean
  • 01:18:32
    when I'm hiring for my executive team
  • 01:18:34
    I'm just like my god I've been looking
  • 01:18:35
    for six months look this person is
  • 01:18:38
    pretty good they're pretty good they're
  • 01:18:39
    pretty good and then if I show it to
  • 01:18:41
    somebody who's way more impartial
  • 01:18:43
    they're like
  • 01:18:45
    really come on and so that I think
  • 01:18:48
    that's what it is it's it's simply that
  • 01:18:50
    Beyond just being a good person what's a
  • 01:18:52
    what's a non-negotiable for someone
  • 01:18:55
    joining dualingo and maybe maybe to the
  • 01:18:57
    good person example if you could also
  • 01:18:59
    tell the the car ride uh of what what
  • 01:19:02
    you all do if people hear this maybe
  • 01:19:04
    they'll know now to be nice to we do a
  • 01:19:06
    bunch of things and you know um we do
  • 01:19:09
    that you know for us it's important that
  • 01:19:12
    people are
  • 01:19:13
    kind um
  • 01:19:16
    and one of the places where that shows
  • 01:19:18
    up a lot is how you treat people who are
  • 01:19:21
    uh lower in the organizational structure
  • 01:19:24
    than you are yeah everybody's pretty
  • 01:19:27
    nice to me when they meet me uh and they
  • 01:19:28
    want to work for dualingo but um you
  • 01:19:31
    know we definitely see how people uh
  • 01:19:35
    treat um some and and this is usually
  • 01:19:39
    for for um more senior roles um where we
  • 01:19:43
    look at you know how they treat much
  • 01:19:45
    more Junior people and you know I mean
  • 01:19:47
    this is one you know we I've told this
  • 01:19:49
    story before where um you know we were
  • 01:19:50
    looking for a CFO for a long time
  • 01:19:53
    and
  • 01:19:54
    um one person had everything looked good
  • 01:19:59
    everything looked good but um they did
  • 01:20:02
    not treat their driver well and the
  • 01:20:04
    driver is a person that has worked with
  • 01:20:05
    us you know not he not a dualing
  • 01:20:07
    employee but it's the same you know we
  • 01:20:09
    have a few drivers that we work with to
  • 01:20:11
    send people to the airport you know to
  • 01:20:13
    to pick them up and you know we asked
  • 01:20:15
    them was this person good I like
  • 01:20:17
    actually no they were pretty pretty
  • 01:20:19
    nasty to me and we didn't hire them and
  • 01:20:22
    that was it was painful for me not to
  • 01:20:24
    hire them because I wanted to hire this
  • 01:20:25
    person because I have been looking this
  • 01:20:27
    is an example I had been looking for a
  • 01:20:28
    long time at that time I proba had been
  • 01:20:30
    looking for like nine months and this
  • 01:20:33
    was a very smart person uh very
  • 01:20:35
    competent but I guess they were not nice
  • 01:20:37
    to our try and and my sense is if that
  • 01:20:41
    happens you know they're probably not
  • 01:20:43
    going to be very nice to a level one you
  • 01:20:45
    know Finance associate or whatever
  • 01:20:48
    beyond that Beyond being kind to people
  • 01:20:50
    are there certain traits that you look
  • 01:20:52
    for
  • 01:20:53
    uh that are non-negotiables or is every
  • 01:20:55
    role kind of specific beyond that it
  • 01:20:57
    depends a lot on the role no it's not
  • 01:20:59
    the role but more the function so you
  • 01:21:02
    know in the case of
  • 01:21:04
    um design you know we really look at
  • 01:21:09
    people's portfolio and we really care
  • 01:21:10
    about polish you know some design
  • 01:21:12
    organizations are okay that some of the
  • 01:21:15
    designers don't have as much polish as
  • 01:21:16
    others for us every designer that we
  • 01:21:18
    hire we want to have polish so we look
  • 01:21:19
    at their portfolio a lot it just kind of
  • 01:21:21
    depends on the function
  • 01:21:23
    uh but I would say companywide you know
  • 01:21:26
    we want people to be kind and obviously
  • 01:21:28
    really good at their Craft um that
  • 01:21:30
    that's just you know kind of obviously
  • 01:21:33
    so there's one other thing that we do
  • 01:21:34
    look for it's it's it's which which we
  • 01:21:37
    try
  • 01:21:38
    um we really want people who are
  • 01:21:42
    um in one way or another exceptional and
  • 01:21:46
    and we we look for that quite a bit and
  • 01:21:48
    exceptional can mean different things I
  • 01:21:49
    mean in some cases you know exceptional
  • 01:21:51
    is the usual kind of tech company
  • 01:21:52
    definition of exceptional which is like
  • 01:21:54
    went to Stanford got up
  • 01:21:56
    4.0 um etc etc that's pretty exceptional
  • 01:21:59
    honestly very few people were able to
  • 01:22:01
    get a 4.0 from Stanford so that's pretty
  • 01:22:03
    exceptional but we look for a broader
  • 01:22:05
    definition of exceptional I mean first
  • 01:22:08
    first person their family to go to
  • 01:22:09
    college it's pretty
  • 01:22:11
    exceptional um comes from a
  • 01:22:14
    country from a very poor country
  • 01:22:17
    actually grew up in a very poor country
  • 01:22:19
    and somehow managed to you know go to
  • 01:22:22
    the university
  • 01:22:23
    France that's pretty
  • 01:22:25
    exceptional uh so stuff like that um you
  • 01:22:28
    know we we we we try to make it so that
  • 01:22:30
    everybody we hire has something
  • 01:22:32
    exceptional what have you learned about
  • 01:22:34
    I guess both leading and managing over
  • 01:22:38
    the course of running dual
  • 01:22:40
    lingo managing definitely I've learned
  • 01:22:42
    how to not become how to not be a
  • 01:22:46
    micromanager I actually believe that if
  • 01:22:48
    you're a small company you know under 20
  • 01:22:51
    employees it really is actually a good
  • 01:22:53
    thing for you to be a micro manager like
  • 01:22:55
    why not um after a certain number you
  • 01:22:59
    just can't beat and be and then after
  • 01:23:01
    you start hiring very senior people you
  • 01:23:03
    piss them off um so I've learned not to
  • 01:23:05
    be a micromanager um you know there a
  • 01:23:08
    number of other things I've learned that
  • 01:23:10
    ultimately when there's a
  • 01:23:12
    [ __ ] not to point fingers to anybody
  • 01:23:15
    but
  • 01:23:16
    yourself and we actually have made that
  • 01:23:18
    pretty good part of the culture doing it
  • 01:23:20
    especially in some functions where you
  • 01:23:22
    know there's fuckups happening every
  • 01:23:24
    day um and we we've gotten pretty good
  • 01:23:27
    at you know it's just like okay well
  • 01:23:30
    keep going and as a leader I think it's
  • 01:23:33
    really important to be very
  • 01:23:36
    very ready to blame yourself um I think
  • 01:23:41
    a lot of leaders I've seen it with you
  • 01:23:43
    know even with people who have in over
  • 01:23:45
    the years report to me were they really
  • 01:23:46
    trying not to get blamed for something
  • 01:23:49
    like no no somebody else did it yeah
  • 01:23:53
    I think people respect it a lot more
  • 01:23:55
    when you say look
  • 01:23:56
    ultimately I'm responsible for it and
  • 01:24:00
    you know by that I'm responsible for all
  • 01:24:02
    the dualingo fuckups and we've had
  • 01:24:04
    hundreds maybe
  • 01:24:07
    thousands it's probably safe to describe
  • 01:24:09
    yourself as a uh accidental entrepreneur
  • 01:24:13
    is that is that fair I did not grow up
  • 01:24:15
    thinking I would be an entrepreneur and
  • 01:24:18
    and and then you capture then then
  • 01:24:20
    recapture then do lingo I guess um when
  • 01:24:24
    recapture sold how many people did you
  • 01:24:26
    have oh not a lot um a dozen got it what
  • 01:24:31
    what what part of the entrepreneurial
  • 01:24:33
    Journey uh do you innately enjoy the
  • 01:24:38
    most I don't know if it's the
  • 01:24:40
    entrepreneurial Journey that I enjoy
  • 01:24:42
    it's more um I enjoy doing things that
  • 01:24:45
    have
  • 01:24:46
    impact that part I enjoy so I enjoy I'm
  • 01:24:48
    enjoying a lot what I'm doing right now
  • 01:24:49
    with do lingo but it's because I'm still
  • 01:24:51
    allowed to touch the product
  • 01:24:53
    um the day I'm no longer allowed to
  • 01:24:54
    touch the product I my job satisfaction
  • 01:24:57
    will goow down quite
  • 01:24:59
    significantly um so I like that I mean I
  • 01:25:01
    like tinkering I like being able to have
  • 01:25:03
    a lot of fun say in
  • 01:25:06
    what in in in what goes into the end
  • 01:25:10
    product of a of a consumer product I've
  • 01:25:12
    heard you say that working out has been
  • 01:25:14
    a life-changing thing for you when did
  • 01:25:17
    that start uh and and how how has it
  • 01:25:21
    actually changed your
  • 01:25:23
    life I mean it started out it started
  • 01:25:26
    about right around when I started do a
  • 01:25:28
    lingo um I just it was a streak I
  • 01:25:31
    decided to have a workout streak and um
  • 01:25:35
    I haven't I haven't lost it actually
  • 01:25:36
    it's been you
  • 01:25:37
    know however many years 13 give or tape
  • 01:25:40
    years every every day or every every day
  • 01:25:43
    every day every day I have not lost it I
  • 01:25:45
    did cheat a couple of times as follows
  • 01:25:47
    these trans these flights to to
  • 01:25:50
    Japan there a problem with the date
  • 01:25:53
    there Che by working out twice the day
  • 01:25:56
    before and uh you know something like
  • 01:25:59
    that but but yeah I haven't lost my
  • 01:26:02
    streak of working out every single day
  • 01:26:04
    um what's the most bare bone thing
  • 01:26:06
    you've done to uh to make it work uh H
  • 01:26:10
    yeah the the New York Times seven minute
  • 01:26:12
    workout yeah got it um look because some
  • 01:26:16
    because you can always say you can go
  • 01:26:17
    running but sometimes it's like horrible
  • 01:26:19
    weather you're in some random hotel that
  • 01:26:22
    has no gym or whatever and all you got
  • 01:26:25
    is you and yeah you do that 7-minute
  • 01:26:29
    workout a few times and that's all you
  • 01:26:30
    do how has it changed you is is it is it
  • 01:26:34
    the energy from it is it the the the
  • 01:26:37
    energy you know I used to get sick quite
  • 01:26:38
    often which is funny which I now I am
  • 01:26:40
    sick but I I get sick once every like
  • 01:26:42
    two years um I used to get sick very
  • 01:26:44
    often before that that's one thing I
  • 01:26:46
    think the other thing is it really
  • 01:26:48
    allows me to regulate my own emotions
  • 01:26:52
    quite a bit like I think I you know I'm
  • 01:26:55
    generally a pretty emotional guy but I
  • 01:26:59
    think that um my wife says I'm I'm Latin
  • 01:27:02
    American Latin American
  • 01:27:05
    men but I think uh um you know it allows
  • 01:27:09
    me to regulate my emotions I think
  • 01:27:11
    someone that uh maybe struggles with
  • 01:27:13
    that or or wants to get going is there
  • 01:27:15
    is there any advice you would have to
  • 01:27:17
    them on like how to just start making
  • 01:27:19
    progress is it just get a streak going
  • 01:27:22
    in a similar way to yeah yeah yeah yeah
  • 01:27:24
    yeah and by the way what that's what
  • 01:27:26
    matters the most just get a street going
  • 01:27:28
    with anything I mean this is this by the
  • 01:27:30
    way it's the same with dualingo look a
  • 01:27:32
    lot of times people are like no no no no
  • 01:27:34
    there's a there's a better method for
  • 01:27:35
    learning a language and it's like look
  • 01:27:37
    ultimately that's like saying the
  • 01:27:39
    treadmill is better than the elliptical
  • 01:27:40
    it probably is but you know what matters
  • 01:27:42
    95% is that you do
  • 01:27:44
    something that's you know did you do it
  • 01:27:47
    with a treadmill with an elliptical or
  • 01:27:49
    one of those weird climbing machines or
  • 01:27:50
    whatever sure probably there there are
  • 01:27:52
    some differences but between not having
  • 01:27:54
    done anything and having done even the
  • 01:27:56
    dumbest one of these it's better to do
  • 01:27:58
    the dumbest one of these I mean and
  • 01:27:59
    that's the thing with doing we're
  • 01:28:00
    optimized for keeping you going is there
  • 01:28:03
    is there another habit or anything
  • 01:28:04
    you've picked up on maximizing
  • 01:28:06
    productivity and balancing your your you
  • 01:28:09
    know work life balance and all of that
  • 01:28:11
    stuff I mean I'm pretty efficient at
  • 01:28:14
    most everything I do and I think that's
  • 01:28:15
    you know a lot of people comment on that
  • 01:28:17
    I'm I do most everything very fast I eat
  • 01:28:20
    very fast so I I don't know if it's a
  • 01:28:22
    habit that I think for some reason I'm
  • 01:28:24
    just very fast at most everything I do
  • 01:28:26
    I'm the same way eating by the way I
  • 01:28:28
    it's always been a problem people think
  • 01:28:30
    that I've been to jail yeah people think
  • 01:28:32
    that I've been to jail because I like
  • 01:28:33
    eat so fast I eat so fast but the truth
  • 01:28:37
    is you know I love food but I I don't
  • 01:28:39
    savor it in my mouth for five hours yeah
  • 01:28:42
    that's funny um one of the uh one of the
  • 01:28:46
    things I'd be curious about is um
  • 01:28:49
    recapture was sold in what was it
  • 01:28:52
    2008 nine 10 nine nine okay and at that
  • 01:28:57
    point in time I I don't think it's ever
  • 01:28:59
    been disclosed but you didn't raise any
  • 01:29:01
    outside money and sold for I I've youve
  • 01:29:04
    said tens of millions of dollars uh and
  • 01:29:07
    so um and now dualingo is an enormous
  • 01:29:10
    success I'm sure we can look up in the
  • 01:29:11
    S1 how uh you know your ownership at
  • 01:29:14
    time of IPO but for safe to say uh you
  • 01:29:17
    you uh you've made more money than you
  • 01:29:20
    can probably spend but between both of
  • 01:29:23
    of these things unless you want to go
  • 01:29:24
    buy a sports team or something uh but I
  • 01:29:27
    assume I don't know if that's in the
  • 01:29:28
    cards for you but what depending on the
  • 01:29:30
    sports team depending on the sports team
  • 01:29:32
    yeah well in that case then then there's
  • 01:29:35
    a lot of stuff there's a lot more that
  • 01:29:37
    you can keep going uh to BU the sports
  • 01:29:39
    team what would be the sports team of
  • 01:29:40
    choice if you
  • 01:29:42
    could I mean I don't know I like
  • 01:29:45
    soccer I'm not that crazy I mean I like
  • 01:29:48
    soccer but I'm not it's not GNA be the
  • 01:29:50
    Steelers or the yeah one of the
  • 01:29:52
    Pittsburgh teams okay well the Pirates
  • 01:29:55
    would be cheap but yeah that's right
  • 01:29:57
    that's right how
  • 01:29:59
    um how has that most changed your life
  • 01:30:03
    or or how you view the the the world uh
  • 01:30:07
    having that level of of success it seems
  • 01:30:09
    like you you still live in Pittsburgh
  • 01:30:11
    presumably a similarish life but is
  • 01:30:13
    there any way that it's changed to the
  • 01:30:15
    positive versus the negative on stuff oh
  • 01:30:19
    yeah I mean you don't worry about some
  • 01:30:20
    of the stuff that you used to I mean I
  • 01:30:22
    remember when I was a graduate student
  • 01:30:23
    you worry about like are you going to
  • 01:30:25
    have enough money to pay the rent and
  • 01:30:26
    stuff like that um you know those those
  • 01:30:30
    worries entirely go away and that I
  • 01:30:31
    think you know that's that's quite a
  • 01:30:33
    nice thing I mean I just don't and and
  • 01:30:35
    not only that the other thing is just
  • 01:30:36
    not having to spend any mental energy on
  • 01:30:39
    life you know should that does this
  • 01:30:41
    toothpaste cost more less than that
  • 01:30:43
    other toothpaste Etc just just I you
  • 01:30:45
    know I basically learn just don't think
  • 01:30:48
    about money unless again unless you're
  • 01:30:49
    going to go doing some crazy thing like
  • 01:30:51
    buy a sports team team kind of just
  • 01:30:52
    doesn't matter uh and you know at least
  • 01:30:56
    I don't spend the um mental energy on
  • 01:30:58
    that now for me I'm you know it's not
  • 01:31:00
    like I go around spending crazy amounts
  • 01:31:03
    of money um so it hasn't changed that
  • 01:31:07
    much how I live my life I mean I have I
  • 01:31:10
    have like three pairs of jeans that's it
  • 01:31:12
    it's kind of what I
  • 01:31:15
    have um yeah hasn't changed that much I
  • 01:31:18
    mean I did you know right now I'm in New
  • 01:31:21
    York in New York and you know I did buy
  • 01:31:23
    a pretty expensive house here but other
  • 01:31:25
    than that I haven't done all that much I
  • 01:31:27
    I've heard you say uh something about
  • 01:31:29
    like to truly understand something is to
  • 01:31:32
    be able to explain it simply or or some
  • 01:31:35
    philosophy like that can can you exp
  • 01:31:37
    spouse or expound on that a little bit
  • 01:31:40
    uh for for why you I learned that's from
  • 01:31:42
    my PhD adviser Manuel Blum I mean it is
  • 01:31:45
    an amazing thing and it's hard to
  • 01:31:48
    explain unless you've actually felt it
  • 01:31:51
    um
  • 01:31:52
    um once you actually understand
  • 01:31:55
    something you can give an amazing clear
  • 01:31:58
    simple explanation of it um and I used
  • 01:32:00
    to think that I just didn't know that um
  • 01:32:05
    and you know I learned that um from my
  • 01:32:09
    advisor and it was as follows it was
  • 01:32:12
    early on in my PhD you know I had to
  • 01:32:14
    explain a scientific paper to him and he
  • 01:32:17
    said go read this paper explain it to me
  • 01:32:19
    because I haven't read it I would come
  • 01:32:22
    you know I came in for a one hour
  • 01:32:23
    meeting and I said like the first
  • 01:32:26
    statement and he said I don't understand
  • 01:32:29
    and I'm like how can you understand it's
  • 01:32:30
    just a simple the simplest statement
  • 01:32:32
    that I he's I I just don't get it why
  • 01:32:34
    did you say that why did you start with
  • 01:32:36
    that why I don't get it and I spent the
  • 01:32:38
    whole hour and I couldn't even go after
  • 01:32:40
    F past the first line of the
  • 01:32:42
    paper um I came back the next week and
  • 01:32:45
    it was the same thing it took me six
  • 01:32:47
    months to finally get to a point where
  • 01:32:49
    he said okay I understand meeting him
  • 01:32:52
    weekly it turned out he was he was
  • 01:32:54
    pretending the whole time actually that
  • 01:32:57
    paper he was a real expert in um but it
  • 01:33:01
    took me really six months and and is
  • 01:33:03
    true I mean the explanation that I had
  • 01:33:04
    at the end of the six months was just
  • 01:33:06
    because I really understood it it was
  • 01:33:08
    just crystal clear and I could explain
  • 01:33:10
    something that at first I was planning
  • 01:33:11
    on taking like 40 minutes to explain it
  • 01:33:13
    to him I was able to explain the idea in
  • 01:33:15
    like a minute like no this is the main
  • 01:33:17
    idea of the paper blah and um yeah but
  • 01:33:21
    that happens once you really understand
  • 01:33:23
    something and that's I mean that's the
  • 01:33:24
    genius of my phc adviser he's he he
  • 01:33:27
    really understands things well I think
  • 01:33:30
    today there's more people learning
  • 01:33:31
    languages in the US through dualingo
  • 01:33:34
    than the education system combined do I
  • 01:33:37
    have that fact right correct there are
  • 01:33:40
    more people learning languages on Dual
  • 01:33:41
    lingual in the US than there are people
  • 01:33:43
    learning languages across all us high
  • 01:33:44
    schools combined but um I I heard you
  • 01:33:48
    say recently that you're not anti the us
  • 01:33:52
    education system uh and in fact may
  • 01:33:56
    maybe Pro of it if Having learned so
  • 01:33:59
    much about how people learn and built a
  • 01:34:03
    business around it at least for uh
  • 01:34:05
    languages and now increasingly music and
  • 01:34:07
    math if you could wage wave a magic wand
  • 01:34:10
    is there anything from a teaching
  • 01:34:13
    standpoint or or how we go about the the
  • 01:34:16
    educational system in the US that you
  • 01:34:19
    would change if there was one thing that
  • 01:34:22
    stuck out it's very different from what
  • 01:34:25
    we do at dual lingo but I mean what
  • 01:34:26
    matters the most in the educational
  • 01:34:28
    system are the
  • 01:34:29
    teachers um this is by far what matters
  • 01:34:32
    the most and um there are some excellent
  • 01:34:35
    teachers but there's also quite a high
  • 01:34:37
    barans and you know what I would wave
  • 01:34:40
    the most is or what I would wave want to
  • 01:34:42
    is make sure that every teacher that is
  • 01:34:46
    out there is
  • 01:34:48
    amazing um that's pretty hard to do uh
  • 01:34:51
    given a system that doesn't pay teachers
  • 01:34:53
    very well given a system where um you
  • 01:34:56
    know there's essentially equivalent of
  • 01:34:57
    tenure in some of these places where you
  • 01:34:59
    know you basically can work and not do
  • 01:35:01
    all that much because of teacher unions
  • 01:35:02
    so there's a lot of stuff that is
  • 01:35:05
    working against that but the thing you
  • 01:35:07
    know you want better education better
  • 01:35:09
    teachers that's you know again very
  • 01:35:11
    different from dualingo uh dualingo we
  • 01:35:13
    don't use humans but the truth of the
  • 01:35:15
    matter is that for the foreseeable
  • 01:35:17
    future humans are going to be needed for
  • 01:35:20
    you know the educational system can you
  • 01:35:21
    tell the story of Bill Gates so Bill
  • 01:35:23
    Gates called you once to try to
  • 01:35:24
    personally recruit you to uh to join
  • 01:35:27
    Microsoft what can you give that
  • 01:35:29
    backstory I picked it up in some I was
  • 01:35:31
    researching and I was like well I have
  • 01:35:33
    to get that story uh or a little bit
  • 01:35:35
    more detail into it clearly it didn't
  • 01:35:37
    work you never went to go work at
  • 01:35:38
    Microsoft right yeah it's when I
  • 01:35:39
    graduated from my PhD I
  • 01:35:43
    um there's this one of thing that
  • 01:35:45
    happens when you and kind of the The
  • 01:35:47
    Winner Takes call kind of thing um I I
  • 01:35:50
    was the kind of most sought after
  • 01:35:52
    canidate after I graduated from my PhD
  • 01:35:54
    every University wanted me to become a
  • 01:35:56
    professor
  • 01:35:57
    there and um usually when you graduate
  • 01:36:01
    from a PhD in computer science you go
  • 01:36:02
    become a professor in computer science
  • 01:36:03
    at our University but there's another
  • 01:36:05
    path which is you could also join
  • 01:36:06
    Microsoft research and I was the most
  • 01:36:09
    sought after candidate you know every
  • 01:36:11
    University every computer science
  • 01:36:12
    department wanted to hire me Microsoft
  • 01:36:14
    research wanted to hire me too and
  • 01:36:16
    Microsoft research I guess at the time I
  • 01:36:18
    don't know they had a thing where the
  • 01:36:20
    person they wanted to hire the most that
  • 01:36:22
    year um they would get Bill Gates to
  • 01:36:24
    call you and personally pitch you for as
  • 01:36:28
    long as you wanted to so I you know I
  • 01:36:30
    ended up talking to them for like an
  • 01:36:31
    hour um mainly not because I had you
  • 01:36:34
    know I I love a lot of the stuff that
  • 01:36:36
    they've done at Microsoft and certainly
  • 01:36:38
    Microsoft research but I I I wanted to
  • 01:36:40
    become a professor so I had no no desire
  • 01:36:42
    but I wanted to talk to Bill Gates for
  • 01:36:44
    an hour so I was happy to do that yeah
  • 01:36:46
    well it's good I if he listen if Bill
  • 01:36:48
    Gates ever calls me I I will gladly take
  • 01:36:50
    his call and I'll spend an hour a smart
  • 01:36:53
    guy he's such a smart guy it's crazy
  • 01:36:55
    when you enter a room and you're like
  • 01:36:57
    he's clearly smarter than me yeah you
  • 01:36:59
    think that with him like every time
  • 01:37:01
    because I've been in a room multiple
  • 01:37:02
    times with him for different things
  • 01:37:03
    different every time you're like yeah
  • 01:37:05
    that guyy smarter than me do do you
  • 01:37:07
    remember what you talked to him about or
  • 01:37:08
    what questions you asked him at that
  • 01:37:10
    time no I don't too much um but I've
  • 01:37:12
    talked to him about a number of things I
  • 01:37:13
    mean he's you know he's a we' talked
  • 01:37:15
    about gling and stuff number of times
  • 01:37:17
    again he's such a smart guy yeah now uh
  • 01:37:20
    last one I'll let hop what's the what's
  • 01:37:22
    the story of you setting up a Honeypot
  • 01:37:25
    website to catch people that were
  • 01:37:27
    cheating on your uh on your tests when
  • 01:37:30
    you were a professor I was a professor
  • 01:37:32
    yes I was a professor of computer
  • 01:37:33
    science um and um uh it wasn't tests I
  • 01:37:37
    you know we were I was assigning um you
  • 01:37:40
    know math problems it was basically
  • 01:37:42
    discret math but it was um math problems
  • 01:37:45
    and we had this rule which was a dumb
  • 01:37:47
    rule but it was a rule that I inherited
  • 01:37:49
    from the guy who T the class before me
  • 01:37:51
    that was you can't use you can't Google
  • 01:37:52
    for
  • 01:37:54
    Solutions um and so we tried every time
  • 01:37:57
    but we never because there's only so
  • 01:37:59
    many really awesome math problems and
  • 01:38:01
    you know we tried to disguise them a
  • 01:38:03
    little bit but you know we thought well
  • 01:38:05
    probably people are Googling for
  • 01:38:06
    Solutions and then it occurred to me
  • 01:38:09
    well I'll teach them not to do that and
  • 01:38:12
    so I made up a problem I specifically
  • 01:38:14
    made up a problem and then I made a
  • 01:38:16
    website for that problem that there a
  • 01:38:19
    fully unique problem made a website that
  • 01:38:21
    had the problem Etc and I owned the
  • 01:38:24
    website and what happened is I assigned
  • 01:38:26
    the problem and I said of course you
  • 01:38:28
    know as usual you can't Google or
  • 01:38:29
    anything but if you Googled that problem
  • 01:38:34
    you got to my website and I could track
  • 01:38:36
    who you were uh on the website and so um
  • 01:38:40
    Turned out 30% of the class this was a
  • 01:38:41
    class with 300 kids so about a 100 of
  • 01:38:43
    them fell on the little
  • 01:38:46
    website and um I found them all and then
  • 01:38:50
    um the next lecture you know I didn't
  • 01:38:53
    say anything I didn't contact them the
  • 01:38:54
    next lecture I just went in I stood in
  • 01:38:57
    front of the class I projected my screen
  • 01:38:59
    I typed you know the query to get to
  • 01:39:02
    that problem I got to the website and
  • 01:39:04
    everybody's just like okay and then I
  • 01:39:06
    said I did a you know to to know who
  • 01:39:09
    owns a website you just do who is and
  • 01:39:12
    the name of the website and I typed in
  • 01:39:13
    in the prompt there who is the name of
  • 01:39:15
    that website and then it said that is
  • 01:39:17
    owned by me and then you see everybody's
  • 01:39:20
    job just drop and then I'm then I Then I
  • 01:39:23
    then I you know I'm using this this
  • 01:39:25
    using Linux I'm I'm showing them you
  • 01:39:27
    know the the server logs and just
  • 01:39:30
    everybody's IP address and I'm like w
  • 01:39:33
    and everybody was just they were um they
  • 01:39:36
    were peeing the pets um and so what did
  • 01:39:40
    you do with it did you did they confess
  • 01:39:42
    or or what happened I mean I knew who
  • 01:39:45
    they were um so I just gave them a zero
  • 01:39:47
    all of them in that assignment I said
  • 01:39:48
    look I'm not going to ruin your life I'm
  • 01:39:50
    just going to give you a zero in that
  • 01:39:50
    assignment Ian you can still get an name
  • 01:39:51
    the class
  • 01:39:53
    but I was not interested in ruining
  • 01:39:55
    anybody anybody's life I just wanted to
  • 01:39:56
    uh have them not do that anymore yeah
  • 01:39:59
    it's a good lesson well thank you for
  • 01:40:01
    doing this this is fun I hope hopefully
  • 01:40:03
    you feel better powering through here uh
  • 01:40:06
    yeah so thanks for doing this it's a
  • 01:40:08
    pleasure and uh yeah this is great thank
  • 01:40:11
    you same pleasure is mine thank
  • 01:40:13
    [Music]
  • 01:40:19
    you
  • 01:40:21
    [Applause]
  • 01:40:28
    he
Tags
  • Luis von Ahn
  • Duolingo
  • language learning
  • A/B testing
  • product iteration
  • user engagement
  • viral marketing
  • freemium model
  • continuous improvement
  • startup culture