DPE Lowdown - How Meta does Developer Productivity Engineering with Adam McCormick

00:59:31
https://www.youtube.com/watch?v=fm2e3w0u5SQ

الملخص

TLDRThe session features insights from Adam McCormick of Meta discussing the intricacies of maintaining a high level of developer productivity within the tech giant, specifically for the Facebook Android app. Adam shares his experiences in productivity engineering, the role of tooling like Buck and custom versions of various development environments, and the systems in place to ensure timely and reliable releases. Key topics include the need for robust testing frameworks, the handling of crashes, and the emphasis on developer happiness as a metric of productivity success. The integration of internal tools to streamline processes, the importance of data-driven decision-making, and Meta's unique approach to developer experience are highlighted.

الوجبات الجاهزة

  • 🏡 Adam recently moved just north of Seattle.
  • 🌧️ The weather in Seattle is often rainy but beautiful.
  • 💻 Meta's team size for productivity engineering is around 1,000 people.
  • 📈 Developer happiness is a key focus for improving productivity.
  • 📅 Major updates for Facebook are released weekly, with minor updates even more frequently.
  • 🔍 The Sapiens system helps simulate human users for testing.
  • 🛠️ Meta uses a customized Android Studio and tools like Buck for builds.
  • 🔄 Rollbacks are handled through a system called unlanding.
  • 📊 Metrics are constantly monitored to drive improvements in developer experience.
  • 💪 Developers at Meta are viewed as key partners in building scalable products.

الجدول الزمني

  • 00:00:00 - 00:05:00

    In the opening segment, Adam shares his new living situation and discusses the rainy weather in Seattle, while reminiscing about past events and introductions made through Jay Zimmerman at various conferences. Justin highlights the purpose of the webcast, which is centered around productivity engineering and the experiences shared by Adam McCormick from Meta. They emphasize the need for enhancing developer productivity, specifically in large systems such as Facebook with nearly 3 billion users.

  • 00:05:00 - 00:10:00

    Adam describes his role on the Facebook app reliability engineering team, explaining how they work to minimize crashes for a massive user base. He explains the complexity of maintaining reliability at scale and the importance of getting code quality assurance integrated into the development process. The team collaborates with multiple product engineers to establish best practices and investigate issues as they arise, particularly for Android applications.

  • 00:10:00 - 00:15:00

    The discussion shifts to how frequently and the nature of app releases at Facebook. Adam details their build and release processes, including the number of builds they manage daily and the stages of releasing major and minor updates. With about 30 to 50 thousand builds created every day, their approach allows for better deployment and testing of all changes, emphasizing the need for a reliable infrastructure to manage such scales.

  • 00:15:00 - 00:20:00

    Adam clarifies the structure and extent of the productivity engineering group at Facebook, mentioning that they have about 1,000 dedicated engineers within a larger workforce, facilitating developers through tools, testing, and environment optimization. The complexity of the monorepo approach is discussed, highlighting how it contributes to the challenge of handling such large code bases and the subsequent investments made to efficiently manage them.

  • 00:20:00 - 00:25:00

    The focus is on the daily experiences of Meta's software engineers as they work on the Android app. Adam discusses the tools and customized systems they have in place, such as Android Studio and their build systems, to streamline coding, testing, and CI/CD processes, showcasing the heavy reliance on simulations and tests to catch issues before reaching broader audiences.

  • 00:25:00 - 00:30:00

    In this segment, Adam emphasizes the significance of productivity engineering as part of their success in software development and shipping at scale, underscoring that it's not merely about adding more engineers but about investing in structured approaches and solutions that improve overall productivity.

  • 00:30:00 - 00:35:00

    They delve into how Meta measures developer happiness and productivity, sharing that they conduct regular surveys and micro-check-ins to gather feedback and analyze the impact of changes on developer efficiency. They also appreciate the importance of consent in data collection, ensuring developers can opt out of specific productivity tools if desired.

  • 00:35:00 - 00:40:00

    Adam shares insights into notable productivity engineering wins at Meta, explaining how they have standardized processes to reduce the complexity of coding and analysis through consolidated tools and services that streamline common tasks, enhancing overall developer experience.

  • 00:40:00 - 00:45:00

    The conversation touches upon employee feedback regarding potential pitfalls of excessive measurement leading to micromanagement perceptions. Adam notes that they focus on the impact of tooling and systems on productivity without imposing numerical goals on individual engineers, emphasizing that the goal is to foster a culture of effective and happy developers.

  • 00:45:00 - 00:59:31

    Audience questions are covered, including one about build and test acceleration technologies at Meta. Adam explains their customized build systems that allow for rapid incremental builds by caching and optimizing compilation processes to ensure code quality while maintaining performance and minimizing resource waste. He highlights the conscious effort to reduce both financial and ecological impacts while still pushing forward with advancement.

اعرض المزيد

الخريطة الذهنية

فيديو أسئلة وأجوبة

  • What is the role of productivity engineering at Meta?

    Productivity engineering at Meta focuses on facilitating developers in releasing code efficiently, improving productivity, and ensuring high quality.

  • How often does Meta release updates for the Facebook app on Android?

    Meta releases major updates weekly and minor revisions multiple times a day, depending on the urgency of fixes.

  • What tools does Meta use for building and testing its code?

    Meta uses a customized version of Android Studio, a system called Buck for builds, and maintains a large monorepo for code management.

  • How does Meta ensure product reliability during rapid development cycles?

    Meta employs a number of measures, including extensive automated testing, monitoring user metrics, and a simulated user testing system called Sapiens.

  • What is the significance of the developer experience in productivity engineering?

    A positive developer experience leads to happier, more productive engineers who can focus on building rather than dealing with distractions.

  • How does Meta handle rollbacks or reversions in their coding process?

    Meta uses a system called unlanding, where a revert commit is created to reverse changes, with automated tools aiding in this process.

  • What types of engineering roles does Meta seek to fill?

    Meta seeks Java, C, and Swift developers, emphasizing that coding for mobile is not the sole focus but rather part of a larger engineering perspective.

  • How does Meta measure developer happiness and productivity?

    Meta uses a survey called 'Pulse' to gauge developer satisfaction, alongside monitoring various metrics related to code quality and production impacts.

عرض المزيد من ملخصات الفيديو

احصل على وصول فوري إلى ملخصات فيديو YouTube المجانية المدعومة بالذكاء الاصطناعي!
الترجمات
en
التمرير التلقائي:
  • 00:00:03
    [Music]
  • 00:00:06
    adam where are you dialing in from today
  • 00:00:09
    so i am in my brand new living room uh
  • 00:00:12
    we just moved house so
  • 00:00:14
    uh congratulations 15 15 miles north of
  • 00:00:17
    seattle and how's the weather in seattle
  • 00:00:20
    so it's a rainy day but on a rainy day
  • 00:00:22
    in seattle around dawn it looks like
  • 00:00:23
    this behind me this is actually taken
  • 00:00:25
    from the transit center near my house
  • 00:00:28
    beautiful like it's this isn't like some
  • 00:00:29
    fancy this isn't like some fancy
  • 00:00:32
    downloaded background this is just a
  • 00:00:33
    cell phone snap
  • 00:00:35
    so
  • 00:00:36
    i i love the story of of how you all met
  • 00:00:39
    while everybody still uh joins here
  • 00:00:41
    we're seeing the participants kind of
  • 00:00:44
    trickle in
  • 00:00:45
    i can i can fill in a little bit of
  • 00:00:47
    backstory there
  • 00:00:48
    uh
  • 00:00:49
    so i've been going to
  • 00:00:51
    events for with no fluff just stuff
  • 00:00:54
    with uh jay zimmerman great great series
  • 00:00:56
    by the way
  • 00:00:57
    uh but i've been going to that for
  • 00:01:00
    like a decade at this point um from back
  • 00:01:03
    when they were a little one show game in
  • 00:01:05
    uh denver
  • 00:01:07
    but so i've been going to these forever
  • 00:01:09
    and i started going to arch comp when it
  • 00:01:11
    moved to florida
  • 00:01:12
    uh
  • 00:01:14
    i don't know what five years ago
  • 00:01:15
    something like that
  • 00:01:16
    uh but so
  • 00:01:18
    we're all we're all hanging out after
  • 00:01:20
    one of the after one of the days
  • 00:01:22
    and jay decides he needs he really needs
  • 00:01:25
    justin to meet me
  • 00:01:27
    so he's a good lady around his bar
  • 00:01:30
    and we're and i'm standing there we've
  • 00:01:32
    been chatting with a bunch of the other
  • 00:01:34
    presenters and and just introduced
  • 00:01:36
    himself
  • 00:01:37
    um and i don't think he thought a lot
  • 00:01:39
    was going to come of it we just chatted
  • 00:01:41
    you know people you know people at a
  • 00:01:42
    conference you're always nice to
  • 00:01:43
    everybody
  • 00:01:45
    but then justin
  • 00:01:46
    i was in one of his sessions later
  • 00:01:48
    and
  • 00:01:50
    i i kept i kept raising my hand like
  • 00:01:52
    every time he wanted input because
  • 00:01:54
    that's kind of who i am i don't know
  • 00:01:56
    just if you want to if you want to talk
  • 00:01:57
    to that a little bit no i mean i think
  • 00:01:59
    you i think you know i mean it was it
  • 00:02:00
    was great it was one of these things
  • 00:02:01
    where we're having the way that you
  • 00:02:03
    really want a talk to go at one of these
  • 00:02:05
    sessions where it's not just you with
  • 00:02:08
    slides talking at the audience but you
  • 00:02:09
    really got participation and uh and adam
  • 00:02:12
    was was like yeah you know into the
  • 00:02:14
    conversation adding a lot of good like
  • 00:02:16
    detail and color but at one point i was
  • 00:02:18
    like wait a minute who are you and where
  • 00:02:19
    do you come from
  • 00:02:21
    and we got kind of more into this you
  • 00:02:23
    know real discussion about the work that
  • 00:02:26
    that adam's been doing for meta
  • 00:02:28
    specifically and um and how that related
  • 00:02:30
    so closely to what we were talking about
  • 00:02:32
    with with productivity and it was like
  • 00:02:34
    you've gotta you gotta come on our
  • 00:02:36
    loadout and so here we are
  • 00:02:38
    and i and i remember uh justin messaging
  • 00:02:41
    me and saying hey i met this gentleman
  • 00:02:43
    we gotta have him so i think he was
  • 00:02:46
    standing in the room at the time and
  • 00:02:48
    he's he's slacking back and forth with
  • 00:02:50
    you while people are like coming up
  • 00:02:51
    after the presentation and you know
  • 00:02:53
    getting swag
  • 00:02:55
    we had some gradle stickers and and a
  • 00:02:57
    couple cool shirts
  • 00:02:59
    that's kind of something
  • 00:03:01
    yeah so we're five minutes after we
  • 00:03:03
    should get started uh and thanks
  • 00:03:05
    everyone for joining uh ruse and justin
  • 00:03:08
    from gradle on the develop productivity
  • 00:03:11
    engineering webcast you know why are we
  • 00:03:14
    here just want to spend a few seconds
  • 00:03:16
    there right from gradle side right we
  • 00:03:19
    wanted to share
  • 00:03:21
    with the whole world about productivity
  • 00:03:23
    engineering
  • 00:03:25
    our wins findings experiences we want
  • 00:03:28
    you all to invest and develop
  • 00:03:31
    productivity engineering developer
  • 00:03:32
    experience
  • 00:03:34
    and why that's important
  • 00:03:36
    right and today we have adam mccormick
  • 00:03:39
    works on the facebook mobile
  • 00:03:42
    reliability engineering team and they
  • 00:03:45
    essentially have one of the hardest
  • 00:03:46
    developer productivity engineering
  • 00:03:48
    problems in developer experience
  • 00:03:50
    problems on the planet uh with their you
  • 00:03:53
    know just facebook alone like almost
  • 00:03:55
    three billion active users right and so
  • 00:03:59
    it's and it's no secret that how fast
  • 00:04:02
    facebook is able to
  • 00:04:05
    innovate and ship software at scale it's
  • 00:04:07
    really hard when you get to that size
  • 00:04:10
    and and that's one of the reasons why
  • 00:04:12
    everyone should invest in productivity
  • 00:04:14
    engineering developer experience to keep
  • 00:04:15
    your top talent and keep your top
  • 00:04:17
    engineers more productive and that's why
  • 00:04:19
    we're here today wanted to share that
  • 00:04:21
    with the whole world
  • 00:04:23
    thank you adam uh for joining
  • 00:04:26
    there's also a gradle community
  • 00:04:29
    slack workspace it's open anybody can go
  • 00:04:32
    there there's a developer productivity
  • 00:04:34
    channel in there
  • 00:04:35
    um we're going to take we're not going
  • 00:04:37
    to get to all the questions right a
  • 00:04:40
    super hot topic everything that we were
  • 00:04:42
    chatting yesterday with that of
  • 00:04:44
    everything they talk about it's gonna be
  • 00:04:45
    super interesting there's gonna be a lot
  • 00:04:46
    of a lot of questions there so we're
  • 00:04:48
    gonna try to
  • 00:04:49
    take adam into that channel and try to
  • 00:04:52
    get him to answer as much as he can
  • 00:04:54
    right you know it's uh you know no no
  • 00:04:57
    promises already given us an hour which
  • 00:04:59
    is more than we can expect but anyway
  • 00:05:02
    adam why don't you uh tell us a little
  • 00:05:04
    bit about what you're working on
  • 00:05:08
    and your team
  • 00:05:10
    at uh at meta
  • 00:05:13
    well sure uh so i'm sure everybody's
  • 00:05:15
    familiar with meta or as people used to
  • 00:05:17
    call it facebook
  • 00:05:19
    uh i work on the facebook app
  • 00:05:21
    reliability and engineering team what
  • 00:05:24
    our big goal is is to minimize the
  • 00:05:26
    number of users who experience crashes
  • 00:05:28
    in a given day
  • 00:05:30
    and that sounds like a simple task but
  • 00:05:33
    when you think that across facebook
  • 00:05:35
    we've got 3.8 billion users uh you have
  • 00:05:38
    to do that at humanity scale uh and so
  • 00:05:41
    that's a question of
  • 00:05:44
    working with a lot of
  • 00:05:45
    different product teams working with a
  • 00:05:47
    whole lot of data
  • 00:05:49
    and trying to optimize build in that
  • 00:05:52
    kind of reliability and
  • 00:05:55
    good practice before the code ever
  • 00:05:58
    reaches the point where it's got to be
  • 00:06:00
    reviewed like we don't have the the
  • 00:06:02
    luxury of
  • 00:06:04
    a lot of soap time or long release
  • 00:06:06
    cycles
  • 00:06:07
    so we we live out we live in that um
  • 00:06:10
    world where you have to
  • 00:06:12
    build in quality my team specifically
  • 00:06:15
    spans a lot of products
  • 00:06:17
    and what we do is we work with our
  • 00:06:19
    product engineers to
  • 00:06:21
    establish those real
  • 00:06:23
    best practices and to help them
  • 00:06:25
    investigate when they mess something up
  • 00:06:27
    part of moving fast which is something
  • 00:06:29
    that we pride ourselves on
  • 00:06:31
    is that you're going to make mistakes
  • 00:06:32
    and so a big thing we do is try to help
  • 00:06:35
    you figure out where you've made
  • 00:06:36
    mistakes
  • 00:06:38
    and then avoid them in the in the future
  • 00:06:41
    i work specifically on android
  • 00:06:44
    for the facebook app
  • 00:06:46
    but our team is kind of the the masthead
  • 00:06:49
    of the reliability organizations so we
  • 00:06:52
    work heavily with the reliability groups
  • 00:06:54
    in instagram in whatsapp in messenger in
  • 00:06:58
    oculus to try and share a lot of that
  • 00:07:01
    knowledge because the facebook app
  • 00:07:03
    people maybe maybe don't realize that
  • 00:07:05
    facebook apps the most complex android
  • 00:07:07
    app there is
  • 00:07:08
    period
  • 00:07:10
    there isn't a more complex world to work
  • 00:07:13
    in in terms of the android space and so
  • 00:07:16
    it takes a certain amount of
  • 00:07:19
    hard work to get to the point where
  • 00:07:20
    you're where we can even release as fast
  • 00:07:22
    as we do
  • 00:07:23
    i was reading one of our dev blog posts
  • 00:07:25
    the other day and it and we used to
  • 00:07:28
    release monthly and it took the better
  • 00:07:30
    part of years
  • 00:07:32
    to get us to the point where we can
  • 00:07:34
    release
  • 00:07:35
    every week
  • 00:07:36
    and then we start butting up against the
  • 00:07:38
    actual infrastructure problems of our
  • 00:07:40
    release systems where google and the
  • 00:07:42
    oems and all the devices won't update
  • 00:07:45
    faster than that
  • 00:07:46
    like there's there's a point at which
  • 00:07:47
    we've pushed those needles as far as we
  • 00:07:49
    can possibly push them
  • 00:07:51
    and that owes to a lot of investment
  • 00:07:54
    when it comes to productivity and you
  • 00:07:57
    know dev
  • 00:07:59
    the actual dev life cycle great
  • 00:08:01
    introduction uh
  • 00:08:03
    one one question about you know just
  • 00:08:05
    releasing when you talk about release
  • 00:08:07
    every week are we talking about major
  • 00:08:08
    releases are we talking about minor
  • 00:08:10
    releases you know how how big are these
  • 00:08:13
    releases well let's put this way uh
  • 00:08:16
    on android alone
  • 00:08:18
    we do between 30 and 50 000 builds a day
  • 00:08:22
    and that's not an exaggeration that's a
  • 00:08:24
    solid number from two years ago it's
  • 00:08:26
    probably more now
  • 00:08:27
    uh
  • 00:08:28
    the
  • 00:08:30
    we do about that many builds
  • 00:08:33
    every particular every particular change
  • 00:08:36
    what we call diff
  • 00:08:37
    to our system
  • 00:08:39
    gets between three and five of those
  • 00:08:40
    builds on its way into the master branch
  • 00:08:43
    we block them at diff time we block them
  • 00:08:45
    at land time and then we roll them out
  • 00:08:48
    slowly into a master branch which we
  • 00:08:50
    then cut
  • 00:08:51
    alpha beta and prod from
  • 00:08:55
    so
  • 00:08:55
    over the course of a week a particular
  • 00:08:58
    revision
  • 00:08:59
    will go into an alpha which goes to a
  • 00:09:02
    lot of
  • 00:09:03
    internal users and a huge automated
  • 00:09:06
    testing community called sapiens
  • 00:09:08
    which is sort of simulated humans
  • 00:09:11
    we do a brick we do a beta cut and then
  • 00:09:13
    that sits for beta for one week
  • 00:09:16
    and then we push to our stores
  • 00:09:18
    that means we push to a bunch of the oem
  • 00:09:21
    stores where they pick up the actual
  • 00:09:24
    newest release and that goes directly
  • 00:09:26
    into their devices that they're selling
  • 00:09:28
    on a given week
  • 00:09:30
    it goes into the google play store and
  • 00:09:31
    it goes into a system we call oxygen
  • 00:09:34
    which is how we work with
  • 00:09:36
    uh android devices that don't have
  • 00:09:38
    access to google play services things
  • 00:09:40
    like amazon devices
  • 00:09:42
    and that lets us roll out those new
  • 00:09:44
    updates once a week
  • 00:09:46
    most oems won't let you update more than
  • 00:09:48
    once every seven days so we're really
  • 00:09:50
    talking about what we look at as a major
  • 00:09:52
    revision
  • 00:09:54
    every week
  • 00:09:55
    minor revisions
  • 00:09:57
    once a day to two or three times a day
  • 00:10:00
    in alpha
  • 00:10:01
    and then up to four and five times a
  • 00:10:03
    week in beta as we discover
  • 00:10:06
    major things that we won't don't want to
  • 00:10:08
    release without getting side events
  • 00:10:10
    and our team is hands-on through that
  • 00:10:12
    entire process we're the ones that
  • 00:10:14
    govern all of the check-ins that want to
  • 00:10:16
    go into beta and all of the hot fixes
  • 00:10:20
    should you ever need to hotfix prod
  • 00:10:23
    and that that's a very hands-on process
  • 00:10:25
    and it requires us to know
  • 00:10:27
    the entire universe of our app
  • 00:10:30
    to a pretty to a pretty deep extent
  • 00:10:33
    and that's
  • 00:10:34
    thanks for sharing that by the way and
  • 00:10:36
    that's a great segue kind of to the next
  • 00:10:38
    topic is
  • 00:10:40
    to
  • 00:10:41
    ship at that scale that frequently with
  • 00:10:44
    that those types of features and
  • 00:10:46
    cadences and whatnot you need support
  • 00:10:48
    you know it's not just it's you can't
  • 00:10:50
    just throw thousands of engineers at it
  • 00:10:54
    tell us about productivity engineering
  • 00:10:57
    uh reliability engineering and developer
  • 00:11:00
    experience kind of orgs behind it
  • 00:11:03
    supporting all these developers to
  • 00:11:05
    getting them to be that successful what
  • 00:11:07
    kind of investment are we talking about
  • 00:11:10
    how many folks are in those
  • 00:11:11
    organizations right and part of what i
  • 00:11:13
    want to share with the community is hey
  • 00:11:17
    there's a reason why facebook can ship
  • 00:11:19
    at this scale
  • 00:11:21
    and innovate this fast you can't just do
  • 00:11:24
    it with just throwing a thousand
  • 00:11:25
    engineers at it you need this investment
  • 00:11:27
    in productivity engineering what are we
  • 00:11:29
    looking at there from here so how is
  • 00:11:31
    so it's actually how is it structured
  • 00:11:33
    how many people are in it how does that
  • 00:11:34
    work at your organization so it's
  • 00:11:36
    actually funny that you say throw a
  • 00:11:37
    thousand engineers at it because that's
  • 00:11:39
    about the size of our dedicated
  • 00:11:42
    productivity engineering group
  • 00:11:45
    and i won't go into exact numbers
  • 00:11:46
    because it fluctuates quite a lot but
  • 00:11:48
    within our infrastructure org which is
  • 00:11:50
    30 000 of our 100 000 employees
  • 00:11:53
    there's a thousand people who we
  • 00:11:55
    dedicate just to facilitating
  • 00:11:58
    developers releasing code that's
  • 00:12:00
    everything from tools that we use for
  • 00:12:02
    testing to our mod our internal
  • 00:12:05
    modifications to all of our development
  • 00:12:07
    environments uh it's sort of an
  • 00:12:09
    interesting world because we run a huge
  • 00:12:12
    monorepo
  • 00:12:13
    we run a world where all the code for
  • 00:12:16
    facebook all of the code for messenger
  • 00:12:18
    all the code for instagram
  • 00:12:20
    and oculus and whatsapp and
  • 00:12:22
    a hundred other little products that
  • 00:12:24
    you've probably never heard of
  • 00:12:26
    all live in this one gigantic repo
  • 00:12:29
    right there's no world in which you can
  • 00:12:31
    build all of that or even download all
  • 00:12:33
    of that
  • 00:12:34
    on most machines hundreds and hundreds
  • 00:12:36
    of gigabytes of code
  • 00:12:39
    right so the kinds of investments that
  • 00:12:41
    we've had to make are everything from
  • 00:12:43
    rewriting our source control system
  • 00:12:47
    because it couldn't because git can't
  • 00:12:49
    handle
  • 00:12:50
    it just cannot handle a repo the size of
  • 00:12:53
    ours working with open source groups in
  • 00:12:55
    this case mercurial
  • 00:12:57
    to fundamentally rebuild
  • 00:12:59
    the source control systems that we use
  • 00:13:01
    and integrate with our internal build
  • 00:13:03
    system a system called buck that again
  • 00:13:06
    we had to rework because of the sheer
  • 00:13:08
    scale of what we're producing
  • 00:13:10
    and then all of these servers and
  • 00:13:12
    clients and systems
  • 00:13:15
    that make it so that you can reliably
  • 00:13:17
    build code
  • 00:13:19
    on your local system or even on what we
  • 00:13:21
    call on-demand servers which are
  • 00:13:23
    pre-warmed images that then you don't
  • 00:13:26
    have to download a lot of code onto you
  • 00:13:29
    start them up and they build
  • 00:13:32
    and
  • 00:13:32
    all of that is is
  • 00:13:36
    a huge investment that nobody else has
  • 00:13:38
    to make at some point
  • 00:13:41
    we have
  • 00:13:42
    a dedicated group who does nothing but
  • 00:13:44
    keep our system
  • 00:13:46
    running on what we call stable which
  • 00:13:48
    follows master by something like two
  • 00:13:50
    hours
  • 00:13:51
    and make sure that we are generating all
  • 00:13:53
    of the various artifacts that go with
  • 00:13:55
    that
  • 00:13:56
    so that when a
  • 00:13:57
    typical dev wants to do a build we're
  • 00:14:00
    only building the very very small parts
  • 00:14:02
    of the system that have to be rebuilt
  • 00:14:05
    for them and all of those other
  • 00:14:07
    artifacts are coming down
  • 00:14:09
    at 50 000 builds a day it is not
  • 00:14:12
    possible for you to follow master as
  • 00:14:14
    yourself you can't constantly keep up
  • 00:14:17
    with master
  • 00:14:18
    you can keep up with stable usually
  • 00:14:20
    because you know up to two hours lag
  • 00:14:24
    but even then you're never going to be
  • 00:14:26
    able to rebase onto master and hope to
  • 00:14:28
    ever merge it
  • 00:14:29
    so we have to have systems that do these
  • 00:14:31
    things automatically that do
  • 00:14:34
    merges when it's possible without you
  • 00:14:36
    having to intervene we have to have
  • 00:14:37
    automated testing we have to know
  • 00:14:39
    exactly how much of the code you touched
  • 00:14:42
    and which artifacts have to be changed
  • 00:14:44
    we have to know
  • 00:14:45
    all of these things just to be able to
  • 00:14:47
    check in one piece of code
  • 00:14:50
    now you add to that that we've got
  • 00:14:52
    thousands and thousands of developers
  • 00:14:53
    all working simultaneously in the same
  • 00:14:55
    repo
  • 00:14:57
    and all moving master even when it's not
  • 00:14:59
    your group or your uh specific piece of
  • 00:15:02
    the baseline
  • 00:15:04
    and the investment involved ends up
  • 00:15:06
    being the full-time job of whole orgs of
  • 00:15:09
    people
  • 00:15:11
    we've got direct we've got director
  • 00:15:12
    level
  • 00:15:14
    uh folks who are
  • 00:15:16
    solely responsible for organizing those
  • 00:15:18
    efforts and there's a reason we put it
  • 00:15:20
    into our infrastructure org
  • 00:15:22
    the 30 000 people in our infrastructure
  • 00:15:24
    or all have the same goal
  • 00:15:26
    and that is to facilitate
  • 00:15:29
    everybody who's building facebook to
  • 00:15:32
    deliver facebook and not just facebook
  • 00:15:34
    but whatsapp and messenger and instagram
  • 00:15:37
    and everything under the meta
  • 00:15:39
    umbrella
  • 00:15:42
    that's a that's 30 of our workforce
  • 00:15:45
    is completely dedicated to making
  • 00:15:48
    everybody else successful it's not one
  • 00:15:51
    guy it's not five guys
  • 00:15:53
    it's not just everybody kind of does a
  • 00:15:55
    piece of it and all the developers have
  • 00:15:57
    to know what they're doing it's
  • 00:15:59
    dedicated workforce
  • 00:16:01
    nice no that's a
  • 00:16:03
    great kind of overview of
  • 00:16:05
    what it takes right and of that scale
  • 00:16:09
    right uh
  • 00:16:10
    you have to build your own in many cases
  • 00:16:13
    to because it hasn't been done before um
  • 00:16:16
    we're getting some good questions justin
  • 00:16:18
    around
  • 00:16:19
    uh building and testing before we get to
  • 00:16:23
    those maybe we jump into this next topic
  • 00:16:25
    of
  • 00:16:26
    wha what's it like in the day in the
  • 00:16:28
    life of a meta software engineer working
  • 00:16:30
    on the android uh the facebook android
  • 00:16:32
    app right yeah they write code
  • 00:16:35
    how do they write builds and tests cicd
  • 00:16:39
    well so here's here's a
  • 00:16:41
    let's take a section out of more of a
  • 00:16:43
    product engineer side of things because
  • 00:16:46
    on reliability on the reliability
  • 00:16:48
    engineering side our day is atypical we
  • 00:16:51
    are in we we do more investigation than
  • 00:16:53
    we do writing code
  • 00:16:55
    we write enough code but when you're
  • 00:16:57
    going to look at when we're looking at
  • 00:17:00
    delivering a piece of the system
  • 00:17:03
    we have
  • 00:17:04
    there's a couple different ways one is
  • 00:17:06
    you can build the code locally you can
  • 00:17:08
    write whatever changes you want deploy
  • 00:17:10
    them to a phone
  • 00:17:12
    and it works
  • 00:17:14
    that that is
  • 00:17:16
    facilitated by the fact that we have a
  • 00:17:19
    customized version of android studio
  • 00:17:21
    that lets you bring in all of these
  • 00:17:24
    artifacts using buck
  • 00:17:26
    we have dedicated configurations of that
  • 00:17:30
    system that bring in all of our lint
  • 00:17:32
    tools because normal linters can't
  • 00:17:34
    handle our code that bring in our
  • 00:17:36
    compilation tools that bring in all of
  • 00:17:39
    our handwritten
  • 00:17:41
    uh
  • 00:17:43
    handwritten modifications of the ast
  • 00:17:47
    and then also integrate with
  • 00:17:50
    an emulator in a way that doesn't break
  • 00:17:52
    facebook
  • 00:17:54
    it's
  • 00:17:55
    because facebook is such a complex
  • 00:17:57
    application it's actually pretty
  • 00:17:59
    difficult to run an emulator that will
  • 00:18:01
    support facebook
  • 00:18:03
    and not crash
  • 00:18:04
    and we're and we're adding in the
  • 00:18:06
    complexity of a bunch of debug
  • 00:18:09
    information and non-optimized builds and
  • 00:18:11
    not minimized code size and that pushes
  • 00:18:14
    the boundaries of what a lot of devices
  • 00:18:16
    even new devices are really capable of
  • 00:18:18
    running
  • 00:18:20
    so as a dev you might
  • 00:18:22
    take a take a new feature you're trying
  • 00:18:23
    to build
  • 00:18:24
    you might build that locally or you
  • 00:18:26
    might check out what we call an
  • 00:18:27
    on-demand server
  • 00:18:29
    an on-demand server is basically
  • 00:18:31
    pre-warmed code
  • 00:18:33
    where you can do whatever development
  • 00:18:34
    you need either mostly in uh our again
  • 00:18:38
    customized version of vs code
  • 00:18:41
    that
  • 00:18:42
    you would then build whatever you need
  • 00:18:44
    to do be able to run your test locally
  • 00:18:46
    run as run your android in server mode
  • 00:18:50
    and then run whatever tests against it
  • 00:18:52
    we have a whole end-to-end testing
  • 00:18:53
    framework wherein any individual test
  • 00:18:56
    can be run against your code or suites
  • 00:18:58
    of tests
  • 00:18:59
    on the server side without explicitly
  • 00:19:01
    having to run an emulator
  • 00:19:04
    because again there's just so much
  • 00:19:06
    infrastructure if every individual had
  • 00:19:08
    to run an emulator we'd have to buy
  • 00:19:10
    everybody mac pros
  • 00:19:11
    and that is you know not really feasible
  • 00:19:14
    but while we're running them all on
  • 00:19:16
    linux
  • 00:19:17
    we can split off our relatively sizable
  • 00:19:19
    infrastructure and all of these
  • 00:19:22
    uh
  • 00:19:23
    all of these these uh
  • 00:19:25
    data warehouses and server
  • 00:19:27
    infrastructures that we have and we can
  • 00:19:29
    shard those out to individual devs now
  • 00:19:32
    something that's interesting i used to
  • 00:19:33
    work at amazon
  • 00:19:34
    uh at amazon every team has to own their
  • 00:19:37
    own machines right you have to know
  • 00:19:40
    about each of your instances in each of
  • 00:19:42
    your clusters and you have to set up
  • 00:19:44
    your own pipelines at facebook
  • 00:19:47
    there's the trade-off of well you don't
  • 00:19:48
    get as much control as you get when you
  • 00:19:51
    own all of your own machines
  • 00:19:53
    but in exchange for that you don't have
  • 00:19:54
    to worry about it if i want a new if i i
  • 00:19:57
    need to work on three projects at once i
  • 00:19:59
    can pull three individual
  • 00:20:02
    on-demand instances
  • 00:20:04
    and they'll get pre-warmed either with a
  • 00:20:06
    diff in progress uh diff being what we
  • 00:20:08
    use
  • 00:20:09
    to talk about a change request or a code
  • 00:20:13
    review
  • 00:20:15
    in progress
  • 00:20:16
    i can pull a diff down i can run it in
  • 00:20:18
    and on demand i can put doesn't matter
  • 00:20:20
    whether it was mine or not
  • 00:20:22
    and that takes you know two three
  • 00:20:24
    minutes as opposed to the hours that it
  • 00:20:27
    takes to download our entire baseline
  • 00:20:29
    when i was in boot camp the
  • 00:20:32
    first five weeks when you're employed at
  • 00:20:34
    facebook is kind of a boot camp to get
  • 00:20:36
    you familiar with all of this tech
  • 00:20:38
    uh it took me eight hours
  • 00:20:41
    to download and compile the android
  • 00:20:44
    baseline that's because i didn't know
  • 00:20:46
    what i was doing and i wasn't leveraging
  • 00:20:48
    any of this
  • 00:20:49
    infrastructure but eight solid hours
  • 00:20:52
    doing nothing but downloading and
  • 00:20:54
    compiling
  • 00:20:55
    where when we compile with the normal
  • 00:20:57
    system five ten minutes
  • 00:21:00
    and you've got it and you've got a
  • 00:21:01
    working system when you pull from a bit
  • 00:21:03
    from the stable baseline
  • 00:21:05
    if you pull it raw
  • 00:21:06
    zero you press go and it just comes up
  • 00:21:10
    because if you don't have changes we
  • 00:21:12
    don't have to rebuild and all those
  • 00:21:13
    artifacts already exist right
  • 00:21:15
    so you've you've built your code you've
  • 00:21:17
    either built it locally and you've run
  • 00:21:18
    it on a hand on your own device
  • 00:21:21
    or you've built it on demand well so
  • 00:21:23
    then you push it
  • 00:21:24
    now we have a system that we call diffs
  • 00:21:27
    uh again completely homegrown because
  • 00:21:30
    everything we do is is just a little bit
  • 00:21:33
    uh more complicated
  • 00:21:36
    because sometimes we run what we call
  • 00:21:38
    code mods and you can have thousands and
  • 00:21:40
    thousands of files touched
  • 00:21:41
    right so that breaks almost every online
  • 00:21:46
    code revision system there is
  • 00:21:48
    so we have our own which we call diffs
  • 00:21:51
    and diffs then can be reviewed by any
  • 00:21:54
    number of people
  • 00:21:55
    um
  • 00:21:56
    every piece of code that a dev touches
  • 00:21:59
    gets reviewed
  • 00:22:00
    uh and that it and we've actually added
  • 00:22:02
    an additional step now where we call
  • 00:22:04
    what we call final review where if you
  • 00:22:06
    modify your code after it's been
  • 00:22:08
    accepted
  • 00:22:09
    you it has to be re-reviewed before it's
  • 00:22:12
    considered okay to have in the baseline
  • 00:22:15
    when you submit a diff
  • 00:22:17
    in the background the infrastructure is
  • 00:22:19
    already going to run a bunch of basic
  • 00:22:21
    tests
  • 00:22:22
    it's going to check linting standards
  • 00:22:24
    it's going to check whether all the all
  • 00:22:26
    of the products that you touched compile
  • 00:22:28
    it's going to check whether you are
  • 00:22:30
    significantly affecting various
  • 00:22:32
    performance metrics and it's going to
  • 00:22:34
    run a basic suite of tests
  • 00:22:37
    then once you have approval and
  • 00:22:39
    someone's reviewed all of your code and
  • 00:22:40
    you've gone back and forth and you've
  • 00:22:42
    done all of the all of the niceties of
  • 00:22:44
    code of code review
  • 00:22:47
    you go to land what we call land uh
  • 00:22:50
    that's the process by which you go from
  • 00:22:51
    a diff to a merge
  • 00:22:53
    right
  • 00:22:54
    and so we've got another more strenuous
  • 00:22:57
    group of tests at that point where then
  • 00:22:59
    we're doing a full compile and we're
  • 00:23:01
    making sure you don't regress app size
  • 00:23:02
    then we're doing
  • 00:23:04
    smoke tests across our entire fleet of
  • 00:23:06
    apps to make sure that you didn't break
  • 00:23:08
    a bunch of other applications
  • 00:23:12
    that's when we're making sure that you
  • 00:23:14
    are significantly regressing production
  • 00:23:16
    metrics as we push that into the into
  • 00:23:19
    the master
  • 00:23:20
    then you've landed that process can take
  • 00:23:22
    anywhere from an hour to four hours and
  • 00:23:24
    we have ways of making that faster
  • 00:23:26
    when we have reason to when we're in the
  • 00:23:28
    midst of a sev when we're trying to fix
  • 00:23:30
    something or move very quickly
  • 00:23:32
    but
  • 00:23:33
    you got to think we're building all of
  • 00:23:35
    this infrastructure and then we're
  • 00:23:36
    running tests against it just like you'd
  • 00:23:38
    run tests on an emulator
  • 00:23:40
    right
  • 00:23:41
    and at 50 000 of those a day
  • 00:23:45
    that's a huge amount of processing power
  • 00:23:48
    that we're throwing at the problem of
  • 00:23:50
    speeding up
  • 00:23:51
    this end to end diff to baseline
  • 00:23:54
    now most of the time that diff to
  • 00:23:56
    baseline is going to run around two
  • 00:23:58
    hours
  • 00:23:59
    but that means that for a
  • 00:24:01
    modest change especially if your team
  • 00:24:03
    already knew about it and the people who
  • 00:24:05
    were affected by it already kind of knew
  • 00:24:07
    it was coming
  • 00:24:08
    you can post a diff
  • 00:24:11
    make a chat make the change get it
  • 00:24:13
    reviewed and deliver
  • 00:24:15
    in two or three
  • 00:24:16
    hours and then we start going live right
  • 00:24:20
    so on the android side
  • 00:24:22
    uh we go we we split out uh alpha
  • 00:24:25
    revision at least once a day often two
  • 00:24:27
    and three times a day depending on
  • 00:24:30
    how
  • 00:24:31
    much how broken the system is right now
  • 00:24:33
    because again we move fast things break
  • 00:24:36
    and we we try to make sure that the most
  • 00:24:38
    recent version of the alpha build is
  • 00:24:40
    working
  • 00:24:41
    at least mostly working because it is
  • 00:24:44
    alpha after all every week we cut a beta
  • 00:24:48
    and then we cherry pick into that beta
  • 00:24:50
    to fix things that are wrong with the
  • 00:24:53
    beta
  • 00:24:53
    and then a week after that we released a
  • 00:24:55
    prod we cut the beta and the prod on the
  • 00:24:57
    same day
  • 00:24:59
    generally
  • 00:25:00
    there are places where we've started
  • 00:25:01
    short of that and the distance from beta
  • 00:25:03
    to prod is
  • 00:25:06
    product to product now but anywhere from
  • 00:25:08
    three days to seven days
  • 00:25:10
    and then once it's in prod that goes up
  • 00:25:12
    to the store and we do a progressive
  • 00:25:14
    rollout one percent 10 100
  • 00:25:18
    now we have a breakdown in that it's not
  • 00:25:20
    quite as simple as rolling all of
  • 00:25:22
    facebook out but that's
  • 00:25:24
    you know that's that's way down in the
  • 00:25:26
    weeds
  • 00:25:27
    but you can see at each of these steps
  • 00:25:29
    it's always gradual it's always
  • 00:25:32
    deliberate every single one of these
  • 00:25:34
    releases is a little bit of a canary
  • 00:25:37
    check the metrics a little bit more
  • 00:25:39
    check all the metrics a little bit more
  • 00:25:42
    check all the metrics
  • 00:25:44
    and at every one of these steps every
  • 00:25:45
    one of these branch cuts when we're
  • 00:25:47
    going to prod
  • 00:25:49
    we've got a lot of
  • 00:25:51
    instrumentation we know exactly how
  • 00:25:53
    often users are crashing we know exactly
  • 00:25:57
    how many stalls there are how many cpu
  • 00:25:59
    spins whether various top line product
  • 00:26:02
    metrics are changing for these users
  • 00:26:05
    and that's why it's critical to have
  • 00:26:07
    these people who are participating in
  • 00:26:08
    our betas and our alphas
  • 00:26:10
    because otherwise you're guessing and we
  • 00:26:13
    don't guess we really don't if we had no
  • 00:26:15
    data we would we would not move
  • 00:26:19
    every single week we have these meetings
  • 00:26:20
    and they're entirely driven by all of
  • 00:26:23
    these measurements and we
  • 00:26:25
    we stick to them until we can just
  • 00:26:28
    until we can actually explain a
  • 00:26:31
    discrepancy we don't ship
  • 00:26:33
    that's
  • 00:26:34
    i mean it's uh
  • 00:26:36
    that's scale i mean this is what you
  • 00:26:38
    need these kind of processes in these
  • 00:26:39
    tools one thing that resonated really
  • 00:26:41
    well with me was when you mentioned hey
  • 00:26:43
    when you first started you try to just
  • 00:26:45
    download and compile this it took eight
  • 00:26:47
    hours
  • 00:26:48
    but then after with facebook's
  • 00:26:51
    investment in these acceleration
  • 00:26:53
    technologies and and tooling they're
  • 00:26:56
    able to do it in five minutes which is
  • 00:26:59
    you know why we brought you here today
  • 00:27:01
    to say hey if you're trying to do things
  • 00:27:03
    the normal way at that scale
  • 00:27:06
    good luck shipping you know good luck
  • 00:27:08
    trying to compete with facebook right
  • 00:27:10
    without investment and in the crazy like
  • 00:27:14
    hearing these stories and kind of just
  • 00:27:15
    seeing it made real you know the
  • 00:27:17
    techniques that you're using i loved
  • 00:27:19
    hearing about that like kind of like
  • 00:27:21
    slicing a little bit more canary at a
  • 00:27:23
    time as opposed to like
  • 00:27:25
    full canary it's like no no no we're
  • 00:27:27
    just introducing a little bit it's
  • 00:27:29
    almost like like like features kind of
  • 00:27:31
    trickling
  • 00:27:32
    um let's take can we take a quick break
  • 00:27:34
    here and kind of hit a few points let's
  • 00:27:36
    say we have so many justin i don't know
  • 00:27:38
    what we're going to get to and we're not
  • 00:27:40
    going to get to them yes um a specific
  • 00:27:43
    one that came in was just a little bit
  • 00:27:45
    more clarity around the sapient system
  • 00:27:47
    that you mentioned uh just just to
  • 00:27:49
    clarify you do mean that these are
  • 00:27:51
    simulated human users yeah and and
  • 00:27:53
    wondering what technology that's based
  • 00:27:55
    on specifically is it based on monkey
  • 00:27:57
    like the chaos well
  • 00:27:58
    so uh and that's the thing is
  • 00:28:01
    i'm sure that the name was originally a
  • 00:28:03
    little bit of a play on chaos monkey
  • 00:28:05
    there's a lot of there's a lot of
  • 00:28:06
    cross-pollination between netflix and
  • 00:28:08
    facebook and all the other
  • 00:28:10
    manga companies
  • 00:28:12
    but the big thing about sapiens right is
  • 00:28:14
    that we have what we call qpl markers
  • 00:28:16
    and qpal markers stands for quick
  • 00:28:18
    performance logger has nothing to do
  • 00:28:20
    with quick or performance anymore uh but
  • 00:28:23
    the point is that what they do is they
  • 00:28:24
    record human action
  • 00:28:26
    when you interact with our application
  • 00:28:28
    in various ways
  • 00:28:30
    we anonymize that data
  • 00:28:32
    and then we can aggregate
  • 00:28:36
    who you how much of which parts of the
  • 00:28:39
    app are used
  • 00:28:41
    and so we take the most firing qpls and
  • 00:28:43
    the biggest workflows
  • 00:28:45
    and we've taught a simulated human
  • 00:28:48
    system and this these aren't humans
  • 00:28:50
    it is a
  • 00:28:51
    effectively an end-to-end test
  • 00:28:53
    that runs against an emulator
  • 00:28:55
    but it's not scripted
  • 00:28:59
    it's trying to tie together all of those
  • 00:29:01
    qpl markers
  • 00:29:03
    and it learns how to walk through them
  • 00:29:05
    and simulate the way a human would use
  • 00:29:08
    the application
  • 00:29:10
    and by doing that we're able to take our
  • 00:29:12
    alpha
  • 00:29:14
    which is a very small group of users
  • 00:29:16
    that's really employees
  • 00:29:18
    uh who have opted to take this somewhat
  • 00:29:20
    broken baseline that is
  • 00:29:23
    really actually often quite broken
  • 00:29:26
    and
  • 00:29:29
    increase the number of users and the
  • 00:29:31
    number of hours used and really we think
  • 00:29:33
    of this in terms of the number of hours
  • 00:29:35
    interacted with when we're talking about
  • 00:29:37
    these alpha releases
  • 00:29:39
    and up it by an order of magnitude
  • 00:29:42
    actually in in the case of what we call
  • 00:29:43
    virtual alpha we're up to three orders
  • 00:29:46
    of magnitude we have a thousand times as
  • 00:29:48
    many hours as we
  • 00:29:50
    from our sapien system as we have from
  • 00:29:53
    humans
  • 00:29:54
    on the on our alpha builds and without
  • 00:29:56
    that
  • 00:29:58
    there are so many features that are
  • 00:29:59
    never used by employees
  • 00:30:01
    or just not used enough
  • 00:30:03
    to drive out bugs and and these changes
  • 00:30:07
    with the sapien system we've been able
  • 00:30:08
    to
  • 00:30:09
    find a lot of those regressions
  • 00:30:12
    far earlier than we would ever have a
  • 00:30:13
    chance to if we waited for our beta
  • 00:30:15
    population and we've got a lot of beta
  • 00:30:17
    we have a public beta that folks can
  • 00:30:19
    sign up for from the google play store
  • 00:30:22
    and that allows you to see some of this
  • 00:30:24
    stuff a little early
  • 00:30:26
    uh it's a little bit crashier than than
  • 00:30:28
    the production version because it is our
  • 00:30:30
    soak in our
  • 00:30:31
    mature it's the maturation phase of the
  • 00:30:34
    app but every week
  • 00:30:36
    we're shipping that and that's where we
  • 00:30:39
    get a
  • 00:30:40
    the lion's share of our of our um
  • 00:30:45
    i don't know i don't know exactly what
  • 00:30:46
    the word is the polish the polish of on
  • 00:30:49
    the on the end of the app but without
  • 00:30:51
    the alpha without sapiens running on
  • 00:30:53
    alpha
  • 00:30:54
    that beta build
  • 00:30:56
    would have to take
  • 00:30:57
    weeks
  • 00:30:58
    because we just don't have the ability
  • 00:31:01
    to force
  • 00:31:03
    people to use the alpha
  • 00:31:05
    build
  • 00:31:06
    and we wouldn't want to at some point
  • 00:31:09
    our goal
  • 00:31:10
    is to try and make a clean experience we
  • 00:31:13
    goal on things like touch responsiveness
  • 00:31:16
    on how fast the app comes up
  • 00:31:18
    on the number of users who ever
  • 00:31:20
    experience a foreground app death the
  • 00:31:22
    number of warm starts which is to say
  • 00:31:25
    how good we are using memory and thus
  • 00:31:27
    how
  • 00:31:28
    infrequently we are killed by the
  • 00:31:30
    operating systems like our goal comes
  • 00:31:33
    down to sad users and bad experiences
  • 00:31:36
    almost 100 percent
  • 00:31:37
    there's one other thing several people
  • 00:31:39
    are wondering about how you handle
  • 00:31:40
    rollbacks in this process and what that
  • 00:31:42
    looks like
  • 00:31:44
    well so i think what we would talk about
  • 00:31:46
    we actually call it unlanding um so
  • 00:31:50
    just the way the same way you would
  • 00:31:51
    revert in a git repo you check in a
  • 00:31:53
    revert commit
  • 00:31:55
    and then those land just like anything
  • 00:31:58
    else
  • 00:31:59
    but the trick is is that because we're
  • 00:32:01
    working on a monorepo and it is
  • 00:32:03
    effectively linear we don't really have
  • 00:32:05
    a proper branching model except for the
  • 00:32:07
    cuts
  • 00:32:08
    for releases
  • 00:32:11
    we roll everything on the end of master
  • 00:32:13
    so if you have a change that needs to be
  • 00:32:16
    reverted we build a inverse change and
  • 00:32:19
    we land that
  • 00:32:21
    now that can get tedious because again
  • 00:32:23
    the master the the tip of master moves
  • 00:32:25
    really fast so again this is a place
  • 00:32:28
    where our productivity engineers have
  • 00:32:29
    come in and helped us build these
  • 00:32:31
    systems that automatically create these
  • 00:32:33
    things
  • 00:32:34
    uh when your
  • 00:32:35
    tech if you have a particular revision
  • 00:32:38
    that breaks a lot of tests even if you
  • 00:32:40
    didn't notice that we're doing something
  • 00:32:43
    we call multisect which will find which
  • 00:32:45
    revision broke a particular
  • 00:32:49
    set of tests on our master branch and it
  • 00:32:52
    will proactively generate revert diffs
  • 00:32:55
    where all all anybody has to do is hit
  • 00:32:58
    go
  • 00:32:59
    and your diff on lands
  • 00:33:01
    it's not
  • 00:33:03
    there's a problem in in a lot of
  • 00:33:05
    uh software that
  • 00:33:07
    is kind of addressed with the monorepo
  • 00:33:09
    by saying
  • 00:33:10
    look breaking changes are going to
  • 00:33:11
    happen there's going to be times when
  • 00:33:14
    the sequence of of changes in master are
  • 00:33:18
    not going to be usable that they're just
  • 00:33:20
    going to be breaking changes
  • 00:33:22
    we don't try to create the fiction that
  • 00:33:24
    master was always stable and that master
  • 00:33:26
    was perfect we don't try to revert out
  • 00:33:28
    the history we don't try to get rid of
  • 00:33:30
    those changes we keep them we keep
  • 00:33:32
    everything
  • 00:33:33
    and we just have this very very long
  • 00:33:35
    linear path that is our master branch
  • 00:33:38
    and so when you break something we
  • 00:33:40
    reverted that we revert out only the
  • 00:33:42
    parts that you touched in a later
  • 00:33:43
    revision
  • 00:33:44
    and that means that every build after
  • 00:33:46
    that since everything is based on all
  • 00:33:49
    these stable artifacts and nobody is you
  • 00:33:52
    know building these long-term branches
  • 00:33:53
    that are based on some ridiculously old
  • 00:33:56
    version of master
  • 00:33:58
    we're constantly doing rebasing we're
  • 00:34:00
    taking everybody's diffs and we're
  • 00:34:02
    constantly bringing them up we're
  • 00:34:04
    constantly moving them forward we don't
  • 00:34:06
    even allow people to land if their syste
  • 00:34:09
    if their current diff is based on a
  • 00:34:11
    master that we don't have those
  • 00:34:14
    artifacts for it's not worth the
  • 00:34:16
    compilation time to try and do that
  • 00:34:18
    everything gets rebased and up and moved
  • 00:34:21
    up to stable versions
  • 00:34:23
    uh one of the beauties of the on-demand
  • 00:34:24
    system that i mentioned is if you pull a
  • 00:34:27
    diff down in an on-demand
  • 00:34:29
    it is rebased as part of that process so
  • 00:34:32
    the first thing you do when that on
  • 00:34:33
    demand comes up is fix any conflicts in
  • 00:34:37
    the rebase which usually there aren't
  • 00:34:39
    many unless you are working on a very
  • 00:34:41
    hot piece of the system
  • 00:34:44
    but that means that as we move forward
  • 00:34:45
    we revert by fixing
  • 00:34:48
    effectively
  • 00:34:50
    and if you've got a fix that can do it
  • 00:34:52
    faster than the revert or that's smaller
  • 00:34:54
    than the revert you fix forward
  • 00:34:57
    and that
  • 00:34:58
    the difference between the two is only a
  • 00:35:00
    question of whether we know that the fix
  • 00:35:02
    is going to revert what you did
  • 00:35:04
    so we live in the we live in this world
  • 00:35:06
    of
  • 00:35:07
    if you regress the metrics we have to
  • 00:35:09
    grow back but also
  • 00:35:11
    we don't want to be in the business of
  • 00:35:14
    everybody doing that manually
  • 00:35:16
    that's i mean
  • 00:35:18
    it's like straightforward hearing the
  • 00:35:19
    answers but then it's like you know you
  • 00:35:21
    think about these problems with scale
  • 00:35:22
    and then it's like okay
  • 00:35:24
    what else are you going to do so anyway
  • 00:35:26
    yeah rumors maybe we go back to the
  • 00:35:29
    so we got three topics that we want to
  • 00:35:31
    go go into and we got
  • 00:35:34
    20 minutes left we're usually we usually
  • 00:35:36
    wrap up at this point and go into q and
  • 00:35:38
    a but we haven't even gotten to our
  • 00:35:40
    favorite topics of productivity
  • 00:35:41
    engineering metrics and whatnot uh
  • 00:35:44
    before we get into that adam in in two
  • 00:35:47
    minutes
  • 00:35:48
    can you talk about the highlights of the
  • 00:35:50
    tooling stack
  • 00:35:52
    you mentioned buck and the you know buck
  • 00:35:55
    two is now you know buck one didn't
  • 00:35:57
    scale up get to have buck two and and
  • 00:36:00
    you know customizing mercurial for for
  • 00:36:03
    version control system
  • 00:36:05
    in two minutes give us you know
  • 00:36:07
    what what other you know highlights of
  • 00:36:09
    the tool i mean i know
  • 00:36:10
    we wanted to go through all the tools
  • 00:36:12
    and probably you should have another uh
  • 00:36:14
    webcast just for that but uh what can
  • 00:36:17
    you give us in two minutes highlights
  • 00:36:19
    well so the biggest thing is
  • 00:36:21
    we have a customized version of vs code
  • 00:36:24
    and it let us integrate all of our
  • 00:36:26
    internal tooling that includes custom
  • 00:36:28
    auth
  • 00:36:29
    uh buck integration uh something we call
  • 00:36:32
    interactive interactive um
  • 00:36:35
    was it
  • 00:36:36
    state logger isl whatever isl stood for
  • 00:36:39
    in our universe
  • 00:36:40
    but that lets us look at branches and
  • 00:36:42
    look at revisions and pull diffs
  • 00:36:45
    uh we have a system called jellyfish
  • 00:36:47
    that is sort of the link between
  • 00:36:50
    uh code revisions and
  • 00:36:52
    diffs and that had to be again custom
  • 00:36:55
    built we've got mercurial that actually
  • 00:36:57
    is tracking the data
  • 00:36:59
    we've got
  • 00:37:00
    customized version of android studio
  • 00:37:02
    that does our actual android builds
  • 00:37:04
    using buck but also understands enough
  • 00:37:07
    about the android ecosystem to let you
  • 00:37:09
    uh debug
  • 00:37:11
    those builds when they go onto a device
  • 00:37:14
    we have on-demand servers
  • 00:37:15
    which are the
  • 00:37:17
    bread and butter of our development
  • 00:37:19
    process
  • 00:37:20
    then you've got all of the other
  • 00:37:22
    ephemera that goes around this stuff so
  • 00:37:24
    this is so then you get into the really
  • 00:37:25
    neat stuff when we do a code search
  • 00:37:28
    in our repo because everybody wants to
  • 00:37:30
    be able to search from code all of our
  • 00:37:32
    code searches are directly integrated
  • 00:37:34
    with those development tools
  • 00:37:37
    so if you're looking at a piece of code
  • 00:37:39
    on the web there's a button that'll take
  • 00:37:41
    you to that same piece of code in vs
  • 00:37:42
    code
  • 00:37:44
    if you're looking at code inside of our
  • 00:37:46
    error reporting tools or inside of our
  • 00:37:50
    search tools or data analytics tools
  • 00:37:53
    you can link directly to that line of
  • 00:37:54
    code in your editor
  • 00:37:57
    and if you've got it on demand up it'll
  • 00:38:00
    be the version of that that's in the on
  • 00:38:02
    demand
  • 00:38:03
    so i mean all taken all together what
  • 00:38:05
    this means is that we don't have 57
  • 00:38:07
    workflows for 57 different tools you
  • 00:38:10
    have the way you do business and we've
  • 00:38:12
    tried to enable as many of those as we
  • 00:38:15
    possibly can
  • 00:38:16
    which just lowers the friction of the
  • 00:38:18
    whole system
  • 00:38:19
    uh jumping into productivity engineering
  • 00:38:22
    and developer happiness
  • 00:38:24
    how does
  • 00:38:26
    meta and maybe specifically your team
  • 00:38:29
    write measure productivity engineering
  • 00:38:31
    and happiness what what data are you
  • 00:38:34
    looking at
  • 00:38:35
    um
  • 00:38:36
    and what
  • 00:38:37
    where does that data come from
  • 00:38:40
    yeah so let's start with really simple
  • 00:38:41
    stuff um we have a
  • 00:38:44
    developer-wide survey that we call pulse
  • 00:38:46
    now everybody does developer surveys and
  • 00:38:49
    they're basically nonsense for most
  • 00:38:51
    people
  • 00:38:52
    right but pulsate meta is different
  • 00:38:55
    we
  • 00:38:56
    actually ask people do you like the way
  • 00:38:59
    that the your the
  • 00:39:01
    tools we have are serving their purposes
  • 00:39:04
    and then we give those that feedback
  • 00:39:07
    directly to those tools team and those
  • 00:39:08
    become actionable we ask people
  • 00:39:11
    why that why they're
  • 00:39:13
    not being able to do things the way they
  • 00:39:15
    want to and then we move those systems
  • 00:39:18
    based on those numbers
  • 00:39:19
    we take our
  • 00:39:21
    broad surveys as the core goaling metric
  • 00:39:24
    of our entire management organization
  • 00:39:26
    they're not gold on features delivered
  • 00:39:28
    they're not gold on how many people do
  • 00:39:30
    how much code or any of that their gold
  • 00:39:32
    on on the pulse on the the literal pulse
  • 00:39:35
    of the system right
  • 00:39:37
    and then throughout the haves
  • 00:39:39
    throughout the half throughout the
  • 00:39:40
    quarter we do what we call micro pulse
  • 00:39:42
    which is just little check-ins
  • 00:39:44
    how how is how are things looking today
  • 00:39:46
    was today better than yesterday was this
  • 00:39:48
    week better than last week
  • 00:39:50
    how how is this looking has the have we
  • 00:39:52
    improved on this that is our thing we're
  • 00:39:55
    trying to work on this half
  • 00:39:57
    constant influx of data right
  • 00:40:00
    but we know that when you ask somebody
  • 00:40:02
    something in a survey you get conflated
  • 00:40:04
    results every time people are going to
  • 00:40:06
    conflate i'm happy with my job with the
  • 00:40:09
    tools are good
  • 00:40:11
    so we're also looking at ways of trying
  • 00:40:13
    to lower the friction of the day-to-day
  • 00:40:15
    development experience we have a bot
  • 00:40:18
    called balancebot which monitors how
  • 00:40:20
    often you're getting notified
  • 00:40:22
    how often things are flowing into your
  • 00:40:25
    your queue be that on our internal chat
  • 00:40:28
    app
  • 00:40:29
    you'll notice the theme here
  • 00:40:30
    everything's everything's custom our
  • 00:40:32
    internal chat app called workplace
  • 00:40:34
    our internal
  • 00:40:36
    uh social media system which
  • 00:40:38
    unlike most companies we actually
  • 00:40:40
    use it as our core communication
  • 00:40:42
    framework whether you're getting too
  • 00:40:44
    many notifications on that whether
  • 00:40:46
    you're getting too many notifications
  • 00:40:47
    via email whether you're
  • 00:40:50
    getting too much getting too many
  • 00:40:51
    reminders set from variant from various
  • 00:40:54
    sources
  • 00:40:55
    and because of this we can it actually
  • 00:40:57
    will turn the tune those down
  • 00:41:00
    we've set it up so that if you've joined
  • 00:41:01
    a lot of groups it'll ask you do you
  • 00:41:03
    want me to just mute these for you you
  • 00:41:06
    you don't really interact with these a
  • 00:41:07
    lot do you want me just turn off these
  • 00:41:08
    notifications for you
  • 00:41:10
    and it try and that automatically sort
  • 00:41:13
    of does a lot of the housekeeping stuff
  • 00:41:15
    that you do for yourself now i don't
  • 00:41:17
    want to hear from this one i want to
  • 00:41:18
    hear from this one i never look at these
  • 00:41:21
    we don't lose any of that productivity
  • 00:41:23
    because we've got a bot that's doing it
  • 00:41:24
    for us
  • 00:41:27
    we then add what we call focus bot which
  • 00:41:29
    is to say we watch
  • 00:41:32
    when our users are not listening to
  • 00:41:34
    their notifications
  • 00:41:36
    not immediately responding in their
  • 00:41:38
    notifications and when they're actively
  • 00:41:40
    working
  • 00:41:41
    and we start
  • 00:41:43
    muting
  • 00:41:44
    those notifications and we let them work
  • 00:41:47
    and we'll establish what we call focus
  • 00:41:49
    blocks
  • 00:41:50
    where if you are really in the groove
  • 00:41:53
    and you're starting to code and you're
  • 00:41:55
    just
  • 00:41:56
    doing the work and you're not focusing
  • 00:41:58
    on any of that stuff
  • 00:42:00
    the volume of all that stuff just goes
  • 00:42:02
    down
  • 00:42:03
    and more than that it communicates to
  • 00:42:04
    others who try to contact you that you
  • 00:42:07
    are in a focus block
  • 00:42:09
    and that maybe they ought to not maybe
  • 00:42:10
    they shouldn't expect a response right
  • 00:42:12
    now because
  • 00:42:14
    you're busy you're doing work
  • 00:42:17
    and what that does is it makes it so
  • 00:42:19
    that doing the right thing is easy
  • 00:42:22
    doing
  • 00:42:22
    software the way that is effective is
  • 00:42:26
    easy
  • 00:42:27
    it's harder to make sure that you're
  • 00:42:29
    answering every email it's harder to
  • 00:42:31
    make sure that you are constantly on the
  • 00:42:34
    the social media or the chat system
  • 00:42:37
    trying to talk to people
  • 00:42:39
    you can still do it
  • 00:42:41
    but
  • 00:42:42
    it emphasizes the the work of building
  • 00:42:45
    the system
  • 00:42:46
    over
  • 00:42:47
    those distractions which is fantastic
  • 00:42:51
    from uh from a sheer overall
  • 00:42:53
    productivity perspective and we measure
  • 00:42:55
    this like we measure everything we know
  • 00:42:58
    when a ch when the group that has
  • 00:43:00
    balancebot
  • 00:43:02
    is producing more and getting better
  • 00:43:04
    ratings
  • 00:43:05
    than the group that doesn't use
  • 00:43:07
    balancebot and i mean all this stuff is
  • 00:43:09
    optional you can turn them all off you
  • 00:43:10
    can say i don't want this
  • 00:43:12
    you can disable these things and they
  • 00:43:14
    actually ask you whether you want to
  • 00:43:16
    because for us consent is everything
  • 00:43:19
    but
  • 00:43:20
    when you have these things enabled we're
  • 00:43:21
    able to show
  • 00:43:23
    you get more code written you spend more
  • 00:43:24
    hours writing code your code has less
  • 00:43:26
    diff comments your code doesn't
  • 00:43:30
    need to be reverted as often on the
  • 00:43:32
    aggregate
  • 00:43:34
    all of these measures that can say can
  • 00:43:36
    you focus
  • 00:43:37
    are you building what we are intending
  • 00:43:39
    to build
  • 00:43:41
    and
  • 00:43:42
    on the aggregate are you a happier
  • 00:43:43
    developer
  • 00:43:45
    because a happy developer is a good
  • 00:43:46
    developer
  • 00:43:48
    and that that means a different thing to
  • 00:43:49
    everybody but
  • 00:43:51
    for most of us i think it means can we
  • 00:43:53
    do the job that we like doing
  • 00:43:55
    nice yeah i mean we
  • 00:43:57
    we know context switching is is a killer
  • 00:44:00
    and
  • 00:44:01
    you know thinking of
  • 00:44:02
    how to decrease that noise is on
  • 00:44:05
    facebook scale very interesting thanks
  • 00:44:07
    for sharing there um
  • 00:44:11
    any
  • 00:44:12
    interesting productivity engineering
  • 00:44:15
    wins that you i know you talked about
  • 00:44:18
    the the
  • 00:44:19
    the no code stuff and the safety and
  • 00:44:21
    stuff any other interesting wins based
  • 00:44:24
    on data that you all have seen
  • 00:44:26
    and things that you implement that you
  • 00:44:28
    can kind of share with us
  • 00:44:30
    well so i think the biggest i think the
  • 00:44:32
    biggest productivity win that i've seen
  • 00:44:34
    here
  • 00:44:36
    is the integration of the apps
  • 00:44:39
    themselves and i mean i know i've
  • 00:44:40
    touched on the monorepo a little bit
  • 00:44:43
    but
  • 00:44:44
    the thing is is that when we're building
  • 00:44:46
    these systems we've kind of everybody
  • 00:44:48
    kind of eventually acknowledges that
  • 00:44:51
    you're not going to reuse as much of the
  • 00:44:52
    code as you think you're going to use
  • 00:44:55
    right you're not going to actually reuse
  • 00:44:58
    that method you wrote 15 years ago
  • 00:45:02
    today for what you're writing
  • 00:45:04
    but a big chunk of what we've tried to
  • 00:45:07
    do
  • 00:45:08
    is to take all of the common
  • 00:45:10
    stuff that we don't want people reusing
  • 00:45:13
    and make it transparent
  • 00:45:16
    so
  • 00:45:17
    we have tools that make it easy to write
  • 00:45:20
    one-off sql scripts
  • 00:45:22
    we have tools that make it really easy
  • 00:45:26
    to build
  • 00:45:27
    complex data flows to do analytics
  • 00:45:30
    without establishing
  • 00:45:32
    new hardware or new infrastructure to do
  • 00:45:35
    it
  • 00:45:36
    we have focused on flexibility over
  • 00:45:40
    purity of ideal
  • 00:45:41
    so we have a system called bento that is
  • 00:45:43
    one of the big things that productivity
  • 00:45:45
    engineering came out with that lets you
  • 00:45:47
    just run data processing and you can run
  • 00:45:50
    it in c sharp or c
  • 00:45:52
    or python you can run a sql query you
  • 00:45:55
    can pull the data into your system and
  • 00:45:57
    you can build these very useful scripts
  • 00:46:00
    with a limited certain
  • 00:46:02
    limited amount of libraries that are
  • 00:46:04
    available
  • 00:46:05
    and do simple visualization and then
  • 00:46:08
    save those and share them
  • 00:46:10
    that keeps us from having people build
  • 00:46:13
    custom data
  • 00:46:14
    data systems trying to download their
  • 00:46:16
    own versions of every single library in
  • 00:46:18
    existence having their need to write
  • 00:46:21
    custom sql code everywhere all those
  • 00:46:24
    integrations to get the data
  • 00:46:26
    unified because it's actually easier
  • 00:46:28
    than them writing themselves
  • 00:46:30
    the code is easy is easy to port and the
  • 00:46:32
    solutions can be shared so we're not
  • 00:46:34
    having everybody run their own
  • 00:46:38
    so that's
  • 00:46:39
    that's one of the biggest wins in my
  • 00:46:41
    opinion is that kind of infra it's infra
  • 00:46:43
    targeted at the facilitation of what
  • 00:46:47
    your users are going to do whether you
  • 00:46:48
    want them to or not
  • 00:46:50
    i mean and by users here i mean the
  • 00:46:51
    developers
  • 00:46:53
    thanks for thanks for sharing that um i
  • 00:46:55
    have one question from our folks justin
  • 00:46:59
    i had to get this one in our folks have
  • 00:47:01
    been
  • 00:47:02
    you said
  • 00:47:03
    don't let adam leave without him
  • 00:47:05
    answering this one question
  • 00:47:08
    you know and of course we're the company
  • 00:47:10
    behind gradle right so obviously
  • 00:47:11
    interested in build test acceleration
  • 00:47:14
    technologies and whatnot
  • 00:47:16
    um you know
  • 00:47:17
    how how
  • 00:47:19
    how long does a
  • 00:47:21
    android pr build take and what are you
  • 00:47:25
    all doing as far as acceleration
  • 00:47:27
    technologies go
  • 00:47:29
    to make this thing and that that process
  • 00:47:32
    and the feedback cycles to be as fast as
  • 00:47:35
    it can be with tools and technologies
  • 00:47:37
    have you done so i think we've i think
  • 00:47:38
    we've talked to that a little bit yeah
  • 00:47:40
    um but here's the thing right if you
  • 00:47:43
    were to check out an on demand
  • 00:47:45
    right now just a blank on demand no did
  • 00:47:48
    no code changes
  • 00:47:49
    and you were to try and build the
  • 00:47:51
    facebook app it's going to be less than
  • 00:47:53
    a minute
  • 00:47:55
    seconds
  • 00:47:56
    now that's building
  • 00:47:58
    the most complex android app in
  • 00:48:01
    existence
  • 00:48:03
    in a few
  • 00:48:04
    seconds now
  • 00:48:08
    getting there
  • 00:48:09
    means that we have a custom build system
  • 00:48:11
    that relies heavily on artifacts that
  • 00:48:14
    are all cached and built constantly into
  • 00:48:17
    what we call the stable and they're
  • 00:48:20
    really checkpoints our stable points are
  • 00:48:22
    checkpoints for each given app where we
  • 00:48:24
    say this is a version that everybody
  • 00:48:26
    should be basing their work on
  • 00:48:29
    and everything past that that you're
  • 00:48:31
    building be that from master or be that
  • 00:48:34
    through your own changes
  • 00:48:36
    has to be recompiled
  • 00:48:38
    so we take as much of stable as we
  • 00:48:40
    possibly can we take the little bit that
  • 00:48:42
    you've added and or changed
  • 00:48:44
    and only that little bit
  • 00:48:46
    is being actually recompiled
  • 00:48:49
    so on a typical change you might do a
  • 00:48:52
    lot of work and you might do
  • 00:48:54
    need to recompile a lot and it might
  • 00:48:56
    take 20 30 minutes to fully recompile
  • 00:48:59
    that but then the next build will be
  • 00:49:00
    five seconds
  • 00:49:03
    because we can't we literally can't
  • 00:49:06
    afford to build all of the code for
  • 00:49:07
    every change that's made
  • 00:49:09
    and i mean that in a literal monetary
  • 00:49:11
    sense running all of these builds from
  • 00:49:14
    scratch
  • 00:49:15
    is financially irresponsible and
  • 00:49:18
    honestly it's it's uh
  • 00:49:20
    an irresponsible thing as stewards of
  • 00:49:22
    the work of the planet
  • 00:49:24
    the sheer i think there was a study by
  • 00:49:26
    google a few years ago that
  • 00:49:28
    their ai system
  • 00:49:30
    was generating something like the entire
  • 00:49:33
    uh heat footprint of a small country
  • 00:49:37
    just to run their ai models
  • 00:49:39
    when we're talking about humanity scale
  • 00:49:41
    data processing
  • 00:49:43
    we can't it isn't
  • 00:49:45
    just
  • 00:49:46
    you know financially bad it's
  • 00:49:48
    irresponsible
  • 00:49:49
    to run that excessive computation
  • 00:49:52
    and so we actually i mean we
  • 00:49:55
    measure those things every single
  • 00:49:56
    feature we look at we look at it in
  • 00:49:58
    terms of how much
  • 00:50:00
    server-side cpu it requires in order to
  • 00:50:04
    do its job we look at how much on basis
  • 00:50:08
    a dev environment is costing in terms of
  • 00:50:11
    how long it's taking to compile how long
  • 00:50:14
    diffs are taking on server-side
  • 00:50:16
    computation hours
  • 00:50:18
    not just human hours but energy usage
  • 00:50:21
    because those are end up being the core
  • 00:50:23
    drivers of success for the system as a
  • 00:50:26
    whole even if it's not the best thing
  • 00:50:29
    monetarily and adam uh this will
  • 00:50:31
    actually help with one of the audience
  • 00:50:32
    questions too
  • 00:50:33
    how much of that time saving is caching
  • 00:50:36
    versus i build caching specifically like
  • 00:50:39
    not dependency cash or anything like
  • 00:50:40
    that how much of that time saving is
  • 00:50:42
    build caching versus some of the other
  • 00:50:44
    accelerating technologies that you've
  • 00:50:45
    got in there
  • 00:50:46
    well so i will admit that we've had
  • 00:50:48
    entire groups uh 10 and 20 people at a
  • 00:50:50
    time who have optimized the actual
  • 00:50:54
    compilers
  • 00:50:57
    to a to an
  • 00:50:58
    obscene degree
  • 00:50:59
    we have custom linting frameworks custom
  • 00:51:02
    compilation frameworks
  • 00:51:04
    we do
  • 00:51:05
    uh app size
  • 00:51:07
    magic that
  • 00:51:09
    nobody else has
  • 00:51:10
    and we we have this problem where we
  • 00:51:12
    clever our way out of everything but at
  • 00:51:14
    some point
  • 00:51:16
    that is really foundational cutting edge
  • 00:51:18
    work that has been done to facilitate
  • 00:51:21
    these
  • 00:51:22
    gigantic products
  • 00:51:25
    being broken down in these tiny little
  • 00:51:27
    bite-sized chunks that can both be
  • 00:51:29
    compiled locally
  • 00:51:31
    developed against
  • 00:51:33
    debugged but also shipped to our you
  • 00:51:35
    shipped to our users without it being
  • 00:51:37
    the only thing their phone can do
  • 00:51:39
    it's i mean it takes a minute to even
  • 00:51:41
    like know how to respond to to say like
  • 00:51:43
    that right because just the complexity
  • 00:51:45
    of each of those little pieces all
  • 00:51:47
    working together but then the fact that
  • 00:51:48
    it actually works is kind of um what's
  • 00:51:51
    what it works so well i think is kind of
  • 00:51:53
    what's so impressive
  • 00:51:55
    um
  • 00:51:57
    when i say it's the most complex
  • 00:51:59
    android code base that i've ever seen
  • 00:52:02
    and i've seen a lot and that was one of
  • 00:52:04
    the first questions actually was
  • 00:52:06
    elaborate and i feel like you've done
  • 00:52:08
    that kind of for the whole
  • 00:52:10
    session it's like yeah there's
  • 00:52:11
    complexity here there's complexity here
  • 00:52:13
    there's complexity here it's not just
  • 00:52:14
    the code it's not just the architecture
  • 00:52:16
    it's the whole process of manufacturing
  • 00:52:18
    and and everything that goes into it
  • 00:52:19
    yeah
  • 00:52:20
    justin how many more can we take before
  • 00:52:22
    oh my god we wrap it up we have i know
  • 00:52:24
    we have so many i know and we're going
  • 00:52:27
    to spill stuff over into the slack so
  • 00:52:28
    again i'll post that before we close out
  • 00:52:30
    again so make sure that and you can
  • 00:52:31
    scroll up in the chat now if you see
  • 00:52:32
    that now um maybe pick uh two of the top
  • 00:52:35
    ones justin and um you know adam will go
  • 00:52:39
    over a bit and we don't want to abuse
  • 00:52:41
    your time we could you know probably
  • 00:52:43
    extend this to all day i could just
  • 00:52:45
    cancel the rest of my meetings and uh
  • 00:52:47
    well i mean it's not the joe rogan
  • 00:52:49
    podcast but
  • 00:52:52
    you know i think this here's one that i
  • 00:52:54
    think is really good because i think it
  • 00:52:55
    ties into some ambiguities that people
  • 00:52:57
    get around the process of developer
  • 00:52:58
    productivity engineering versus kind of
  • 00:53:01
    management um and i think this is a good
  • 00:53:03
    opportunity to disambiguate that what
  • 00:53:05
    we're talking about here is using
  • 00:53:06
    technology to make productivity better
  • 00:53:09
    through happiness and less frustration
  • 00:53:12
    but one of the questions was you know
  • 00:53:13
    don't you think too much measurement
  • 00:53:15
    feels like micromanagement for engineers
  • 00:53:17
    maybe you could tell me a little bit
  • 00:53:18
    about culturally yeah so i've got i've
  • 00:53:21
    got the perfect answer this one if you
  • 00:53:22
    haven't ever heard of goodheart's law
  • 00:53:24
    um you should look it up um what good
  • 00:53:27
    heart's law says
  • 00:53:28
    is that as soon as a metric
  • 00:53:31
    becomes a goal
  • 00:53:33
    it ceases being a good metric
  • 00:53:36
    now we take that to heart and we take it
  • 00:53:39
    to a to an extreme really
  • 00:53:41
    where we don't say
  • 00:53:43
    you need to write this much code or
  • 00:53:46
    spend this number of hours we don't say
  • 00:53:49
    there are this many requirements for how
  • 00:53:51
    many lines of code get written or how
  • 00:53:53
    many diffs get put through what we do is
  • 00:53:56
    we look at
  • 00:53:57
    the effect of a given change
  • 00:54:00
    on all of those metrics
  • 00:54:02
    and we don't say they have to be a
  • 00:54:03
    certain number or arbitrarily say they
  • 00:54:05
    need to improve
  • 00:54:07
    we look at it from an evolutionary
  • 00:54:08
    perspective where those things are
  • 00:54:10
    positive direction
  • 00:54:12
    so when we put in a change like balanced
  • 00:54:14
    bot we say okay
  • 00:54:16
    are these things that are generally
  • 00:54:18
    neutral goods
  • 00:54:20
    improving are we getting
  • 00:54:23
    gifts through faster are diffs landing
  • 00:54:25
    better
  • 00:54:26
    are there being
  • 00:54:28
    more thorough review and less reverse
  • 00:54:31
    are our pro or
  • 00:54:33
    the average regression that a diff
  • 00:54:36
    causes against all the all of the
  • 00:54:38
    company financial
  • 00:54:41
    active user those kinds of product
  • 00:54:43
    metrics
  • 00:54:44
    is the average effect of a diff getting
  • 00:54:46
    better
  • 00:54:48
    we don't care
  • 00:54:49
    how it gets better we don't really care
  • 00:54:52
    if the way it got better was that you
  • 00:54:54
    had to stand on one foot and dance under
  • 00:54:56
    the moonlight
  • 00:54:58
    that doesn't matter what matters is that
  • 00:55:00
    you ran an experiment that shows that by
  • 00:55:03
    doing this thing whatever it is
  • 00:55:06
    all of those other things
  • 00:55:08
    net went better and better is the whole
  • 00:55:11
    point
  • 00:55:12
    better is the way we get to functional
  • 00:55:15
    and the way we get too efficient
  • 00:55:18
    even little steps if you do them long
  • 00:55:21
    enough will get you where you're going
  • 00:55:23
    now that always brings up the problem of
  • 00:55:24
    how do you choose
  • 00:55:26
    which thing will give you the most
  • 00:55:27
    impact and that's a whole problem in and
  • 00:55:30
    of itself and it's sort of the the most
  • 00:55:32
    difficult thing that we ever have to ask
  • 00:55:35
    but rather than asking how can i
  • 00:55:36
    possibly do that we're asking which of
  • 00:55:39
    these things that i could do
  • 00:55:41
    are going to have that that impact
  • 00:55:44
    and because we're constantly measuring
  • 00:55:47
    we can decide whether we were right
  • 00:55:50
    and that helps us tune that whole
  • 00:55:52
    process
  • 00:55:53
    but we never goal
  • 00:55:55
    we never goal never never never never
  • 00:55:58
    goal on
  • 00:55:59
    productivity
  • 00:56:00
    or on individual pro on individual
  • 00:56:04
    output as a thing a person needs to do
  • 00:56:08
    that's not how we measure our engineers
  • 00:56:10
    it's not how we measure our people
  • 00:56:12
    we measure them on the impact that they
  • 00:56:14
    have to the organization and to the
  • 00:56:17
    products and to our users
  • 00:56:19
    that's
  • 00:56:20
    quite possibly a perfect answer i mean i
  • 00:56:23
    i love that it's like no these metrics
  • 00:56:25
    are not here because we want to drive
  • 00:56:26
    them necessarily in any particular
  • 00:56:28
    direction these metrics are here so that
  • 00:56:31
    we actually know what we're doing we can
  • 00:56:32
    make informed decisions so i know we're
  • 00:56:34
    up on time i just posted the community
  • 00:56:37
    slack channel again to the main chat
  • 00:56:39
    uh
  • 00:56:40
    and and we can fold in there i'm gonna
  • 00:56:42
    do my best to get these things the
  • 00:56:44
    remaining questions in there and then uh
  • 00:56:46
    adam can asynchronously as he has time
  • 00:56:50
    respond to so many of these
  • 00:56:52
    amazing questions i mean really we need
  • 00:56:54
    another session i feel like uh because
  • 00:56:56
    there's so much that we can still
  • 00:56:57
    discuss so but why don't i head on back
  • 00:56:59
    over to you for and
  • 00:57:02
    just wanted to thank adam for his time
  • 00:57:04
    and of course you know their their team
  • 00:57:06
    is hiring right thank you for facebook
  • 00:57:08
    engineering for allowing
  • 00:57:11
    adam to
  • 00:57:12
    uh speak and share all this with us
  • 00:57:15
    right there there's a link
  • 00:57:17
    qr code to link for the jobs within
  • 00:57:21
    uh adam's
  • 00:57:24
    org
  • 00:57:25
    right if you're interested and adam
  • 00:57:27
    thanks again i mean we we go all day and
  • 00:57:29
    thanks for you know giving your time to
  • 00:57:32
    go into the gradle community slack and
  • 00:57:34
    and share all that there and with that
  • 00:57:37
    said we'll
  • 00:57:38
    uh you got anything for our folks before
  • 00:57:41
    we go
  • 00:57:42
    i'll just say that uh
  • 00:57:44
    the way we think about android and ios
  • 00:57:47
    is not like the way everybody else
  • 00:57:49
    thinks of android and ios
  • 00:57:51
    we want talented java and c developers
  • 00:57:56
    the fact that that our code is running
  • 00:57:58
    on mobile is not the critical part of
  • 00:58:00
    any of this as much as that's valuable
  • 00:58:04
    if you've ever wanted to work at meta
  • 00:58:06
    and you know java or you know c or you
  • 00:58:08
    know swift
  • 00:58:09
    mobile is a great way of coming in
  • 00:58:12
    because we look at mobile the way most
  • 00:58:15
    people look at servers
  • 00:58:18
    that's awesome and folks
  • 00:58:21
    the moral of the story if you don't
  • 00:58:24
    invest in tooling productivity
  • 00:58:26
    engineering developer experience
  • 00:58:28
    and one day you ended up competing
  • 00:58:30
    against facebook
  • 00:58:33
    you know send thoughts and prayers right
  • 00:58:35
    yeah that's perfect and that's why we're
  • 00:58:37
    here today and thanks again everyone for
  • 00:58:40
    joining and spending this time with us
  • 00:58:42
    and thank you adam thank you justin
  • 00:58:44
    thank you the great old team and we'll
  • 00:58:46
    see you all on the next one which um
  • 00:58:50
    android x how the android the google
  • 00:58:53
    android x productivity uh well how the
  • 00:58:56
    google android x team does productivity
  • 00:58:58
    engineering uh we're doing a webcast on
  • 00:59:01
    that with armis from the android x
  • 00:59:04
    libraries team they're the folks behind
  • 00:59:07
    jetpack thanks all thanks adam and we'll
  • 00:59:11
    next time
  • 00:59:30
    you
الوسوم
  • Meta
  • Facebook
  • developer productivity
  • engineering
  • Android
  • Sapiens
  • Buck
  • reliability engineering
  • developer experience
  • team structure