Agents, Lawyers, and LLMs

00:42:16
https://www.youtube.com/watch?v=ZESTYyGZ7Y4

概要

TLDRThe video features an interview discussing Harvey, a specialized AI platform for legal and professional services aimed at enhancing efficiency within law firms. The speaker outlines the types of legal workflows Harvey can automate, emphasizing the need for collaboration between AI agents and humans. The conversation touches on how the AI boom, initiated by tools like ChatGPT, has pressured law firms to adopt AI technologies despite previous skepticism. Additionally, it explores data security measures implemented by Harvey, the changing business models within the legal sector, and the ongoing development of user-friendly AI interfaces. The speaker asserts that while AI's potential in law is significant, understanding specific workflows and instilling trust in AI systems are crucial for successful integration in enterprise environments.

収穫

  • 🤖 Harvey automates legal workflows, enhancing efficiency for lawyers.
  • 📈 The legal sector is adapting to AI, pressured by client demands.
  • 🔒 Data security is crucial; Harvey adheres to strict policies.
  • 🤝 Collaboration between AI agents and humans is vital to success.
  • 🧠 AI alone won't ensure success; understanding human workflows is essential.
  • 🚀 There is a shift from skepticism to advocacy for AI in legal firms.
  • 🛠️ Harvey uses model evaluations to measure effectiveness.
  • 🩺 The healthcare industry is next for AI expansion following legal.
  • 👩‍⚖️ The UX for legal professionals is evolving towards AI-native experiences.
  • 🏢 Enterprises need tailored AI solutions to improve workflows.

タイムライン

  • 00:00:00 - 00:05:00

    The discussion begins with the importance of collaboration between AI agents and humans in the legal field, defining the role Harvey plays in assisting law firms with automation in legal documents and advisory processes.

  • 00:05:00 - 00:10:00

    An overview of the legal landscape is shared, identifying key categories of work, such as transactional, litigation, and in-house legal functions, detailing how Harvey supports different workflows across these areas.

  • 00:10:00 - 00:15:00

    The conversation addresses market dynamics in legal tech, noting that the rise of AI technologies like ChatGPT has spurred law firms to adopt innovative solutions due to increased competition and client demand.

  • 00:15:00 - 00:20:00

    Harvey's strategy is to integrate legal expertise into its product development process by employing lawyers in sales and product teams, allowing for better alignment with customer needs and effective communication of technical offerings to legal professionals.

  • 00:20:00 - 00:25:00

    The discussion shifts to the implications of AI in legal workflows, emphasizing the collaborative nature of legal work where human lawyers guide AI outputs in a co-pilot fashion rather than fully automating tasks.

  • 00:25:00 - 00:30:00

    There’s a focus on the pricing models for AI products in legal services, with Harvey's approach being flexible to market demands while the company aims for future outcome-based pricing as AI adoption matures.

  • 00:30:00 - 00:35:00

    The role of customer education and engagement in realizing the value of AI is emphasized, highlighting strategies Harvey employs for onboarding users and gamifying the usage of AI in legal tasks to encourage adoption.

  • 00:35:00 - 00:42:16

    Lastly, the conversation touches on the future direction of AI in legal services, predicting that the landscape will continue to evolve and result in deeper integrations and specialized applications, while also noting the long-term challenges and opportunities in building trust and competence in AI deployments within law firms.

もっと見る

マインドマップ

ビデオQ&A

  • What does Harvey do?

    Harvey is a domain-specific AI for legal and professional services, automating tasks like drafting documents and providing strategic advice.

  • What types of legal work does Harvey support?

    Harvey supports transactional work, litigations, and in-house legal processes.

  • How has AI adoption changed in law firms?

    Law firms are increasingly pressured to adopt AI due to client demands and competitive dynamics.

  • How does Harvey ensure data security?

    Harvey has strict policies regarding data access and does not use client data for training models.

  • What is the significance of the generative AI boom?

    The generative AI boom, initiated by ChatGPT, has prompted legal professionals to recognize the potential of AI in transforming legal practices.

  • How does Harvey assess model performance?

    Harvey uses evaluations that measure the percentage of work completed by AI compared to human outputs.

  • What is the future of AI in the legal sector?

    AI will continue to evolve, but firms need to focus on understanding specific workflows and building trust with their AI solutions.

  • How does Harvey plan to expand beyond legal services?

    Harvey aims to naturally expand into other sectors while continuing to focus on legal expertise.

  • Does Harvey have plans to create its own AI models?

    No, Harvey prefers to collaborate with established AI model providers instead of building their own.

  • What is the biggest challenge for AI adoption in enterprises?

    The challenge lies in overcoming human bottlenecks, trust issues, and effectively integrating AI into existing workflows.

ビデオをもっと見る

AIを活用したYouTubeの無料動画要約に即アクセス！
字幕
en
オートスクロール:
  • 00:00:05
    oan is not going to oneshot your S1
  • 00:00:07
    agents have to collaborate well with
  • 00:00:10
    humans to get the work done we really
  • 00:00:13
    want law firms to collaborate with their
  • 00:00:15
    clients inside of Harvey and inside with
  • 00:00:18
    Harvey's
  • 00:00:19
    [Music]
  • 00:00:25
    agents so I'm a I lead product at Harvey
  • 00:00:28
    uh I've been at Harvey for year and a
  • 00:00:30
    half now when we uh were around 30
  • 00:00:32
    people and since we've scaled to 250
  • 00:00:35
    people so it's been quite a journey
  • 00:00:36
    going through that growth um and
  • 00:00:38
    generally my background and where I
  • 00:00:40
    spend most of my career is actually in
  • 00:00:42
    hypergrowth AI startups and so before
  • 00:00:44
    this uh I was at scale of four four and
  • 00:00:46
    a half years as a product leader and
  • 00:00:47
    before that I was at Shield AI also a6z
  • 00:00:50
    portfolio company um and really it's a
  • 00:00:53
    privilege to be part of Harvey for a
  • 00:00:55
    third time through hypergrowth um
  • 00:00:58
    because it's such a PIV moment in human
  • 00:01:00
    history and I think a lot of things are
  • 00:01:02
    going to change and I'm excited to kind
  • 00:01:03
    of be at the Forefront maybe for people
  • 00:01:05
    who aren't familiar in the audience or
  • 00:01:07
    people who are listening online what
  • 00:01:08
    exactly does Harvey do because I think a
  • 00:01:10
    lot of people like know that Harvey
  • 00:01:12
    exists but we might not know in extreme
  • 00:01:13
    detail like what the product offering is
  • 00:01:15
    so Harvey is domain specific AI for
  • 00:01:18
    legal and Professional Services um our
  • 00:01:20
    product basically helps users and
  • 00:01:22
    lawyers automate drafts entheses
  • 00:01:24
    strategic advice memos and more got it
  • 00:01:26
    and before we get like just a little bit
  • 00:01:28
    deeper into the practice of building
  • 00:01:29
    applied AI
  • 00:01:30
    are there like specific use cases that
  • 00:01:32
    Harvey tackles the most knowing that
  • 00:01:33
    there are a lot of different workflows
  • 00:01:35
    one could theoretically do for legal or
  • 00:01:37
    Professional Services yeah so broadly
  • 00:01:40
    for for legal there's like um maybe two
  • 00:01:43
    or three types of legal work so there's
  • 00:01:44
    transactional work which is essentially
  • 00:01:46
    for um mergers Acquisitions uh Venture
  • 00:01:49
    funding you know large transactions that
  • 00:01:52
    involve you know tremendous amounts of
  • 00:01:53
    money and then there's litigations which
  • 00:01:55
    is you know if someone sues someone if
  • 00:01:57
    there's a case uh in court um you know
  • 00:02:00
    also often involving a lot of money um
  • 00:02:02
    and then probably the third is really uh
  • 00:02:04
    focus on in-house so Enterprise uh
  • 00:02:07
    councils and Enterprise inhouse teams um
  • 00:02:10
    so these three are the the larger
  • 00:02:11
    buckets um we serve all these in various
  • 00:02:14
    ways and so um you know if you think
  • 00:02:16
    about what you need in a merger or
  • 00:02:19
    acquisition you need to do due diligence
  • 00:02:21
    in that you need to understand all the
  • 00:02:24
    liabilities you understand the
  • 00:02:25
    financials you understand uh you know
  • 00:02:27
    where the gotches are are gotches are of
  • 00:02:30
    the Target and the acquire um so the
  • 00:02:33
    each each like like a due diligence can
  • 00:02:35
    be broken up into you know almost 10 to
  • 00:02:38
    12 different workflows and we help in
  • 00:02:40
    different ways in those workflows and
  • 00:02:42
    then same thing in litigation um so
  • 00:02:44
    there's there's that like high level and
  • 00:02:46
    and really focused on different steps in
  • 00:02:48
    that that Journey for a long time in
  • 00:02:50
    Silicon Valley circles people believed
  • 00:02:52
    that selling to law firms or selling
  • 00:02:54
    Professional Services just wasn't the
  • 00:02:56
    most fruitful area given they weren't
  • 00:02:58
    known for adopting technology quickly a
  • 00:03:00
    lot of people thought the billing model
  • 00:03:02
    wasn't allying to increasing efficiency
  • 00:03:04
    or using technology I'm curious like
  • 00:03:07
    what is Harvey seen in that regard yeah
  • 00:03:09
    so I think there's there's like two
  • 00:03:10
    things there's like the market and what
  • 00:03:12
    Harvey has specifically done so I think
  • 00:03:15
    Market timing for any startup is
  • 00:03:16
    incredibly important if you look overall
  • 00:03:19
    you know when Chach gbt November 2022
  • 00:03:22
    came out um that really kind of you know
  • 00:03:24
    Unleash the Power of gen for a lot of
  • 00:03:27
    people so you know lawyers uh inhouse
  • 00:03:30
    Council managing Partners uh cios really
  • 00:03:33
    started understanding this technology
  • 00:03:34
    and saying oh wow this is actually can
  • 00:03:36
    change a lot of things it wasn't really
  • 00:03:38
    a hidden thing before before Chach PT AI
  • 00:03:41
    was just like maybe this hidden thing
  • 00:03:43
    that we don't really know how to apply
  • 00:03:45
    but because it put it uh in people's
  • 00:03:47
    hands uh the kind of the cat cat was out
  • 00:03:49
    of the bag in terms of you know the
  • 00:03:51
    practice of law was going to change and
  • 00:03:53
    so the cat was out of the bag everyone
  • 00:03:55
    knows that this going to happen and
  • 00:03:57
    because everyone knows that it's going
  • 00:03:58
    to happen a lot of Enterprises are
  • 00:04:01
    saying hey Law Firm my Law Firm X you
  • 00:04:04
    know we we use AI I've seen AI in action
  • 00:04:07
    um you all should use AI to become more
  • 00:04:09
    efficient do more work etc um so there
  • 00:04:12
    law firms started feeling pressure from
  • 00:04:13
    clients and then the the law firm market
  • 00:04:17
    and legal Market in general is very
  • 00:04:19
    competitive um you know in any region
  • 00:04:21
    there's uh four or five major players
  • 00:04:23
    really going tooth and nail to each each
  • 00:04:25
    other and so it's important for uh a law
  • 00:04:29
    from to signal that they're Innovative
  • 00:04:31
    because they'll get more clients and
  • 00:04:32
    they're more efficient and and so
  • 00:04:34
    because of this competitive Dynamic now
  • 00:04:36
    everyone really wanted to adop
  • 00:04:37
    technology and I think it's this perfect
  • 00:04:40
    storm of of Market timing and where
  • 00:04:42
    where Harvey was and so there was these
  • 00:04:46
    macro kind of movements and and
  • 00:04:48
    pressures from the market and then I
  • 00:04:50
    think what Harvey early on I think and
  • 00:04:52
    we still do I think really well is
  • 00:04:54
    really embedding the legal expertise
  • 00:04:57
    across all different functions so
  • 00:05:00
    what that meant was you know early on we
  • 00:05:02
    actually had lawyers selling the product
  • 00:05:03
    so lawyers as account Executives and our
  • 00:05:05
    CEO is a lawyer and our head of legal
  • 00:05:07
    research actually is also a lawyer which
  • 00:05:09
    I'll go into in a second but yeah we
  • 00:05:11
    have we had lawyers selling the product
  • 00:05:12
    and so they would go to a law firm and
  • 00:05:15
    speak the language speak the lingo be
  • 00:05:16
    super empathetic and they would actually
  • 00:05:18
    come from a lot of the customers that uh
  • 00:05:21
    we were serving so they knew exactly how
  • 00:05:23
    things worked and that really allowed us
  • 00:05:25
    to get the distribution and really get
  • 00:05:26
    the GTM going and then on the product
  • 00:05:28
    and a side we also have lawyers embedded
  • 00:05:32
    uh in our product and AI teams we have
  • 00:05:34
    like a legal research uh function
  • 00:05:36
    actually that works hand inand with
  • 00:05:38
    product managers and and AI engineers
  • 00:05:40
    and what they really do is convert
  • 00:05:43
    basically legal process into algorithms
  • 00:05:46
    so the best way to think about Harvey
  • 00:05:48
    kind of under the hood is we have like
  • 00:05:50
    an agentic or compound AI system that
  • 00:05:53
    basically functions how a law firm would
  • 00:05:55
    function so if in a law firm if a
  • 00:05:57
    partner gets a deal or litigation they
  • 00:06:00
    break it up into multiple different
  • 00:06:02
    pieces maybe give it to the junior
  • 00:06:03
    Partners Junior Partners break it up
  • 00:06:05
    further give it to Associates and it's
  • 00:06:07
    kind of passed down the chain and then
  • 00:06:09
    because law firms are fairly like
  • 00:06:10
    hierarchical organizations the
  • 00:06:12
    associates do the work then they you
  • 00:06:14
    know pass it up for approval and checks
  • 00:06:16
    and then ultimately the partner delivers
  • 00:06:18
    the end product to to the client and our
  • 00:06:22
    lawyers who work with their Engineers
  • 00:06:23
    actually just basically replicate that
  • 00:06:25
    same model for different types of tasks
  • 00:06:27
    and convert and lit whiteboard out
  • 00:06:30
    different processes so that AI Engineers
  • 00:06:31
    can convert it into uh kind of model
  • 00:06:33
    systems do you consider like these
  • 00:06:35
    different agentic workflows then do you
  • 00:06:36
    consider them replacing any kind of
  • 00:06:39
    Labor that people were previously doing
  • 00:06:41
    or do you view it more in like the
  • 00:06:42
    classic like agentic like labor
  • 00:06:44
    replacement versus co-pilot model yeah
  • 00:06:46
    no it's a it's a good question I think
  • 00:06:48
    it's a it's a bit of a narrow take I
  • 00:06:50
    think you know the the legal landscape
  • 00:06:52
    overall is very complex and getting even
  • 00:06:55
    more complex and honestly very costly to
  • 00:06:57
    navigate you globalization the internet
  • 00:07:00
    AI has increased legal work you know
  • 00:07:02
    exponentially over the last few decades
  • 00:07:04
    and so you have basically infinite
  • 00:07:05
    demand for legal work because companies
  • 00:07:07
    are are you know wanting to do different
  • 00:07:10
    transactions litigations Etc so you have
  • 00:07:12
    infinite demand and then what that means
  • 00:07:14
    is that the supply is very constrained
  • 00:07:17
    and the unfortunate human cost of Supply
  • 00:07:20
    constraints is very long hours often
  • 00:07:22
    doing very mundane kind of boring tasks
  • 00:07:26
    uh you know we talk to lawyers who we've
  • 00:07:28
    hired our customers
  • 00:07:30
    they haven't become lawyers to write the
  • 00:07:31
    fifth draft of the same document the
  • 00:07:33
    fifth time uh or ask the same legal
  • 00:07:36
    research question right uh they became
  • 00:07:38
    lawyers to apply law the law in creative
  • 00:07:41
    ways publish opinions uh kind of shape
  • 00:07:43
    the fabric of society and so we hear
  • 00:07:46
    this from customers all the time like
  • 00:07:47
    Harvey gives 30% 40% of their time back
  • 00:07:50
    because it really helps them automate
  • 00:07:52
    that you know mundane root work you know
  • 00:07:54
    actually the the other day one of our
  • 00:07:56
    customers said Harvey allows them to go
  • 00:07:58
    home to their family in time because
  • 00:08:00
    it's you know been able to accelerate a
  • 00:08:02
    lot of things so infinite demand a lot
  • 00:08:04
    of Supply constraints and you know it's
  • 00:08:06
    a great place for for a health yeah c
  • 00:08:08
    can you talk more about that a little
  • 00:08:09
    bit what would that interaction pattern
  • 00:08:11
    actually look like yeah so this is a a
  • 00:08:13
    general question with I think generative
  • 00:08:15
    AI like what is the the human component
  • 00:08:17
    how much is it fully automated um I
  • 00:08:19
    think the reality is like let's say
  • 00:08:21
    you're drafting an S4 or like an S1 um
  • 00:08:25
    S1 is when you you know go go public
  • 00:08:28
    like you're not going to one shot that
  • 00:08:29
    into the bigest biggest reasoning model
  • 00:08:31
    and say hey write me an S1 and you're
  • 00:08:33
    done right uh it requ are safe yeah all
  • 00:08:36
    the makers are safe you're not going to
  • 00:08:37
    write uh 01 is not going to oneshot your
  • 00:08:40
    S1
  • 00:08:42
    um the the process of doing an S1 or
  • 00:08:45
    process of doing a merger um is is
  • 00:08:48
    really interactive with both parties
  • 00:08:50
    both the law firm the client and any
  • 00:08:52
    other parties involved and so we think
  • 00:08:55
    basically these agents have to
  • 00:08:57
    collaborate well with humans to get the
  • 00:09:00
    work done because humans may have um
  • 00:09:03
    some particular intent that they haven't
  • 00:09:05
    told the agent or um you know they'll
  • 00:09:08
    they may have some data that the agent
  • 00:09:09
    doesn't actually have and so um we think
  • 00:09:13
    about building these agents in a in a
  • 00:09:15
    nice like kind of AI native ux way so
  • 00:09:17
    that they can actually like collaborate
  • 00:09:19
    with different organizations to actually
  • 00:09:21
    get the work done and say Hey you know I
  • 00:09:23
    wrote this draft is am I on the right
  • 00:09:25
    track you know give me this more
  • 00:09:27
    information because I don't know what to
  • 00:09:28
    do about this decision here so I think
  • 00:09:31
    we'll start to see more of proactive
  • 00:09:34
    agents that really ping different people
  • 00:09:35
    at the each firm and really collaborate
  • 00:09:37
    effectively uh to get something
  • 00:09:39
    something done got it yeah and maybe
  • 00:09:41
    circling back a little to the question I
  • 00:09:42
    had just asked which is around you know
  • 00:09:43
    how the market has changed overall and
  • 00:09:45
    how chachu BT really was this moment for
  • 00:09:47
    Enterprises to to realize that the cat
  • 00:09:49
    was out of the bag has that changed how
  • 00:09:52
    legal or law firms think about charging
  • 00:09:55
    Etc because one of the things that
  • 00:09:57
    people believed about legal for a long
  • 00:09:58
    time was because of the billable model
  • 00:10:01
    it actually didn't matter from a profit
  • 00:10:02
    perspective how how many hours you spent
  • 00:10:04
    in it even if you know maybe people just
  • 00:10:05
    wanted to go home to their to their kids
  • 00:10:07
    I think this goes back to again the the
  • 00:10:09
    market Dynamic where you have infinite
  • 00:10:11
    demand you you just have to get more
  • 00:10:13
    efficient to service all that demand um
  • 00:10:15
    you know we we started in a seat based
  • 00:10:18
    model um you know we charge basically on
  • 00:10:21
    a on a per seat basis and it's not
  • 00:10:23
    because we don't believe outcome based
  • 00:10:25
    pricing or or paying for the work is is
  • 00:10:27
    the future it's really just because
  • 00:10:29
    we want to make it understandable for
  • 00:10:32
    Enterprise buyers like I think there's
  • 00:10:34
    this uh you know VC statement outcome
  • 00:10:37
    base pricing is a future it's happening
  • 00:10:39
    like I think it will happen um but I
  • 00:10:42
    think what people have to understand is
  • 00:10:44
    Enterprises don't really know how to
  • 00:10:45
    reason about buying outcome based work
  • 00:10:48
    especially for such a experimental
  • 00:10:50
    product like AI um and so I think it'll
  • 00:10:52
    happen over over time I know one thing
  • 00:10:54
    also about deploying um AI into the
  • 00:10:57
    Enterprise for maybe the first time ever
  • 00:10:59
    some of these customers people might not
  • 00:11:00
    know how to use it like it's sort of a
  • 00:11:02
    new UI ux experience um people don't
  • 00:11:05
    really know how to prompt agents A lot
  • 00:11:07
    of the time how do you guys think about
  • 00:11:09
    the types of things that you need to do
  • 00:11:10
    to actually get an Enterprise to like
  • 00:11:12
    meaningfully get value out of an AI
  • 00:11:14
    product yeah so our utilization has
  • 00:11:17
    grown from 40% earlier uh last year to
  • 00:11:21
    70% now what the metric of um so it's uh
  • 00:11:25
    active users over a number of seats on a
  • 00:11:27
    monthly basis basically okay yeah I
  • 00:11:29
    think a lot of that growth has been uh
  • 00:11:31
    driven by you know good old fashioned
  • 00:11:34
    discipline across different functions um
  • 00:11:36
    so maybe starting with uh the GTM sales
  • 00:11:39
    team as I mentioned we have lawyers
  • 00:11:42
    embedded in the in the sales team and
  • 00:11:45
    they really because they come from this
  • 00:11:47
    field because they come from a lot of
  • 00:11:48
    our customer archetypes they put a a lot
  • 00:11:51
    of emphasis into a very specific kind of
  • 00:11:54
    like onboarding program and use case
  • 00:11:56
    building where um you know they speak
  • 00:11:59
    the lingo they speak like um exactly how
  • 00:12:02
    to accomplish a certain use case and so
  • 00:12:04
    it makes it a lot more uh approachable
  • 00:12:07
    for uh for users um so that's that's one
  • 00:12:10
    on the sales and GTM side on the
  • 00:12:12
    customer success side um we've really
  • 00:12:15
    tried to actually gamify a lot of
  • 00:12:17
    deployments um internally um so our
  • 00:12:20
    customer success team often does big
  • 00:12:22
    launches or like use case contests and
  • 00:12:25
    law firms love to post on LinkedIn and
  • 00:12:27
    so uh if if we say hey this person is
  • 00:12:30
    the best you know AI prompt engineer or
  • 00:12:32
    whatever they love to talk about that on
  • 00:12:34
    LinkedIn and creates a real nice kind of
  • 00:12:35
    healthy competitive mentality yeah and
  • 00:12:38
    then the other question is like as you
  • 00:12:39
    expand to other Industries you're two
  • 00:12:41
    years or so into the company now and you
  • 00:12:43
    actually want to expand Beyond legal so
  • 00:12:45
    would love to maybe first understand
  • 00:12:46
    just the rationale behind doing that
  • 00:12:48
    versus maybe going deeper into legal and
  • 00:12:50
    then how applicable do you think the
  • 00:12:52
    product set as well as the go to market
  • 00:12:53
    strategies would be for the New Vertical
  • 00:12:55
    we have a lot of uh legal customers but
  • 00:12:57
    we don't want to rest on a lore and and
  • 00:12:59
    become complacent uh we actually have a
  • 00:13:01
    a cultural principle that says you know
  • 00:13:03
    job's not finished uh it's referencing
  • 00:13:05
    the the Kobe quote I aware of it um and
  • 00:13:08
    so we don't I wasn't but now I we never
  • 00:13:11
    want to be complacent and so a lot of
  • 00:13:13
    our effort is still focused on legal but
  • 00:13:15
    I think overall if you look at
  • 00:13:18
    transactions if you look at litigation
  • 00:13:20
    if you look at lawyers and legal work
  • 00:13:22
    overall there's often times a lot of
  • 00:13:24
    professions involved that are not just
  • 00:13:26
    legal like in a in a transaction if
  • 00:13:28
    you're doing a m&a there's tax people
  • 00:13:30
    involved there's Financial people
  • 00:13:31
    involved uh there's HR people involved
  • 00:13:33
    to to combine the two teams and
  • 00:13:36
    so in general I think it would be
  • 00:13:39
    disservice to say oh only Harvey uh only
  • 00:13:41
    lawyers can use the Harvey and and be uh
  • 00:13:44
    and take advantage of it in inside of
  • 00:13:45
    this transaction and so the way we think
  • 00:13:48
    about it is like as we're doing these
  • 00:13:49
    like larger project based workflows um
  • 00:13:52
    using that to expand to hey maybe the
  • 00:13:55
    tax professional needs to know the same
  • 00:13:56
    thing as the legal person with one maybe
  • 00:13:58
    incremental thing on top and so we're
  • 00:14:01
    really using the lawyers and the
  • 00:14:03
    projects that they work on to expand
  • 00:14:05
    kind of naturally to these verticles and
  • 00:14:08
    um there there's like a few ways to do
  • 00:14:10
    it I mean generally we we take a very
  • 00:14:12
    customer driven approach um so not only
  • 00:14:15
    uh you know a lot of our Enterprise
  • 00:14:16
    customers actually already have their
  • 00:14:18
    compliance and HR teams on Harvey
  • 00:14:20
    because um you know if you're reviewing
  • 00:14:22
    employment contracts like the HR team is
  • 00:14:24
    obviously going to be very involved and
  • 00:14:26
    so um that's like one Avenue is kind of
  • 00:14:30
    organically expanding inside of
  • 00:14:31
    Enterprises uh and then being very
  • 00:14:33
    customer-driven and partnering with kind
  • 00:14:35
    of leading firms so uh we work with PWC
  • 00:14:38
    um to build basically custom tax and uh
  • 00:14:43
    Financial diligence systems um because
  • 00:14:46
    you know uh especially internationally
  • 00:14:48
    they're the experts in in tax law
  • 00:14:49
    they're experts in financial diligence
  • 00:14:51
    and they've really helped us um learn a
  • 00:14:54
    lot about those domains and really push
  • 00:14:55
    us in that direction um and so we've
  • 00:14:57
    been kind of laying the seeds for that
  • 00:14:59
    expansion for a bit and over the next 2
  • 00:15:02
    3 years really going to have naturally
  • 00:15:03
    expand to those areas what do you mean
  • 00:15:04
    when you say like custom models or
  • 00:15:07
    custom workflows for those domains like
  • 00:15:08
    is that custom as in PWC specific and
  • 00:15:12
    therefore like you actually actively
  • 00:15:13
    don't want to bring it into maybe
  • 00:15:15
    similar customers or particularly for
  • 00:15:17
    the tax work tax attorneys across the
  • 00:15:19
    world ask a lot of questions about
  • 00:15:22
    certain tax laws how it can be appli
  • 00:15:24
    applied to their clients and so a lot of
  • 00:15:27
    that knowledge is actually just in PWC
  • 00:15:30
    you know the the world's leading tax
  • 00:15:32
    experts in UK law or UK tax law are
  • 00:15:34
    actually at PWC and so when we say we're
  • 00:15:37
    building custom systems there we're
  • 00:15:39
    actually using a lot of the data that
  • 00:15:40
    they've curated as well as uh using the
  • 00:15:43
    expertise and uh evl from their experts
  • 00:15:46
    to improve that system so we you know
  • 00:15:49
    build uh various fine tune models rag
  • 00:15:51
    systems that incorporate that data and
  • 00:15:53
    eval from those customers so I think PWC
  • 00:15:56
    is unique in that sense but you know
  • 00:15:58
    over time we may start to work with
  • 00:15:59
    other professional service providers as
  • 00:16:00
    well so I I do want to talk a little bit
  • 00:16:02
    more about the product building and how
  • 00:16:03
    you guys think about evals how you think
  • 00:16:05
    about selecting model providers Etc but
  • 00:16:07
    maybe one last point on this is you you
  • 00:16:09
    talk about how PWC has been a great
  • 00:16:11
    partner in designing some of these more
  • 00:16:13
    like custom projects that you guys
  • 00:16:15
    didn't have previously I imagine that
  • 00:16:17
    that required first a lot of trust on
  • 00:16:19
    pwc's part because they're giving you
  • 00:16:21
    very sensitive data and then a lot of
  • 00:16:23
    open questions that I think anyone
  • 00:16:24
    building for the Enterprise or any
  • 00:16:26
    Enterprise buyers have around how is my
  • 00:16:27
    data actually being used both in this
  • 00:16:30
    context is it getting fed back to the
  • 00:16:32
    models is it going to go to you know
  • 00:16:33
    some of my competitors Etc so I'm
  • 00:16:35
    curious how you guys think about those
  • 00:16:37
    questions I think this this is a under
  • 00:16:40
    discussed Topic in in Enterprise
  • 00:16:42
    software in general not just AI like
  • 00:16:44
    Enterprise Readiness goes Way Beyond
  • 00:16:46
    just sock 2 um it is I think a culture
  • 00:16:49
    you have to build with particular your
  • 00:16:51
    product and Engineering teams really
  • 00:16:52
    from the beginning and so you know
  • 00:16:54
    examples of what we've done really from
  • 00:16:56
    the beginning because we started with
  • 00:16:58
    the hardest customer first they work on
  • 00:17:00
    extremely sensitive work across the
  • 00:17:02
    world and it's a big thing for them to
  • 00:17:04
    actually trust a small startup
  • 00:17:06
    relatively to to do that so A few things
  • 00:17:09
    that um we Implement from the beginning
  • 00:17:10
    is I think one you know a strict no
  • 00:17:13
    training policy for for data that's sent
  • 00:17:15
    so by default all our paperwork
  • 00:17:18
    everything you know doesn't allow Harvey
  • 00:17:20
    certainly not to even train that data
  • 00:17:22
    but people at Harvey can't even look at
  • 00:17:23
    the data we call this the term eyes off
  • 00:17:25
    no one in Harvey can even access most of
  • 00:17:27
    our customer data because because it's
  • 00:17:29
    such a sensitive you know set of uh set
  • 00:17:32
    of data another part of it is we have a
  • 00:17:34
    very strict external vendor list we're
  • 00:17:37
    only allowed to use for example Azure
  • 00:17:39
    deployed models to improve our system
  • 00:17:41
    and and and uh Power our product and
  • 00:17:43
    it's because again Azure has a lot of
  • 00:17:46
    trust in the Enterprise like all our
  • 00:17:48
    customers they're all uh on huge Azure
  • 00:17:50
    deployments uh and so they do trust
  • 00:17:53
    Azure a lot and what that also means
  • 00:17:55
    though is you know if a new model comes
  • 00:17:57
    out Google anthropic or a new you know
  • 00:18:00
    fancy tool comes out on Twitter or
  • 00:18:02
    something like we we can't use it right
  • 00:18:05
    away we have to be very strict about
  • 00:18:07
    that and I think again this goes back to
  • 00:18:09
    product and Engineering culture we
  • 00:18:11
    really have to make sure Engineers
  • 00:18:13
    understand that you can't actually just
  • 00:18:14
    you know use the product or deploy it uh
  • 00:18:16
    we really strict about that I think the
  • 00:18:18
    last thing is we really hired a security
  • 00:18:20
    team very early on um like our head of
  • 00:18:22
    security was hired I think as the first
  • 00:18:25
    15 employees or something and and he's
  • 00:18:28
    really helped us develop a really robust
  • 00:18:30
    security program and when he goes in
  • 00:18:32
    front of a CIO or a C Level person at a
  • 00:18:34
    bank they know we are legitimate and we
  • 00:18:36
    we don't sound like a startup basically
  • 00:18:38
    so I think a lot of those mix of things
  • 00:18:40
    has been really crucial to gaining that
  • 00:18:42
    trust and what is your philosophy around
  • 00:18:45
    building applied AI products on the one
  • 00:18:46
    hand you know you get to own the
  • 00:18:47
    customer and that's great and on the
  • 00:18:49
    other hand there's new fun things coming
  • 00:18:51
    out on Twitter every single day there's
  • 00:18:52
    new models basically every month
  • 00:18:55
    nowadays and I imagine that's a very
  • 00:18:57
    tough Foundation to be able to build a
  • 00:18:59
    consistent product on top of yeah so I
  • 00:19:02
    think there's like a few ways there's
  • 00:19:03
    another question also uh we often get
  • 00:19:05
    like how how much do you focus on
  • 00:19:07
    existing workflows and existing surface
  • 00:19:09
    areas for lawyers versus like like a net
  • 00:19:11
    new a native ux I think the the one
  • 00:19:14
    thing we need to highlight is uh there
  • 00:19:16
    is no IDE for lawyers um there's no like
  • 00:19:19
    VSS code or cursor or whatever for
  • 00:19:20
    lawyers um the the two tools that they
  • 00:19:23
    use the most are word and email or
  • 00:19:26
    basically Outlook uh and
  • 00:19:29
    we we are intergrating with both of
  • 00:19:30
    those on email and and word But
  • 00:19:32
    ultimately we didn't really have a
  • 00:19:34
    choice to um build on top of existing
  • 00:19:37
    tools or existing software because there
  • 00:19:38
    really isn't one and so we've really
  • 00:19:40
    opted for a NBI native ux um and and a
  • 00:19:44
    an app and I think there's a few things
  • 00:19:47
    so I think yeah like what what does that
  • 00:19:49
    mean like what is Ani native ux yeah so
  • 00:19:52
    ultimately like one of the main
  • 00:19:53
    principles is we want Harvey to feel
  • 00:19:55
    like a coworker and not just a AI or our
  • 00:19:59
    software we want it to feel like a human
  • 00:20:01
    and if you're working with a human um at
  • 00:20:05
    a law firm or an Enterprise you you can
  • 00:20:08
    basically talk to them and go back and
  • 00:20:10
    forth a lot if you you know give them
  • 00:20:11
    work so if let's say I go up to someone
  • 00:20:13
    and say hey can you draft me this you
  • 00:20:15
    know one-on-one disclosure they if
  • 00:20:18
    you're they're a good cooworker they
  • 00:20:19
    will ask you hey I need more information
  • 00:20:22
    you know can you give me what is the
  • 00:20:23
    information Source like what should I
  • 00:20:25
    base the format and the tone on or what
  • 00:20:27
    deal are we even doing
  • 00:20:29
    um and then you know they may write a
  • 00:20:32
    draft of it and say hey can you check my
  • 00:20:34
    work am I on the right track and I think
  • 00:20:36
    that's really how we want um Harvey to
  • 00:20:39
    feel like is you're going this back and
  • 00:20:41
    forth and you're being guided to do that
  • 00:20:44
    work I think is it like a chatbot UI
  • 00:20:47
    still or like what is the actual UI that
  • 00:20:49
    people are using here yeah so it's like
  • 00:20:51
    a it's like a kind of like a chat UI
  • 00:20:53
    with like kind of dynamic UI components
  • 00:20:56
    that are uh that are surfaced um and I
  • 00:20:59
    think the the other principle that we
  • 00:21:03
    really want to take into account here is
  • 00:21:05
    there's this principle called the Ikea
  • 00:21:06
    effect so um which is basically the idea
  • 00:21:10
    that people feel a lot more responsible
  • 00:21:13
    for what they do if they help build it
  • 00:21:15
    and Ikea really took advantage of this
  • 00:21:18
    right they have they've really kind of
  • 00:21:19
    made the the process of building their
  • 00:21:22
    Furniture very delightful uh and fun and
  • 00:21:24
    you know really invested a lot in the
  • 00:21:25
    manuals and everything and people
  • 00:21:27
    there's like a cult like following for
  • 00:21:29
    Ikea because uh people assemble it
  • 00:21:31
    themselves maybe now nowadays they don't
  • 00:21:33
    as much but uh that they used to they
  • 00:21:35
    used to yeah and so I think for us this
  • 00:21:38
    goes back to you can't you can't onot an
  • 00:21:41
    S1 with 01 um like there's so much back
  • 00:21:45
    and forth that goes into this like
  • 00:21:46
    actual legal work it's complex you need
  • 00:21:48
    humans uh you you know unique data sets
  • 00:21:51
    where um if if we were just like hey you
  • 00:21:55
    know draft this disclosure schedule and
  • 00:21:57
    it and Harvey did it no one would trust
  • 00:21:59
    it because they had no idea what
  • 00:22:00
    actually went into creating that um and
  • 00:22:03
    so we we want to bake in these like
  • 00:22:04
    nudes um and kind of uh we call it
  • 00:22:08
    shoulder Taps uh so that uh Harvey asked
  • 00:22:11
    for feedback ask for data ask for intent
  • 00:22:13
    um before actually producing alcome and
  • 00:22:15
    can you talk through like if I'm an
  • 00:22:17
    individual lawyer like what does that
  • 00:22:18
    look like in practice because I know
  • 00:22:19
    like one of the ux experiments a lot of
  • 00:22:21
    people are trying to figure out is while
  • 00:22:23
    the agent is doing work it'll like
  • 00:22:25
    Expose and it'll tell you what it's
  • 00:22:26
    doing but there's also like some level
  • 00:22:27
    of downtime that happen happens there
  • 00:22:29
    like does the lawyer get like a little
  • 00:22:31
    notification it's like oh come back I
  • 00:22:32
    have a question like how do they
  • 00:22:34
    integrate that with their day-to-day
  • 00:22:36
    work so that it's not just sitting there
  • 00:22:38
    like monitoring the agent one
  • 00:22:40
    interesting thing for for our user base
  • 00:22:41
    and our product is that we're not very
  • 00:22:43
    latency constrainted I think for a lot
  • 00:22:46
    of chat products or you know consumer AI
  • 00:22:49
    products most people expect an instant
  • 00:22:51
    answer yeah but because the the quality
  • 00:22:54
    of the output is so good and so
  • 00:22:56
    humanlike people are okay waiting 2
  • 00:22:59
    minutes 3 minutes 4 minutes to actually
  • 00:23:02
    get an outcome and because of that we're
  • 00:23:05
    able to basically shove more
  • 00:23:06
    intelligence into every single pass uh
  • 00:23:09
    more model calls more you know
  • 00:23:10
    algorithms and so people can wait and
  • 00:23:13
    are fine waiting and we're starting to
  • 00:23:15
    add basically asynchronous agents that
  • 00:23:17
    work where it'll email you when it's
  • 00:23:19
    done or you know ping you when it's done
  • 00:23:22
    and so that latency constraint is just
  • 00:23:24
    not a big constraint for us which allows
  • 00:23:26
    us a lot of freedom to work on as long
  • 00:23:28
    as the the agent is basically providing
  • 00:23:31
    some transparency of what it's doing and
  • 00:23:33
    it's not just you know endless spinning
  • 00:23:35
    I think it works out for our user base
  • 00:23:36
    do you think we've arrived at the point
  • 00:23:38
    that we know like what is the best AI
  • 00:23:40
    native UI or uux experience yet and if
  • 00:23:43
    the answer is yes I'd love to know like
  • 00:23:44
    what it is um and if the answer is no
  • 00:23:46
    what do you think are the experiments
  • 00:23:48
    still being run or what are the the
  • 00:23:49
    types of workflows you think people
  • 00:23:50
    haven't quite figured out yet sure
  • 00:23:52
    answer is no I think uh CH chat is the
  • 00:23:55
    command line of AI I think when m dos
  • 00:23:58
    first came out you were just typing into
  • 00:24:00
    a terminal to move things around like
  • 00:24:02
    that's that's where we are with with AI
  • 00:24:04
    actually I think hopefully in 2025 we
  • 00:24:06
    see a lot more Innovation on uh AI
  • 00:24:08
    native ux Dynamic ux ways to interact
  • 00:24:10
    with the model that is not just text
  • 00:24:12
    based so I think I think that that first
  • 00:24:14
    of all I think what people have to
  • 00:24:17
    realize is you know most users and and
  • 00:24:21
    certainly our users have very
  • 00:24:24
    unspecified queries or prompts it's it's
  • 00:24:27
    interesting people how how comfortable
  • 00:24:30
    people have gone with AI where they just
  • 00:24:33
    assume the AI knows everything like you
  • 00:24:36
    know we we get a lot of uh support
  • 00:24:38
    tickets saying go into my email and
  • 00:24:40
    search up this thing and you know
  • 00:24:41
    produce this outcome or hey do you
  • 00:24:43
    remember when I talked about this last
  • 00:24:45
    time like you know use that to come up
  • 00:24:47
    with with the answer I think it's an
  • 00:24:48
    educational thing but also I think AI
  • 00:24:50
    really has to work collaborate again
  • 00:24:52
    with with the individual to actually
  • 00:24:53
    extract the intent from the individual
  • 00:24:55
    versus just relying on the one shot
  • 00:24:57
    prompt to get it exactly right I'm
  • 00:25:00
    hoping to see more unique back and
  • 00:25:03
    forths and guidance that the agent can
  • 00:25:05
    provide instead of just at Tech space
  • 00:25:07
    prompt I think with with Enterprises you
  • 00:25:09
    actually kind of need this AI native ux
  • 00:25:11
    even more because the work is so complex
  • 00:25:13
    and difficult and often times the work
  • 00:25:16
    is is being done by teams of people or
  • 00:25:18
    or or humans and so you do need a more
  • 00:25:21
    full-fledged kind of natural ux versus I
  • 00:25:24
    think consumer because the use cases are
  • 00:25:26
    so varied and because uh there's so many
  • 00:25:29
    ways to use AI like maybe the best use
  • 00:25:32
    case or best UI is a chat right because
  • 00:25:36
    it's so openend you can capture the
  • 00:25:38
    whole uh Market with just an open-end UI
  • 00:25:40
    and it's kind of what we're seeing so I
  • 00:25:42
    I do think Enterprises there should be
  • 00:25:45
    more experimentation inter with uh a na
  • 00:25:47
    uxs because the workflows are so
  • 00:25:49
    specific because the work is so
  • 00:25:51
    difficult and you know again never one
  • 00:25:52
    shot yeah makes sense maybe switching
  • 00:25:55
    gear slightly I'd love to know like to
  • 00:25:57
    the extent that guys can talk about it
  • 00:25:59
    how do you think about the
  • 00:25:59
    infrastructure under under the hood like
  • 00:26:01
    are you primarily using one model and if
  • 00:26:03
    so what is that how do you think about
  • 00:26:05
    swapping out models as new capabilities
  • 00:26:08
    come out Etc yeah so as I mentioned
  • 00:26:11
    previously Harvey is consist of you know
  • 00:26:14
    hundred of different model calls Bing
  • 00:26:16
    aent or compound AI system to produce
  • 00:26:19
    the output and um you know currently we
  • 00:26:22
    uh primarily use uh open AI models
  • 00:26:24
    either open directly or open AI through
  • 00:26:26
    through Azure uh in
  • 00:26:29
    um and that's particularly because well
  • 00:26:30
    one uh models are really good both the
  • 00:26:33
    Open Eye and Azure infrastructure is
  • 00:26:35
    really good and fast and security and
  • 00:26:37
    customer trust like you know as I
  • 00:26:38
    mentioned earlier people really really
  • 00:26:40
    want to make sure Azure is the kind of
  • 00:26:42
    default cloud of choice for us um and
  • 00:26:45
    that's really how we've been able to
  • 00:26:46
    gain trust but in general we're not
  • 00:26:48
    really tied to open ey we work with all
  • 00:26:50
    the major Labs already actually to
  • 00:26:53
    basically evalve their products and
  • 00:26:54
    provide guidance on how they should
  • 00:26:56
    think about legal reasoning um and you
  • 00:26:58
    know sharing data sets sharing um you
  • 00:27:00
    know insights that we've gleaned and so
  • 00:27:03
    we are certainly open to using all sorts
  • 00:27:05
    of different models um it's just um
  • 00:27:08
    because of security and infrastructure
  • 00:27:10
    constraints we haven't gotten to that
  • 00:27:11
    yet how easy is it to swap a model for
  • 00:27:13
    you guys because you can have like
  • 00:27:14
    because they're non-deterministic you
  • 00:27:15
    can imagine like something weird happens
  • 00:27:17
    like how do you run evals on that
  • 00:27:18
    afterwards to make sure that the
  • 00:27:20
    experience is still consistent if you do
  • 00:27:21
    swap out a model so from an AI
  • 00:27:23
    infrastructure perspective again I think
  • 00:27:25
    from for early on we really tried to
  • 00:27:28
    emphasize modularity um so that we can
  • 00:27:31
    swap model strings in and out and end
  • 00:27:32
    points in and out um the more difficult
  • 00:27:35
    thing is actually the evl as you
  • 00:27:36
    mentioned um each model has a different
  • 00:27:39
    personality characteristic Behavior
  • 00:27:42
    maybe the same prompts um or data for
  • 00:27:44
    fine tting don't work the same way for
  • 00:27:46
    different models uh and so um swapping a
  • 00:27:49
    model in and out uh does require a lot
  • 00:27:52
    of eval because we want to make sure it
  • 00:27:53
    doesn't degrade uh quality so do you
  • 00:27:55
    have you guys built out internal eval
  • 00:27:57
    infrastructure to do this Evo is is a
  • 00:27:59
    big Focus for us you know I'm come from
  • 00:28:01
    scale like uh I know human expert data
  • 00:28:04
    is extremely important to building AI
  • 00:28:06
    systems I think there's like two kind of
  • 00:28:09
    aspects to eval that we think about one
  • 00:28:11
    is basically internal eval to improve
  • 00:28:13
    our AI systems and then there's external
  • 00:28:15
    eval to communicate the value so on the
  • 00:28:18
    internal side we have have uh basically
  • 00:28:20
    a mix of human experts that um we have
  • 00:28:24
    internally or uh that we contract um so
  • 00:28:26
    like lawyers and um all different
  • 00:28:29
    countries all different practice areas
  • 00:28:31
    uh to be able to do all sorts of kind of
  • 00:28:33
    absolute or relative eval so absolute is
  • 00:28:35
    like look at this piece of content and
  • 00:28:38
    um you know rank it based on this rubric
  • 00:28:40
    or whatever it is and then side by side
  • 00:28:41
    is like okay look at two different
  • 00:28:43
    verions that are algorithm and then rank
  • 00:28:45
    it you know side by side um and we
  • 00:28:48
    really invested a lot in that um and you
  • 00:28:51
    know have kind of scaled that up as
  • 00:28:52
    we've as we've grown on the external
  • 00:28:54
    side the the difficulty is is um you
  • 00:29:00
    know a lot of legal work is actually
  • 00:29:02
    applying subjective opinions on
  • 00:29:04
    objective facts and judging subjective
  • 00:29:06
    opinions is very hard there's no there's
  • 00:29:09
    certainly no objective truth like you
  • 00:29:11
    know did you apply the law in this way
  • 00:29:13
    or does your interpretation worse or
  • 00:29:16
    better than the other um so eval overall
  • 00:29:19
    externally and communicating that is
  • 00:29:20
    really hard um and then generally legal
  • 00:29:24
    tasks um externally there's just so many
  • 00:29:28
    like if you look at the legal taxonomy
  • 00:29:30
    of Tas out there there's almost like
  • 00:29:32
    10,000 Leaf nodes and you know lawyers
  • 00:29:35
    have actually um you map this out um and
  • 00:29:38
    so and I think part of the challenge
  • 00:29:41
    here is how do you communicate to
  • 00:29:44
    customers that Harvey is good or
  • 00:29:46
    accurate or doesn't hallucinate or
  • 00:29:47
    whatever it is um so we we spent a lot
  • 00:29:50
    of time we we released this um Benchmark
  • 00:29:53
    called Big law bench um earlier last
  • 00:29:55
    year where it basically um presents
  • 00:29:59
    tasks for that represent real billable
  • 00:30:02
    work that lawyers do on a daily basis
  • 00:30:04
    and it's the first Benchmark of its kind
  • 00:30:06
    like all public legal benchmarks so far
  • 00:30:09
    have been multiple choice I would love
  • 00:30:11
    if legal was multiple choice but legal
  • 00:30:13
    is not multiple choice uh is very
  • 00:30:15
    open-ended and messy um and so that one
  • 00:30:17
    is really saying that that The Benchmark
  • 00:30:19
    we produced is really saying here's real
  • 00:30:21
    work that we know lawyers do and here's
  • 00:30:24
    how Harvey performs and I think the one
  • 00:30:26
    other unique thing that we did is um
  • 00:30:28
    we're not measuring necessarily accuracy
  • 00:30:30
    we are measuring the percent of work
  • 00:30:34
    that the model does compared to a 100%
  • 00:30:38
    human response you mean like time is the
  • 00:30:40
    metric um more more like uh the total
  • 00:30:43
    work so got it maybe it gets you 85 90%
  • 00:30:46
    of the we there to drafting a disclosure
  • 00:30:48
    schedule and maybe the human just gets
  • 00:30:50
    has to get it to 10% um the reason is
  • 00:30:53
    because if you just frame things uh
  • 00:30:56
    based on accuracy people like
  • 00:30:58
    no one wants a 90% accurate agentic
  • 00:31:01
    system right um I think it just it's not
  • 00:31:04
    the exact right um kind of framework to
  • 00:31:06
    the think about uh communicating value
  • 00:31:08
    because um even if you get a 90%
  • 00:31:11
    Complete product that is still helpful
  • 00:31:13
    than starting from zero and then one
  • 00:31:14
    last question on this front which is a
  • 00:31:15
    little bit of a tangent but I was
  • 00:31:17
    thinking as you were talking about the
  • 00:31:19
    infrastructure around swapping out
  • 00:31:20
    models but doing evals to make sure that
  • 00:31:22
    you know that the experience is
  • 00:31:23
    consistent and the product doesn't
  • 00:31:25
    degrade what are your thoughts on the
  • 00:31:26
    new open AI reasoning models cuz I
  • 00:31:28
    imagine like legal is actually one of
  • 00:31:30
    the use cases that is probably on the
  • 00:31:32
    Spectrum more reasoning heavy than a lot
  • 00:31:34
    of other use cases have you seen that to
  • 00:31:36
    be a dramatic difference and like how
  • 00:31:38
    has that applied to like you guys
  • 00:31:39
    thinking about Which models you would
  • 00:31:40
    actually want to use it's been a huge
  • 00:31:43
    unlock for a product and our customers
  • 00:31:45
    one nice thing as I mentioned earlier is
  • 00:31:47
    our customers are like latency is in a
  • 00:31:50
    big constraint and um you know the one
  • 00:31:52
    downside of these reasoning models is
  • 00:31:53
    that they T take time to think um and um
  • 00:31:57
    you know kind of show their their
  • 00:31:59
    thought process and Chain of Thought um
  • 00:32:01
    and so our customers are already used to
  • 00:32:03
    that so kind of putting in these
  • 00:32:05
    reasoning models has actually um been
  • 00:32:07
    very natural uh because of the way we've
  • 00:32:09
    designed our product um and then on the
  • 00:32:12
    the AI side um the models have been
  • 00:32:15
    they're actually really really good at
  • 00:32:17
    long form Drafting and long form
  • 00:32:19
    reasoning like drafting a whole you know
  • 00:32:21
    motion dismiss argument um based on
  • 00:32:24
    pulling from various different facts um
  • 00:32:26
    wouldn't wouldn't have been possible
  • 00:32:28
    before these reasoning models maybe this
  • 00:32:30
    is like getting a little bit too in the
  • 00:32:31
    weeds but I'm trying to think of like
  • 00:32:32
    some of the nice things about like you
  • 00:32:34
    said like seat based is that it's you
  • 00:32:36
    know it's a very clean metric or like
  • 00:32:38
    usage base is kind of a clean metric too
  • 00:32:39
    so for like support tickets it's like a
  • 00:32:41
    ticket is the unit of metric how are you
  • 00:32:43
    defining like unit of work being done
  • 00:32:45
    for one of these eval sets because I
  • 00:32:46
    imagine like people have a hard time
  • 00:32:48
    given that this is relatively new also
  • 00:32:50
    grocking like exactly what that means
  • 00:32:52
    it's incredibly difficult in general and
  • 00:32:54
    it does vary a lot based on the task I
  • 00:32:57
    think there's is it based on the task
  • 00:32:59
    but also based on our customers like the
  • 00:33:02
    way you you create a chronology for a
  • 00:33:04
    case might be very different from Law
  • 00:33:06
    Firm to Law Firm um and so I think we've
  • 00:33:09
    thought about like let's try to
  • 00:33:10
    standardize the names and the taxonomies
  • 00:33:13
    of these tasks first and then devise
  • 00:33:15
    rubrics for like okay um you know maybe
  • 00:33:18
    Law Firm a and Law Firm B have the the
  • 00:33:21
    date column in a chronology in a
  • 00:33:23
    different place but it at least has the
  • 00:33:24
    date right and so I think we've actually
  • 00:33:27
    developed a whole rubric and this is
  • 00:33:28
    where a lot of our internal expertise
  • 00:33:30
    comes in um for each kind of major task
  • 00:33:33
    that we've evaluated that is unique that
  • 00:33:35
    rubric is unique uh to that task uh and
  • 00:33:38
    we've tried to standardize it but um
  • 00:33:40
    there there is so much variance has
  • 00:33:42
    Harvey built Its Own Foundation model or
  • 00:33:43
    do you guys have any plans to the short
  • 00:33:45
    answer is no we have not built a Our Own
  • 00:33:47
    Foundation model um instead we've um got
  • 00:33:51
    of work really closely with opening ey
  • 00:33:52
    to uh fine tune to Post train to prompt
  • 00:33:55
    engineer to do rag uh to build up these
  • 00:33:58
    agent to compound AI systems got it do
  • 00:34:00
    you guys want to build your own
  • 00:34:02
    Foundation model eventually and I'm just
  • 00:34:04
    curious whatever the answer is like what
  • 00:34:06
    was your rationale behind either yes or
  • 00:34:07
    no so short answer is no we don't want
  • 00:34:09
    to build Our Own Foundation model um the
  • 00:34:12
    I think the compute stats are out there
  • 00:34:14
    but it's extremely expensive um and we'd
  • 00:34:17
    rather leave it you rais a lot of money
  • 00:34:19
    yeah it do they did raise a lot of money
  • 00:34:22
    billions uh and we'd rather leave that
  • 00:34:24
    to The Experts um and really focus on
  • 00:34:26
    delivering our end customer value and
  • 00:34:28
    and kind of the products around that
  • 00:34:30
    okay so you guys don't want to build
  • 00:34:30
    your own Foundation model I'm curious
  • 00:34:32
    then as you think about the foundation
  • 00:34:34
    models getting better and better you
  • 00:34:36
    know a lot of people are like oh AGI is
  • 00:34:37
    almost three to five years away or
  • 00:34:39
    whatever do you view the foundation
  • 00:34:41
    models as ultimately competitors as they
  • 00:34:44
    generally get better at reasoning
  • 00:34:45
    capabilities there is the ability to do
  • 00:34:47
    more domain specific things now we have
  • 00:34:49
    to assume the models just get you know
  • 00:34:51
    better and better and so what does that
  • 00:34:53
    mean for us we have to accumulate
  • 00:34:55
    different types of advantages um and not
  • 00:34:58
    just the model itself and so few few of
  • 00:35:00
    those advantages are um you know product
  • 00:35:04
    data uh Network and brand um and so few
  • 00:35:08
    of these things are uh on the product
  • 00:35:10
    side so there's ux and kind of the
  • 00:35:12
    Enterprise platform so I think most
  • 00:35:14
    people again uh underestimate what it
  • 00:35:17
    takes to actually deploy products in the
  • 00:35:19
    Enterprise um I think even AGI is
  • 00:35:21
    probably going to underestimate what it
  • 00:35:23
    takes to go through security checks at
  • 00:35:24
    at a bank and and so again we've built a
  • 00:35:27
    lot of this these Security checks
  • 00:35:28
    permissions audit logging um usage
  • 00:35:31
    dashboards all this Enterprise and admin
  • 00:35:34
    functionality that's uh really required
  • 00:35:36
    and you know companies like sap service
  • 00:35:39
    now workday they've invested decades in
  • 00:35:41
    this stuff and this is why Enterprises
  • 00:35:43
    you know like them and enjoy them so I
  • 00:35:45
    think investing in Enterprise platform
  • 00:35:47
    is important ux is also extremely
  • 00:35:49
    important as I mentioned the ux that AI
  • 00:35:52
    is going to use to collaborate with
  • 00:35:54
    whole organizations is not going to be a
  • 00:35:56
    chat-based product and um so we need to
  • 00:36:00
    really innovate on the ux and how you do
  • 00:36:03
    workflow specific ux um that uh you can
  • 00:36:07
    collaborate with AI on um so that's
  • 00:36:09
    another one and then data sets I think
  • 00:36:11
    is really important so you know AGI is
  • 00:36:13
    not going to have the data that is
  • 00:36:16
    sitting on some on-prem server at a law
  • 00:36:18
    firm right and and this happens a lot of
  • 00:36:20
    law firms have on-prem servers and and
  • 00:36:23
    so what really makes the law firm unique
  • 00:36:25
    is a lot of the historic deals and cases
  • 00:36:28
    and data that they actually have and so
  • 00:36:30
    we're starting to basically have Harvey
  • 00:36:33
    be able to use that data and tailor
  • 00:36:35
    outputs workflows uh based on that data
  • 00:36:38
    so I I think overall there's these like
  • 00:36:41
    product ux um kind of advantages when
  • 00:36:43
    you accumulate how much has all the AI
  • 00:36:47
    Zeitgeist all the things that we hear
  • 00:36:49
    about coming out weekly how much has
  • 00:36:50
    that actually permeated in the
  • 00:36:52
    Enterprise and um what do you think like
  • 00:36:55
    is the latency there of like us hearing
  • 00:36:57
    about it versus something actually
  • 00:36:58
    getting deployed there in a similar way
  • 00:37:00
    to how Silicon Valley gets information
  • 00:37:02
    you know often times through X now um a
  • 00:37:05
    lot of our law fir customers get
  • 00:37:07
    information through Linkedin and so the
  • 00:37:09
    best way for me to understand our
  • 00:37:11
    personas is actually to look at a lot of
  • 00:37:13
    LinkedIn post from a lot of our personas
  • 00:37:15
    and see what they're liking see what who
  • 00:37:16
    they're following uh because that's
  • 00:37:18
    really where the zeist and and
  • 00:37:19
    conversation happens um I think overall
  • 00:37:22
    um maybe like this time last year we
  • 00:37:24
    would we would actually go to customers
  • 00:37:27
    and they they would have never heard
  • 00:37:28
    about Chach BT like sure AI but never
  • 00:37:31
    heard about Chach BT like end of 2023
  • 00:37:33
    beginning of 2024 uh no beginning of
  • 00:37:35
    20124 yeah exactly yeah um they would
  • 00:37:37
    have never heard of Chach BT um they
  • 00:37:39
    never even used it um and and that was
  • 00:37:43
    like a wake call for me because again
  • 00:37:45
    coming from scale I was surrounded by a
  • 00:37:46
    i for a long time I'm like okay wow this
  • 00:37:48
    has not actually permeated that as much
  • 00:37:50
    as I thought um I think fast forward to
  • 00:37:52
    now most people have heard about trbt um
  • 00:37:56
    but you know often times people don't
  • 00:37:58
    use it like I think now if you ask
  • 00:38:00
    anyone in Tech like why don't you use
  • 00:38:01
    chbt you're you're at a disadvantage but
  • 00:38:04
    most of uh our law from customers and
  • 00:38:06
    and that World um often don't use it but
  • 00:38:09
    they've at least heard about it um and
  • 00:38:11
    then I think for the
  • 00:38:14
    Enterprises they have you know it's been
  • 00:38:17
    like two-ish two and a half years since
  • 00:38:19
    the since PT like they have at least
  • 00:38:22
    deployed in some intern uh internal
  • 00:38:25
    chapot or have just co-pilot um and
  • 00:38:29
    maybe um maybe use it to draft emails or
  • 00:38:32
    or whatever but we haven't really seen
  • 00:38:35
    even even in leading Enterprises and not
  • 00:38:38
    just law firms we haven't really seen
  • 00:38:40
    workflow specific adoption of AI in the
  • 00:38:43
    way that Harvey is trying to trying to
  • 00:38:45
    push um so I just think like this goes
  • 00:38:48
    back to my bottom like question I just
  • 00:38:49
    think we're so so early um like AGI
  • 00:38:53
    takeoff can happen and the LinkedIn law
  • 00:38:55
    firms are never going to hear about it
  • 00:38:57
    for five years so I think that's that's
  • 00:39:00
    honestly been a um a good empathy test
  • 00:39:03
    for a lot of our team is most people
  • 00:39:05
    don't know that this is happening um and
  • 00:39:07
    so another reason for a lot more apply
  • 00:39:10
    to ass startups to really really go into
  • 00:39:12
    these you know quotequote hidden markets
  • 00:39:14
    because um it is just wide open so then
  • 00:39:17
    I guess that my next question you you
  • 00:39:20
    may have already answered it is have
  • 00:39:22
    they thought about how their business
  • 00:39:24
    model or Staffing model needs to adapt
  • 00:39:26
    as a result of Ai and maybe the answer
  • 00:39:28
    is like no because on LinkedIn you're
  • 00:39:29
    not seeing people talk about impending
  • 00:39:31
    AGI um but at least in Silicon Valley
  • 00:39:33
    people talk about that a lot when it
  • 00:39:34
    comes to Professional Services or
  • 00:39:36
    billing based models yeah I think this
  • 00:39:38
    the the mindset has it has it actually
  • 00:39:41
    changes maybe every 3 to six months and
  • 00:39:43
    that's probably the the leading or
  • 00:39:45
    lagging time that um of information but
  • 00:39:48
    you know like six months ago um clients
  • 00:39:51
    of law firms basically would have said
  • 00:39:55
    don't use AI on my projects um you know
  • 00:39:58
    because X Y and Z trust concerns risk
  • 00:40:00
    concerns um but end of last year now
  • 00:40:03
    they're just like you have to use AI on
  • 00:40:05
    our projects because it's going to be
  • 00:40:06
    more efficient and so I think this this
  • 00:40:09
    is evolving uh quite a bit and this
  • 00:40:11
    understanding is evolving quite a bit um
  • 00:40:13
    I think there are um more bleeding edge
  • 00:40:17
    companies and and and customers that
  • 00:40:19
    we've partnered with have really leaned
  • 00:40:20
    into hey we think AI is going to fully
  • 00:40:23
    change how our practice works we we
  • 00:40:25
    should get on and try to try to drive
  • 00:40:27
    and control it so I think there are the
  • 00:40:29
    the more Visionaries uh who are who are
  • 00:40:31
    thinking about this but in general
  • 00:40:33
    people people know something is going to
  • 00:40:35
    happen um but they don't know what and
  • 00:40:37
    they don't know how it's going to change
  • 00:40:39
    neither do we yeah neither do we yeah it
  • 00:40:41
    gets better seemingly every single day
  • 00:40:43
    um and you know there's new capabilities
  • 00:40:45
    companies popping up all the time now
  • 00:40:47
    how do you guys think about or how do
  • 00:40:49
    you think about you know the next couple
  • 00:40:51
    of years like if you have any
  • 00:40:52
    predictions on like where are do you
  • 00:40:54
    think most people are actually going to
  • 00:40:55
    find Value in the enter PR in particular
  • 00:40:58
    in AI like what do you think are still
  • 00:41:00
    the unlocks that need to happen such
  • 00:41:02
    that more places can actually see Roi
  • 00:41:04
    Etc I think you know in Silicon Valley
  • 00:41:06
    we talk a lot about you know AI takeoff
  • 00:41:08
    or AGI takeoff that you're going to the
  • 00:41:11
    model's going to get so good and you're
  • 00:41:12
    just it's ramp and then everyone's going
  • 00:41:14
    to live happily and never have to work
  • 00:41:15
    again and just retire again I I just
  • 00:41:18
    think like intelligence isn't the only
  • 00:41:20
    thing you need um you run into human
  • 00:41:22
    bottlenecks deploying this stuff like I
  • 00:41:24
    think you've run into you quotequote
  • 00:41:26
    softare B like trust like um you know
  • 00:41:30
    the ability to work well with the model
  • 00:41:32
    and so I think I would encourage and I
  • 00:41:35
    hopefully we see this more in 2025 of
  • 00:41:38
    encourage more Enterprise AI companies
  • 00:41:40
    to get really really deep with their
  • 00:41:42
    customers and understand their workflows
  • 00:41:44
    at a pretty deep level so that they can
  • 00:41:46
    bring AI to them in very specific ways
  • 00:41:50
    and build kind of the product and and ux
  • 00:41:52
    around it and establish that Enterprise
  • 00:41:54
    trust and so I I don't I don't believe
  • 00:41:56
    at least in the next like two three
  • 00:41:58
    years we're going to reach AGI Heaven um
  • 00:42:00
    it is contining going to be uh really
  • 00:42:02
    customer focused Builders applying Ai
  • 00:42:04
    and unique ways through Enterprise
  • 00:42:06
    workflows as well
  • 00:42:08
    [Music]
タグ
  • AI
  • LegalTech
  • Automation
  • Harvey
  • Law Firms
  • Generative AI
  • Enterprise AI
  • Data Security
  • Product Development
  • Market Trends