00:00:03
[Applause]
00:00:13
okay excellent uh good afternoon
00:00:15
everybody so um we're here today to to
00:00:17
talk to you about a actual live and
00:00:20
delivered project that we both worked on
00:00:24
from a contract po perspective and also
00:00:26
uh at
00:00:27
PWC um the premise of the project um is
00:00:31
using and focus specifically on
00:00:33
generative AI um it's using our
00:00:36
Discovery tool um at contrap um and the
00:00:40
way that we went about this was looking
00:00:42
at what is effectively a very highly
00:00:45
complex legal use case um around the
00:00:48
analysis of very complex and a large
00:00:50
amount of data um Jeff do you want to
00:00:52
start off by just giving a bit of
00:00:53
background on the on the client sure
00:00:55
great thanks Mark it's great to be here
00:00:57
as as Mark mentioned this is a live
00:01:00
project that we actually delivered and
00:01:02
you're going to see some metrics later
00:01:04
on on the accuracy of the the Leah
00:01:06
platform but uh PW our client pwc's
00:01:09
client um is private Equity Firm Global
00:01:13
also has a venture capital arm and they
00:01:15
have approximately 500 portfolio
00:01:17
companies and each month each quarter
00:01:20
there's a responsibility to extract
00:01:22
information from a reporting standpoint
00:01:25
not only for legal but for accounting
00:01:27
boards so there was this cross-section
00:01:29
of of legal and accounting that was very
00:01:33
critical and multiple jurisdictions the
00:01:36
PWC team was du conducting this very
00:01:39
manually uh you through our teams across
00:01:41
the globe so we have a lot of you time
00:01:43
and motion study that we we looked at
00:01:45
and um Wanted a solution that could
00:01:48
quickly get us accuracy on basic
00:01:51
extractions but then generate a
00:01:53
conclusion because the our client has a
00:01:55
thinly staffed legal department and
00:01:57
compliance function and it was taking up
00:01:59
time and and taking them away from
00:02:00
strategic work so uh our our goal was
00:02:04
twofold and uh we brought it to our
00:02:06
partner contract po Ai and and jointly
00:02:09
architected the
00:02:10
solution so so let's talk to you guys a
00:02:13
little bit about just what that process
00:02:15
looks like first of all the the impact
00:02:17
that we can make using generative I
00:02:20
about delivering these highly complex
00:02:22
projects is very different to where we
00:02:24
were before um uh that starts with uh
00:02:28
how we interacted with PW see and Jeff
00:02:30
you want to just talk a little bit about
00:02:32
the scoping piece at the start of this
00:02:33
project yeah so as as mentioned it was a
00:02:35
very manual process and and and we had
00:02:38
you crossb teams working on this so the
00:02:41
first step was looking at what our
00:02:43
client's goal was and how we needed to
00:02:46
conform with the reporting requirements
00:02:48
for the chief legal officer but also the
00:02:50
chief compliance officer and then the
00:02:52
accounting board standard so built out a
00:02:55
workflow hand inand with contract pod Ai
00:02:59
and then de developed a a a a test case
00:03:02
where we looked at roughly 10 portfolio
00:03:06
companies to build a data model and I
00:03:08
think one of the key takeaways for us is
00:03:10
W with what was exciting about this
00:03:12
project our ability to focus on what is
00:03:15
uh a very interesting one-off project
00:03:18
actually the true success of this is can
00:03:20
we turn this into a repeatable model
00:03:24
that impacts the client on a quarterly
00:03:26
basis um and for the long term so if you
00:03:28
think about the initial scoping being
00:03:30
done by PWC that then comes over to our
00:03:33
team at at contractpodai again we're
00:03:35
using our Lear Discovery module and
00:03:37
we're using multiple large language
00:03:39
models now the way we go about this is
00:03:42
first of all we are actually getting the
00:03:44
the relevant expectations in terms of
00:03:46
what the actual answers look like um and
00:03:49
then we're running that through our
00:03:51
generative AI solution and we're
00:03:53
starting to then do a quality control
00:03:56
and testing of the actual results now as
00:03:59
Jeff rightly mentioned one of the
00:04:02
biggest impacts of this was it's one
00:04:04
thing being able to actually just get
00:04:06
the answer um from Leah um what is very
00:04:09
important for the project and for the
00:04:11
client's success was not only the answer
00:04:14
but then actually how Le has got to that
00:04:16
answer so it gives the relevant analysis
00:04:19
um it gives the relevant detail around
00:04:21
how that conclusion has been reached
00:04:24
because ultimately moving forward and
00:04:25
why this is a co-pilot project moving
00:04:28
forward the the law on the client side
00:04:31
need to be able to interpret the results
00:04:33
and they will always come down to some
00:04:35
form of interpretation of of law now
00:04:38
once we were then Happy on a contract
00:04:40
pod perspective in terms of the quality
00:04:42
and we' then pass that back to the PWC
00:04:45
team um they would run further QA
00:04:48
against it and then that would come back
00:04:49
in uh where we would then do um
00:04:52
refinement of the prompts that we're
00:04:54
using to make sure that the accuracy is
00:04:57
as high as possible you'll you'll notice
00:04:59
on this Slide the full project for full
00:05:02
delivery was within the 6 week period
00:05:05
actually we delivered the the the
00:05:07
initial phase of the project actually
00:05:08
within a 3 we period right yeah it was
00:05:10
actually 25 portfolio companies in the
00:05:13
first phase the first three weeks and
00:05:15
the the next slide you'll see the the
00:05:17
results which were for that first phase
00:05:19
but we had such great success in the
00:05:21
first three weeks we doubled the size of
00:05:23
the sample set to 50 portfolio companies
00:05:27
and uh we're extremely pleased with the
00:05:30
results as was our as was our client
00:05:32
yeah absolutely so so let's talk a
00:05:34
little bit about the the the results
00:05:36
themselves so again this is talking
00:05:38
about what we did in that initial 3-week
00:05:41
phase um so thinking about the amount of
00:05:44
pages that were reviewed and again this
00:05:46
is what why we have the ability to
00:05:49
perform these types of projects at the
00:05:51
moment our our focus on not just relying
00:05:54
on a single llm but also having the
00:05:57
ability to to buy pass any restriction
00:06:01
constraints so we don't have
00:06:03
restrictions on the amount of data
00:06:04
that's coming in and importantly we
00:06:07
don't have restrictions on the output
00:06:09
when we're thinking about the results
00:06:11
themselves um as Jeff mentioned the
00:06:13
initial phase was around 20 entities um
00:06:16
and the accuracy results even within the
00:06:18
first phase as you can see it just over
00:06:21
98% was exceptionally high now that goes
00:06:24
up in terms of the second phase delivery
00:06:26
that that we talked about um but the the
00:06:29
True Result was that from a client's
00:06:31
perspective and and from our perspective
00:06:34
Leah was able to outperform a lot of the
00:06:37
human manual tasks that were being
00:06:40
achieved within this project there were
00:06:42
some very interest examples where when
00:06:46
both lawyers on both of our sides and
00:06:48
legal Engineers were reviewing some of
00:06:50
the results initially we actually
00:06:52
questioned whether what Le was passing
00:06:55
out in terms of the result was accurate
00:06:57
um and there was a few examples where
00:06:59
where actually both lawyers on both
00:07:01
sides and our legal Engineers actually
00:07:03
changed their conclusion based on the
00:07:06
information that LE was was putting out
00:07:09
um so uh yeah a highly successful but
00:07:13
now an ongoing project yeah and I think
00:07:14
the impact to our client was uh it was
00:07:18
multipronged and in in essence they had
00:07:20
that thinly staff team who there was a
00:07:23
morale issue that was not measurable
00:07:25
until after this seeing the results
00:07:28
where they uh felt comfortable they were
00:07:30
then be a they were able to be uh
00:07:32
deployed on more strategic initiatives
00:07:34
it helped our team it helped reduce
00:07:36
their spend with us with PWC uh so it's
00:07:40
growing our relationship and trust that
00:07:42
we're bringing best to breed technology
00:07:44
and you know I will also say I think
00:07:46
they were of the mindset they needed to
00:07:49
be with one llm I think what was really
00:07:52
appealing and and a selling point for
00:07:54
them when they saw these results was the
00:07:56
strategy that contract pod deploys with
00:07:58
Leah which is the multiple llm
00:08:01
strategy okay excellent we'll we'll
00:08:03
leave you with a few takeaways just in
00:08:05
terms of what was successful with this
00:08:07
project I'll I'll just call a couple out
00:08:10
I think the scalability angle so we you
00:08:12
know we've started the relevant work
00:08:14
with this client as an example but how
00:08:16
do we then scale this uh across their
00:08:18
business um and also just identifying
00:08:21
what is there's a lot of very cool stuff
00:08:23
being discussed obviously around geni
00:08:25
the difference between what is possible
00:08:28
versus what can actually be delivered
00:08:30
today we obviously wanted to focus on
00:08:32
what could be delivered for the client
00:08:33
so um thank you for your time and enjoy
00:08:35
the rest of the
00:08:37
[Applause]
00:08:42
event
