00:00:00
[Music]
00:01:01
[Applause]
00:01:02
[Music]
00:01:21
[Music]
00:01:24
n
00:01:35
[Music]
00:01:41
[Music]
00:01:48
[Music]
00:01:57
hello everyone welcome to today's pleas
00:01:59
presentation on generative AI basics for
00:02:02
B2B product managers my name is Punit
00:02:05
Goyle I've been in product management
00:02:08
for over 10 years and in the tech
00:02:10
industry for over
00:02:11
20 most recently I was a product lead at
00:02:14
Google where I worked on a number of
00:02:16
products including Google Chat other
00:02:19
social media products security products
00:02:22
and HR tooling before Google I let
00:02:25
product at several midsize companies
00:02:27
I've also done product marketing at
00:02:30
Adobe and I started my career as a
00:02:33
software engineer at
00:02:35
Oracle the reason I wanted to do this
00:02:37
session is because gen is so new and
00:02:41
many product managers don't know how to
00:02:43
use it within their own products most
00:02:46
people have used chat GPT and have seen
00:02:48
other chat Bots but don't quite know how
00:02:50
to use geni in Enterprise applications
00:02:54
My Hope Is that after the session you
00:02:56
will know the basics of how you can use
00:02:58
gen in business business
00:03:00
applications in this session we'll cover
00:03:02
what geni is the capabilities that it
00:03:05
offers how you can make it work in your
00:03:08
applications how you can measure success
00:03:11
and also cover the challenges and
00:03:13
limitations of gen so let's dive
00:03:17
in generative AI or gen refers to a type
00:03:22
of AI that can create new content based
00:03:25
on patterns it learns from training data
00:03:29
this is in contrast to traditional AI
00:03:32
which is typically designed to perform
00:03:34
specific predefined tasks with Genai the
00:03:38
system learns from large data sets of
00:03:41
existing content such as documents
00:03:43
images and audio after this training
00:03:46
process the AI can then generate new
00:03:50
unique content that reflects the
00:03:52
characteristics of the training data
00:03:55
this could include writing an original
00:03:57
poem composing music or producing
00:04:00
realistic
00:04:01
images in other words gen can create
00:04:05
content at scale this makes it a really
00:04:08
powerful tool for a wide range of
00:04:10
applications including business
00:04:15
applications let's look at how gen
00:04:17
differs from traditional AI systems
00:04:21
traditional AI is typically designed to
00:04:23
perform specific well-defined tasks
00:04:26
based on predefined rules and patterns
00:04:29
in contrast gen is much more flexible
00:04:32
and adaptive it doesn't require
00:04:35
extensive training for each individual
00:04:38
task and it can create entirely new
00:04:41
content that resembles human created
00:04:45
work this flexibility and generative
00:04:47
capability makes gen well suited for
00:04:50
tasks that have variation rather than
00:04:52
those that require strict consistency
00:04:55
and
00:04:56
precision it opens up a wider array of
00:04:58
use cases compared to traditional AI for
00:05:02
product managers a big benefit is that
00:05:04
you can experiment with geni yourself
00:05:07
without having to find AI experts on
00:05:09
your team this helps you incrementally
00:05:12
add AI capability to your
00:05:16
products so what are the some of the key
00:05:18
capabilities of gen that can be
00:05:21
leveraged in business
00:05:23
applications when we talk about gen the
00:05:25
first example that comes to mind is a
00:05:27
chatbot this is mostly because chat GPT
00:05:30
was likely the first ni product you used
00:05:34
extensively I want you to think more
00:05:36
broadly about how gen can be
00:05:39
used personally I like to think of three
00:05:41
broad categories that gen can be used in
00:05:45
it can create content it can modify
00:05:48
content it can also analyze
00:05:52
content let's start with the generative
00:05:55
capability gen can be used to generate a
00:05:58
variety of content in including product
00:06:00
descriptions marketing copy user manuals
00:06:03
and other text based
00:06:05
content it can also create audio
00:06:08
recordings for voiceovers podcasts and
00:06:11
audio
00:06:12
Snippets additionally jni can be used to
00:06:14
generate product images designs and
00:06:18
visualizations this can be particularly
00:06:20
useful for creating assets at scale for
00:06:23
example in e-commerce
00:06:25
applications Beyond just generating new
00:06:27
content gen can also Al be used to
00:06:30
modify existing content for example it
00:06:33
can generate summaries of lengthy
00:06:36
documents reports or customer
00:06:39
conversations it can also rephrase
00:06:41
existing text to achieve a different
00:06:43
tone such as converting bullet points to
00:06:46
paragraphs in an
00:06:48
email gen can also be used to localize
00:06:51
content for different regions by
00:06:52
translating and culturally adapting
00:06:54
materials like user
00:06:56
manuals finally gen can be used to
00:06:59
analyze existing content it can turn
00:07:03
unstructured content into structured
00:07:05
data extract metadata from documents and
00:07:09
identify Trends and patterns in customer
00:07:11
feedback or other data sources you can
00:07:15
hopefully start to see how you might use
00:07:17
one of these three different categories
00:07:19
of capabilities in your applications of
00:07:22
course you can also mix and match these
00:07:26
capabilities let's walk through an
00:07:28
example from my own work work I was
00:07:30
responsible for improving the experience
00:07:32
users had with the performance
00:07:34
management system it's essential to
00:07:37
start with the problem that you want to
00:07:39
solve the problem we had was that it was
00:07:42
timec consuming for managers to do
00:07:44
performance
00:07:45
reviews we analyzed where managers were
00:07:48
spending their time one area that we saw
00:07:51
where it took them a lot of time was to
00:07:53
read through peer reviews and one-on-one
00:07:56
notes the solution was to summarize peer
00:07:58
reviews and one-on-one
00:08:00
notes in terms of roll out we wanted to
00:08:04
make sure we ran experiments to validate
00:08:06
our hypothesis so we rolled out the
00:08:09
feature as an optin we looked at opt-in
00:08:12
rates and for the users who opted in we
00:08:14
looked at the time spent on performance
00:08:17
reviews compared to other users this
00:08:20
helped us understand the value of the
00:08:21
solution and whether we were actually
00:08:24
providing enough value to the
00:08:28
users now let's dive into how you can
00:08:31
make geni work in your
00:08:33
applications a key aspect of this is
00:08:36
called prompt engineering or more simply
00:08:39
crafting effective prompts prompts are
00:08:42
how you instruct gen to produce the
00:08:46
desired
00:08:47
output the quality of the prompt
00:08:49
significantly influences the quality of
00:08:51
the model's response right now it's an
00:08:54
art not a science and requires
00:08:57
experimentation to get things right
00:09:00
the goal is to guide the AI towards a
00:09:02
specific type of output by providing
00:09:05
relevant variables and context for
00:09:08
example you might prompt the system to
00:09:10
create a personalized email sequence for
00:09:12
re-engaging past clients in an
00:09:15
industry if you're targeting someone in
00:09:18
the manufacturing industry you can
00:09:20
specify that as a variable and because
00:09:23
you're specifying the industry each time
00:09:25
the response from the llm is likely to
00:09:27
be much more relevant for that
00:09:32
industry another technique for making
00:09:35
gen work effectively is called retrieval
00:09:38
augmented generation or simply
00:09:40
rag let's take the example of a support
00:09:43
bot the user in this example asks the
00:09:45
question how do I create a new account
00:09:49
the llm of course knows nothing about
00:09:50
your system because it's been trained on
00:09:53
public internet data how do we enable
00:09:55
the llm to answer the question about
00:09:57
your
00:09:58
system the first step is to look through
00:10:01
support documentation and find the
00:10:03
relevant section that contains the
00:10:05
answer this kind of retrieval is enabled
00:10:09
through the use of a technique called
00:10:11
embedding embeddings are representations
00:10:13
of objects like text images and audio
00:10:17
that are designed to be consumed by AI
00:10:19
models they translate objects into a
00:10:22
mathematical form according to the
00:10:25
factors or traits an object may or may
00:10:27
not have and the cat categories that
00:10:29
they might belong to essentially
00:10:32
embeddings enal AI models to find
00:10:35
similar
00:10:36
objects given a user query you can find
00:10:39
the relevant section or a similar
00:10:41
section that might contain the answer to
00:10:44
the
00:10:46
question you can now feed the relevant
00:10:48
section to the llm as part of the prompt
00:10:51
with rag gen can access and incorporate
00:10:55
relevant information from external
00:10:57
sources such as documentation or a
00:10:59
knowledge base and this allows the llm
00:11:02
to provide more contextual and accurate
00:11:05
responses and frankly responses that are
00:11:09
helpful back to the question that the
00:11:11
user asked how do I create a new account
00:11:13
the llm can retrieve the relevant
00:11:15
section from the support documentation
00:11:18
and incorporate that information into
00:11:20
its
00:11:23
response in addition to the support
00:11:26
documentation you can also feed an
00:11:28
additional contact actual information so
00:11:30
for example if a user is logged in and
00:11:33
has a question you might want to look up
00:11:35
the user profile uh for that logged in
00:11:38
user to give them more personalized
00:11:42
responses you will need to add this
00:11:44
personalized context to the llm prompt
00:11:48
and depending on your use case you will
00:11:50
have to figure out what information the
00:11:51
llm needs to provide a good response
00:11:54
back to the
00:11:58
user as as you implement gen in your
00:12:00
applications it's really important to
00:12:02
establish metrics for measuring
00:12:05
success and this is really important
00:12:07
because it's very very hard to come up
00:12:10
with absolute objective metrics that
00:12:14
evaluate the quality of an llms
00:12:17
response the first type of metrics to
00:12:19
look at are quality metrics the ai's
00:12:22
output should closely match the expected
00:12:24
results and you can measure this by
00:12:26
looking at error rates or success rates
00:12:28
in the achieving the desired
00:12:30
outcome the generated content must be
00:12:33
pertinent to the user's context and
00:12:35
needs and relevance can be Quantified
00:12:37
through user engagement metrics or task
00:12:40
completion
00:12:41
rates the AI should be able to deliver
00:12:44
stable and reliable performance over
00:12:45
time which you can monitor through
00:12:47
longitudinal studies or regular quality
00:12:50
checks the AI should also be able to
00:12:53
handle a diverse set of user
00:12:54
requirements and
00:12:56
inputs you can measure measure a
00:12:59
performance across different use cases
00:13:01
to understand the diversity
00:13:05
metric finally your product or feature
00:13:08
should make the users happy you can run
00:13:11
user satisfaction surveys to get
00:13:12
insights into the perceived value and
00:13:15
the effectiveness of the AI you should
00:13:17
also look at usability as this can
00:13:19
impact adoption this can be measured you
00:13:22
through user testing time spent on tasks
00:13:26
and error rates during interaction with
00:13:28
the AI system
00:13:30
and as your Gathering metrics you should
00:13:32
continuously improve the product your
00:13:34
feedback loops should inform AI
00:13:36
improvements and you should be measuring
00:13:38
these metrics continuously not just one
00:13:41
time uh and based on what you learn uh
00:13:44
you should be updating the AI system and
00:13:47
make sure you're able to update the AI
00:13:49
system in response to the user feedback
00:13:52
and changing the
00:13:56
requirements of course even the most
00:13:58
advanced generative AI systems can
00:14:00
encounter failures and produce
00:14:02
inaccurate or nonsensical outputs uh
00:14:06
this is often referred to as
00:14:08
hallucinations this is something to be
00:14:10
prepared for and have a plan in place to
00:14:13
address a typical approach is a humanin
00:14:16
the loop approach where human experts
00:14:19
review and correct the ai's outputs um
00:14:22
as a way to deal with these
00:14:24
failures the human feedback can be used
00:14:26
to further train and improve the AI
00:14:28
system to reduce future errors in
00:14:31
practice what this often might include
00:14:33
is changing your prompts or changing the
00:14:35
data in the context you're feeding to
00:14:37
the
00:14:40
llm and while gen AI offers so many new
00:14:44
and exciting possibilities you should be
00:14:47
aware of some key
00:14:50
challenges identifying valid use cases
00:14:53
that provide significant value is
00:14:55
crucial not every problem needs to be
00:14:58
solved with Gen
00:15:00
ensuring the availability and quality of
00:15:02
data required to train the models can be
00:15:04
a challenge especially if you're going
00:15:06
to find tune your
00:15:08
model the cost associated with the
00:15:10
compute power required for Gen can be
00:15:13
high so it's important to carefully
00:15:15
evaluate the ROI you don't want to use
00:15:18
gen in situations where the ROI is
00:15:23
low you might need to build some
00:15:25
expertise to implement gen and that can
00:15:29
involvement learning curve for many
00:15:31
organizations finally make sure that the
00:15:34
policies and regulations in your company
00:15:36
and in your industry allow for usage of
00:15:41
geni in addition to these challenges
00:15:43
there's also some inherent limitations
00:15:45
of gen that you should consider the Gen
00:15:50
outputs are going to be unpredictable
00:15:52
the outputs are not deterministic and
00:15:55
even with the same input you might get
00:15:58
different outputs
00:15:59
the models are often going to be opaque
00:16:02
so you're not quite going to know why a
00:16:05
model responded in certain way even
00:16:08
though your output or your input seem to
00:16:11
be the right one and then because these
00:16:15
models are trained on the public
00:16:16
internet there's the potential for
00:16:18
perpetuating biases and and you need to
00:16:21
manage all of these factors
00:16:23
carefully it's essential that as you're
00:16:26
designing your features or products you
00:16:28
keep in mind these limitations and have
00:16:30
proactive measures to address
00:16:33
them as product managers you're often
00:16:36
going to see that there's a lot of
00:16:37
pressure on you to use AI even when it
00:16:40
doesn't make sense I'll again reiterate
00:16:44
that you should use gen where it makes
00:16:46
sense it should really be solving a
00:16:48
problem and it should not be a
00:16:50
gimmick don't treat Genai as a magic
00:16:54
solution gen is incredibly powerful but
00:16:57
it's not a magic wand that solves every
00:16:59
problem and like any tool it has limits
00:17:03
that we need to be aware of and and deal
00:17:05
with and treating it as a Panacea is
00:17:08
just setting ourselves up for
00:17:11
disappointment so maintain realistic
00:17:13
expectations for around what gen can do
00:17:16
and what it cannot
00:17:17
do don't overlook data
00:17:20
quality uh one of the biggest concerns
00:17:22
is the massive impact data quality has
00:17:26
on gen we've heard about garbage garbage
00:17:29
out and it's definitely true here if we
00:17:32
feed these models low quality or bias
00:17:35
data or in our prompts we give it
00:17:39
context that isn't relevant you can
00:17:41
expect garbage
00:17:43
outputs and and you need to make sure as
00:17:46
you're building your products You're
00:17:47
Building robust data pipelines and
00:17:50
you're curating the inputs that are
00:17:52
going into the
00:17:54
model don't rely on gen for critical
00:17:57
decisions at least not right now and
00:18:00
when it comes to high stakes domains
00:18:02
like healthcare or Finance uh we need to
00:18:05
be careful about how we deploy
00:18:09
geni right now I would caution against
00:18:11
using it for autonomous critical
00:18:14
decision making these systems should
00:18:16
augment and Empower human experts not
00:18:19
replace them on on areas where there's a
00:18:23
huge consequence of these decisions and
00:18:26
human oversight remains GR crucial uh in
00:18:29
the use of
00:18:32
gen so to summarize gen is a very very
00:18:36
flexible tool set you can use it to
00:18:38
generate new content modify existing
00:18:40
content or to analyze data its
00:18:43
deployment should be strategic and it
00:18:45
should Target genuine user needs rather
00:18:48
than it being used
00:18:50
indiscriminately the success of gen is
00:18:52
gauged by the quality diversity and user
00:18:54
reception of its outputs making
00:18:57
continuous enhancement process crucial
00:18:59
to the success of an eii product and
00:19:02
while gen holds immense promise it's not
00:19:05
infatable occasional failures or biases
00:19:08
necessitate a human Centric approach to
00:19:11
supervise and fix outputs to use ji you
00:19:14
will need to overcome these key
00:19:17
challenges to be successful with your
00:19:20
product with that I want to thank you
00:19:24
[Music]
00:20:01
[Music]
