00:00:14
All right.
00:00:14
Hi, everyone.
00:00:15
Welcome to the session on building AI apps, technical use
00:00:19
cases and patterns.
00:00:21
I'm Devanshi Joshi, Senior Product Marketing Manager for the Azure
00:00:24
Application Platform.
00:00:26
And joining me today for this session will be Mandy
00:00:29
Whaley, Partner Director for Product for Developer Tools and Dan
00:00:33
Gartner, Director Specialist for App Innovation.
00:00:37
We will start with exploring the technical patterns and use
00:00:41
cases for AI powered app development, then explore the tools
00:00:45
to help you get started easily and finally close with
00:00:48
how to operationalise these AI apps in production.
00:00:53
All right, So before we begin into the deep depths
00:00:56
of technical technology use cases with a show of hands
00:01:00
here in person and also virtually, can you tell me
00:01:04
how many of you interact with AI today?
00:01:09
Wow, like 9599% of the room and I'm sure virtually
00:01:13
as well.
00:01:14
It could be in your personal lives with home assistance
00:01:19
or in your professional lives with AI powered tools using
00:01:23
Microsoft Copilot, ChatGPT and other routines that are now part
00:01:28
of both your work and daily home lives.
00:01:32
So AI is pretty much revolutionizing everything in our lives
00:01:37
personally and professionally.
00:01:40
It's been around for a couple of decades.
00:01:42
The last few years have been radically transformative, transforming also
00:01:47
the app development journey that we have seen our customers
00:01:50
on board and join us along.
00:01:53
So every Business Today across every industry is reinventing their
00:01:58
current application landscape and building new apps right at the
00:02:03
centre with AI.
00:02:07
OK, so how about delivering this transformation, getting real with
00:02:13
your organization?
00:02:14
What are some of the key consideration areas for you
00:02:17
to address?
00:02:18
You start with looking at the represented use cases that
00:02:22
are aligned to your business value.
00:02:24
Discuss when and how to modernize your existing apps or
00:02:28
data assets, and also highlight key operational imperatives and opportunities
00:02:34
to ensure you can securely scale as your business grows.
00:02:39
Discovering your AI app strategy with internal processes or experiences
00:02:44
for external customers.
00:02:47
It can be a whole layer of end to end
00:02:51
experiences you build.
00:02:54
So understanding where to focus your AI investment and remembering
00:02:58
that your ROI is not just about cost.
00:03:01
It could vary from time savings to customer satisfaction as
00:03:05
well as employee engagement.
00:03:07
And depending on your business.
00:03:09
Consider these areas for your AI investments, whether it be
00:03:13
reshaping business processes to operate more effectively, enriching the employee
00:03:19
experiences or reimagining your new customer experiences, accelerating existing product
00:03:26
innovations or new ones to deliver completely transformed experience for
00:03:31
both your solution in house and end customers.
00:03:35
So one way to think about it is that you
00:03:38
consider cases that are mapped to internal experiences and then
00:03:43
external experiences.
00:03:45
So reinventing experiences for your customers can include personalization and
00:03:50
product discovery.
00:03:51
I see someone go yeah, in the crowd.
00:03:53
OK, I see you're there already.
00:03:54
Nice content generation and marketing service and support or building
00:04:00
your own assistance with build your own copilot experiences like
00:04:04
here TomTom is re imagining the digital cockpit by understanding
00:04:09
already 95% of their complex requests, improving response time from
00:04:14
12 seconds to 2.5 seconds.
00:04:16
And when we reflect on internal experiences, the business processes
00:04:21
side of it, extending to include again building internal assistance,
00:04:25
AI assistance, Co pilots or having high volume transaction processing
00:04:30
and anomaly detection, information discovery and knowledge mining or even
00:04:35
document intelligence and summarisation.
00:04:38
Manulife here has data scientists take only days now to
00:04:42
set up their environment to make fraud detection much easier.
00:04:47
So to get all of this started for our customers,
00:04:50
Microsoft's Azure AI Application platform provides the framework for you
00:04:56
to build your own AI powered apps.
00:04:59
In fact, it's the framework that Microsoft Copilot is built
00:05:02
on.
00:05:03
It's the most advanced platform for creating AI capabilities and
00:05:08
solutions, whether you're extending and building on top of Microsoft
00:05:12
Copilot or building your own Copilot, or innovating and automating
00:05:17
with AI throughout your business processes.
00:05:20
The Azure AI Application Platform provides that comprehensive and integrated
00:05:25
platform to deliver on these applications performing better, taking you
00:05:30
to market faster.
00:05:32
The platform includes AI infrastructure providing computing power with CPUs
00:05:37
and GPUs, application services to deploy and scale your apps,
00:05:42
foundation models from AI innovators across the industry, your own
00:05:46
business data, and the developer tooling to build these solutions.
00:05:51
All of this wrapped up in our industry leading approach
00:05:55
to AI privacy, safety and security.
00:05:59
Now Azure AI platform, we heard about it a lot
00:06:02
today starting from the very keynote.
00:06:04
It's this unified AI platform representing the Azure AI Portal
00:06:08
and unified SDK experience is in addition to the prebuilt
00:06:13
app templates and access to the third party ISV and
00:06:16
to ISV tools and services.
00:06:19
So earlier today, Satya shared that we're evolving the Azure
00:06:23
AI Studio into this fully enterprise grade management console to
00:06:27
give organisations the visibility they need on cost, quality, performance
00:06:32
and safety.
00:06:33
We're introducing the new Azure AI Agent service to connect
00:06:37
knowledge, memory, action, and models, all of it together for
00:06:42
creating robust and sophisticated apps together with the Azure AI
00:06:47
Model catalog, open AI service content safety, all of this
00:06:52
is together powering the Azure AI Foundry.
00:06:56
Developers will now be able to stay in their flow
00:06:59
right where they are, directly access Azure AI services from
00:07:03
the world's most popular developer tools, move quickly from idea
00:07:07
to code and Microsoft back AI app templates, agent orchestration
00:07:12
service, as well as the curated selection of integrated third
00:07:16
party solutions.
00:07:17
Discovering the best of models out there.
00:07:20
the IT admins, operations and compliance teams can now leverage
00:07:24
the management and government's tools to run their AI apps
00:07:27
effectively and with confidence.
00:07:30
So now every generation of applications has brought a changing
00:07:34
set of needs.
00:07:36
So just as web, mobile and cloud drove the rise
00:07:40
of new application platform, AI will also evolve how we
00:07:45
build, run, govern and optimize our apps that are defined
00:07:50
by AI.
00:07:51
With the power of the new models that are coming
00:07:55
out every day, their power to solve reason complex problems,
00:07:59
there's this real opportunity to automate complex business processes and
00:08:04
human workflows to drive business growth at the end.
00:08:07
So data here forms the key to get you those
00:08:11
insights and value creation for your businesses.
00:08:16
Data forms the key here, driving us into decision making,
00:08:20
better product design, better customer service and better acquisitions.
00:08:26
The opportunity for you here as customers, we seek to
00:08:30
use AI to automate virtually anything and every step of
00:08:34
the way in your journey.
00:08:36
Workflows, business processes, ingesting signals to business processes and workflow
00:08:43
automation and to outputs, even actions.
00:08:47
So there are challenges however, and the developer has a
00:08:51
steep learning curve, learning curve here from functions to new
00:08:55
tools coming out, particularly with the Gen.
00:08:57
AI models coming out every other day with newer revisions.
00:09:02
So as well as the AI engineer has to work
00:09:05
on integrating to the existing systems, AI is bringing in
00:09:09
new waves.
00:09:10
You get to modernize, but you also have this whole
00:09:13
set of existing capabilities to make it work with your
00:09:16
technical debt in a different in a whole new light.
00:09:20
So the AI is also hallucinating at the same time.
00:09:24
It's hard.
00:09:25
And of course, the IT professional, we can never forget
00:09:29
that the IT professional leads have also transformed for the
00:09:32
management of the tools that understand their security, privacy, safety,
00:09:37
all of the aspects today along with Gen.
00:09:39
AI coming in way to manage costs and optimize resources.
00:09:44
Now I would like to invite Mandy to walk us
00:09:47
through the end to end developer experience for AI app
00:09:50
development and overcome, help you overcome these challenges and get
00:09:54
started easily.
00:09:54
Thank.
00:09:55
You so much.
00:09:56
Hi everybody.
00:09:57
How's it going?
00:09:57
Thanks for being here last session of the day.
00:10:00
Hope you've had a great day.
00:10:01
One, I'm Andy Whaley, I lead our Azure dev tools
00:10:04
and Azure SDK team and I'm really excited to share
00:10:07
with you some of the things that we've been building
00:10:10
to help you build AI apps for a lot of
00:10:12
these great use cases that Devante was sharing.
00:10:15
So at Microsoft, we take an end to end approach
00:10:18
for developer experience for helping you build cloud and AI
00:10:21
applications.
00:10:22
It starts in your IDE, in your editor where you're
00:10:25
writing your code, connecting to your DevOps tools like GitHub
00:10:28
and Azure DevOps and then all the way to Azure,
00:10:30
the cloud platform.
00:10:32
And our goal is for any developer working in any
00:10:35
language to be successful.
00:10:37
So we build libraries in pythonjavascriptjava.net and Go.
00:10:42
And GitHub Copilot is there to help you along the
00:10:44
whole journey.
00:10:45
I wanted to take a minute to share about some
00:10:48
of the new things happening with our Azure AISDKS because
00:10:51
these are really at the core of how we're enabling
00:10:54
you to build AI applications.
00:10:56
A big part of this is our Azure Open AISDKS.
00:10:59
We have these in python.net, JavaScript, Java, and Go.
00:11:02
These bring the incredible powers of the Open AI models
00:11:06
to Azure with Azure Security and Azure Enterprise Promises.
00:11:10
You can check these out with the URL, the QR
00:11:12
code that's there.
00:11:13
This is a great way to get started.
00:11:15
Tons of samples and really easy to use SDKS.
00:11:19
And then we're really excited for the net community because
00:11:23
Open AI just announced their stable release of the Open
00:11:27
AI Net SDK.
00:11:28
And this is really great because it allows.net developers to
00:11:32
use the latest Open AI features, and it pairs with
00:11:35
the Azure Companion SDK libraries that we build so that
00:11:38
you can easily move to Azure Open AI and use
00:11:41
things like Azure Identity and all of the Azure features
00:11:44
that you're used to.
00:11:45
So definitely go give this a try if you're a.net
00:11:48
developer and then today you've probably been hearing a lot
00:11:52
about Azure AI Foundry.
00:11:53
We're really excited to have the Azure AI Foundry SDK
00:11:57
in preview.
00:11:58
This is a new experience focused on a really simplified
00:12:01
developer experience that brings everything you need to build an
00:12:05
AI application together.
00:12:07
Evaluations, tracing models.
00:12:10
And one of the really cool things about it is
00:12:13
that you can explore different models in the really expansive
00:12:17
model catalogue that we offer and make very few code
00:12:20
changes.
00:12:21
And that's one of the things that this Foundry SDK
00:12:23
enables.
00:12:24
So definitely go take a look at that.
00:12:26
We are going to share some today, some demos, some
00:12:29
tooling, some things that can help you on this whole
00:12:32
journey of building an AI application.
00:12:34
As we've been building AI applications and talking with customers
00:12:38
and community members, we've kind of broken down into these
00:12:42
these steps of that taking an idea of your AI
00:12:45
powered application and getting it to production.
00:12:48
So you start with exploration, you're trying out different models,
00:12:51
you're trying them in the playground, you're starting to collect
00:12:54
your prompts, you're trying to see what does successful output
00:12:57
start to look like.
00:12:59
And then very quickly you're into this development phase where
00:13:02
you're connecting the LLM part of your app to the
00:13:04
rest of your application.
00:13:06
Maybe existing resources that you have.
00:13:08
You may be starting to put your prompts in source
00:13:11
control so you can start to iterate on them and
00:13:13
just connecting and building out your user interactions and really
00:13:16
starting to look like what is your proof of concept
00:13:19
look like.
00:13:20
And then from there, you're starting to move towards pilot
00:13:23
and production and all of your normal software engineering best
00:13:27
practices apply.
00:13:28
You want to get your CICD pipeline set up, but
00:13:30
there's some new things to think about in the pipeline,
00:13:33
like how do you bring your prompt evaluations into your
00:13:36
CICD pipeline so that you can be running your prompt
00:13:39
evaluations just like you run your other automated tests.
00:13:43
And we've built some great tooling to help you with
00:13:45
that.
00:13:45
We're going to be demoing some of that today.
00:13:48
And then from there, you're moving to pilot and production
00:13:50
and users are starting to really interact with your application
00:13:53
and the ways that you're getting customer feedback and doing
00:13:56
experimentation become really important.
00:13:58
So we're going to take a look at this.
00:13:59
I'm going to show you some of the tooling and
00:14:01
experiences you can that it can help you during exploration
00:14:04
and development.
00:14:05
And then Dan is going to come up and cover
00:14:07
some of the things that help you operationalize your app
00:14:09
when you're heading for production.
00:14:11
So we're going to build an application.
00:14:13
We're on a team.
00:14:14
We want to use GPT 4 O and we want
00:14:16
to use the image processing capabilities of this.
00:14:19
We're going to build an application where we can upload
00:14:22
images and then chat with those images.
00:14:25
We're going to use two new tools, the AI app
00:14:28
template gallery and also GitHub copilot for Azure.
00:14:33
We're going to start in the AI app template gallery.
00:14:35
So this is a new resource that we've just launched
00:14:38
for developers, and it's a gallery of application templates that
00:14:42
cover different AI use cases, application patterns.
00:14:46
You can filter by the kind of task that it's
00:14:48
doing.
00:14:49
You can try out all the different languages, you can
00:14:52
filter by model.
00:14:53
And each one of these is an application that includes
00:14:56
all of the source code needed to get it up
00:14:58
and running, as well as the infrastructure as code files
00:15:01
that help you provision the Azure resources.
00:15:04
You can see what the app's going to look like
00:15:06
when you deploy it, an architecture diagram showing you what
00:15:10
Azure resources are used.
00:15:11
And many of them include videos that can help you
00:15:14
get started.
00:15:15
All of them you can open directly in GitHub code
00:15:18
spaces, get started with one command, or just view the
00:15:21
code on GitHub.
00:15:23
So we're going to use a template from the gallery
00:15:25
to help us build our application.
00:15:28
And we're also going to use that together with GitHub
00:15:30
Copilot for Azure.
00:15:31
GitHub copilot for Azure is now in public preview.
00:15:34
We're super excited.
00:15:35
This is a copilot chat extension that you can use
00:15:39
in VS Code to help you with all things Azure.
00:15:42
So, you know, looking for information about Azure resources, learning
00:15:46
about new Azure services.
00:15:48
Now you have the latest Azure documentation right inside of
00:15:51
your IDE that you can access just by asking questions.
00:15:54
It also has information about your Azure context, your subscription,
00:15:58
your tenants.
00:15:59
So it can help you get information about your resources,
00:16:02
troubleshoot issues, and even build and deploy applications.
00:16:06
So we're going to use these two things together, GitHub
00:16:08
copilot for Azure and the AI App Template Gallery.
00:16:11
All right, we're going to move into VS Code.
00:16:15
So here I'm in VS Code and I'm going to
00:16:17
see if Copilot can help me find a template to
00:16:20
get started on my application using GPT 4 O.
00:16:23
So it's going out, it's querying and looking at that
00:16:26
Azure AI template gallery for something that matches what I'm
00:16:30
looking for.
00:16:31
And here it is.
00:16:32
It's found a template.
00:16:33
It's the chat plus vision template.
00:16:35
It gives me the URL.
00:16:38
I can just type that init command into my terminal
00:16:41
and this is going to start to set up my
00:16:44
local environment.
00:16:46
So I'm going to give it an environment name.
00:16:48
And now I have the application code here ready for
00:16:51
me to work with.
00:16:53
You can see that we've got the application code.
00:16:55
Here's where the chat is happening.
00:16:59
And then I've also got infrastructure as code files that
00:17:02
are helping provision all the different Azure resources that are
00:17:05
used.
00:17:06
So that looks great.
00:17:07
This is going to help me a lot.
00:17:09
One of my favorite copilot features is the workspace feature.
00:17:14
And so you can ask at workspace and this helps
00:17:17
copilot look at the entire repo, the whole code base,
00:17:20
not just a few files.
00:17:22
So I think about this as like maybe sort of
00:17:24
an expert programmer on this code base.
00:17:26
If I'm ramping up on a new project, it can
00:17:28
help me learn about it.
00:17:30
So this is great.
00:17:31
This is telling me, this is Python, we're using Azure
00:17:34
Open AI.
00:17:34
It's going to deploy to container OPS.
00:17:36
This is looking like the application that I want to
00:17:39
use, but I know we're also using Bicep to specify
00:17:42
the resources.
00:17:43
So I'm going to ask at Azure to look a
00:17:45
little bit deeper to take a look at the actual
00:17:48
Bicep files and just confirm which model we're using.
00:17:52
Great.
00:17:52
It took a look.
00:17:53
It knows that we're using GPT 4 O, that matches
00:17:56
what I want to do.
00:17:57
And so now I'm ready to start deploying to Azure,
00:18:01
but I want to check the availability of GPT 4
00:18:04
O in certain regions.
00:18:06
So we're always adding new models to the AI Foundry
00:18:09
catalogue and they roll out to different regions.
00:18:13
So as you're thinking about where to put your application,
00:18:15
where to put your compute resources, you can use at
00:18:18
Azure to help you know the latest information on regional
00:18:21
availability for different models.
00:18:23
You can also use it to get information about AI
00:18:25
resources that you've already deployed.
00:18:28
OK, great.
00:18:28
So this is letting me know this and E US
00:18:31
two, I'm going to use that.
00:18:33
So next I'm going to ask GitHub copilot to help
00:18:35
me deploy this application.
00:18:38
So this is great.
00:18:39
It's knows I'm using the Azure developer CLI.
00:18:41
It lets me know I can just use AZD provision
00:18:44
and AZD deploy.
00:18:46
Now, this template takes about three to four minutes to
00:18:49
deploy the whole thing, provisioning all the resources, packaging the
00:18:53
application code and deploying it.
00:18:56
So in the time that you could maybe go grab
00:18:57
a coffee, your application's up and running.
00:19:00
This is the resource group that it will create.
00:19:03
I've got Azure Open AI resources, container apps, container apps
00:19:07
environment, my managed identity so I can do keyless authentication
00:19:11
and my log analytics workspace.
00:19:13
And I could dive into, you know, each one of
00:19:15
those resources and take a closer look.
00:19:17
And here's my application that was deployed so we can
00:19:20
try it out.
00:19:21
We'll pick a graph because one of the things that
00:19:24
our team may be interested in is can we chat
00:19:27
and get information from charts and graphs.
00:19:30
So this is an example we're asking about is this
00:19:33
tree coverage loss, which region has the most?
00:19:35
And it can tell that tropical has the most and
00:19:38
it's the dark green area.
00:19:40
Another thing we're interested is in interpreting diagrams.
00:19:43
So this is an architecture diagram.
00:19:46
I'm going to ask it to list the services used
00:19:50
in this app.
00:19:51
And so it's going to take a look at that
00:19:53
architecture diagram and it returns.
00:19:55
These are all the Azure services that were used in
00:19:57
that grade.
00:19:58
So we did it.
00:19:59
We built our application.
00:20:00
We've got our we're using GPT 4 O, we've got
00:20:03
our image processing application.
00:20:05
From here we could start to evolve that and add
00:20:07
more resources and take our application to the next steps.
00:20:11
There's a couple more things I want to show you
00:20:13
with GitHub copilot for Azure that can help you as
00:20:16
you are evolving your application.
00:20:19
So we're back in VS Code and this is the
00:20:21
Azure Container OPS VS Code extension, same resources we were
00:20:24
just looking at.
00:20:26
And one of the things that's really handy is you
00:20:28
can focus on a resource and then say ask at
00:20:30
Azure.
00:20:31
And this lets the agent know that you want to
00:20:33
talk about this specific resource.
00:20:35
So I'm going to ask it to go get the
00:20:37
most recent logs for this resource.
00:20:39
This is something really common where I'm working on my
00:20:42
application.
00:20:42
And maybe I just want to check, you know, is
00:20:44
there anything going on?
00:20:45
I could ask it to get errors.
00:20:48
I can just take a quick look at what's going
00:20:50
on with my application.
00:20:51
So it's using my Azure context again, my subscription, my
00:20:54
tenant, it knows what resource we're talking about.
00:20:57
It's going out and querying those logs and it's even
00:20:59
going to bring them back and put them in a
00:21:01
nice table for me.
00:21:02
And one of the things that I really like about
00:21:05
it is that I get a quick look at what's
00:21:07
going on, but it will also give me the link
00:21:10
that if I want to go continue looking deeper into
00:21:13
the logs in the portal, I'll get a link to
00:21:15
continue that in the portal.
00:21:17
It's coming, There we go.
00:21:19
So that's it.
00:21:20
That is a super time saver.
00:21:22
One other time saver that I really like to use
00:21:25
is the fact that GitHub Copilot for Azure knows about
00:21:28
my Azure Resource graph.
00:21:30
So I can ask it something like this for that
00:21:33
resource group we just created when we deployed our application.
00:21:37
Just give me a list of all those resources and
00:21:39
give me the link to go directly to them in
00:21:41
the portal.
00:21:41
This is like a really great time saver.
00:21:44
It just gets all of those resources right in my
00:21:46
coding environment, super accessible for me to if I need
00:21:49
to go check a setting in the portal or dig
00:21:52
deeper on something.
00:21:54
Now one other thing that you want to think about
00:21:57
as you're going in those early phases of exploration and
00:22:00
development and starting to move towards pilot are your cost.
00:22:04
It's great to keep an eye on your cost as
00:22:06
you're exploring different models, different application architectures and get a
00:22:10
copilot for Azure can help you with that too.
00:22:13
We have a cost management assistant.
00:22:15
You can ask it to break down cost for certain
00:22:17
time ranges by resource group by service and just get
00:22:21
a quick view of that right inside of of VS
00:22:23
Code.
00:22:25
OK, so we did it.
00:22:27
We built our application.
00:22:29
It's up and running and everything that I showed you
00:22:32
is available for you to try today.
00:22:34
You can, you can go use all of this right
00:22:36
now.
00:22:37
The AI app template gallery is live.
00:22:39
This QR code will get there.
00:22:40
Go try out.
00:22:41
You can deploy the app we were just looking at.
00:22:44
And then you can also suggest use cases.
00:22:47
If there's use cases that you would love to see
00:22:49
in the application gallery, please let us know.
00:22:52
And then GitHub copilot for Azure, available in the VS
00:22:55
Code Marketplace.
00:22:56
Go install it, give it a try, start finding out
00:22:59
which things help save you time and which things are
00:23:02
really important to have in your editor.
00:23:04
Really excited to be here with you today.
00:23:06
Next, we're going to go ahead and look at the
00:23:09
next phases through evaluations, through production, and Dan's going to
00:23:12
show you some great demos on that.
00:23:14
All right.
00:23:16
All right.
00:23:16
Thank you, Mandy.
00:23:22
All right, so I'm going to close this out by
00:23:25
walking through how we can operationalize our AI applications using
00:23:30
Azure AI Foundry and the Azure AI Foundry SDKS.
00:23:33
And what we're going to use to do this is
00:23:35
we're going to use an AI app template.
00:23:37
So these templates are really fantastic.
00:23:39
There's tons of different use cases out there there.
00:23:42
There's chat over images, content creation, enterprise chat, enterprise search,
00:23:47
and then some of them contain everything that you need
00:23:50
to fully operationalize your app for production and are fully
00:23:54
connected to AI Foundry.
00:23:55
So let's dive into one of those, the app template
00:24:00
that we're going to look at.
00:24:03
It's called Contoso Creative Writer.
00:24:06
Yeah, it's super simple to install, just do AZD init
00:24:09
dash T Contoso Creative Writer.
00:24:12
You'll pull down the template onto your machine.
00:24:14
But this is a very interesting example application.
00:24:18
One, it's a multi agent AI application.
00:24:21
So it's doing a bunch of different things.
00:24:23
It's orchestrating across multiple different AI agents.
00:24:27
We've got one researcher agent that's using Azure Open AI.
00:24:30
It's using tools like the Bing search service, It's using
00:24:33
Azure AI Search.
00:24:35
There's another writing a writer agent that creates marketing copy.
00:24:39
And then there's an editor agent.
00:24:40
So if you're looking for an example of a multi
00:24:43
agent application, this is where to go.
00:24:45
But this also includes a lot of the different services
00:24:48
and resources that you need to operationalize an AI application.
00:24:52
And we're going to dive into those.
00:24:55
So when I meet, when I say operationalize, this is
00:24:57
what I mean, this is what I'm going to focus
00:25:00
on.
00:25:00
This is what we're going to show on today.
00:25:02
AI governance, right?
00:25:04
Most enterprises really, one of the barriers is they're not
00:25:07
going to adopt generative AI unless they've got the ability
00:25:11
to govern the different AI models, to govern the data
00:25:14
that your AI applications are going to be using.
00:25:17
And so we can implement all of that with Azure
00:25:20
AI Hubs.
00:25:20
So we're going to look at those that's deployed with
00:25:23
this AI app template.
00:25:25
We're going to look at the developer experience with AI
00:25:28
Foundry Playgrounds, so your developers can work with these different
00:25:32
models.
00:25:32
There's multimodal playgrounds that they can work with.
00:25:36
We're going to dive into AI evaluations.
00:25:39
We're going to look at CICD, continuous integration with GitHub
00:25:43
Actions, and you have the ability to run AI evaluations
00:25:47
as part of that.
00:25:48
And then we're going to look at all the baked
00:25:51
in app monitoring, observability tracing, as well as experiments.
00:25:56
So I'm going to switch over to my machine and
00:26:00
we're going to get started.
00:26:03
First thing that I want to show you is this
00:26:06
is how we can get Contoso Creative writer onto your
00:26:09
machine.
00:26:10
It's super simple, just AZD init Contoso Creative writer.
00:26:14
It's going to ask you to create a new environment
00:26:16
name.
00:26:17
I'm just going to give it a name.
00:26:19
Three, I've got a project installed.
00:26:22
I want to deploy this to Azure.
00:26:23
This is what I do AZD up, done.
00:26:28
Pick a subscription, pick a region and that Contoso Creative
00:26:32
Writer AI template is going to be deployed to my
00:26:35
Azure subscription.
00:26:36
Now this one has a bunch of different resources in
00:26:39
it, so it takes about 15 minutes.
00:26:41
It's provisioning, AI search, all that sort of stuff.
00:26:45
So if I've already pre provisioned it, this is a
00:26:48
resource group that eventually gets created.
00:26:51
Now you'll see there's some very interesting things here.
00:26:54
It's not just the app pieces, it's not just container
00:26:57
apps, but we've we've got an AI hub, we've got
00:27:00
an AI project, we've got application insights for monitoring.
00:27:04
So this template contains basically everything that I need to
00:27:08
to implement all of that operate all of all those
00:27:11
operationalization capabilities.
00:27:14
This is the application, this is the Contoso Creative Writer
00:27:18
application, and I can go ahead and and show you
00:27:21
what it does.
00:27:21
So basically if we look at this debug view over
00:27:25
here, you'll see that we've got multiple agents working together
00:27:30
to create ad copy for a particular blog post.
00:27:34
First, there's a researcher task that's going out.
00:27:37
It's querying the web, it's querying Bing search to look
00:27:40
up things based on the prompt that we have for
00:27:42
our article.
00:27:43
And then it's querying open AI search to retrieve product
00:27:48
information.
00:27:49
So the point of this is I want to author
00:27:52
A blog post that infuses kind of marketing speak for
00:27:55
products that I have in my store.
00:27:58
And then after that there's an editor agent and the
00:28:01
editor goes and it it evaluates the blog post, it
00:28:04
gives kind of review notes and those sorts of things,
00:28:07
whether or not they should accept this.
00:28:09
So this is a very interesting multi agent application.
00:28:13
Now how do we operationalize it?
00:28:17
So the first thing that we want to take a
00:28:20
look at is the Azure AI Hub.
00:28:23
So you could tell by the sort of random name
00:28:26
here that this was provisioned by the infrastructure as code
00:28:30
that was brought down when I ran AZD and Net.
00:28:34
The hub is the place where I can define different
00:28:37
projects for my developers.
00:28:39
This is where I can control the different models that
00:28:42
our developers have access to.
00:28:44
This is where I can control the data that our
00:28:47
developers have access to.
00:28:49
If you've been to other sessions, you may have heard
00:28:52
there's over 1800 models available to you in Azure AI.
00:28:55
This is where you can constrain that.
00:28:56
This is where you can say, all right, developers, you
00:28:59
have access to these models, you have access to these,
00:29:01
this data, you have access to these projects.
00:29:03
Go ahead and go nuts, right?
00:29:05
You can apply the governance at the hub level and
00:29:08
you still give your developers the flexibility to be productive
00:29:12
and build AI applications.
00:29:15
Another great thing about hubs is this is where I
00:29:18
can manage my quota.
00:29:19
Big concern for most enterprises is how do I manage
00:29:22
and control costs and those sorts of things.
00:29:25
This is sort of a single pane of glass where
00:29:29
I can view quota across my entire subscription.
00:29:32
So you'll see here in my subscription, I've got a
00:29:35
couple dozen models deployed, right?
00:29:37
And I can see the quota allocated to each of
00:29:39
them.
00:29:40
I can see quota allocated by model as I can
00:29:43
see quota allocated by resource, right?
00:29:46
And if I want to change the quota for one
00:29:48
particular model, it's just a click right here and I
00:29:51
can increase quota for particular models.
00:29:53
So this is where you can have centralized sort of
00:29:56
management of quota of cost and all those sorts of
00:29:58
things.
00:29:59
So those are AI hubs and that was deployed from
00:30:02
that AI app template.
00:30:04
Now your developers, they're going to work with projects.
00:30:08
So I've clicked into one of the projects that was
00:30:11
deployed by that Azure AI template app template, and they're
00:30:15
going to spend a lot of time in playgrounds.
00:30:18
Now you'll see here, once this playground loads, this is
00:30:22
where I can interact with, with the different models that
00:30:26
were defined at the hub level, right?
00:30:28
So I don't have access to every single model in
00:30:31
the in the model catalogue.
00:30:34
I only have access to those models that were were
00:30:37
basically assigned to my project at the Azure AI hub.
00:30:41
So again, the point is you can control the models
00:30:44
that your developers have access to.
00:30:47
And then they can go in here and they can
00:30:48
experiment with prompts.
00:30:50
They can write their own custom prompts.
00:30:52
They can use AI to generate prompts that they want
00:30:55
to use to interact with this particular model.
00:30:58
They can add their own data, but they can only
00:31:01
add the data that was assigned at that hub level.
00:31:04
And, and so we can enforce permissioning across all of
00:31:07
this as well.
00:31:09
So this is the developer experience.
00:31:11
This is Azure AI hubs and Azure AI projects.
00:31:15
This enables you to enforce the governance that you need.
00:31:20
Let's talk about some other operationalization concepts.
00:31:25
One of those is tracing, right?
00:31:28
Most AI applications are going to use multiple models and
00:31:32
you're going to need good observability in order to see
00:31:36
how those models are interacting with each other, right?
00:31:40
And so we can use the Azure AI Foundry SDKS
00:31:44
to send our telemetry up to Azure AI Foundry, as
00:31:49
you can see right here.
00:31:51
So I can drill in to telemetry for my application.
00:31:57
Looks like I had an error so I can drill
00:31:59
in here.
00:31:59
I can see there's different models interacting with each other.
00:32:03
I can see what model calls the other model, I
00:32:06
can see what APIs are called, and I can drill
00:32:08
into the tracing right here in Azure AI Foundry.
00:32:12
I can also, if I wanted to open this up
00:32:16
in Azure Application Insights.
00:32:20
So if I want to do deeper log analysis, we
00:32:22
can use App Insights to query the logs as well,
00:32:26
right?
00:32:26
And so we just launched this into a, into a
00:32:28
workspace.
00:32:29
I can see all my traces here.
00:32:31
And there's even other queries where I can delve into
00:32:36
the LLM telemetry.
00:32:37
So if I wanted to see things like tokens used
00:32:40
over time, average model duration, all these queries were a
00:32:44
part of that AI app template that we deployed.
00:32:47
So from a monitoring perspective, you can use the Azure
00:32:51
Foundry SDKS to send your telemetry to app sites and
00:32:56
to your Azure AI project.
00:32:59
The last thing that I want to show is evaluations.
00:33:02
So we can create evaluations from Azure AI Foundry.
00:33:08
You may have seen these demos, but it's super simple.
00:33:11
It's just a button you click, you pick your metrics
00:33:14
and you've got a new and you've got a new
00:33:16
evaluation.
00:33:17
But what I want to show is you can also
00:33:20
run those from code.
00:33:22
And so if I drill in, this is the code
00:33:26
repo that we pulled down when I ran AZD init.
00:33:31
I'm using the latest Azure AI Foundry SDKS to run
00:33:35
evaluations.
00:33:36
So I can run these on demand.
00:33:37
And what you're going to see here is I'm configuring
00:33:40
the different metrics that I want to include in this
00:33:43
particular evaluation.
00:33:44
I've got a connection string up to my Azure AI
00:33:48
project.
00:33:48
So that's where I can stream the evaluation results.
00:33:52
And so I can kick this off locally, right?
00:33:55
So this is just Python.
00:33:56
I'm using the Python libraries.
00:33:57
I can run it locally, but it's also very easy
00:34:00
for me to include this as part of the CICD
00:34:03
pipeline.
00:34:05
So this is a GitHub action.
00:34:06
I'm calling the evaluate method and so as part of
00:34:10
my CICD and this is good practice.
00:34:13
Anytime you tweak a prompt, anytime you change the model,
00:34:16
anytime you, you know, change the data that your model
00:34:19
is going to be using with, with Rag, it's a
00:34:21
good idea to run evaluations against it so you can
00:34:24
validate whether or not you've improved or degraded performance.
00:34:28
Well, now it's very easy to run these as part
00:34:30
of your, your Git of actions as part of your,
00:34:33
your CICD pipeline.
00:34:35
So just to take a quick peek, if you look
00:34:38
at my GitHub, we'll see I've got evaluations loaded here.
00:34:43
I can see the, the different scores for some of
00:34:46
the different metrics that I, that I configured in that
00:34:49
code.
00:34:50
I can also view evaluation results right here in Azure
00:34:55
AI Foundry as well.
00:34:57
So you can, you can stream your results from your
00:35:00
emails in multiple locations, CICD and AI Foundry.
00:35:03
So those are the different capabilities, the different operationalization capabilities
00:35:10
that we wanted to show.
00:35:12
All that stuff was included in that Azure AI template,
00:35:15
ACV Init, ACV up.
00:35:17
I've got a really good example for how I can
00:35:20
build sort of a production ready enterprise AI application.
00:35:25
That's it.
00:35:25
I'm going to hand it back to Devanshi to close
00:35:27
this out.
00:35:28
So thank you everybody.
00:35:36
All right.
00:35:40
OK, I think I'm stuck.
00:35:56
Just bear with me.
00:36:09
OK, so I think thank you, Dan and Mandy for
00:36:12
walking us through those amazing demos on how to get
00:36:16
started across our technical patterns and use cases when we
00:36:21
talk about starting to build AI apps.
00:36:24
So for you to get started, of course, check out
00:36:27
the AI App template gallery.
00:36:28
Start today with the VS Code extension for GitHub Copilot
00:36:32
for Azure.
00:36:33
You can download it, use the same demo that we
00:36:36
showed here today in your own environment, and start building.
00:36:42
I'm going to just wait five more seconds for everyone
00:36:45
to take a photo of this or go to the
00:36:47
QR code.
00:36:48
All right, OK, moving on.
00:36:55
So Microsoft offers a variety of programs for you to
00:36:58
get started and accelerate that journey for cloud and AI
00:37:02
adoption like Azure Innovate, Migrate and Modernize as well as
00:37:06
Azure Essentials built for both enterprise customers as well as
00:37:10
partners, ensuring that the cloud solutions that are being built
00:37:14
are both reliable, secure, as well as performant at scale.
00:37:18
So take advantage of those programs as well as the
00:37:22
new code with program where Microsoft is offering code development
00:37:26
with customers for a two to three-week Sprint to accelerate
00:37:30
your AI up dev journey.
00:37:32
The development team here is helped with adopting these patterns
00:37:37
and practices we talked about, and it pretty much accelerates,
00:37:42
speeds up that realisation of your ideas into productionised features
00:37:47
and overall helping your velocity to adapt to market changes
00:37:51
with AI.
00:37:53
This spans all use cases we talked about today, all
00:37:56
technical patterns right from code generation, documentation analysis, image analysis,
00:38:02
document processing and summarisation, video processing, building your own AI
00:38:07
assistance across all of the industries, healthcare, media, entertainment, professional
00:38:13
and financial services as well as retail.
00:38:17
So for getting your hands dirty, getting hands on, try
00:38:21
out the Lab Mastering Azure Container Apps and Gen.
00:38:25
AI for Intelligent Solutions Lab 413 tomorrow, as well as
00:38:30
take this AI app template demo more upfront on Thursday,
00:38:35
Theatre session 616 with Kickstarter AI app development journey.
00:38:41
And lastly, on Friday morning, 8:30 AM, we have another
00:38:45
breakout that breaks down these use cases with customer insights.
00:38:49
Choose the right AI app use case for your business.
00:38:53
So we'll see you there today at tomorrow and also
00:38:56
on Friday.
00:38:57
This brings us to the end, and thank you for
00:38:59
joining us today.
00:39:00
Have a great rest of Microsoft Ignite.
