00:00:06
hello my name is Hannah matron I'm so
00:00:09
excited to have the opportunity to
00:00:12
present a project that I've been working
00:00:14
on at the University of Texas at Austin
00:00:17
libraries it is one of the AI Focus
00:00:20
projects that I'm investigating with Aon
00:00:23
Cho the director of research and
00:00:25
strategy this project explores how we
00:00:29
can potentially enhance Library services
00:00:32
with conversational AI the output is the
00:00:36
things we've learned along the way and a
00:00:39
testable proof of concept for this idea
00:00:41
a library assistant
00:00:45
chatbot today I'm going to run through
00:00:48
some background information and talk
00:00:50
about some of the research that went
00:00:52
into the design of the chatbot and then
00:00:55
I'm going to show you the design in the
00:00:57
voice flow platform and I'll end with a
00:01:00
live
00:01:03
demo why this project there was a space
00:01:07
that might be useful and an opportunity
00:01:09
to
00:01:12
experiment we have an ASA librarian chat
00:01:15
service that's been in operation for
00:01:17
about 10 years it's staffed by
00:01:20
Librarians and Gras and is very valuable
00:01:24
to our community however it is not
00:01:28
available overnight or on holidays so
00:01:31
there was an opportunity to extend the
00:01:34
hours of operation with AI we also
00:01:38
wanted to get a better idea of how and
00:01:41
whether our values and the high
00:01:44
standards of service that we have could
00:01:46
be incorporated into a chatbots
00:01:51
design we began this project with a
00:01:54
topic analysis of our historical chat
00:01:56
logs we looked at the first and last two
00:02:00
weeks of the Fall 2022 semester we were
00:02:03
looking for Trends in questions and
00:02:07
responses the big takeaways were that
00:02:10
though there is a lot of variety in the
00:02:13
questions they fit into just a few broad
00:02:17
categories 51% are research related 34%
00:02:23
are General Library questions and 133%
00:02:26
are related to accounts or technical
00:02:29
support
00:02:31
also most of the responses about 70%
00:02:35
include referrals to websites to subject
00:02:38
area Librarians or to library
00:02:41
departments this gave us a good idea of
00:02:44
the information that our chat bot needed
00:02:47
in order to be useful and also of the
00:02:49
response Norms of the
00:02:55
service in February 2024 we held
00:02:58
interviews with five Librarians over
00:03:01
Zoom all five had extensive experience
00:03:05
in both the field of librarianship and
00:03:08
also with the Asal librarian service we
00:03:12
are so grateful to them for sharing
00:03:14
their thoughts with us and uh after
00:03:17
analyzing the interviews using
00:03:20
qualitative coding and pulling out
00:03:22
themes that stood out four key insights
00:03:27
really came to the Forefront so I'm
00:03:29
going to talk about
00:03:30
how we incorporated those into our
00:03:32
design the first was
00:03:35
transparency that means being clear
00:03:38
about who or what is interacting with
00:03:40
the user on the other end of the
00:03:43
chat and how the conversations will be
00:03:47
reviewed and used to improve the
00:03:50
services so we built that into our
00:03:52
design in our demo by including a
00:03:56
disclaimer and giving a viable
00:03:58
alternative in this case it's a blog
00:04:01
post with demo videos for users who
00:04:04
aren't comfortable with the terms as
00:04:06
we've outlined
00:04:08
them secondly
00:04:11
accuracy it's a well-known Pitfall of
00:04:14
large language models which we address
00:04:17
by instructing the large language model
00:04:19
to give resources instead of
00:04:23
answers in addition to avoiding some of
00:04:26
the people pleasing hallucinations that
00:04:29
llms are given to this has the added
00:04:32
benefit of costing Less in inference and
00:04:36
using less energy than maintaining an
00:04:38
integrating an enormous knowledge base
00:04:41
to cover the large amount of variety in
00:04:45
questions that come to the
00:04:47
chat so I'll show you what that looks
00:04:49
like in the
00:04:51
demo third The Librarians we interviewed
00:04:55
wanted to scope the chatbot so that it
00:04:57
would help with tasks that the library
00:04:59
is designed to help with but not for
00:05:03
example write a paper for a student to
00:05:07
address this and increase our control
00:05:10
over the conversations we used a
00:05:12
traditional conversation design with
00:05:15
large Lang language model
00:05:19
steps and last we wanted to Foster the
00:05:22
human connection that is such an
00:05:24
important part of the educational
00:05:26
experience at a tier one re research
00:05:29
Unity University like UT Austin so to do
00:05:32
this we still want the chatbot to refer
00:05:36
students and researchers to subject
00:05:39
specialist Librarians on our campus who
00:05:42
are here to help them deeply investigate
00:05:45
research
00:05:49
queries to keep us on track and set
00:05:51
ourselves up for Success we used rubrics
00:05:54
in our design
00:05:56
process first the Ala code of ethics
00:06:00
as you know it provides guidance
00:06:03
on uh professional values and ethical
00:06:07
responsibilities like intellectual
00:06:09
freedom and Equitable
00:06:12
service the other rubric we used was
00:06:15
Microsoft's guidelines for human AI
00:06:17
interaction which are best practices in
00:06:20
AI user
00:06:22
design that includes for example
00:06:26
matching relevant social norms and
00:06:29
making clear why the system did what it
00:06:33
did our practice with both of these
00:06:36
rubrics was to go through Point by point
00:06:39
and answer the question of what each
00:06:42
guideline or ethical principle meant in
00:06:44
the context of this chatbot
00:06:49
project so now I'm going to show you the
00:06:52
voice flow
00:06:55
platform this is where we built our
00:06:58
conversational experience erience we
00:07:01
have designed a general Library flow a a
00:07:05
research flow and a mental health flow
00:07:09
the mental health flow is designed to
00:07:11
address any time where a student might
00:07:14
Express something concerning and should
00:07:16
be referred to campus Health
00:07:19
Resources at a couple of different
00:07:21
points in the
00:07:23
conversation user inputs will be matched
00:07:26
to intense which will trigger these
00:07:28
flows
00:07:31
each of these boxes represents a step in
00:07:35
the conversation as you can see there
00:07:39
are
00:07:40
arrows that are linking the steps taking
00:07:44
the users on a preset Journey it's a
00:07:48
kind of Choose Your Own Adventure style
00:07:50
chat with the intents and the buttons
00:07:54
here helping users to call their own
00:07:57
shots the gray step here do not call
00:08:02
llms these could be
00:08:05
functions like
00:08:07
um input steps like buttons or text
00:08:11
capture or output steps like pre-written
00:08:16
text the green boxes are the llm
00:08:22
steps
00:08:24
okay this is a response AI step it's a
00:08:28
place in the conversation where voice
00:08:30
flow sends a message to a large language
00:08:34
model API and then receives a response
00:08:37
back the response can then be printed
00:08:40
for the user in the conversation or
00:08:43
stored as a
00:08:44
variable this is where you set the
00:08:50
configurations you can see here that I
00:08:53
have chosen to use the memory of the
00:08:56
conversation in addition to the prompt
00:09:02
that's important to give the context of
00:09:03
the
00:09:05
chat we can choose here between
00:09:09
different large language models voice
00:09:11
flow has integrated anthropic and open
00:09:14
AI models as well as Google's
00:09:17
Gemini we can also set the temperature
00:09:20
or the variability and the max tokens or
00:09:22
the
00:09:25
length in the system message I've
00:09:29
created the chatbot Persona and set some
00:09:32
rules for the
00:09:34
response and I've also given various
00:09:37
Specific Instructions on what to return
00:09:40
the to the user and how to create the
00:09:43
links that I wanted to
00:09:45
provide and as you can see
00:09:49
[Music]
00:09:51
here I've provided a example of a good
00:09:58
response so
00:10:00
let's try this thing
00:10:01
out we're going to click Start
00:10:04
conversation and here's our disclaimer
00:10:07
we will agree to
00:10:10
that now thus far nothing has
00:10:14
been handled by
00:10:17
AI we have some buttons here that we
00:10:20
could click on to take us to different
00:10:22
flows but we're just going to type a
00:10:25
question
00:10:27
um where can can I reserve a
00:10:32
room also what computer
00:10:37
labs of
00:10:41
matlb and now the large language model
00:10:44
is going to send us back links to Pages
00:10:47
where we can find the answers to these
00:10:50
questions and so it's sent us to a page
00:10:52
where we can find out about reserving a
00:10:55
study room and it sent us another link
00:10:57
for um checking on available
00:11:01
software and the third here is contact
00:11:04
information for the the service desks at
00:11:07
the different branches of our
00:11:09
libraries um
00:11:13
so I'll just show you this so as you can
00:11:17
see this is a page where we have all the
00:11:19
software listed for each Library the
00:11:21
great thing about using our website like
00:11:25
this instead of answering it directly in
00:11:27
the chat bot is that if something
00:11:29
changes you know if some software is no
00:11:33
longer available or something is added
00:11:35
we don't have to worry about that in
00:11:36
voice flow or with our chatbot we just
00:11:40
are going to get those correct answers
00:11:43
by sending people to the website it also
00:11:46
really helps when questions are maybe a
00:11:48
little bit more outside of the norm of
00:11:51
what we deal with there's also um the
00:11:53
chat bot it can hand back a link to a
00:11:57
Google search of our website so that
00:11:59
that students or researchers could find
00:12:01
information in that way as
00:12:04
well okay so now we could click on one
00:12:07
of these buttons again we could start
00:12:09
over but we're just going to go straight
00:12:11
into the research part of this demo so
00:12:17
um I will type I'm
00:12:20
researching um the
00:12:22
effects of
00:12:26
microplastics on health
00:12:30
now we're being sent to our research
00:12:35
flow and asked if we want to give any
00:12:37
additional context and I will say uh
00:12:42
it's for
00:12:46
a
00:12:50
environmental science
00:12:56
class we could get resources right away
00:13:00
or we can brainstorm some topic ideas
00:13:03
this is a great way to use something
00:13:05
that large language models are really
00:13:07
good
00:13:08
at okay um so we have our research pads
00:13:13
there are some more choices down here
00:13:15
for us we could get new ideas we could
00:13:17
research a different topic if we changed
00:13:20
our mind we could say you've
00:13:22
misunderstood the topic just in case you
00:13:25
know it's not exactly what we
00:13:27
wanted um
00:13:29
but let's say that we want to research
00:13:34
um microplastic contamination in
00:13:37
drinking water
00:13:39
sources so now we're sending all of this
00:13:42
information about what we have in this
00:13:44
conversation to a large language model
00:13:46
Claude in this case and it's going to
00:13:49
send us
00:13:50
back the resources that I have specified
00:13:54
I wanted to to give in for this specific
00:13:59
research
00:14:00
topic
00:14:03
so we have a UT Austin Library search so
00:14:08
this is our
00:14:12
catalog so that's great they've gotten
00:14:14
to our
00:14:17
catalog we have um a search of our
00:14:21
databases so there are a couple of
00:14:24
different things that the large language
00:14:27
model has suggested as
00:14:30
um possible keywords to use to search
00:14:33
our
00:14:38
databases and we have a lip guide
00:14:42
search so the great thing about this is
00:14:46
that we're taking advantage of the
00:14:48
search that we already have on our
00:14:50
website and we're also bringing people
00:14:54
to resources that they might not have
00:14:56
found on their own because you do have
00:14:58
to spend a little bit of time looking
00:15:00
for them on our
00:15:02
website you know it takes a a minute
00:15:05
it's a little bit
00:15:07
deeper than just one search um so now
00:15:11
going outside of our website we have a
00:15:13
couple of Google Scholar searches and
00:15:15
again it's about building the searches
00:15:18
for the user and then they're here so
00:15:20
then you could change it to something
00:15:22
else whatever you know you wanted to
00:15:24
follow
00:15:26
um in your research
00:15:30
we have a semantic scholar search we
00:15:32
have site searches which I think are
00:15:34
another great way of taking advantage of
00:15:36
something that large language models are
00:15:38
good at which is knowing a lot about
00:15:41
sort of large areas of study so site
00:15:46
search is a great way to filter search
00:15:49
to only bringing back resources
00:15:52
from the website that you specified but
00:15:56
in this case um it's going to have the
00:15:59
added Boost from the large language
00:16:02
model of um being websites that it
00:16:06
thinks will be useful for academic
00:16:09
research in this area so we have a World
00:16:13
Health Organization site search and the
00:16:17
the search is for microplastics drinking
00:16:20
water and it will only bring up World
00:16:23
Health Organization
00:16:24
resources or we have the Environmental
00:16:27
Protection Agency or the National
00:16:30
Institute of
00:16:31
Health plastic particles in bottled
00:16:34
water nanoplastics may help set stage
00:16:37
for Parkinson's risk so it's a good
00:16:39
place to start um there are things that
00:16:43
you know people might not think of that
00:16:45
come up here that can be interesting
00:16:48
American Chemical
00:16:50
Society
00:16:52
perhaps um and then we have a couple of
00:16:56
links for writing and citation help and
00:16:59
we have a web page here where um there's
00:17:03
an email form where a student or
00:17:06
researcher could ask for additional
00:17:08
Assistance or they can click here to
00:17:11
contact a
00:17:13
librarian now this is a separate message
00:17:15
that is sent to a large language model
00:17:17
with again all of this
00:17:19
context and in addition to that it has a
00:17:22
list of our Librarians in their
00:17:24
Specialties so it's going to give
00:17:27
back one or to Librarians that it thinks
00:17:30
might be helpful in this research and
00:17:34
it's going to also give an explanation
00:17:36
of
00:17:37
why and something that I really like is
00:17:40
that if you click on the website link
00:17:44
that is given
00:17:46
here um we come to a librarian's web
00:17:50
page on our website in this case Hannah
00:17:52
Chapman trip and you can schedule an
00:17:55
appointment right here or email her
00:17:57
right here so it's very low
00:18:04
friction thank you so much for sticking
00:18:07
with me through this
00:18:09
presentation if you are interested in
00:18:11
our project or in starting one of your
00:18:14
own or have questions or comments please
00:18:19
get in touch we'd love to talk to
00:18:23
you you can also test out the chatbot at
00:18:26
the link here we learned something new
00:18:28
from each chat and it helps us to make
00:18:31
it better so don't hold back and thank
00:18:34
you again
