00:00:00
Hi and welcome back!
00:00:01
In the previous lectures,
00:00:03
we’ve learned how to write golang codes
to perform CRUD operations
00:00:06
on each individual table of the simple bank
database.
00:00:10
But in a real world application,
00:00:12
we often have to perform a transaction
00:00:14
that combines some operations from several
tables.
00:00:17
Today we will learn a how to to implement
it in Golang.
00:00:21
Before we jump into coding,
00:00:22
Let’s talk a bit about transaction.
00:00:24
What is a database transaction?
00:00:25
Well, basically,
00:00:26
It’s a single unit of work
00:00:28
That’s often made up of multiple database
operations.
00:00:32
For example,
00:00:33
In our simple bank,
00:00:35
we want to transfer 10 USD from account 1
to account 2.
00:00:39
This transaction comprises 5 operations:
00:00:42
First, we create a transfer record with amount
equals to 10.
00:00:46
Second, we create an account entry record
for account 1
00:00:50
with amount equals to -10,
00:00:52
since money is moving out of this account.
00:00:55
Third, we create another account entry record
for account 2,
00:00:59
but with amount equals to 10,
00:01:01
because money is moving in to this account.
00:01:04
Then we update the balance of account 1 by
subtracting 10 from it.
00:01:09
And finally we update the balance of account
2 by adding 10 to it.
00:01:13
This is the transaction that we’re going
to implement in this video.
00:01:17
We will come to that in a moment.
00:01:19
But why do we need to use database transaction?
00:01:22
There are 2 main reasons:
00:01:24
First, we want our unit of work to be reliable
and consistent,
00:01:28
even in case of system failure.
00:01:31
And second, we want to provide isolation between
programs
00:01:34
that access the database concurrently.
00:01:37
In order to achieve these 2 goals,
00:01:39
a database transaction must satisfy the ACID
properties.
00:01:43
Where A is Atomicity,
00:01:45
which means either all operations of the transaction
complete successfully,
00:01:50
or the whole transaction fails, and everything
is rolled back,
00:01:53
and the database is unchanged.
00:01:55
C is Consistency,
00:01:57
which means the database state should remains
valid after the transaction is executed,
00:02:03
More precisely, all data written to the database
must be valid
00:02:07
according to predefined rules, including constraints,
cascades, and triggers.
00:02:13
I is Isolation,
00:02:15
meaning all transactions that run concurrently
should not affect each other.
00:02:19
There are several levels of isolation
00:02:22
that defines when the changes made by 1 transaction
can be visible to others.
00:02:26
We will learn more about it in another lecture.
00:02:29
The last property is D, which stands for Durability.
00:02:33
It basically means that all data written by
a successful transaction
00:02:37
must stay in a persistent storage,
00:02:39
and cannot be lost, even in case of system
failure.
00:02:43
OK, so how to run a SQL database transaction?
00:02:46
It’s pretty simple.
00:02:48
We start a transaction with the BEGIN statement.
00:02:51
Then we write a series of normal SQL queries
(or operations).
00:02:55
If all of them are successful,
00:02:57
We COMMIT the transaction to make it permanent,
00:03:00
The database will be changed to a new state.
00:03:02
Otherwise, if any query fails,
00:03:05
we ROLLBACK the transaction,
00:03:07
thus all changes made by previous queries
of the transaction will be gone,
00:03:11
and the database stays the same as it was
before the transaction.
00:03:15
Alright,
00:03:16
Now we has some basic understanding about
database transaction.
00:03:19
Let’s learn how to implement it in Golang.
00:03:21
I’m gonna create a new file store.go inside
the db/sqlc folder.
00:03:27
In this file, let’s define a new Store struct.
00:03:31
This store will provide all functions
00:03:33
to run database queries individually,
00:03:36
as well as their combination within a transaction.
00:03:39
For individual queries,
00:03:41
we already have the Queries struct generated
by sqlc
00:03:45
that we have learn in previous lectures.
00:03:47
However, each query only do 1 operation on
1 specific table.
00:03:52
So Queries struct doesn’t support transaction.
00:03:56
That’s why we have to extend its functionality
00:03:59
by embedding it inside the Store struct like
this.
00:04:02
This is called a composition,
00:04:04
and it is the preferred way to extend struct
functionality in Golang instead of inheritance.
00:04:11
By embedding Queries inside Store,
00:04:13
All individual query functions provided by
Queries will be available to Store.
00:04:19
And we can support transaction by adding more
functions to that new struct.
00:04:24
In order to do so, we need the Store to have
a sql.DB object.
00:04:28
because it is required to create a new db
transaction.
00:04:32
OK, now let’s add a function to create a
new Store object.
00:04:35
It will takes a sql.DB as input, and returns
a Store.
00:04:41
Inside, we just build a new Store object and
return it.
00:04:45
Where db is the input sql.DB,
00:04:48
and Queries is created by calling the New
function with that db object.
00:04:53
This New function was generated by sqlc
00:04:56
as we’ve already known in the previous lectures.
00:04:59
It creates and returns a Queries object.
00:05:03
Next, we will add a function to the Store
00:05:06
to execute a generic database transaction.
00:05:09
The idea is simple,
00:05:10
It takes a context and a callback function
as input,
00:05:14
Then it will start a new db transaction,
00:05:17
Create a new Queries object with that transaction,
00:05:19
And call the callback function with the created
Queries,
00:05:22
And finally commit or rollback the transaction
00:05:25
based on the error returned by that function.
00:05:28
Alright, let’s implement this.
00:05:31
First, to start a new transaction,
00:05:33
we call store.db.BeginTx()
00:05:36
Pass in the context,
00:05:38
And optionally a TxOption.
00:05:41
This option allows us to set a custom Isolation
Level for this transaction.
00:05:45
If we don’t set it explicitly,
00:05:48
then the default isolation level of the database
server will be used,
00:05:51
which is read-committed in case of Postgres.
00:05:55
We will learn more about this in another lecture,
00:05:58
For now, let’s just pass nil here to use
the default value.
00:06:01
The BeginTx() function returns a transaction
object or an error.
00:06:06
If error is not nil, we just return it immediately.
00:06:10
Otherwise, we call New() function with the
created transaction
00:06:14
And get back a new Queries object.
00:06:16
This is the same New() function that we used
in the NewStore() function,
00:06:21
The only difference is, instead of passing
in a sql.DB,
00:06:25
We’re now passing in a sql.Tx object.
00:06:28
This works because the New() function accepts
a DBTX interface
00:06:32
as we’ve seen in the previous lecture.
00:06:35
OK, now we have the queries that runs within
transaction,
00:06:39
We can call the input function with that queries,
00:06:41
And get back an error.
00:06:43
If the error is not nil,
00:06:45
Then we need to rollback the transaction
00:06:48
We do that by calling tx.Rollback().
00:06:51
It also returns a rollback error.
00:06:55
If the rollback error is also not nil,
00:06:58
then we have to report 2 errors.
00:07:00
So we should combine them into 1 single error
before returning.
00:07:04
To do that, we use fmt.Errorf command,
00:07:08
First we add the transaction error,
00:07:10
Then the rollback error.
00:07:13
In case the rollback is successful,
00:07:15
we just return the original transaction error.
00:07:18
Finally,
00:07:19
If all operations in the transaction are successful,
00:07:22
We simply commit the transaction with tx.Commit(),
00:07:27
and return its error to the caller.
00:07:29
And we’re done with the execTx() function.
00:07:32
Note that this function is unexported
00:07:35
(it starts
with a lowercase letter e),
00:07:37
because we don’t want external package to
call it directly.
00:07:41
Instead, we will provide an exported function
for each specific transaction.
00:07:46
Now let’s go a head and add a new TransferTx()
function
00:07:50
to perform the money transfer transaction
example
00:07:53
that we discussed at the beginning of the
video.
00:07:56
To recall, it will create a new transfer record,
00:07:59
add new account entries,
00:08:02
and update accounts’ balance
00:08:04
within a single database transaction.
00:08:07
The input of this function will be a context
00:08:11
And an argument object of type TransferTxParams
00:08:15
And it will return a TransferTxResult object
or an error.
00:08:20
Now let’s define the TransferTxParams.
00:08:23
This struct contains all necessary input parameters
to transfer money between 2 accounts,
00:08:30
FromAccountID is the ID of the account where
money will be sent from.
00:08:36
ToAccountID is the ID of the account where
money will be sent to.
00:08:41
And the last field is the amount of money
to be sent.
00:08:45
The TransferTxResult struct contains the result
of the transfer transaction.
00:08:51
It will have 5 fields:
00:08:53
The created transfer record.
00:08:56
The from account after its balance is updated.
00:09:01
The to account after its its balance is updated.
00:09:05
The entry of the from account which records
that money is moving out.
00:09:11
And the entry of the to account which records
that money is moving in.
00:09:16
Alright, now we can implement the transfer
transaction.
00:09:20
First we create an empty result.
00:09:24
Then we call the store.execTx() function that
we’ve written before
00:09:28
to create and run a new database transaction.
00:09:31
Pass in the context
00:09:33
and the callback function.
00:09:35
For now let’s just return nil.
00:09:37
Finally we return the result and the error
of the execTx() call.
00:09:42
Now let’s come back to implement this callback
function.
00:09:46
Basically, we can use the queries object
00:09:48
to call any individual CRUD function that
it provides.
00:09:52
Keep in mind that this queries object is created
from 1 single database transaction,
00:09:57
So all of its provided methods that we call
will be run within that transaction.
00:10:02
Alright, let’s create the transfer record
by calling q.CreateTransfer()
00:10:06
Pass in the input context, and a CreateTransferParams
00:10:11
Where FromAccountID is arg.FromAccountID
00:10:15
ToAccountID is arg.ToAccountID
00:10:20
And Amount is arg.Amount
00:10:22
The output transfer will be saved to result.Transfer
00:10:26
And we also have to declare this error variable.
00:10:31
Then here we check
00:10:32
If error is not nil, just return it right
away.
00:10:36
Now here you can see that
00:10:37
we’re accessing the result variable of the
outer function
00:10:41
from inside this callback function.
00:10:43
similar for the arg variable.
00:10:46
This makes the callback function become a
closure.
00:10:49
Since Go lacks support for generics type,
00:10:51
Closure is often used when we want to get
the result from a callback function,
00:10:56
because the callback function itself doesn’t
know the exact type of the result it should return.
00:11:02
OK so the 1st step to create a transfer record
is done.
00:11:06
Next step is to add 2 account entries,
00:11:09
1 for the from account, and 1 for the to account.
00:11:13
So result.FromEntry, error equals to q.CreateAccountEntry()
00:11:19
We pass in the context
00:11:20
and a CreateAccountEntryParams,
00:11:23
where AccountID is arg.FromAccountID,
00:11:27
and Amount is -arg.Amount because money is
moving out of this account.
00:11:33
And just like before,
00:11:35
If error is not nil, we just return it
00:11:38
so that the transaction will be rolled back.
00:11:41
We do similar thing to create an account entry
for the to account.
00:11:45
This time, it is result.ToEntry
00:11:49
the AccountID is arg.ToAccountID,
00:11:52
And the amount is just arg.Amount since money
is moving in to this account.
00:11:58
And we’re done with the account entries
creation.
00:12:02
The last step to update account balance will
be more complicated
00:12:06
because it involves locking and preventing
potential deadlock,
00:12:10
So I think it’s worth a separate lecture
to talk about it in details.
00:12:15
For now, let’s add a TODO comment here,
00:12:17
and we will come back to implement it in the
next video.
00:12:21
Now let’s say our TransferTransaction is
done
00:12:23
with 1 transfer record and 2 account entries
are created.
00:12:28
We have to test it to make sure that it’s
working as expected.
00:12:32
I’m gonna create a new store_test.go file.
00:12:35
It’s in the same db package as our store.go
00:12:39
Then let’s define a new unit test for the
TransferTx() function.
00:12:44
First we need to create a new Store object.
00:12:47
The NewStore() function requires a sql.DB
object,
00:12:51
If you still remember, in the previous lecture,
00:12:54
we have already created a sql.DB object
00:12:57
in the main_test.go file
00:12:59
with this sql.Open() call.
00:13:01
So in order to reuse it,
00:13:04
here instead of assigning the result to the
connection variable,
00:13:08
we will declare a new global variable: testDB
00:13:11
And store the result of the sql.Open() command
in it.
00:13:15
We should remove this colon because testDB
is not a new variable.
00:13:20
And therefore, we must declare the error up
here to make it an existed variable.
00:13:26
Finally we have to change this connection
variable to testDB
00:13:30
when creating the testQueries object.
00:13:32
OK, now we can come back to our unit test
00:13:35
and pass the testDB into this function to
create a new Store.
00:13:40
Next, we will create 2 random accounts using
the function we wrote in the previous lecture.
00:13:45
We will send money from account 1 to account
2.
00:13:49
From my experience,
00:13:51
writing database transaction is something
we must always be very careful with.
00:13:55
It can be easy to write,
00:13:57
but can also easily become a nightmare
00:13:59
if we don’t handle the concurrency carefully.
00:14:03
So the best way to make sure that our transaction
works well
00:14:06
is to run it with several concurrent go routines.
00:14:09
Let’s say I want to run n = 5 concurrent
transfer transactions
00:14:14
And each of them will transfer an amount of
10 from account 1 to account 2.
00:14:19
So I will use a simple for loop with n iterations
00:14:23
And inside the loop,
00:14:25
we use the go keyword to start a new routine.
00:14:28
Make sure to have this bracket at the end
of the function to run it.
00:14:32
Now inside the go routine,
00:14:34
We call store.TransferTx() function with a
background context
00:14:39
And a TransferTxParams object,
00:14:42
where FromAccountID is account1.ID
00:14:46
ToAccountID is account2.ID
00:14:50
And Amount is 10 as we declared above.
00:14:54
This function returns a result or an error.
00:14:57
Now we cannot just use testify require to
check them right here
00:15:01
because this function is running inside a
different go routine
00:15:05
from the one that our TestTransferTx function
is running on,
00:15:09
So there’s no guarantee that it will stop
the whole test if a condition is not satisfied.
00:15:14
The correct way to verify the error and result
is to send them back
00:15:19
to the main go routine that our test is running
on,
00:15:22
and check them from there.
00:15:24
To do that, we can use channels.
00:15:26
Channel is designed to connect concurrent
Go routines,
00:15:30
and allow them to safely share data with each
other without explicit locking.
00:15:35
In our case, we need 1 channel to receive
the errors,
00:15:39
And 1 other channel to receive the TransferTxResult.
00:15:43
We use the make keyword to create the channel.
00:15:46
Now, inside the go routine,
00:15:49
we can send error to the errors channel using
this arrow operator,
00:15:53
The channel should be on the left,
00:15:55
and data to be sent should be on the right
of the arrow operator.
00:16:00
Similarly, we send result to the results channel.
00:16:04
Then, we will check these errors and results
from outside
00:16:09
We simply run a for loop of n iterations,
00:16:13
To receive the error from the channel,
00:16:15
we use the same arrow operator,
00:16:17
But this time, the channel is on the right
of the arrow,
00:16:21
and the variable to store the received data
is on the left.
00:16:26
We require no errors here,
00:16:28
which means the received error should be nil.
00:16:31
Likewise, we receive result from the results
channel
00:16:36
Check that result is not an empty object.
00:16:40
As result contains several objects inside,
00:16:42
Let’s verify each of them.
00:16:45
Start with the result.Transfer.
00:16:48
We require this transfer object to be not
empty
00:16:53
Then the from account ID field of transfer
should equal to account1.ID
00:16:59
The to account ID field of transfer should
equal to account2.ID
00:17:06
And transfer.Amount should equal to the input
amount.
00:17:10
The ID field of transfer should not be zero
because it’s an auto-increment field
00:17:16
And finally transfer.CreatedAt should not
be a zero value
00:17:20
because we expect the database to fill in
the default value,
00:17:24
which is the current timestamp.
00:17:26
Now to be sure that a transfer record is really
created in the database,
00:17:31
We should call store.GetTransfer() to find
the record with ID equals to transfer.ID.
00:17:38
Here you can see that,
00:17:39
because the Queries object is embedded inside
the Store,
00:17:43
the GetTransfer() function of Queries is also
available to the Store.
00:17:47
If the transfer really exists,
00:17:49
this function should not return an error.
00:17:52
So we require no error.
00:17:54
OK, next we will check the account entries
of the result.
00:17:59
First, the from entry.
00:18:01
Just like before,
00:18:02
We check that it should not be empty.
00:18:06
The account ID should be account 1
00:18:11
And the amount of the entry should equal to
-amount because money is going out.
00:18:18
Finally the ID and created at fields of the
entry should be not zero.
00:18:24
We also try to get the account entry from
the database
00:18:27
to make sure that it’s really got created.
00:18:35
Checking the to entry is similar.
00:18:38
So I will copy this block of code
00:18:41
And change these variable and field names
to toEntry.
00:18:46
This account ID should be account 2 instead
of account 1.
00:18:50
And the amount should be positive instead
of negative because money is going in.
00:18:58
In the end, we should get the toEntry record
instead.
00:19:02
Now keep in mind that we should also check
the accounts’ balance as well,
00:19:06
But since we haven’t implemented the part
to update accounts’ balance yet,
00:19:11
Let’s add a TODO comment here for now,
00:19:13
and we will complete it in the next lecture.
00:19:16
Alright, now the test is ready
00:19:20
Let’s try to run it.
00:19:23
It passed!
00:19:24
Excellent!
00:19:27
Let’s run the whole package tests.
00:19:30
All passed!
00:19:32
The coverage is about 80%,
00:19:34
Which is very good.
00:19:36
And that wraps up today’s lecture about
00:19:38
how to implement database transaction in Golang.
00:19:42
I hope you enjoy it,
00:19:43
You can try to implement the update account
balance yourself
00:19:47
while waiting for the next video.
00:19:50
Happy coding and see you in the next lecture!