In Part 3, we built out the API and learned how to access a database from inside of our Play application. In this post we will take a deeper look at concurrent requests, utilizing the asynchronous tools at our disposal to fix some concurrency bugs that were introduced in the last section.
As before, the source code for this tutorial series is available on github with the code specifically for this post here. Since the code builds on the previous post, you can grab the current state of the project (where part three finished) here.
Racing to failure
We will start this post by looking at a concurrency problem that exists in our current implementation of Ticket Overlords. When a user attempts to place an order, the following steps are performed:
- The JSON body is parsed into an
- The current amount of available tickets is queried
- An order is saved to database if enough tickets are available
- The response in converted to JSON
We can represent this graphically while tracking the state of our available tickets.
That diagram is a bit simplistic, as our server can easily be expected to handle multiple connections at the same time. Ignoring the asynchronous nature of the Slick calls to the database, we can illustrate two connections again to see what happens in a best case scenario.
As before, the orders are processed as expected, with the first order removing five tickets from the available pool causing the second order to fail when it checks if there are six tickets still available for purchase. This is the best case scenario, but not the only possibility.
In this flow, we can see that the logic to check how many tickets were available ran in succession. This resulted in a race condition allowing tickets to be oversold. Since we are not running an airline, this is a problem. We have ignored the fact that our Slick requests are asynchronous only for simplicity in the diagram. In reality, the async nature amplifies this race condition.
Locking it down
You may be thinking “Gee, thanks for teaching me standard comp sci fundamentals, could we cover basic data structures next?” to which the answer is YES! Kind of. We are going to use actors as queues. Since the default mailbox of an actor functions as a queue, we can use this to ensure that the critical section consisting of Step 2 and Step 3 from before is performed atomically.
Start by creating a class named
TicketIssuer in the
com.semisafe.ticketoverlords package that extends
This is a fairly straight forward example of an actor skeleton. The
function only expects one type of message, an
Order class indicating the order
to be placed. We can now fill out the
placeOrder function. It will look very
familiar, as the logic is nearly identical to what is written in the
Before we do this, we need to create a new
representing the error condition when no tickets are available.
Now for the rest of the
The major difference between the original code from the controller and this
version is that we are not returning the responses directly. We are assuming
ask is utilized to call this actor and we either respond with
Order that was created, or an
ActorFailure to avoid ambiguity with the
Failure associated with
The fact that
sender is reassigned to a local value of
origin at the
placeOrder is also important. This is because
sender(), a function. It is a common mistake to assume it is a value
that is safe to reference at any point. When
origin is utilized to send the
createdOrder message back, it is actually two
Futures deep. This means that
the value returned by
sender() inside of
createdOrderResult.map() is most
likely not the
ActorRef that would be assigned when we saved the value into
We can now open up the
Orders controller to make our changes that support the
new method of placing an order. We start by initializing an actor directly in
Now that we know the actor will exist when we need it, we can replace the
create function in
Orders with code that utilizes the
TicketIssuer actor. We could request an
ActorSelection from the path
/user/ticketIssuer which is where our previously declared
val ends up being
located, but since the
ActorRef is stored locally, there is no need.
We set a default
timeout and proceed to send an
ask message to the
ask returns a
Future[Any] as a result. This is what
people are complaining about when they say that Akka actors are not type safe.
orderFuture is not just a
Future[Any], it is most definitely a
Future[Order]. We can force this with the
mapTo method on the future.
If something in our code changes later and for some reason the future can not be
cast to a
mapTo future becomes a failure and can be
handled by normal error recovery code.
While we are changing things, the value of 5 seconds is totally arbitrary and
probably too short. People used to camp out overnight for concert tickets, they
will wait at least 30 seconds on a website for an order to process. Open the
conf/application.conf and add the following lines to the end.
Orders controller, we can remove the line with the hardcoded value and
replace it with logic to use the configuration value if present. Every
configurable value is an
Option, so you will still need that arbitrary default
We are now ready to tackle the actual response from the actor. As before, we can
map on the future to transform it from a
Future[Order] to a
This is looking good, but let’s be honest with each other. Not all futures are
sunshine and rainbows. Some result in a dystopian society where dogs and cats
live together and machines have enslaved us all. Our new robot overlords force
us to vacuum their apartments for them while they are at work, chanting over and
over “How do you like that now?” in disembodied robotic voices. Then there are
the futures that are really just caused by exceptions being thrown. We can
handle the latter case with the
recover method allows us to transform the
Throwable content of the
failure into a different type of
Future. In our case, there is the expected
failure when there are not enough tickets to process the order. There is also
a potential case where something unexpected occurs, which we will log and wrap
in a standard JSON response.
Our code no longer suffers from the same race condition during orders from concurrent requests, but it is still present in the async actions from the database access library. Although the race condition exists when two concurrent actions operate on the same ticket block, there is no problem if concurrent actions are operating on different ticket blocks.
Create a new class in the
com.semisafe.ticketoverlords package named
TicketIssuerWorker. The base definition of this actor will differ slightly, as
we will add a constructor parameter for the ticket block ID.
The logic inside of the
TicketIssuerWorker is almost identical to the original
TicketIssuer, with extra logic that guarantees placing orders only for
the ticket block that was assigned at instantiation.
We now have the building blocks to add these workers in to our
actor. We will start by creating a
var member of
TicketIssuer that is a
ActorRefs and a utility function for adding a child actor worker for
a given ticket block ID. In general, we try to avoid the use of
vars in scala,
but when inside of an actor, it can be acceptable within reason.
This function checks to see if there is already a local reference to that ticket block. If the reference does not exist, a new actor is instantiated and has it’s reference added to the local mapping for retrieval later.
With this logic defined, we can now add a
preStart method on our
preStart method is called every time the actor is started
and restarted making it suitable for setting up the initial state of our actor.
To utilize our new
TicketIssuerWorker classes, we need to replace the
placeOrder method in
placeOrder method extracts the
ActorRef for the correct worker and
passes along the actual order to be placed. You will notice that instead of a
!) or an ask (
?), we use
forward. This performs a
but passes along the original
sender’s reference. This is what allows our
worker actors to reply to the originator of the request by using their local
We still have an unhandled error condition, so we will create the class for that exception now.
And update the
placeOrder code one more time.
To finish this up, the only change remaining is to add this new error condition
Orders controller, inside the
recover block (right before the
This requires a new
ErrorResponse value to be defined. Once again, any value
will work, the example code uses 1002.
Updating the worker pool
One thing you may have noticed is that the system will only create worker actors
TicketIssuer actor is started (or restarted). This means that we are
not be able to access or place an order on a new ticket block without restarting
the system, which is inconvenient.
We will create a case class named
TicketBlockCreated and place it just above
the class definition for
We will also modify
TicketIssuer’s receive function to accept a
TicketBlockCreated message using the contents as the argument to the
createWorker function we defined earlier.
Now we have a way to respond to signals after a new
TicketBlock has been
created. We can update the
create method of
TicketBlock to send this
message. Since the
TicketBlock object has no local reference to the issuer
ActorRef, we need to use the
If your application is running, stop it. Now start it again. (If it was not
running, start it now). Before you do anything to interact with the application,
try creating a fresh ticket block with
Based on the response from
curl, it appears to have been successful. We
received a good response back and the ticket block has an ID, but if we look in
the activator log window, it tells a different story.
This happened because we only instantiate the
TicketIssuer actor inside of the
Orders controller, but since nothing has interacted with that object yet, none
of the member values have been instantiated, including our actor.
A solution for this is to encapsulate all references to the
inside of it’s own companion object. While we are creating a companion object,
we will follow another best practice for actors, creating a
props method unifies how all actors are created instead of requiring
differing code depending on if the actor requires instantiation parameters or
This guarantees that our
TicketIssuer actor will have been instantiated before
any other code calls it, while also removing the need for other code to know
where it resides (the
/user/ticketIssuer part). We can replace the
getSelection call in
TicketBlock and the local reference in
the following code.
A downside to this is that instantiating the actor is still on-demand the first time a code path is hit that requires access to the ticket blocks. Depending on how much is required for the actor to start (how many ticket blocks need to have workers created), there may be some delay in the actual response. We will cover the actual 100% correct and recommended way in a later post. This method is good enough for now.
The separation of dogma and state
In Scala, it is generally not advised to use a
var when a
val will suffice.
If you define one in eclipse, it even colors it red as a way of saying “You
probably should not be doing this”. That said, there are times when a
appropriate. We already saw this with our map of
references. In the current
TicketIssuerWorker code, every time an order is
placed there is a database query to gather the amount of available tickets,
followed by a second database query to create the order. The async calls here
are the root of why we still have a race condition. Now that
TicketIssuerWorker is the canonical source for ordering tickets, we can
actually maintain the ticket count internally within the worker.
Inside of the
TicketIssuerWorker class, but outside of any function
definition, move the availability query from
make an availablilty
placeOrder method to decrement
availability by the amount of
the order. Now that the availability Future has been moved to the
method, every access to the availability value is queued and locked by the
actor’s receive path.
This is a safe use for a
var since it is being used to represent the internal
state of the actor and it is not accessible to anything externally. The fact
that there is only one single concurrent path accessing the
safety from concurrency issues.
We can do better than this
TicketIssuerWorker is functioning properly, but we can probably find a
safe way to get rid of the
var while also having it return a more appropriate
message while it is starting up. Currently, if a request comes in after
preStart has run, but before the database call finishes, the response will
appear that the ticket block has sold out, as opposed to a more correct message
that it is currently unavailable.
We can do all of this through the use of the actor’s
become functionality. Our
actor really has three possible states: initialization, normal operation and
sold out. We can define these as three separate
Actor.Receive functions that
accept either the
Order message or a new case class
AddTickets for when
tickets are to be added to our ticket block. We will also factor out the routing
placeOrder as each receive method will utilize this.
Notice at the end, we have replaced the previous definition of
receive with a
statement indicating that the actor begins with the
initializing behavior. We
context.become() method to change behavior and communicate the current
availability value. The removes the need for the
var, which can now be
We need to modify our
placeOrder method one more time to account for this.
And finally update our
preStart method to safely pass the initial availability
to our actor.
This give us a fully operational concurrency safe actor that manages an internal
state without the use of a
Until next time…
We now have a system that can safely handle many concurrent requests where each
ticket block is isolated from the other ones. We covered reading
configuration parameters from the
conf files, handling failure cases in
Futures. We have also learned about handling asynchronous actions and managing
state with actors, which is good for the general overall performance of a Play
In part 5, we will take a quick look at composing futures from multiple sources.