When it comes to features to deal with concurrent threads or processes, it is not enough just to write unit tests to verify the accuracy of our code.
In this post, you will find a short but effective way to create an integration test able to cover the possibility of dealing with more than one process at the same time and to verify that each one gets the expected response.
We had a table to manage a few important configurations. Modifying those tables would trigger file-related transactions that take some time to complete. So, if somebody else tried to modify the configs before the files were updated, it would create a big problem for the application.
The first solution for this was to use row-level locking using a read-committed isolation level interior transaction that could not be tested correctly because of the surrounding transactions created by DatabaseCleaner. So, we used a "hack" to avoid the problem in testing, but it meant not testing the code that is being used in production.
To make better code without requiring that read-committed isolation level, we decided to move the locking onto Redis, away from PG, which would lower the PG load level a bit, and also improve the testing coverage of our code and quality.
The first point to tackle was to move the attribute used to lock (boolean) the current record from a column in the database to an attribute stored in Redis. This new attribute will be saved under a key created by concatenating a prefix MY_MODEL_CONFIG_LOCKER plus the record ID.
The method was modified so that we first check for this attribute in Redis and decide whether it should be modified or not. If so, we would call the method that modifies or deletes the Redis attribute.
The value stored in the Redis attribute has got to be the UUID of the current thread, so we can know if the locker of the model is our current thread (meaning that we can modify or unlock the record).
At this point, our methods are supposed to do what we need them to do, but, in the unit tests, we can only be sure that it would work if only one process tries to modify the record. This is our real problem to solve because this feature was made to actually work with multiple clients requesting the service at the same time.
We can test the method with more than one process by starting more than one rail console and holding the record on one of them. The expected result would be that the rest of the consoles don’t have the ability to modify the requested record. But, when we want to add the test to the automated tests of the entire application, we would have to create an integration test and somehow keep an eye on each result and verify that only one of the created processes could modify the requested record.
For this test, we decided to use two variables so the user can decide if he/she wants to see the detail of each iteration or context.
The first test we have to write is to be sure that we at least have a connection to a Redis server. We did that in the same way we would do it in a rails console, with the command ping.
The idea here is to run a few scenarios where different numbers of processes try to modify the same record. So I decided to make a shared example and tell them how many processes it will generate and insert into the race to change the record. The name I used for this one parameter is num_procs.
In the next block, we will use the method Fork to create a new thread for each iteration. For each Thread, we are going to create a temp file where the logs and results of each process will be stored. Later, we are going to read the file and get the data to verify if it is what we expected.
The line my_model_company_config.reset is a method in the model that removes any blocking by deleting the related attribute in Redis.
You will also see that we use sleep to leave a gap between each process.
In this part, we had a little problem with the PostgreSQL connection. Each time a Thread finished its work, it would close the connection to the database, causing an ActiveRecord error. The solution here was to request a connection from the pool by using the method ActiveRecord::Base.establish_connectio".
At the end of this block, we are going to read the generated files and parse the logs to generate a JSON with the data we want.
Now let’s check out what we have in the method my_subject.
In this method, we collect the information about the lock previous to requesting the locking of the record, then we request the locking and collect the information again. We are putting that info together into a JSON and pushing the results to the output variable.
That is the data that we read at the end of the previous block.
In these two blocks, we do the actual validation of the results. In the first block, we define a method for showing the summary of the tests. I will leave the method below.
Also, we map the response we got in the "all_proc_data" in order to get only the column that tells us if the request had a successful result or not.
In the second block (the it clause), we ensure that all but one of the results were FAILED and that only one of the processes succeeded in blocking the record.
This way, we can be sure that only one of N processes was able to block and modify a single record, and can be sure that our code will work as expected when it is needed.
This is the method that shows the user the collected data in a formatted table.
And this is the shown data:
We’d love to learn more about your project.
Engagements start at $75,000.