Current Situation
Several times a week, and occasionally multiple times a day, I push code to production. The product is an in-house portfolio management solution for a small Danish fund. This post is about the devops setup and some broader considerations on requirements for this kind of work. I am writing about this as a small contribution to dispel myths about best practices in software development.Today, any new project should start with git and continuous deployment. Here the task is to find the minimum viable setup, one that let's you push to a branch and voilà the code is in production. In our case, the code for all apps and database migrations is hosted in separate repositories and deployed to various servers by pushing to a branch with a special name within each repository. We have no staging branches, backend tests or end-to-end tests in the deployment pipeline whatsoever. It’s the most simple setup I’ve ever worked with, and that is a mindful choice.
How on earth can you push un-tested code directly to production and sleep at night? A fundamentalist test-first developer might ask. The answer lies in thinking about resources and acting within constraints.
Most importantly, I’m the sole developer. This means that in order to be effective, I have to make sure that all my efforts go into adding value. In a team of 5, if one person is 50% productive, and all others are at their capacity of 100%, the overall productivity loss is 10%. If I’m 50% productive, the overall productivity loss is 50%. In addition, I’m building an in house platform for a small company. This gives rise to an entirely different situation than building a user facing system for a large number of users scattered around the world. The feedback loop of an in-house platform in a small company is exceptionally fast, so bugs that make it into production, are short lived. If anything breaks, I will know it soon after it’s been discovered, and in 9 out of 10 cases it’s fixed within working hours the same day.
Also, the fact that there are no automatic tests in the deployment pipeline does not mean that I don’t use TDD. I like testing for the sake of development, and occasionally use it to identify potential bugs or make sure known bugs are fixed. I write tests, use them, and forget about them.
And last, our situation is such that the most important numbers, the numbers we as a hedge fund, just need to get right, are provided to us by our depositor. Without going into technical details, this means that we always have numbers with a very high degree of certainty to compare with the non-audited numbers of our internal platform. This provides a constraint, which quickly exposes critical misalignment of data on the platform.
So in conclusion, I use tests in various ways during development, but don’t strive to build a systematic framework for testing. This is not because I have anything against employing a systematic testing scheme, but mainly because my particular situation as described above lends itself to a less rigid approach.
A Better Approach
Best practices are situational. So what if I had more resources at my disposal? Then these are some of the measures I would employ:Manual code reviews. I’ve worked places where I’ve done reviews and had my own code reviewed. It doesn’t just improve the code of the given commit, but improves the overall coherence, naming conventions, employed standards and so of the entire code base. It makes people think extra hard. And it’s fun.
Integration tests. Testing the entire backend from the HTTP request to the response is by far my favorite layer to test. It’s not even a layer, it just means all of it. I do that from time to time during development, by spinning up a new database and seeding with minimum required data before each test suite is run. I’m not a fan of running unit tests as a step in the deployment process, since we should allow ourselves to assume that the units of our code just works. Just as we don’t typically test external libraries, as we can assume that they work for our purpose of usage. Furthermore, integration tests should be fine grained enough to catch malfunctioning units in the execution path.
In our particular case, unit tests are no that relevant, since all of the complex calculations within the applications that I’m building happen in the database layer. By definition, if a database is a dependency, we're no longer in the realm of unit testing. This definition however is a convenience more than anything, since why not rely on a database? The database of my occasional tests does not exist any longer than the tests run. So we could argue that they play the role of RAM of the tests, somewhere to keep state during execution. If so, it would maybe dawn upon us that RAM itself is a dependency of unit tests, and hence unit tests themselves are not real unit tests. And through that route we have arrived at a paradox, and we should throw away the concept of unit tests all together. The counter argument is that you need RAM to just run the code, so RAM is inherent in the smallest meaningful unit of a program. And the counter argument to that, is that you really don’t. RAM is just convenience, and you can build CPUs that can execute programs without the massive bloat of things like RAM and operating systems. Sure, very limited programs encoded into the registers. Nevertheless, let’s just say language is convention.
End-to-end tests. I’ve developed other web facing applications where I’ve prioritized end-to-end tests. Cypress is good for that. Of course the frontend is volatile, but still, with an adequate amount of resources, I would not mind a comprehensive Cypress test suite to run on each PR.
Staging and production branches. Currently, I don’t even use that. Not having a staging branch might for some developers, even solo developers of an in-house system sound too extreme. However, again it’s a based on a tradeoff between spending time deploying and spending time building features. For a staging branch to make any sense, it has to add value, and that value either stems from manual or automatic testing. If you don’t intend to embark on that, then adding a staging branch is just adding one more step in your deployment process. Of course I have set up automatic rollbacks in case anything does not compile or the database does not migrate properly or whatever. So far results have been good, and not once over several years has a commit caused an app to crash in production. If we would one day start to open up our platform for other funds, branches are on of the first thing I would do differently. But I would probably just have an internal and an external branch, where the internal branch would be a couple of weeks ahead of the external branch, and hence when the external branch is updated, it would be well tested from the daily use within our own fund.
Systematic manual tests. Lastly I would not mind real people continuously test new features. This already happens directly on the production branch, since multiple end users do work every day on the platform. Ideally, I would post a brief summary on an internal messaging board whenever the staging branch is updated, allowing someone with intricate knowledge of the platform to manually review potentially affected workflows.