01 Feb 2023 - by 'Maurits van der Schee'
Whether you deploy using git push, rsync or even sftp you never want your site to be down or inconsistent during updates. If your site has high traffic you may not only do frequent updates, you may also have significant traffic during these updates. With "zero downtime" deployments you can assure that all traffic can keep flowing and that there is little chance that anyone notices your deployment.
A naive deployment of an Apache website may go like this:
It should be clear that this can lead to downtime between the removal (often instantaneous) and the completion of the upload of the new version. A popular way to deal with this downtime is to place the site "under maintenance" by redirecting all requests to a single HTML page during the update. This isn't ideal, especially on larger sites with frequent updates as the downtime may be considerable.
A rsync with "--delete-after" goes directory by directory and does:
An zero downtime deployment takes place in the following steps:
As the server replaces the old version with the new version any new request that comes in after the switch is handled consistent. Any request that is handled before the switch is also handled consistent. Requests that are handled in a time-frame in which a switch took place may be handled inconsistently. E.g. a request that started 100 milliseconds before the switch and retrieved it's last asset 500 milliseconds after the switch is such a request. And obviously it is only inconsistent if the requests have actually hit some updated code. So depending on the size of the change that chance may vary.
The following script can deploy a git repository with zero downtime:
#!/bin/bash # set primary to the existing and # set secondary to the target directory if [[ -d green ]]; then PRIMARY=green SECONDARY=blue else PRIMARY=blue SECONDARY=green fi # make the target directory mkdir $SECONDARY # do the deployment if git -C app.git archive --prefix=$SECONDARY/ | tar x then # the deployment succeeded create a new symlink ln -s $SECONDARY public_html_new # replace the old symlink with the new symlink (atomic) mv -fT public_html_new public_html # remove the old files from disk rm -Rf $PRIMARY else # remove the failed deployment rm -Rf $SECONDARY fi
Note that the command "
git -C app.git archive --prefix=$SECONDARY/ | tar x" may also be a Hugo build or any other long lasting deployment process. And this Bash script may be put into the "post-receive" hook of your git repository (when pushing to production). This is what the directory looks like after deployment:
drwxrwxr-x 7 maurits maurits 4096 Nov 21 12:02 app.git drwxr-xr-x 9 maurits maurits 4096 Feb 1 17:09 green lrwxrwxrwx 1 maurits maurits 5 Feb 1 17:09 public_html -> green
Note that the Apache "DocumentRoot" is set to "public_html" directory, which is actually a symlink. Nine minutes later (after the next deployment) the directory looks like this:
drwxrwxr-x 7 maurits maurits 4096 Nov 21 12:02 app.git drwxr-xr-x 9 maurits maurits 4096 Feb 1 17:18 blue lrwxrwxrwx 1 maurits maurits 5 Feb 1 17:18 public_html -> blue
You can see that the existence of the blue and green directories alternate.
Even though zero downtime deployments can guarantee 100% uptime during deployments, they cannot guarantee 100% consistent replies during deployment. You can trade consistent replies for some forced downtime (and terminated connections) by stopping Apache before and starting it after the zero downtime deployment, but whether or not you want to make that trade-off is entirely up to you. I have had a lot of success with very frequent small incremental deployments using this zero downtime strategy and I hope you will too...