We at adjust recently started to use Elixir. We built a couple of small services using the Phoenix framework which successfully went live. In this blogpost I’d like to talk about, I’d say, the most undiscussed topic when it comes to Elixir — deployment.
Although you can find some blog posts about deploying Elixir applications, usually after reading them, it still remains unclear how to get the desired command which would deploy your code to production - and which would automate all the routines.
Capistrano way
The first thing we’ve tried was mina
. I’d say, trying to use Capistrano
or Mina
is an obvious choice if you come from the Ruby world. However, it becomes clear very quickly that the Capistrano way doesn’t fit well for Elixir apps. As you probably know, the preferred way to deploy Elixir applications is to use releases, which means you need a place where a release should be built. It’s possible to write a Capistrano
or Mina
recipe to clone a project to the production host and build the release there, but that wouldn’t be very good idea. Compiling and building a release will take some resources (especially memory) which you don’t want to share on production.
Another option would be to build a release locally using the cross-compiling feature and copy it to production. There are a few gotchas with such approach:
- there might be some differences in environment (dependency versions, elixir version, etc) between different developers’ computers, so two developers might build two different builds based on the same codebase;
- it would be quite tricky to write such a recipe for
Capistrano
(although much easier forMina
); generally, using Capistrano just to copy one tarball to a server, unpack it and start it looks like overkill.
So using releases means that there should be a machine where every developer can build a release. Right, a build server! And the problem is that the concept of a build server isn’t something familiar for Capistrano
or Mina
. So there should be a tool which is aware of the concept of a build server, which maybe even knows how to work with Elixir releases…
Thankfully such a tool does indeed exist.
Edeliver
Edeliver is a deployment tool for Elixir and Erlang projects. It knows how to work with releases and how to apply hot-upgrades, it’s aware of a build host and helps you to automate the deployment workflow. Edeliver
has very good and comprehensive documentation, including several wiki pages describing some edge cases as well. I don’t want to review edeliver
s README in this blogpost, but rather I’d like to cover some of those edge cases and gotchas which we’ve discovered while using it.
Auto Versioning
There is a small issue with release names — they must be unique, so every time the mix edeliver build release
command finishes, a unique release should be generated. Edeliver
solves this issue by having a special config parameter with which it’s possible to append a Git revision, Git branch, build date, etc to a release name. So you don’t need to go to the mix.exs
file and change version
in project/0
function – edeliver
does it for you. We found that AUTO_VERSION=git-branch+git-revision
generates sufficiently unique release names. With this combination a release name would be something like “awesome_adjust_app_0.0.1+master-01b4601.release.tar.gz”.
Custom environments
By default edeliver
provides only two environments to which it’s possible to deploy — staging and production. There is no easy way to add custom environments, but as it turned out it’s still possible to achieve that by overriding STAGING_HOSTS
and STAGING_USER
variables in .deliver/config
.
Let’s say we want to add beta
and qa
environments. To do so .deliver/config
should look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
As you can see, the ENVNAME_NODES
variables should be added and then based on $DEPLOY_ENVIRONMENT
, staging related variables should be overridden.
Also, it’s important to add the .deliver/help
file where these new environments should be added:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
With this config, it would be possible to deploy a release to the beta
and qa
hosts (in addition to staging
and production
) and to maintain these custom hosts. For example, in order to check the version of the beta
host, you’d run a command like this: mix edeliver version beta
.
Deploy notifications
It’s quite common to send notifications about successful deployments. For example, we might display such notifications in a Slack channel. edeliver
has hooks which can be implemented as bash functions. For example, there are two hook functions: pre_upgrade_release()
and post_upgrade_release()
. They are called exactly before applying an upgrade
and right after an upgrade
has been applied, respectively. Notifications about deployment usually contain information about the person who deployed, the Git branch and revision, and the environment name (staging/production).
The issue here is that you can’t get a Git branch and Git revision out of a release since a release is just a binary. With Capistrano, you can just run a couple of git commands on the target host to get the necessary data. With edeliver
it becomes a bit more tricky. The current workaround we use is to include the Git revision and Git branch into a release name using the following config: AUTO_VERSION=git-branch+git-revision
. This is as I described in the previous section on Auto-Versioning. Then in the project itself a Notifier
module might look as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
Then the pre_upgrade_release()
and post_upgrade_release()
hooks might look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
However, there are two flaws here. First, it works only when applying upgrades - not for releases. And second, when calling Elixir.MyApp.Notifier
from pre_upgrade_release
, Edeliver.release_version
returns a git revision of the currently deployed release. So ‘deploying’ notification would have a git revision of the currently deployed version and the ‘deployed’ notification would have a git revision of the new version.
Different configurations on different deploy hosts
Most probably, your application has different settings for staging and production environments. Which means that you need either to build a release for each environment separately or somehow provide different settings on different hosts for the same release. Edeliver
, following a philosophy “build once, deploy everywhere” suggests to solve this problem by using LINK_SYS_CONFIG
or LINK_VM_ARGS
config variables as described on this wiki page.
I’ll describe briefly how it works with LINK_VM_ARGS
variable. The logic is the same for LINK_SYS_CONFIG
. So it works as follows: you need to create a file which should have the same path on both staging
and production
hosts with config values specific for the target host. This could be /home/deploy_user/my_app/vm.args
, for example. Then in .deliver/config
you can specify LINK_VM_ARGS=/home/deploy_user/my_app/vm.args
.
When making a release or an upgrade, edeliver
would put a symlink inside a release (instead of the real generated vm.args
) which will point to /home/deploy_user/my_app/vm.args
. So this tricky and sophisticated approach solves the issue. In theory. I couldn’t actually make it work. After a release deployment I see a symlink as expected, but on release start, my custom symlinked vm.args
file should replace vm.args
from running-config
which does not happen. However, if I remove the running-config
folder first and start a release afterwards, it works.
So since this approach didn’t fully work, we decided to build a release per environment, which is also suboptimal:
- you need to build a release per environment
- it violates a release philosophy: build once, deploy everywhere
- error prone: somebody can by mistake deploy a release on production, which has been built for staging
To partially fix the last bullet from the list above it’s possible to add a mix-env
parameter to the AUTO_VERSION
config value: AUTO_VERSION=git-branch+git-revision+mix-env
. So every build would have -environment
in its name to indicate for which environment a release has been built.
Usually, for Phoenix applications secret production settings (like database connection credentials for production DB) are stored in prod.secret.exs
. This file is not under version control, but it should be inside a release. To achieve that you might want to put this file manually into the build host, but the issue here is that a folder where a project is built is cleaned by edeliver
before every release build. The ‘cleaning’ means that everything which is not under version control will be removed before every build, so config/prod.secret.exs
will be gone. To avoid that there is an option to explicitly instruct edeliver
which folders should be cleaned. Having the config option GIT_CLEAN_PATHS="_build rel deps"
tells edeliver
to clean _build
, rel
and deps
folders before every release build, so config
folder stays untouched and therefore prod.secret.exs
stays alive between release builds.
Bonus: Change font color output
For light terminal themes edeliver
output by default looks as follows:
There is an option to change that by overriding the color of the font:
1 2 3 |
|
With the fix the output looks as follows:
Alternatives
Currently, there are not so many alternatives to edeliver
. But there is at least one: dicon. It’s in the early stages of development, it doesn’t have comprehensive readme, it’s not aware of build host and it does not support hot-upgrades yet. However, Digital Conveyor
has some niceties: it’s written completely in Elixir, it’s small and it supports configurations per target host out of the box. It will be interesting to see how dicon
will be evolving.
Conclusion
Edeliver
is a great, ready-to-use deployment tool packed with a lot of useful features. It works with releases, supports hot-upgrades and build host concept, has very good documentation and gives you simple commands to automate deployment routines. Importantly, the project is in active development. I’d like to thank bharendt for amazing support, almost every tip or trick I’ve described in the post is a result of a detailed answer from him to an opened issue. Sometimes I had a feeling that I’m literally chatting with him in the Issues
tab, that’s amazing.
That’s it for today. Happy deploying!