Honing the craft.

Dup-composer automated releasing with GitHub Actions is round' the corner

Last week, I have closed some pretty lucrative issues on my list, although I fell short on closing them all. In the meantime I have done the research on GitHub Actions and the initial release workflow should be ready by the end of this week. In this post, I share some details about two small features that got introduced this week: the config change safeguard and the dependency check for Duplicity.

The config change safeguard

Although YAML is an easily readable structured data format, hence my choice of language for Dup-composer configuration files, sometimes it is too easy to make a mistake when writing YAML, especially when changing existing configuration. You have to pay attention to indentation and some features of the syntax are rather subtle.

Hence, even though Duplicity has pretty good protection in place against doing anything harmful with your existing backup chains, I wanted to put in some form of rudimentary safeguard for Dup-composer as well, so that the user is aware of any changes to existing backup groups. This doesn't include changing the group name, as accurately detecting changes of names and group configuration in the meantime would be too much of an effort for this stage of development. We still have many other lower hanging fruits we can go for.

To keep things simple, all the implemented safeguard does is caching the configuration state for each run, so that the current state of all the backup groups' configuration can be cross checked with the previous run. If there is a difference, the user is informed about the changed groups and the execution is aborted. Once the user has double checked the configuration, and found the changes were desired and in order, it can re-run the tool with the -s option, that skips the safeguard.

Implementing this feature was a piece of cake. From a testing perspective however, things got a bit tricky, as the state of the last run had to be cached in a file, which has to be cleaned up after each test run, except for the tests of the safeguard functionality. We want to leave the files untouched for that, so this required some isolation, including moving the safeguard test fixture files in a separate subdirectory.

The Duplicity dependency check

Since the Duplicity package is not maintained on PyPI, I could not add it as a dependency to setup.py, so that it automatically installs when Dup-composer does. The user has to install Duplicity manually as a prerequisite.

As there is no guarantee, that the right version, or even any version of Duplicity is installed before the dupcomp command is executed, I wanted to add a verification step to make sure, that the duplicity script is:

• on the PATH.
• can be run with normal (0) exit code with the --version option.
• of the correct (0.7+) version.

To test these scenarios, I have created mock implementations of the Duplicity script, that emulate:

• a Duplicity version that is too old.
• a non-zero return code when reading the version is attempted.

Releasing to PyPI with GitHub Actions

I have looked into GitHub Actions to automate the release of Dup-composer to PyPI. The workflow to implement this is fortunately pretty straightforward if you follow the Python Packaging Guide. This covers about 80% of what I need, so this should be implemented this week.

But what is GitHub Actions? In a nutshell, you can define workflows right in your GitHub repository, that describe how you want to automate the testing, integration, build, packaging and release of your project. The workflow is described in a YAML configuration file that is divided into jobs, which are built up from steps. These steps can either be as simple as running a command in the shell, or be a more complex action, expressed in JavaScript code. GitHub provides various JS toolkit libraries, that give you access to many GitHub context goodies. You can build your own custom actions if need be, but most of the commonly imaginable tasks are covered by community released actions that you can plug right into your workflow. The coolest thing is, that you can run this whole process on GitHub's infrastructure, automatically triggered by your specifically tagged commits pushed.

The documentation on GitHub Actions is vast and pretty good, although I felt I have to jump back and forth between documents to follow the information logically. The selection of actions in the GitHub Marketplace is phenomenal. Even if you don't find a canned action for your needs, you will probably find something, that will be a good starting point for a modified version suiting your workflow.

This week

• There are still five open issues, that I want to close this week.
• Testing backup and restore in production will continue.
• The GitHub Actions workflow for automated PyPI releases will be implemented.

If you give Dup-composer a try and you run into any problems, please open an issue in the Dup-composer GitHub repository.