Honing the craft.

Dup-composer is published on PyPI

With the latest release of Dup-composer, the project's package is now published on PyPI; new features, like keyring support and SFTP backups have also been added. Read on, or just grab the latest copy with pip right now!

There have been quite a few things going on in my world lately, including starting a project with a customer (Yay!), which is always a priority, and moving with my family to a different place. As a result, the Dup-composer project had to be put on the back burner for a while, but I am back again with most of the features planned the last time I blogged finally released.

PyPI packaging

The ability to install the utility from the command line using pip is something I am really excited about. First, this is my inaugural open source project on PyPI and it is a fantastic feeling to just issue the pip install command and see all the dependencies pulled in and your software be installed within a few seconds. Second, if I want to find out if some people find this tool useful, I have to spread the word in places where Duplicity users are around. But having to install Python software and its dependencies manually might be a hurdle for most to try it out. A new user doesn't want to invest a lot of time in trying out a tool that might or might not fit her needs. But if you can install with a single command, the user might give a shot. That's just my logic anyways.

You can install Dup-composer with:

pip3 install dup-composer

The steps to get Dup-composer ready for packaging turned out to be a lot less complicated than I have expected. Briefly:

• Create the package's setup.py install script.
• Move my command script used for launching the utility into the dupcomposer package and have setuptools generate the entry point script upon installation.
• A few of the imports had to be moved around to accommodate this new structure.
• Update functional tests to call the entry point script at the new location.
• Setup an account with the test PyPI repository as well the production one.
• Install the twine package from PyPI
• Publish to the test PyPI and see if things go well, it gets published, and you can install the package with pip.
• Publish to the production PyPI.

This Real Python article has been a great guide in the process.

Since Duplicity is the main dependency of Dup-composer, at first, I have included this dependency in the setup.py script. Sadly, Duplicity's PyPI repository has only a single release published in 2017., so it looks like it is not maintained. Hence I had to remove this dependency. I don't think that this will be a huge hurdle for most really, as the majority of users who will want to try out Dup-composer are assumably working with Duplicity already. I will add some form of Duplicity dependency check to Dup-composer*, but that is all I am planning to do about this for the time being.

SFTP support

Implementing SFTP support was very easy, considering, that the Duplicity command line and environment API is essentially identical with that of SCP. From the perspective of configuring Dup-composer, the only difference is in the URL: just replace scp:// with sftp://, make sure, that SFTP is allowed and configured on the remote host and you are good to go.

Keyring support

This was the biggest chunk of work and it involved a big refactor of the otherwise small keyring interface code and multiple refactors of the unit tests. The interface was originally written to handle a single keyring configuration for the whole configuration file, but then I realized, that this is silly as the backup groups were exactly created, so that the configuration for multiple environments can be included. Consequently, the code had to be refactored to have each backup group have its own keyring interface. Each one of these discrete keyring configurations generate their own context and make their own changes to the runtime environment, thus, their effects had to be isolated.

Supported backends

Dup-composer uses the keyring Python library with the SecretStorage backend that supports the following implementations:

• GNOME Keyring (version 2.30+)
• KSecretsService (KDE keyring)

Other keyring backends, like the macOS Keychain will be supported later, if there is demand for them.

Configuring keyring originated credentials

Currently, there are three different credentials in the Dup-composer configuration, that can be read from a keyring:

• GPG passphrases
• SCP and SFTP passwords
• AWS secret keys

To have Dup-composer read a credential from the keyring, you just have to provide a list containing, the keyring service and account names. This will go into the configuration where you would normally provide the passphrase, password, secret key. For instance, to read the password stored under the backupserver1 service and backup account, you would provide:

password: ['backupserver1', 'backup']

This assumes, that the entry has already been added to the keyring and that the keyring is unlocked when Dup-composer is started.

Using the keyring integration in a desktop session

In the GNOME or KDE desktop environment DBUS is automatically started, the keychain is unlocked when you log in the session. Hence using the keyring in a desktop environment doesn't require additional steps, other than adding the credentials to the keyring and then specifying the keyring service and account name in the Dup-composer configuration.

Using the keyring integration on a headless server

In case you have no desktop environment installed, on a server for example, you have to install the keyring software and start the DBUS session manually, then run Dup-composer inside that session. This is how this would look like with GNOME Keyring on Red Hat Enterprise Linux and CentOS 8:

Install the keyring:

# yum install gnome-keyring

Start the DBUS session:

# dbus-run-session -- sh

Start the GNOME Keyring:

gnome-keyring-daemon --unlock

At this point, you can run Dup-composer and it should be able to read the credentials added to the keyring beforehand. Please check the Dup-composer keyring documentation for a more detailed explanation.

Using the keyring integration for unattended backup

So if the keyring is only unlocked when the user logs into the desktop environment, or if the user starts and unlocks the keyring manually, how to handle scheduled backups, where the user is not logged in? My solution to this problem that is probably the easiest to use but a bit hard to set up was to configure gnome-keyring-daemon as a systemd service, that is started on boot and automatically unlocks the keyring for the user in question.

This allows Dup-composer to access the credentials when needed:

• as long as it is running as the user owning the keyring,
• or as a superuser, that has the ability to change the effective user id, to the keyring owner's id and these details has been included in the Dup-composer configuration (see below).

The keyring documentation includes a step by step guide on setting up the systemd unit to make this work, please refer to that document for help and feel free to open up a GitHub issue if the information you find there has to be corrected or extended in your opinion.

To accomodate more special scenarios with a dedicated Dup-composer keyring, it is possible to specify the keyring's owner user and DBUS socket address directly in the Dup-composer configuration. This will cause the utility to communicate with the keyring as the specified user on the specified socket address. For example:

backup_groups:
   backup_server:
    keyring:
       username: keyringuser
       bus_address: /run/user/1002/bus
     backup_provider:
       url: sftp://backupuser@backuphost.example.com/
       password: ['backuphost', 'backup']
     ...

This will tell Dup-composer to do the following when reading the credentials ['backuphost', 'backup'] from the keyring:

• Change the effective user id (EUID) to the user id of keyringuser.
• Communicate with the keyring (DBUS) over the socket /run/user/1002/bus.

Plans for the week

• Work through my GitHub issue list and close all of the ones I have opened until now.
• Extend the documentation, as there are some missing facets that need to be included about the new features.
• Try Dup-composer in production with a few of my less critical regular backups.
• Look into GitHub Actions to automate the release process.

That's it for this week's post. If you use Duplicity and maybe even Duply at the moment, please install Dup-composer, try it out and let me know if you encounter any problems or you miss any features by opening a GitHub issue.