Honing the craft.

Migrating ownCloud CE to Docker

As part of my effort to move all my production services and development projects deployed on my home microserver into Docker containers, I have started to migrate the ownCloud Community Edition I am running for my personal data into a new container based installation. I have encountered some problems and the migration required some trial and error, hence, to facilitate the iteration, I have written a shell script to handle most of the standard migration steps. This post explains, what the script does and how the migration process looks like using it, while also giving a bit of a background.

Migration

Is this really for you?

To clarify a bit more on the target group of this script, this was created and used to migrate my absolutely standard installation of ownCloud Community Edition 10.1. ownCloud was installed from the official repository for Debian Jessie and nothing was really changed from the default configuration apart from doing some hardening work based on the ownCloud documentation. The configuration of the target ownCloud Docker stack is also almost exactly the same as the example configuration shared in the ownCloud docs, docker-compose.yml just has been customized, to allow it to run on Docker Swarm properly. If you are using ownCloud Enterprise, or using anything different from a single server (i.e. cluster) setup, you definitely won't be able to use this script as-is. You are encouraged to review the script source however, and to customize it to your needs.

What can the migration script do for you?

Once the prerequisites are completed and you execute the script, you will get a few basic questions about the location of your current deployment, the port number you wish to use and the domain name you will use to access the ownCloud service. You can also override the default stack name owncloudd if you want. After the information is provided, the script uses this and additional information pulled from the existing ownCloud deployment, to build a similar configuration for the official Docker image, deploy it on Docker Swarm and migrate your ownCloud database and files data. You will end up with a new ownCloud environment running on Docker Swarm with all your original data living in it.

Prerequisites

The official ownCloud Docker image is rather opinionated about a few things; this and the aim to keep things simple have mandated a few prerequisites, that have to be fulfilled in order to make the migration possible with the script.

• Your current ownCloud deployment has to use MySQL or MariaDB as its database backend.
• Your ownCloud files data has to be located in the /mnt/data/files directory. More on this later, including how the data can be moved to this location.
• As a precaution, create a manual backup of your current system:
  • Create a backup of the data and config directories of your current ownCloud deployment.
  • Create a backup of the ownCloud database.
• Your single node Docker Swarm has to be initialized on the local (ownCloud) host.

You can get the script from my GitHub project. You can also find a more detailed explanation of how you can use it in the project README.

The migration process

Once you have verified, that the prerequisites for running the migration are fulfilled and you are ready to start the migration process, you can expect the following.

First of all, the script will require you to provide information and make choices, that will govern the migration process. The migration script doesn't touch your current deployment, it will just deploy ownCloud into Docker using the official image and migrate your data into the appropriate Docker volumes. The two deployments can co-reside on the same host without interference - unless you want to use the same port for the new ownCloud installation, in which case, you need to stop the current (non-containerized) production service at least temporarily. Running the two in parallel while you experiment with the migration is highly encouraged though to minimize potential downtime.

The script will request the following information when the migration is initiated:

• The absolute path of your current ownCloud root directory. (This was /var/www/owncloud in my case.)
• The port number you want to use for accessing the ownCloud service.
• The domain name you wish to use with ownCloud has to be picked. (A list is pulled from the existing configuration in config.php, but the Docker image only supports a single domain, so you need to select one if you had multiple configured.)
• The stack name can also be overridden, but that is optional.

The following will be verified by the script:

• The current database engine used is MariaDB or MySQL.
• Docker Swarm is initialized.
• config.php exists on path <USER_PROVIDED_OC_ROOT_PATH>/config/config.php
• The .ocdata marker file exists in /mnt/data/files.
• The port number picked by the user is not in use.

Once all the information has been collected and the ownCloud database is backed up, the script downloads the docker-compose.yml configuration file from my GitHub project, then exports the environment variables used in the compose file. The exports are also saved into the .env file, you can use it later, in case you need to upgrade or redeploy the stack. After this, the ownCloud stack containing an ownCloud, a MariaDB and a Redis service (container) is deployed. ownCloud needs a few minutes to install and stabilize and when that is finished, the migration continues with turning maintenance mode on.

In the next stage, the database backup is copied on the MariaDB container's data volume and the database restore is executed. Then, the files data is also copied to the ownCloud container's data volume. This is the stage, that will probably take the longest if you have lots of stuff in ownCloud.

The final stage encompasses running a database upgrade inside the ownCloud container with the occ upgrade command and some additional maintenance commands. The last step is to disable maintenance mode if all the steps were completed successfully.

In case you need to try again, don't forget to do this.

It is possible, that something will go wrong during the migration and you need to try again. This shouldn't be a problem, as long as you clean up the ownCloud Docker stack and the volumes that belong to it. That is, if you want to start with a clean slate.

Why I have created this script the first place - the background story.

When my first few attempts to migrate have failed - chiefly due to my lack of ownCloud specific knowledge - I have decided to distill the known steps into this script to facilitate the trial and error process. Many of the steps are rather repetitive, need copying data from one place to another, or, like copying the data to the volumes required lengthy commands hard to memorize. So it was easier to collect all this information into the script and gain both consistency and greater execution speed. Many of the problems encountered during this trial and error process have been completely solved in the script, however, there are a few caveats and compromises.

The problems I ran into.

The official ownCloud Docker image is quite opinionated about a few things, one of them being the path of the files data directory. With the default configuration, the ownCloud container expects to find your files data at /mnt/data/files and although it might be possible to customize this using the undocumented environment variables OWNCLOUD_VOLUME_ROOT and OWNCLOUD_VOLUME_FILES, you need to consider a few things before trying to use those. On one hand, I have never tried to override these environment variables and also, this isn't documented anywhere, so I assume changing these settings is not officially supported. On the other hand, many times, ownCloud root is located at /var/www/owncloud for regular deployments, with the files data being at /var/www/owncloud/data . However, if you mount a volume to /var/www/owncloud inside the container, it will mask the application and configuration files, that form the ownCloud deployment. Theoretically, you could mount two different volumes: one for the files data, mounted at /var/www/owncloud/data and another one for everything else, which can just be at the default path /mnt/data. But I felt that it is too risky to rely on an undocumented feature. It is also complicated and diverges too much from the default setup, which must be well tested and working fine. I can only assume, but probably this was the reason why /mnt/data is being used as default; normally there is little risk, that someone hosts their data on such a path - definitely much less, than /var/www/owncloud anyways.

After considering all what's above, I have decided to keep the default configuration, so I have copied my files data into the files directory of the data Docker volume used by the ownCloud container and of course...drum rolls: it didn't work!

Since the ownCloud container comes with its own configuration file generated from a template and populated with the environment information using gomplate, I have just used the documented environment variables and that's it, there was no need to copy the configuration file. However, since I have restored the data from a different deployment, there were severe hiccups and nothing was working as the secret and passwordsalt values were missing from the original configuration. Again, ownCloud didn't work once the migration was completed.

Running occ upgrade is required after the migration, so I have done that, but it has failed, saying that my apps - I have downloaded a bunch of pretty standard ones from the Marketplace, like Pdfviewer - are not upgradeable, as they can't be upgraded or downloaded from Marketplace. Interesting...

After the deployment on Docker Swarm was done, the script continued with the migration steps, however the container was not in the right state yet, so I encountered some issues.

As mentioned earlier, I kept things simple with my new ownCloud deployment, deviating from the example configuration only where absolutely necessary. The example docker-compose.yml file was part of the GitHub project for the Docker image, so I have decided, that the script will just download that and only do the necessary modifications needed for it to run on Docker Swarm. Once that was set up, Murphy's Law striked and the file has been removed from the project, probably to improve consistency; the example now only exists in the official documentation in one place.

Solutions, caveats

The problem with copying the files to /mnt/data/files was, that the files data path is also set in the ownCloud database, so you need to carry out additional steps to update the relevant entries there. What works is following the documentation on moving the file data location before doing the migration.

To address the issue with the secret and passwordsalt mismatch in the configuration file, I have simply added a step to the migration script to copy the information from the original configuration file to the config.php file in the container.

Regarding the failing occ upgrade command, fortunately the error message shown for the app upgrades is pretty descriptive; it gives a hint on how you can disable each app before you run the upgrade. Since all my apps were from the Marketplace - which means, that I could install them from the web interface later - I have simply disabled each and ran the database upgrade script again, which completed successfully this time.

To replace the official docker-compose.yml file, I had no other choice, but to create my own version of the file in the migration script's GitHub project, that is automatically downloaded and used during the migration process.

Feedback

Any feedback, especially constructive criticism is very welcome. You can reach me on Twitter or via email with your feedback.