Copy docker container data volume to another host

The article outlines a method to transfer Docker container data volumes between hosts.

Posted on Dec. 5, 2019 in docker.

Recently I had to migrate different docker based projects from the old machine to the new one. The easiest thing that came to my mind was to install the same version of Docker on the new machine and copy the whole /var/lib/docker/ docker directory. This did not work.

Projects that were needed to be copied were different web applications (mostly Django and 2 php projects). All of them shared a property that they were defined in a docker-compose file, and as such were easily rebuilt on the new machine by copying directories to the new machine and running docker-compose up --no-start for each of them. Except for the data.

Idea is to write a command that would automate the copying of data inside docker containers. Nice thing is that the container name is predictive (see note).

Here is the command that will copy data to a container on the remote machine.

Command

(CONTAINER=myproject_db_1 REMOTE_HOST=newhost DIR=/var/lib/mysql; \
    docker run --rm \
    --volumes-from "$(docker inspect --format={{.Id}} $CONTAINER)" \
    busybox tar -czv --to-stdout $DIR  \
    | ssh $REMOTE_HOST docker run --rm -i \
    --volumes-from "\$(docker inspect --format={{.Id}} $CONTAINER)"\
    busybox tar -xzv \
    )

This command can be divided in three parts: set variables, create archive, upload and expand archive. Bellow are explanations for parts of the command so it can be understood and adapted for different needs.

1. Set variables

CONTAINER=myproject_db_1 REMOTE_HOST=newhost DIR=/var/lib/mysql;

CONTAINER - container name that we want to copy data from

REMOTE_HOST - remote host we are copying data to

DIR - location of data inside of container, ie: for MySql it is /var/lib/mysql, and for postgres it is /var/lib/postgresql/data

2. Create archive to standard output

docker run - rm \
 --volumes-from "$(docker ps -aqf "name=$CONTAINER")" \
ubuntu tar -czv --to-stdout $DIR

run - Run a command in a new container

--rm - Automatically remove the container when it exits

--volumes-from - Mount volumes from the specified container

docker inspect --format={{.Id}} $CONTAINER - get container id by its name

busybox - we use BusyBox docker image to run command (see Notes)

tar -czv --to-stdout $DIR - (c)reate g(z)ipped (v)erbose archive of data directory to standard output

3. ssh to target host and extract files in destination container volume

| ssh $REMOTE_HOST docker run --rm -i \
    --volumes-from "\$(docker inspect --format={{.Id}} $CONTAINER)"\
    busybox tar -xzv

| - pipe output of previous command

ssh $REMOTE_HOST - connect to remote host

docker run --rm - run docker on remote host, removing the container when it exits

-i - keep STDIN open even if not attached

--volumes-from "\$(docker inspect --format={{.Id}} $CONTAINER)" - mount volume from remote container. Notice that dollar sign is escaped, so command substitution occurs on the remote machine and not on the local machine.

busybox tar -xzv - e(x)tract g(z)ip archive (v)erbose.

Notes:

  • everything is wrapped inside parentheses making variables visible only inside the subshell,
  • BusyBox is used because it is lightweight, it could be replaced with ubuntu or some other distribution
  • docker-compose 1.23.0 introduced a change that appended random strings to container names created by docker-compose up (that was reverted in 1.23.2
Share on Reddit