April 22, 2016

Migrating your Gitlab Infrastructure into Docker

by Pedro Fortuna

Migrating your Gitlab Infrastructure into Docker

To see how quickly you can annoy your developers you just need to bring your Gitlab down.

If you did that already, you know how sensitive your build environment is and you should do everything you can to make it healthy and stable. Unfortunately, that sometimes means that you might think twice before deciding to upgrade your Gitlab configuration, or before upgrading your CI server.

However, when using Docker those troubles basically go away. In this article we are going to show you step by step how can you migrate your current Gitlab infrastructure into Docker.

Docker 101

If you have been around in the last couple of years you probably already heard of Docker. Docker is basically a Linux Container management toolset. Linux Containers provide resource limitation and prioritization (CPU, memory, block I/O, network) and application isolation in terms of the execution environment (filesystem, process trees, networking, user IDs). This has obvious benefits in terms of security, but an equally important benefit is making the management of your CI infrastructure much more enjoyable and less painful.

Docker containers are not persistent because they are based on images that are immutable. If you restart a container, you’ll restart all the processes running inside it and all changes made in previous executions will be lost because the container is created from the same original image.
To persist data you need to separate the state (configuration files you want to be able to edit frequently, databases, logs, etc) from the rest (system and static application files). The static parts should live inside the docker container and the state should be moved out to host volume folders that you can mount into your containers. This way, your state will be safe across multiple executions of the container.

Creating the initial state

Assuming that you already installed Docker in your server, the first step is selecting a Docker image that suits our needs. It’s always best to choose an official image as those are well structured and tend to be more up to date. For this post we selected the Gitlab Community Edition Docker image, which is based on the Omnibus package.

Before running the container we need to create the host volumes and set up the default configuration files that we’ll need in the first execution. There are three host volume folders that we will use:

Host Volume Container location Usage
/docker/gitlab/data /var/opt/gitlab Application data
/docker/gitlab/data /var/log/gitlab Logs
/docker/gitlab/data /etc/gitlab GitLab configuration files

To set our default configuration first create Gitlab’s configuration file /docker/gitlab/gitlab.rb. Let’s start with something simple:

# Change the external_url to the address your users will type in their browser
external_url 'https://gitlab.yourcompany.com'  
nginx['listen_port'] = 443  
nginx['listen_https'] = true  
# SMTP settings
gitlab_rails['smtp_enable'] = true  
gitlab_rails['smtp_address'] = "smtp.yourcompany.com"  
gitlab_rails['smtp_port'] = 587  
gitlab_rails['smtp_user_name'] = "gitlab"  
gitlab_rails['smtp_password'] = "supersecurepassword"  
gitlab_rails['smtp_domain'] = "yourcompany.com"  
gitlab_rails['smtp_authentication'] = "login"  
gitlab_rails['smtp_enable_starttls_auto'] = true  
gitlab_rails['gitlab_email_from'] = '[email protected]'  

For security purposes, we should enable SSL by default. This gitlab image will look for the ssl certificate in the /ssl/gitlab.yourcompany.com.crt.
Generate your own self-signed certificate or even better, purchase a certificate for your gitlab sub-domain and install the files in:

/docker/gitlab/ssl/gitlab.yourdomain.com.crt
/docker/gitlab/ssl/gitlab.yourdomain.com.key

Don’t forget to do a chmod 600 /docker/gitlab/ssl/gitlab.yourdomain.com.*. The easiest way to break your crypto is to give your key away!
  

Creating the container

We don’t need to create a Dockerfile to setup our container, but it’s best if we do! We can later tweak things to our taste and it’s easier to remember exactly what settings we used and a convenient starting point to upgrade our configuration. If you are migrating your current non-Docker installation of Gitlab into Docker you’ll need to install the same version. Don’t worry, you can upgrade later. To specify a version you need to add an existing tag to the image name. In our case we are using 8.5.7-ce.0.

We’ll use supervisord to wrap our service. Supervisor is basically a daemon wrapper that can automatically restart your services if for some reason they become down. This is the look of our Dockerfile:

FROM gitlab / gitlab - ce: 8.5.7 - ce.0  
MAINTAINER [email protected] yourdomain.com

## Setup##
RUN apt - get update &&  
    apt - get install - y supervisor &&
    mkdir /
    var / log / supervisor
COPY. / supervisor.conf / etc / supervisor / conf.d /

    EXPOSE 22 443
VOLUME['/etc/gitlab', '/var/log/gitlab', '/var/opt/gitlab']  
CMD["/usr/bin/supervisord"]  

The FROM instruction specifies which image we are using as starting ground to build our own. RUN instructions indicate commands that will be executed on the container filesystem when building the image. You can have multiple RUN instructions or a single one separating commands with &&. It’s best to merge into a single one as each instruction will result in a new layer that will patch the previous one. More layers will be slightly slower in terms of I/O latency. The COPY instruction copies the file from the build directory to a directory inside the container. The EXPOSE instruction sets the ports that will be accessible outside the container.

The VOLUME instruction indicates which directories can be mounted as host or data volumes on the container. The CMD tells Docker which binary to execute when the container starts.
In this case we’ll tell it to execute the supervisor daemon, which then starts and monitors Gitlab execution. The base image that we are using executes as foreground /assets/wrapper. That’s what we need to set on our supervisor.conf file:

[supervisord]
nodaemon = true

    [program: gitlab]
command = /assets/wrapper  
numprocs = 1  
autostart = true  
autorestart = true  

The nodaemon instructs supervisord to execute in foreground, and the second block tells supervisor to execute gitlab’s boot script. autostart will tell it to start automatically when you run the container, and autorestart instruct supervisor to restart gitlab automatically if it comes down for some reason.

So right now you should have the following files in your build directory:

  • Dockerfile
  • supervisor.conf

To build your image, execute this command in the build directory:

docker build - t yourcompany / gitlab - ce: 8.5.7 - ce.0.  

Running the container

To launch containers, we recommend that you use docker-compose and write down a docker-compose.yml configuration file. Again it’s a better way to manage it as later on chances are that you won’t remember the settings you used to start your containing.
You can think of the Dockerfile as the place where you should describe the building blocks of your container and the docker-compose.yml as the place where you describe how a particular instance of the container should be executed and, if need be, where you override some configuration parameters.

gitlab:  
    image: 'yourcompany/gitlab-ce:8.5.7-ce.0'
restart: always  
hostname: 'gitlab.yourcompany.com'  
environment:  
    -GITLAB_OMNIBUS_CONFIG: |
    external_url 'https://gitlab.yourcompany.com'#
Add any other gitlab.rb configuration options  
ports:  
    -'443:443' - '22:22'
volumes:  
    -'/docker/gitlab/conf:/etc/gitlab' - '/docker/gitlab/logs:/var/log/gitlab' - '/docker/gitlab/data:/var/opt/gitlab'

It’s very easy to understand what these options do, but you can always peek the docker-compose.yml reference to learn more.

You don’t need to pass Gitlab variables using this file, but it is a convenient way to override your default configuration if you are deploying multiple instance of this Docker image.

To run our container: docker-compose up -d

The first time it runs it will take some time to create all state like databases, etc.

If you are setting up Gitlab for the first time in Docker, this is it. Create your first user and start adding your mates. You can jump the next section as well.

Importing existing Gitlab data

This step is really easy as Gitlab already has a tool to export/import all data in bulk.

In your existing gitlab execute the steps below. It will only work if your previous setup is an Omnibus Package installation. If you installed from source see here. Make sure your Gitlab is running when you do this.

sudo gitlab - rake gitlab: backup: create  

It can take some time depending on how many projects you have.
In the end it will write a file with a name like <unixtimestamp>_gitlab_backup.tar e.g. 1460911132.

To restore move this backup file into /docker/gitlab/data/backups.
Then, enter the container by typing:

docker exec - it `docker ps | grep "yourcompany/gitlab-ce:8.5.7-ce.0" | cut -d " " -f1`  
bash  

Now enter these commands:

# stop services
gitlab-ctl stop unicorn  
gitlab-ctl stop sidekiq  
cd /var/opt/gitlab/backups  
chown git:git *_gitlab_backup.tar  
# Overwrite the contents of your GitLab database. It may take some time to complete, depending on how big your database is.
gitlab-rake gitlab:backup:restore BACKUP=<DATE> (e.g. 1460911132)  
gitlab-ctl start  
# Check if everything is ok. Watch out for errors and warnings
gitlab-rake gitlab:check SANITIZE=true  

And that’s it! Access https://gitlab.yourcompany.com, login with your user and make sure everything is in there!

Backups

To automate backups it’s just a matter of running an instance of the docker machine

docker exec `docker ps | grep "yourcompany/gitlab-ce:8.5.7-ce.0" | cut -d " " -f1` / opt / gitlab / bin / gitlab - rake gitlab: backup: create  

Install it in the host’s cron (say everyday at 4am) using a more generic grep regex to avoid the need to change this cron job when you upgrade the version.

0 4 * * * docker exec `docker ps | grep "yourcompany/gitlab-ce:8.5.7-ce.0" | cut -d " " -f1` / opt / gitlab / bin / gitlab - rake gitlab: backup: create  

This will only backup your database, so remember to backup your configuration files as well (/docker/gitlab/conf) and the logs in (/docker/gitlab/logs), in case you want to keep them.
  

Upgrading your Gitlab

First stop your current docker instance:

docker stop `docker ps | grep "yourcompany/gitlab-ce:8.5.7-ce.0" | cut -d " " -f1`  

Delete the container:

docker rm `docker ps -a | grep "yourcompany/gitlab-ce:8.5.7-ce.0" | cut -d " " -f1`  

To upgrade you need to edit your Dockerfile and docker-compose.yml files. You only need to edit the tag, e.g. from 8.5.7-ce.0 to 8.5.8-ce.0.

Build it again:

docker build - t yourcompany / gitlab - ce: 8.5.8 - ce.0.  

And then start it:

docker - compose up - d  

If all went well, you should now have a new version up and running. Please keep in mind that new versions might break stuff sometimes. So test it before deploying it and it can’t hurt to run gitlab-rake gitlab:check SANITIZE=true to check for issues.   

Wrapping up

Migrating your Gitlab to Docker is the first step into turning your CI more manageable. You can easily move it to another server if you need and you can easily upgrade to newer versions without running the risk of breaking things. Gitlab is the core of your team’s development infrastructure and you’ll want it to run smoothly. Docker can help you achieve just that.

Of course, if you want even more stability, you need a High-availability (HA) containers setup (e.g. with HAProxy) or to setup live-migration of containers and data.Those are advanced topics that we may explore on a future blog post.

Further reading & Resources: