In the last post, we had a look at how to set up a private Docker registry. Setting up a Docker registry requires some steps:
- Install the Docker software itself. In turn, that requires to add the official Docker repository to the list of allowed system repositories.
- Run the Docker registry image. That is quite straightforward.
- Check that everything works fine, by pushing images to and pulling from the newly installed repository.
We left considering how the the setup was incomplete and unacceptable, as it leaves a lot of security holes.
Let’s see how to harden our private docker registry and take full advantage of it!
There are two main issues with the current situation:
- There’s no guarantee the registry we will push to is the correct one: this is less likely to happen with plain IP, but imagine that the DNS resolving the registry domain name is compromised, pointing to a registry under the control of a third-party. You could actually push images to it, and the people behind could have access to your proprietary code. TLS is required to fix this, and will have the nice side-effect of preventing Man-In-The-Middle attacks.
- Anybody who knows the registry address can access it and can also push images to it. This can be fixed by adding an authentication layer.
Adding TLS to the Docker registry
In order to add TLS on top of the registry, a valid certificate is necessary, either a self-signed certificate or a certificate signed by a trusted authority. We will use Let’s Encrypt, which is a trusted CA and has the advantage of being free of charge.
Let’s Encrypt’s certificates can be obtained in different ways depending on how and where you need to use them. We will use Certbot to obtain a standalone copy of the certificates to use with our registry.
Packages are available depending on the exact Ubuntu version, but a Certbot Docker image is also available. Let’s take advantage of that:
$ sudo docker run -it --rm --name certbot -p 80:80 \
-v "/etc/letsencrypt:/etc/letsencrypt" \
-v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
certbot/certbot certonly
The command will start an interactive program to configure the certificate:
How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c' to cancel): docker.exoscale.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for docker.exoscale.com
Waiting for verification...
Cleaning up challenges
A couple of things are worth noting here:
- Certbot cannot deliver certificates to bare IPs.
It requires a full-fledged domain name e.g.
docker.exoscale.com. - To deliver the certificate, Certbot needs the port 80 to be open. On Exoscale you can take advantage of security groups for that, and open the firewall for Certbot to verify your website.
- Because Docker Certbot is executed with
sudo, the owner of the created files/folder will beroot. This is not an issue because the Docker Registry is also executed withsudo, but if the current user is added to thedockergroup as some documentation proposes, ownership needs to be changed accordingly. - Let’s Encrypt certificates expire after 90 days. You have to ensure their renewal, and ideally automate the process.
Once installed, your certificates can be found in the /etc/letsencrypt/live/ folder.
$ sudo ls -l /etc/letsencrypt/live/docker.exoscale.com
total 4
-rw-r--r-- 1 root root 682 Sep 12 12:36 README
lrwxrwxrwx 1 root root 48 Sep 12 12:36 cert.pem -> ../../archive/docker.exoscale.com/cert1.pem
lrwxrwxrwx 1 root root 49 Sep 12 12:36 chain.pem -> ../../archive/docker.exoscale.com/chain1.pem
lrwxrwxrwx 1 root root 53 Sep 12 12:36 fullchain.pem -> ../../archive/docker.exoscale.com/fullchain1.pem
lrwxrwxrwx 1 root root 51 Sep 12 12:36 privkey.pem -> ../../archive/docker.exoscale.com/privkey1.pem
It’s now possible to start the secured version of the registry:
$ sudo docker run -d --name registry --restart=always \
-p 443:5000 -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/letsencrypt/live/docker.exoscale.com/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/etc/letsencrypt/live/docker.exoscale.com/privkey.pem \
-v /etc/letsencrypt:/etc/letsencrypt \
-v /var/lib/docker/registry:/var/lib/registry \
registry:2
Let’s check that it works:
$ curl -X GET https://docker.exoscale.com/v2/_catalog
{"repositories":["hello-world"]}
Great! Our private docker registry is now protected by TLS, meaning that all communication is encrypted and we have the guarantee of talking with the correct registry!
Setting up basic authentication for the private registry
Now that our communications with the registry are secured, it’s time to let only authorized users access it.
Out-of-the-box, Docker registry allows a single authentication option:
file-based login/password matches with the htpasswd command.
More advanced setups require a web server proxy (e.g. Nginx, Apache, etc.) in front of the registry, offering a lot more options, including integration of already existing authentication datastores (e.g. LDAP, etc.).
Here we’ll focus on the simplest option.
To create an entry into a new .htpasswd file in the home directory:
$ htpasswd -cB ~/.htpasswd nicolas
Then, to add an entry into an existing file:
$ htpasswd -B ~/.htpasswd david
Now is time to launch the authentication-enabled registry:
sudo docker run -d --name registry --restart=always \
-p 443:5000 -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/letsencrypt/live/docker.exoscale.com/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/etc/letsencrypt/live/docker.exoscale.com/privkey.pem \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Docker Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/htpasswd \
-v /etc/letsencrypt:/etc/letsencrypt \
-v /var/lib/docker/registry:/var/lib/registry \
-v ~/.htpasswd:/htpasswd \
registry:2
Let’s try to pull with no prior authentication:
$ docker pull docker.exoscale.com/hello-world
Docker returns accordingly:
Using default tag: latest
Error response from daemon:
Get https://docker.exoscale.com/v2/hello-world/manifests/latest:
no basic auth credentials
Let’s authenticate using the credentials provided during .htpasswd setup:
$ docker login docker.exoscale.com
Username: nicolas
Password:
Login Succeeded
Yay! The registry is now secured! Only people you gave access to can push and pull from it, granting full privacy and security of your images.
If a message similar to WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json is output, that means there’s no encryption configuration on the client side.
To store your credentials on the local machine in a more secure way, please check the relevant documentation.
And now:
$ docker pull docker.exoscale.com/hello-world
Using default tag: latest
latest: Pulling from hello-world
d1725b59e92d: Pull complete
Digest: sha256:1a6fd470b9ce10849be79e99529a88371dff60c60aab424c077007f6979b4812
Status: Downloaded newer image for docker.exoscale.com/hello-world:latest
Toward more industrialization
While everything works as intended, the number of instructions to type in order to get there is quite long. A more robust setup would involve putting all those parameters into a dedicated configuration file, in order to store it in a version control system.
Docker offers this possibility by letting you mount an external configuration file for the registry as a volume. Starting from the official Docker configuration template, the result is the following:
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
tls:
certificate: /etc/letsencrypt/live/docker.exoscale.com/fullchain.pem
key: /etc/letsencrypt/live/docker.exoscale.com/privkey.pem
auth:
htpasswd:
realm: "Docker Registry Realm"
path: /htpasswd
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
And the associated launch command:
sudo docker run -d --name registry --restart=always \
-p 443:5000 \
-v `pwd`/config.yml:/etc/docker/registry/config.yml \
-v /etc/letsencrypt:/etc/letsencrypt \
-v /var/lib/docker/registry:/var/lib/registry \
-v ~/.htpasswd:/htpasswd \
registry:2
Going further
We now have a full-fledged secured Docker registry, with all the included advantages:
- You can use images to build artifacts for proprietary code knowing where the code is stored and ensure who has access to it
- You have fine-grained access control of your images, that can be further improved by integration with your corporate AD/LDAP and SSO
- Improved latency, keeping your images close to your application
- Compliance to regulations demanding full control over data, e.g.: PCI
Although the registry is now secured and can safely be used for production purposes, in a real world deployment scenario some more steps would be required to make this setup convenient from an operational point of view, e.g.:
- Configure a web server proxy like nginx in front of the registry, to take care of authentication and take advantage of an existing identity store
- Auto-renewal of the certificates, obtainable by installing Certbot instead of using a Docker image
- Apply an Infrastructure-as-Code approach to the above, and manage it through relevant tools e.g. Terraform, Packer, Ansible, Puppet, Chef, etc.
Also is worth nothing that the storage of the actual images is decoupled from the registry itself: you may for example have multiple registries, close to your consumers, and a centralized image store, as detailed in one of our previous articles: Private Docker registry with Exoscale object storage.
A setup of this kind would be surely interesting for a Jenkins server, keeping the load low on the virtual machine and reading directly from the Object Storage instead.
