GitLab SSH

Over the weekend I migrated my home server from an aging Mac mini with far too little RAM to a new server in a handmade wooden case.

Arguably, the bigger part of the move involved migrating all of the various services to Docker to help make management & backups easier. I settled on using nginx-proxy to handle the reverse proxy and SSL for all of the containers, and for the most part everything went smoothly with one big exception: SSH for GitLab.

The whole point of Docker is to lock everything down as much as possible. Most of the containers¹ don’t have ports bound to the system and are only accessibly by their own Docker network. This includes port 22, which is an important one.

There are a few obvious options, all with problems:

• Bind port 22 of the host to the container

sshd is on the host is already bound to port 22, so I’d either be able to SSH into the host or the container, but not both.

• Have the container use the host’s network

This defeats the purpose of running GitLab in a container in the first place and it means nginx-proxy can’t connect.

• Run GitLab’s SSH over a different port

I’d need to open another port in my firewall and all of my existing clones would need to be updated.

The Solution

Figuring other people had to have run into this problem before I did some searching around and finally found an article by someone who had the same list requirements I did.

The author did some excellent sleuthing to figure out that you can set up a git user on the host to forward the SSH requests the to git user in the container. The guide is fairly straight-forward and explains the reasoning behind each step, however there were a a couple of small problems I ran into.

Authentication Refused

The most perplexing issue was this line in /var/log/auth.log:

Authentication refused: bad ownership or modes for directory /srv

I’ve seen the error before, but for the .ssh directory or specific files, but never on a top-level directory.

It turns out sshd is resolving the symlink and traveling up the hierarchy to the root of the drive, which can’t have such limited permissions.

After a bit of trial and error the the solution I came up with is to hard link the files in .ssh so sshd is happy with their paths:

ln /srv/gitlab/data/.ssh/* /home/git/.ssh/

This allows both the GitLab container to update the contents as keys are added or removed while keeping the paranoid sshd happy.