Tag: Security

The GitLab Runner defaults to having read access only to the current repository. Git pushes such as tagging a commit won’t work. There are some suggestion out there, such as issue #23894, but I didn’t find anything more straight forward than what I’m writing here.

GitLab Personal Access Tokens are one way of using git pushes. However, they are tied to a user and needs changes in case the user leaves the company or simply changes the team. And if you use a shared “service” user, it’ll consume a license in the Enterprise Licensing model.

GitLab deploy keys are an alternative. This article shows how to use the GitLab deploy keys to push tags to a git repository.

Create and configure deploy keys

The long anwer can be found on the GitLab and SSH keys documentation. The short answer is:

ssh-keygen -o -t rsa -b 4096 -C "me@example.com"

1	ssh-keygen -o -t rsa -b 4096 -C "me@example.com"

Copy the content of the public key (by default named id_rsa.pub) to your project. It’s located at GitLab Project -> Settings -> Repository -> Deploy Keys. Once added, the SSH keys fingerprint is displayed on the user interface. You can double check for a match by extracting the fingerprint from your private key locally via:

ssh-keygen -lf id_rsa

1	ssh-keygen -lf id_rsa

Encrypt deploy keys in your repository

I recommend encrypting the private key in your repository, and decrypt it at runtime. I’m using AWS Key Management Service (KMS), but there are many alternatives available, and also corresponding implementations at different cloud providers. Anyway, here’s how to get the private key encrypted:

aws kms encrypt --key-id &lt;guid of key from AWS KMS> --plaintext fileb://id_rsa --output text --query CiphertextBlob --region eu-west-1 | base64 --decode > gitlab-deploy-key-encrypted

1	aws kms encrypt --key-id <guid of key from AWS KMS> --plaintext fileb://id_rsa --output text --query CiphertextBlob --region eu-west-1 \| base64 --decode > gitlab-deploy-key-encrypted

Build file

The trick part of the build file, besides decrypting the private key, is configuring the git push URL and comment correctly. See inline comments of the following file:

#!/usr/bin/env bash

set -e
set -u

# decrypt the private key
aws kms decrypt --ciphertext-blob fileb://gitlab-deploy-key-encrypted --output text --query Plaintext --region eu-west-1 | base64 --decode > ./gitlab-deploy-key
chmod 400 ./gitlab-deploy-key

# set the git username/email to be able to perform git operations
echo git version: $(git --version)
git config --global user.name "${GITLAB_USER_NAME}"
git config --global user.email "${GITLAB_USER_EMAIL}"

# set the push URL differently to leverage SSH protocol
git remote set-url --push origin git@cimpress.githost.io:&lt;your-team>/&lt;your-project>.git
git remote -v

# set the git tag to the current version
VERSION=1.0.$CI_PIPELINE_ID
git tag -a $VERSION -m "Setting version as tag during build."

# push the git tag
# leverage GIT_SSH option to use a dedicated SSH key
# see https://git-scm.com/docs/git#git-codeGITSSHcode for documentation of this feature
echo 'ssh -i ./gitlab-deploy-key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*' > ssh
chmod +x ssh
GIT_SSH='./ssh' git push origin $VERSION

#!/usr/bin/env bash

set -e

set -u

# decrypt the private key

aws kms decrypt --ciphertext-blob fileb://gitlab-deploy-key-encrypted --output text --query Plaintext --region eu-west-1 | base64 --decode > ./gitlab-deploy-key

chmod 400 ./gitlab-deploy-key

# set the git username/email to be able to perform git operations

echo git version: $(git --version)

git config --global user.name "${GITLAB_USER_NAME}"

git config --global user.email "${GITLAB_USER_EMAIL}"

# set the push URL differently to leverage SSH protocol

git remote set-url --push origin git@cimpress.githost.io:<your-team>/<your-project>.git

git remote -v

# set the git tag to the current version

VERSION=1.0.$CI_PIPELINE_ID

git tag -a $VERSION -m "Setting version as tag during build."

# push the git tag

# leverage GIT_SSH option to use a dedicated SSH key

# see https://git-scm.com/docs/git#git-codeGITSSHcode for documentation of this feature

echo 'ssh -i ./gitlab-deploy-key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*' > ssh

chmod +x ssh

GIT_SSH='./ssh' git push origin $VERSION

.gitlab-ci.yml

Eventually, the script needs to be invoked on in the .gitlab-ci.yml file. Additionally, all other stages need to be excluded when tags are pushed to avoid an infinite loop of builds when each build pushes a tag and triggers a build (yes, I did it).

stages:
  - build
  - push_git_tag

build:
  stage: build
  script: ./build.sh
  except:
    - tags

push_git_tag:
  stage: push_git_tag
  only:
    refs:
      - master
  except:
    - tags
  script:
    - ./push_git_tag.sh
  dependencies:
    - build

stages:

- build

- push_git_tag

build:

stage: build

script: ./build.sh

except:

- tags

push_git_tag:

stage: push_git_tag

only:

refs:

- master

except:

- tags

script:

- ./push_git_tag.sh

dependencies:

- build

Once figured out, it’s straight forward to create an SSH key, encrypt it, store it in source code, update the git remote push URL and put those components together.

The AWS Well-Architected framework is a recommendation by AWS, summarized in an 80 page PDF document. After focusing on cost optimization in my first article, this article looks at one specific aspect of the security pillar.

Passwords are bad

Yes, passwords are bad. I don’t need to repeat that, right? Anyways, a few words on that: Managing passwords is a challenge. Especially when you have to manage and hand them out as a central team. First, you end up spending a lot of time resetting passwords, and potentially even managing the secrets in some “secure” store. Second, you have a security risk by keeping passwords active after employees left the company or simply by having the headache on how to protect a central credentials store.

Instead of using AWS IAM users, use AWS IAM Roles. Roles are a central piece of the AWS infrastructure, and every AWS service supports them. Notably EC2 can have an attached IAM profile. Once you attach an IAM instance profile, all calls to AWS services from that EC2 machine are invoked with the specified IAM role.

Custom applications

I often experience teams discussing how to securely store AWS Secret Keys in their development environment or tool they configure. Discussions are usually around how to pass them along to the build server and the production server. The answer is almost always: You don’t. Just ensure the EC2 machine uses an IAM Instance Profile (limited to the required permissions).

But wait, what about local development? I can’t assign an IAM Instance Profile to my machine. Again, don’t do anything in code. Instead, rely on well-documented credential configuration outside of your application (see example documentation for Node.js). Short version is to simply configure your user’s AWS credentials (~/.aws/credentials) and auto-rotate them on a schedule (mirri.js is a good tool to do that).

If you use federated logins to your AWS account, an alternative is to leverage AWS STS and automatically generate a temporary key every time you need one. This eliminates key rotation completely.

External services

There is also the case where you need to grant access to external services. For example, an external build server like Travis CI, a log collector like SumoLogic, etc. Some might have an option to configure an IAM Role with an enterprise subscription, but often the only way is to actually use access keys. So you’re tied to simply rotate them regularly. The key is to automate the log rotation. Felix is a tool that supports some external services, and definitely gives a baseline on how automation can be written.

References

Two weeks after I wrote this blog post, the AWS Security team came up with a great summary or a related topic. See Guidelines for protecting your AWS account while using programmatic access.

Push Git Tag from GitLab Runner