Using multiple repositories in your CI builds

Djinn CI makes working with multiple repositories in a build simple via the sources parameter in the build manifest. This allows you to specify multiple Git respositories to clone into your build environment. Each source would be a URL that could be cloned via git clone. With most CI platforms, a build's manifest is typically tied to the source code repository itself. With Djinn CI, whilst you can have a build manifest in a source code repository, the CI server itself doesn't really have an understanding of that repository. Instead, it simply looks at the sources in the manifest that is specified, and clones each of them into the build environment.

Defining multiple sources in a manifest

Like with the previous post, we're going to use djinn-ci/imgsrv as an example of using multiple sources in a build manifest. If we look at the top of the manifest file, we will see that it requires three repositories to build. These are, the source code for djinn-ci/imgsrv itself, golang/tools, and valyala/quicktemplate, defined like so,

sources:
- https://github.com/djinn-ci/imgsrv.git
- https://github.com/golang/tools
- https://github.com/valyala/quicktemplate

during execution of the manifest, all of the above source repositories will be cloned. If submitting this through the user interface for the build server to run, then each of these will have its own clone.n job associated with it, where n is the number counting up from 1.

SSH keys and private repositories

As previously mentioned, the URLs specified in the sources parameter will be given to git clone for cloning in the build environment. This means you can use the SSH format when specifying a source URL,

git@github.com:org/private-repo

this would be preferable in instances where you would want to clone from a private repository. This would however require creating an SSH key to use as a sort of deployment key when cloning into the environment.

Let's consider the above example, when cloning from github.com. We would create a deploy key for that repository, and upload it to Djinn CI. When doing so, we will be able to specify some custom SSH configuration to associate with that key, this is what will allow us to make use of it when it is cloned in the build,

Host github.com
    User git
    IdentityFile /root/.ssh/id_deploy_private_repo

with the key in place, along with the config, we would then be able to use the repository org/private-repo in our builds. However, an issue can arise here, assume we want to clone from another private repository, we could add the same deployment key there, but this might not be preferable if you're concerned about the security implications. So instead, you would create another key for that repository. With two deploy keys in place, one for each repository, this raises the question, how would you associate each key with each repository?

This could be done by making use of the .ssh_config file format and Host matching. Assume the two private repositories are private-repo and acme-repo, both exist under org on github.com. And they each have a deployment key named id_deploy_private_repo and id_deploy_acme_repo. Then we could give each key the following SSH configuration,

Host private-repo.github.com
    User git
    IdentityFile /root/.ssh/id_deploy_private_repo

Host acme-repo.github.com
    User git
    IdentityFile /root/.ssh/id_deploy_acme_repo

then in the build manifest we could give the following sources,

sources:
- git@private-repo.github.com:org/private-repo
- git@acme-repo.github.com:org/acme-repo

it's a bit of a hacky work around, but it works if you would prefer to use different deploy keys for different private repositories.

More information

To learn more about Djinn CI, visit the about page which gives an overview of the features of the platform, and be sure to checkout the user documentation too.