Github is the most popular online service for hosting and collaborative development of open source projects. And Redmine is the most popular personal and enterprise open source project hosting and management solution. So, no wonder, that these two systems are used together quite often.

Thus, Github’s Redmine service hook can be used to trigger fetching of repository updates by Redmine, when new commits are made to the Github repository. Unfortunately, it’s useless for the pure Redmine, as Redmine is not able to work with Github repositories directly (only through the local Git mirror, which needs to be created and updated manually)… Moreover, I am not aware of any solution, which could be used for this (even through the mirror).

Certainly, Redmine (especially its community) does much more to support Github. Thus, there is the plenty of tutorials describing how to mirror and keep Github repositories in sync for Redmine (e.g., this or this). Also there exist special plugins for making Github repositories easier usable under Redmine. The most noticeable such plugin is Github hook authored by Jakob Skjerning.

The Github hook plugin allows local mirror repositories to be automatically updated by Github through its special hook (which should be registered manually). But, this hook is not the Redmine service hook, which has been mentioned above – it’s another, generic one.

Anyway all such Redmine plugins (at least, ones known to me) help only with a part of the job, needed to make Github repositories work smoothly under Redmine. All other stuff is still needed to be done manually… That’s when you may find SCM Creator useful. The version 0.5.0b (beta for now) of this Redmine plugin provides the automation solution for Github. And this is the solution we will discuss in this article.

Easy mirroring

Redmine comes with out of the box Subversion and Git SCMs support (in addition to others). But, Git repositories may be local only (i.e., located at the same server, where Redmine runs). So, in fact, you can’t use Github repositories under Redmine (directly)! On the other side, Subversion repositories may be remote…

This is a known issue, that makes Git fans frustrated. And this is an especially significant issue as Github – a Git repository hosting service – is extremely popular. It would be great, if Github could be used the same way the remote Subversion repository is – i.e., just by pasting repository URL in the text box and clicking “Create”… And it does can be used this way now! With SCM Creator 0.5.0b.

Generally, the flow looks and feels the same as for remote Subversion repositories – you select the “Github” SCM type (not Git), paste the repository URL, optionally specify login and password (which are needed for private repositories) and click “Create”. After that SCM Creator does everything required (and described by Redmine guys, e.g., here) to create a local mirror of the Github repository and register it (the mirror) in Redmine.

For private repositories to be able to use login and password you need to specify the HTTP URL of the Github repository, i.e., the one, that starts with “https://github.com/”.

Configuring the Github SCM

As SCM Creator actually makes a local mirror of the Github repository (unlike remote Subversion repositories, which are accessed directly), it needs to know the path, where such mirrors can be stored. In fact, this is the only configuration option, which is required for Github mirroring to work.

The configuration of the SCM Creator lies in the “scm.yml” file under the “config” directory of the Redmine installation. This is the YAML file containing a section for each SCM supported by SCM Creator. For mirroring the corresponding github section should look like:

production:
  github:
    path: /var/lib/github

Here, /var/lib/github is the path, where local mirrors are going to be stored. This path should exist and be writtable by Redmine.

Check also this page for more details on configuring SCM Creator.

Easy creating

However, the main goal of the SCM Creator (as it comes from its name) is to allow creating new repositories and not just cloning them locally. Easy morroring is just a nice bonus to this greater feature.

When SCM Creator is properly configured, the new repository dialog contains the additional “Create new repository” button (which is well-known to SCM Creator users). In case of Github this button lets you create new remote Github repositories:

In other words, when you specify the name of the repository and click the “Create new repository” button, SCM Creator automatically creates new remote repository on Github, makes its local mirror and adds it to Redmine!

This is made possible by the Github API. SCM Creator uses the Github’s official Octokit library to make use of this API.

Configuring the Github creator

Certainly, this also should be configured. The configuration is done in the same “scm.yml” file under the new subsection api as follows:

production:
  github:
    path: /var/lib/github
    api:
      username: login
      password: password
      token: 36857b12efbccc64ce725ea8674fab6ecbd05c03

SCM Creator needs to authenticate with Github to use its API and it needs either login and password or the API key/token to do this. The latter should be preferred!

To get the API key for your Github user in your Github account go to Edit profile → Applications → Personal access tokens → Generate new token:

To be able to create new public and private repositories using the generated API token check the repo scope in this form. To be able to create only public repositories select the public_repo scope instead.

You will, probably, also need the admin:repo_hook scope, that allows registering service hooks using the Github API (see below).

Note, that the generated API key can be seen only once, right after the generation (you can regenerate it later though):

So, be sure to copy the generated API key and paste it into the token option of your “scm.yml” file.

After that (and after the restart of Redmine) you will see the “Create new repository” button enabled for the “Github” SCM. By specifying the repository name and clicking this button you will be able to create new Github repositories in your Github account from within Redmine!

Be sure to limit who can create repositories using the “Manage repository” permission of Redmine (Administration → Roles and permissions).

Keeping repositories in sync

To display commits (and to do this faster) Redmine uses the database, not the (local) repository directly. So, before showing latest commits to the user it first needs to import them into the database.

This means, that in the case of Github the local mirror of the repository should be first updated with latest changes from the original remote repository! This is, in fact, the additional step specific to the Github SCM. Luckily, SCM Creator does this step automatically before Redmine imports data from the local mirror. So, you do not need to worry about that.

But, what you may need to worry is when Redmine updates the database with the latest commits. This is controlled by the special “Fetch commits automatically” option under Administration → Settings → Repositories tab:

If this option is enabled, Redmine updates the repository each time you load anything under the “Repository” tab of the project. This option is enabled by default, as it does not require any additional configuration. But, in fact, this is the worst configuration, as it may take a time for Redmine to update the repository (especially for Github, when it will also need to update the local mirror). Considering, that this is going to be done for every request, this may appear to be a problem…

The better and more advanced configuration is to disable the automatic fetching and to configure each repository to update Redmine automatically as soon as a commit happens. For other repository SCMs to do this you just need to add a “hook” script on the SCM server as described, e.g., here. But for Github you can’t access the server! Luckily Github provides the web interface to add such hooks. So, normally to update the local Github mirror one installed the Github Hook plugin of Jakob Skjerning, then created and configured the Github generic hook as described here. This way, when a commit was pushed, Github instructed Redmine (through the Github Hook plugin) to update the repository. But, with SCM Creator 0.5.0b this is made even easier…

In SCM Creator the “Github” virtual SCM comes with the additional “Register hook for fetching commits” option:

This option is available only when the “Fetch commits automatically” option is disabled and the “Enable WS for repository management” option is enabled. Both these options can be configured under Administration → Settings → Repositories tab:

The “Enable WS for repository management” option enables the special remote API endpoint, which can be used to let Redmine know, which project’s repositories need to be updated (e.g., because a new commit was pushed to one of them). But, to be used this endpoint requires the API key, which can be (re-)generated at the same page (using the “Generate a key” link). And this is the endpoint, which is used by the Github Redmine service, which has been mentioned at the begining of this article.

So, as SCM Creator injects the local mirror update into the commits fetching procedure (before the actual fetching) this makes the Github Redmine service usable with Github SCM repositories of SCM Creator (by the way, it seems like this is the first plugin, that supports this Github hook). But, to make it work you need to create the service hook in your Github repository (as described here)… But, Github also makes it possible to create the hook using the Github API – the same API, which is already used by SCM Creator to create repositories. So, yes, it’s possible to create this hook automatically as well! And that’s what the “Register hook for fetching commits” option does!

If the “Register hook for fetching commits” option is checked, SCM Creator will attempt to create the Redmine service hook on Github for the repository automatically. This can be done for new repositories and for already existing ones – in the first case it will use API credentials from the “scm.yml” file and in the second case it will use credentials specified in the repository form (which should grant the administrative access to the repository).

Please note, that in order to be able to create the hook using the API token, you must enable the admin:repo_hook scope for this token on Github (see above).

Configuring hook creation

The hook creation can be controlled through the special register_hook option in the github → api section of the “scm.yml” file.

production:
  github:
    path: /var/lib/github
    api:
      token: 36857b12efbccc64ce725ea8674fab6ecbd05c03
      register_hook: true

This option accepts one of the four values:

• forbid – the hook registration is denied, no “Register hook for fetching commits” option will be available.
  This value can be set, if you do not want any automatic hook registration.
• false – by default the “Register hook for fetching commits” option is unchecked.
  So, users will need to explicitly check it to try registering the Redmine service hook.
• true – by default the option is checked.
• force – the hook is automatically registered for each newly created Github repository.
  For already existing repositories you will still have to check the “Register hook for fetching commits” option explicitly.