Tag Archive for 'git'

Providing git:// (protocol) access to repos using GitLab

I mirror a bunch of open source projects in a local GitLab instance which works well.

By default, GitLab only provides https and ssh access to repositories, which can be a pain for continuous integration (especially if you were to use self-signed certificates).

However, it’s relatively easy to configure your GitLab server to run a git daemon and provide read-only access to anyone on any repos that you choose.

On my CentOS box, I installed git-daemon which includes systemd git@.service and git.socket files. I copied these to make a new service called git-daemon, like so:

[root@gitlab ~]# cp /usr/lib/systemd/system/git@.service \
[root@gitlab ~]# cp /usr/lib/systemd/system/git.socket \
[root@gitlab ~]# systemctl daemon-reload

Then I edited the git-daemon.socket to point it to the git repositories, /var/opt/gitlab/git-data/repositories/, which is the default location when using the GitLab omnibus package.
Description=Git Repositories Server Daemon
ExecStart=-/usr/libexec/git-core/git-daemon \
--base-path=/var/opt/gitlab/git-data/repositories/ \
--syslog --inetd --verbose

Now start and enable the service:
[root@gitlab ~]# systemctl start git-daemon.socket
[root@gitlab ~]# systemctl enable git-daemon.socket

As per the git-daemon.service systemd file, you should now have git-daemon listening on port 9418, however you may need to open the port through the firewall:

[root@gitlab ~]# firewall-cmd --permanent --zone=public --add-port=9418/tcp
[root@gitlab ~]# systemctl reload firewalld

Now, to enable git:// access to any given repository, you need to touch a file called git-daemon-export-ok in that repo’s git dir (it should be owned by your gitlab user, which is probably git). For example, a mirror of the Linux kernel:

-sh-4.2$ touch /var/opt/gitlab/git-data/repositories/mirror/linux.git/git-daemon-export-ok

From your local machine, test your git:// access!

[12:15 chris ~]$ git ls-remote git://gitlab/mirror/linux.git |head -1
46e595a17dcf11404f713845ecb5b06b92a94e43 HEAD


If you wanted to, you could set up a cron job to make sure that any new mirrors that come along are exported without manual intervention.

First, create an executable script somewhere, like /usr/local/bin/export_git-daemon_repos.sh (note, this excludes any wiki git repos).

set -eo pipefail
if [[ "$USER" != "git" ]]; then
    echo "Only run this as the git user."
    exit 1
cd /var/opt/gitlab/git-data/repositories/mirror
for x in $(ls -d * |grep -v \.wiki\.git) ; do
    pushd ${x}
    if [[ ! -e "git-daemon-export-ok" ]]; then
        touch git-daemon-export-ok

Then add it as a cron job for the git user on your gitlab server to run every two hours, or whatever suits you, e.g.:
-sh-4.2$ crontab -l
0 */2 * * * /usr/local/bin/export_git-daemon_repos.sh >/dev/null

Mirroring git repositories (to GitLab)

There are several open source git repos that I mirror in order to provide local speedy access to. Pushing those to a local GitLab server also means people can easily fork them and carry on.

On the GitLab server I have a local posix mrmirror user who also owns a group called mirror in GitLab (this user is cannot be called “mirror” as the user and group would conflict in GitLab).

In mrmirror’s home directory there’s a ~/git/mirror directory which stores all the repos that I want to mirror. The mrmirror user also has a cronjob that runs every few hours to pull down any updates and push them to the appropriate project in the GitLab mirror group.

So for example, to mirror Linux, I first create a new project in the GitLab mirror group called linux (this would be accessed at something like https://gitlab/mirror/linux.git).

Then as the mrmirror user on GitLab I run a mirror clone:
[mrmirror@gitlab ~]$ cd ~/git/mirror
[mrmirror@gitlab mirror]$ git clone --mirror git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Then the script takes care of future updates and pushes them directly to GitLab via localhost:
# Setup proxy for any https remotes
export http_proxy=http://proxy:3128
export https_proxy=http://proxy:3128
cd ~/git/mirror
for x in $(ls -d *.git) ; do
    pushd ${x}
    git remote prune origin
    git remote update -p
    git push --mirror git@localhost:mirror/${x}"
echo $(date) > /tmp/git_mirror_update.timestamp

That’s managed by a simple cronjob that the mrmirror user has on the GitLab server:
[mrmirror@gitlab mirror]$ crontab -l
0 */4 * * * /usr/local/bin/git_mirror_update.sh

And that seems to be working really well.

Permanently fixing permissions on a shared git repo

When creating a shared git repository (perhaps on a central server) it’s good to use the –shared option:
git init --bare --shared

If you don’t, then you may find that repository permissions get clobbered each time a different person commits and no amount of umasks, chmods and sticky bits seem to help long term.

For your next shared repo that’s fine, but if you have an existing repository you can still fix this (assuming git is your group for write access):

ssh server
chown -Rf root:git /path/to/bare/git/repo
cd /path/to/bare/git/repo
git config core.sharedRepository group
find /path/to/bare/git/repo -type f | xargs chmod 664
find /path/to/bare/git/repo -type d | xargs chmod 775
find /path/to/bare/git/repo -type d | xargs chmod g+s

Enjoy some sanity!

Delete local and remote git branches

Just a quick one for reference..

Deleting one or more local branches is trivial:
git branch --delete branch branch2

However if you want to delete regardless of the merge state:
git branch -D branch branch2

To delete a remote branch you need to push the delete:
git push remote --delete branch

The –delete option is newish, so if your git is old you can use the original syntax:
git push remote :branch

That’s all.

Using Git with Vim

Matt just sent me an article on using Git with Vim, which looks pretty awesome.

Git.vim is a more comprehensive plugin that allows the user to perform a lot more Git operations from within the Vim environment.

Now if only the plugin for Eclipse was ready..


Sourceforging ahead

At work we develop two open source Java applications, Xena and DPR, both of which we host on Sourceforge under CVS. I’ve been pushing to move away from CVS for quite some time now, but it hasn’t gained much traction. This has been mostly due to the lack of a decent Eclipse plugin and partly because of developer apathy. The other day I noticed that Sourceforge enabled support for Git, my favourite SCM system. Today I came across an article saying that they will now provide support for Bazaar and Mercurial also. Sweet.