Fortiter Et Recte
I mirror a bunch of open source projects in a local GitLab instance which works well.
By default, GitLab only provides https and ssh access to repositories, which can be a pain for continuous integration (especially if you were to use self-signed certificates).
However, it’s relatively easy to configure your GitLab server to run a git daemon and provide read-only access to anyone on any repos that you choose.
Continue reading ‘Providing git:// (protocol) access to repos using GitLab’
There are several open source git repos that I mirror in order to provide local speedy access to. Pushing those to a local GitLab server also means people can easily fork them and carry on.
On the GitLab server I have a local posix mrmirror user who also owns a group called mirror in GitLab (this user is cannot be called “mirror” as the user and group would conflict in GitLab).
In mrmirror’s home directory there’s a ~/git/mirror directory which stores all the repos that I want to mirror. The mrmirror user also has a cronjob that runs every few hours to pull down any updates and push them to the appropriate project in the GitLab mirror group.
So for example, to mirror Linux, I first create a new project in the GitLab mirror group called linux (this would be accessed at something like https://gitlab/mirror/linux.git).
Then as the mrmirror user on GitLab I run a mirror clone:
[mrmirror@gitlab ~]$ cd ~/git/mirror
[mrmirror@gitlab mirror]$ git clone --mirror git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
It’s pretty easy to configure postfix on a local Linux box to forward emails via an external mail server. This way you can just send via localhost in your programs or any system daemons and the rest is automatically handled for you.
Here’s how to forward via GMail using authentication and encryption on Fedora (23 at the time of writing). You should consider enabling two-factor authentication on your gmail account, and generate a password specifically for postfix.
Install packages:
sudo dnf install cyrus-sasl-plain postfix mailx
Basic postfix configuration:
#Only listen on IPv4, not IPv6. Omit if you want IPv6.
sudo postconf inet_protocols=ipv4
#Relay all mail through to TLS enabled gmail
sudo postconf relayhost=[smtp.gmail.com]:587
#Use TLS encryption for sending email through gmail
sudo postconf smtp_use_tls=yes
#Enable authentication for gmail
sudo postconf smtp_sasl_auth_enable=yes
#Use the credentials in this file
sudo postconf smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
#This file has the certificate to trust gmail encryption
sudo postconf smtp_tls_CAfile=/etc/ssl/certs/ca-bundle.crt
#Require authentication to send mail
sudo postconf smtp_sasl_security_options=noanonymous
sudo postconf smtp_sasl_tls_security_options=noanonymous
By default postfix listens on localhost, which is probably what you want. If you don’t for some reason, you could change the inet_interfaces parameter in the config file, but be warned that then anyone on your network (or potentially the Internet if it’s a public address) could send mail through your system. You may also want to consider using TLS on your postfix server.
By default, postfix sets myhostname to your fully-qualified domain name (check with hostname -f) but if you need to change this for some reason you can. For our instance it’s not really necessary because we’re forwarding email through a relay and not accepting locally.
Check that our configuration looks good:
sudo postconf -n
sudo postfix check
Create a password file using a text editor:
sudoedit /etc/postfix/sasl_passwd
The content should be in this form (the brackets are required, just replace your username@gmail.com address and password):
[smtp.gmail.com]:587 username@gmail.com:password
Hash the password for postfix:
sudo postmap /etc/postfix/sasl_passwd
Tail the postfix log:
sudo journalctl -f -u postfix.service &
Start the service (you should see it start up in the log):
sudo systemctl start postfix
Send a test email, replace username@gmail.com with your real email address:
echo "This is a test." | mail -s "test message" username@gmail.com
You should see the email go through the journalctl log and be forwarded, something like:
Feb 29 04:32:51 hostname postfix/smtp[4115]: 87BE620221: to=
I’m sure there are lots of howtos on the Internet for this, but…
Say you are running a web server like nginx and your log files are in a non-standard location, you may have problems starting the service because SELinux is blocking nginx from reading or writing to the files.
You can set the context of these files so that nginx will be happy:
[user@server ~]$ sudo chcon -Rv --type=httpd_log_t /srv/mydomain.com/logs/
That’s only temporary however, and the original context will be restored if you run restorecon or relabel your filesystem.
So you can do this permanently using the semanage command, like so:
[user@server ~]$ sudo semanage fcontext -a -t httpd_log_t "/srv/mydomain.com/logs(/.*)?"
Now you can use the standard selinux command to restore the correct label and it will use the new one you set above.
[user@server ~]$ sudo restorecon -rv /srv/
Say you’re using FreeIPA (or perhaps you’ve generated your own CA) and you want to have your machines trust it. Well in Fedora you can run the following command against the CA file:
# trust anchor rootCA.pem
I should have posted this ages ago. There is a well known problem which may affect Nexus 4 devices where it powers off and won’t power on again. When you plug it into the USB charger you get a solid red light and it never recovers.
If you search for the problem the main advice is to return the phone for a factory fix, however there’s an easier trick that’s worked for me; using a wireless charger.
Every time I’ve had this problem (on a few different Nexus 4 phones) I’ve been able to bring the phone back by sitting it on my Nexus wireless charger for a few minutes, then pressing the power button and it springs to life.
After that, you can charge and use the phone as normal. I am writing this now because I was reminded after using this trick to fix a friend’s phone tonight (he’d been googling the problem for a while with no luck).
Maybe someone still has one in their bottom drawer and can make use of it again by using this trick!
Edit: I made some changes which have dropped the temps to around 40 degrees at idle (haven’t tested at load yet). The case has potential, but I still think it’s slightly too cramped and the airflow is not good enough.
Here’s what I changed:
So I’m building a NAS (running Fedora Server) and thought that the Silverstone DS380 case looked great. It has 8 hot-swappable SATA bays, claims decent cooling with filters, neat form factor.
It requires an SFX PSU, but there are some that have enough juice on the 12v rail (although avoid the SilverStone SX500-LG, it’s slightly too long) so that it’s not a major problem (although I would prefer standard ATX).
So I got one to run low-power i3, C226 chipset mainboard and five HGST 3TB NAS drives. Unfortunately the cooling through the drives is pretty much non-existent. The two fans on the side draw air in but blow onto the hotswap chassis and nothing really draws air through it.
As a result, many of the drives run around 65 degrees Celsius at idle (tested overnight) which is already outside of the drives’ recommended temperature range of 0-60 degrees.
I’ve replaced the case with my second choice Fractal Design NODE 304 and the drives at idle all sit at around 35 degrees.
It has two smaller fans at the front to bring air directly over the drives and a larger one at the rear, with a manual L/M/H speed controller for all three on the rear of the case. As a bonus, it uses a standard ATX power supply and has plenty of room for it.
The only downside I’ve found so far is the lack of hot-swap, but my NAS isn’t mission-critical so that’s not a deal breaker for me.
Your mileage might vary, but I won’t buy the DS380 for a NAS again, unless it’s going to run full of SSDs or something (or I heavily mod the case). It’s OK for a small machine though without a bunch of disks (shame!) and that’s what I’ve re-purposed it for now.
-c
Some jobs may fail in Jenkins when running concurrently because they don’t like the @ symbol in the path.
For example, you may get a jobs at something like:
This can be easily changed to something else, as per the Jenkins system properties page by modifying the -D arguments sent to Java. I’ve changed it to _job_ at the moment.
echo 'JAVA_ARGS="$JAVA_ARGS -Dhudson.slaves.WorkspaceList=_job_"'\
>> /etc/default/jenkins
systemctl restart jenkins
Now concurrent jobs will be something like:
Which seems much nicer to me.
Update 2016-08-26: A nasty bug was found in the RAID5/6 Btrfs parity calculation, so I recommend using RAID 10 for now. Where I use raid6 below you may want to change this to raid10. See this post for how to migrate to RAID 10.
I’m building a NAS and given the spare drives I have at the moment, thought I’d have a play with Btrfs. Apparently RAID 6 is relatively safe now (update: turns out, it’s not), so why not put it through its paces? As Btrfs doesn’t support encryption, I will need to build it on top of dm-crypt.
Boot drive:
Data drives:
I installed Fedora 24 Server onto /dev/sda and just went from there, opening a root shell.
# Install the btrfs and crypt packages (if not already there) so that this will actually work.
dnf install -y btrfs-progs cryptsetup
WARNING WARNING WARNING
The following cryptsetup commands will wipe any drives you specify below. Please make sure you are specifying the correct drives.
# Setup dm-crypt on each data drive
# and populate the crypttab file.
for x in b c d e f ; do
cryptsetup luksFormat /dev/sd${x}
UUID="$(cryptsetup luksUUID /dev/sd${x})"
echo "luks-${UUID} UUID=${UUID} none" >> /etc/crypttab
done
# Rebuild the initial ramdisk with crypt support
echo "add_dracutmodules+=crypt" >> /etc/dracut.conf.d/crypt.conf
dracut -fv
# Verify that it now has my crypttab
lsinitrd /boot/initramfs-$(uname -r).img |grep crypttab
# Reboot and verify initramfs prompts to unlock the devices
reboot
# After boot, verify devices exist
ls -l /dev/mapper/luks*
OK, so now I have a bunch of encrypted disks, it’s time to put btrfs into action (note the label, btrfs_data):
# Get LUKS UUIDs and create btrfs raid filesystem
for x in b c d e f ; do
DEVICES="${DEVICES} $(cryptsetup luksUUID /dev/sd${x}\
|sed 's|^|/dev/mapper/luks-|g')"
done
mkfs.btrfs -L btrfs_data -m raid6 -d raid6 ${DEVICES}
See all our current btrfs volumes:
btrfs fi show
Get the UUID of the filesystem so that we can create an entry in fstab, using the label we created before:
UUID=$(btrfs fi show btrfs_data |grep uuid |awk '{print $4}')
echo "UUID=${UUID} /mnt/btrfs_data btrfs noatime,subvolid=0 0 0"\
>> /etc/fstab
Now, let’s create the mountpoint and mount the device:
mkdir /mnt/btrfs_data
mount -a
Check data usage:
btrfs filesystem df /mnt/btrfs_data/
This has mounted the root of the filesystem to /mnt/btrfs_data, however we can also create subvolumes. Let’s create one called “share” for shared network data:
btrfs subvolume create /mnt/btrfs_data/share
You can mount this specific volume directly, let’s add it to fstab:
echo "UUID=${UUID} /mnt/btrfs_share btrfs noatime,subvol=share 0 0"\
>> /etc/fstab
mkdir /mnt/btrfs_share
mount /mnt/btrfs_share
You can list subvolumes easily by referencing our mounted Btrfs volume:
btrfs subvolume list -p /mnt/btrfs_data/
If you want to delete a subvolume, first unmount it, then remove it from fstab, delete the Btrfs subvolume and finally remove the mount point.
umount /mnt/btrfs_share
sed -i /btrfs_share/d /etc/fstab
btrfs subvolume delete /mnt/btrfs_data/share
/mnt/btrfs_share
Now I plugged in a few backup drives and started rsyncing a few TB across to the device. It seemed to work well!
There are lots of other things you can play with, like snapshots, compression, defragment, scrub (use checksums to repair corrupt data), rebalance (re-allocates blocks across devices) etc. You can even convert existing file systems with btrfs-convert command, and use rebalance to change the RAID level. Neat!
Then I thought I’d try the rebalance command just to see how that works with a RAID device. Given it’s a large device, I kicked it off and went to do something else. I returned to an unwakeable machine… hard-resetting, journalctl -b -1 told me this sad story:
Nov 14 06:03:42 localhost.localdomain kernel: ------------[ cut here ]------------
Nov 14 06:03:42 localhost.localdomain kernel: kernel BUG at fs/btrfs/extent-tree.c:1833!
Nov 14 06:03:42 localhost.localdomain kernel: invalid opcode: 0000 [#1] SMP
Nov 14 06:03:42 localhost.localdomain kernel: Modules linked in: fuse joydev synaptics_usb uas usb_storage rfcomm cmac nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtab
Nov 14 06:03:42 localhost.localdomain kernel: snd_soc_core snd_hda_codec rfkill snd_compress snd_hda_core snd_pcm_dmaengine ac97_bus snd_hwdep snd_seq snd_seq_device snd_pcm mei_me dw_dmac i2c_designware_platform snd_timer snd_soc_sst_a
Nov 14 06:03:42 localhost.localdomain kernel: CPU: 0 PID: 6274 Comm: btrfs Not tainted 4.2.5-300.fc23.x86_64 #1
Nov 14 06:03:42 localhost.localdomain kernel: Hardware name: Gigabyte Technology Co., Ltd. Z97N-WIFI/Z97N-WIFI, BIOS F5 12/08/2014
Nov 14 06:03:42 localhost.localdomain kernel: task: ffff88006fd69d80 ti: ffff88000e344000 task.ti: ffff88000e344000
Nov 14 06:03:42 localhost.localdomain kernel: RIP: 0010:[
Nov 14 06:03:42 localhost.localdomain kernel: RSP: 0018:ffff88000e3476a8 EFLAGS: 00010293
Nov 14 06:03:42 localhost.localdomain kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: RDX: ffff880000000000 RSI: 0000000000000001 RDI: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: RBP: ffff88000e347728 R08: 0000000000004000 R09: ffff88000e3475a0
Nov 14 06:03:42 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000002 R12: ffff88021522f000
Nov 14 06:03:42 localhost.localdomain kernel: R13: ffff88013f868480 R14: 0000000000000000 R15: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: FS: 00007f66268a08c0(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 06:03:42 localhost.localdomain kernel: CR2: 000055a79c7e6fd0 CR3: 00000000576ce000 CR4: 00000000001406f0
Nov 14 06:03:42 localhost.localdomain kernel: Stack:
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000000 0000000000000005 0000000000000001 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000001 ffffffff81200176 0000000000270026 ffffffffa0925d4a
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000002158 00000000a7c0ba4c ffff88021522d800 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: Call Trace:
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: [
Nov 14 06:03:42 localhost.localdomain kernel: Code: 10 49 89 d9 48 8b 55 c0 4c 89 7c 24 10 4c 89 f1 4c 89 ee 4c 89 e7 89 44 24 08 48 8b 45 20 48 89 04 24 e8 5d d5 ff ff 31 c0 eb ac <0f> 0b e8 92 b7 76 e0 66 90 0f 1f 44 00 00 55 48 89 e5
Nov 14 06:03:42 localhost.localdomain kernel: RIP [
Nov 14 06:03:42 localhost.localdomain kernel: RSP
Nov 14 06:03:42 localhost.localdomain kernel: ---[ end trace 63b75c57d2feac56 ]---
Bummer!
Looks like rebalance has a major bug at the moment. I did a search and others have the same problem, looks like I’m hitting this bug. I’ve reported it on Fedora Bugzilla.
Anyway, so I won’t do a rebalance at the moment, but other than that, btrfs seems pretty neat. I will make sure I keep my backups up-to-date though, just in case…
Normally I just download the developer image tarball, verify the checksum and extract it, boot my phone to the bootloader (volume down and power buttons), install android-tools on Fedora and run “fastboot oem unlock“, then run the “flash-all.sh” script from the image tarball, followed by “fastboot oem lock” once I get back to the bootloader.
With a Nexus 6P this has changed a little. First, the command is now “fastboot flashing unlock“ so you need the latest version of fastboot utility (which Fedora does not have). I did this by downloading the basic Android SDK tools only (android-sdk_r24.4.1-linux.tgz), extracting it and running the SDK Manager (./tools/android binary), and installing latest SDK Platform Tools.
Then I could run fastboot to put boot it to bootloader:
sudo ./platform-tools/fastboot reboot bootloader
I also needed to use the new fastboot to flash the default, and the flash-all.sh script from the developer image uses fastboot from the user’s PATH.
In Fedora fastboot is installed to /usr/bin/fastboot but also /bin/fastboot – a user gets the former, root gets the latter, so I moved both of these out of the way and copied in the fastboot binary from
sudo mv /bin/fastboot{,-fedora}
sudo mv /usr/bin/fastboot{,-fedora}
sudo cp ./platform-tools/fastboot /bin/
sudo cp ./platform-tools/fastboot /usr/bin/
Secondly, once you have that the flash-all.sh script still fails with a cryptic message about being unable to remotely unlock.
You need to also boot the phone to Android, activate developer settings (by browsing to Settings -> About Phone and tapping on build 7 times) and then under Settings -> Developer options turn on the option to allow OEM unlocking.
Now I was finally able to flash the phone.
sudo ./flash-all.sh
Boot back to bootloader and re-lock.
sudo adb reboot bootloader
sudo fastboot flashing lock
Hope this helps someone else out there!