Archive for the 'FOSS' Category

Page 2 of 35

Changing Jenkins concurrent job token from @ to something else

Some jobs may fail in Jenkins when running concurrently because they don’t like the @ symbol in the path.

For example, you may get a jobs at something like:

  • /var/lib/jenkins/jobs/cool-project
  • /var/lib/jenkins/jobs/cool-project@2

This can be easily changed to something else, as per the Jenkins system properties page by modifying the -D arguments sent to Java. I’ve changed it to _job_ at the moment.


echo 'JAVA_ARGS="$JAVA_ARGS -Dhudson.slaves.WorkspaceList=_job_"'\
 >> /etc/default/jenkins
systemctl restart jenkins

Now concurrent jobs will be something like:

  • /var/lib/jenkins/jobs/cool-project
  • /var/lib/jenkins/jobs/cool-project_job_2

Which seems much nicer to me.

Btrfs RAID 6 on dm-crypt on Fedora 23

I’m building a NAS and given the spare drives I have at the moment, thought I’d have a play with Btrfs. Apparently RAID 6 is relatively safe now, so why not put it through its paces? As Btrfs doesn’t support encryption, I will need to build it on top of dm-crypt.

Boot drive:

  • /dev/sda

Data drives:

  • /dev/sdb
  • /dev/sdc
  • /dev/sdd
  • /dev/sde
  • /dev/sdf

I installed Fedora 23 Server onto /dev/sda and just went from there, opening a shell.
# Setup dm-crypt on each data drive
# and populate the crypttab file.
for x in b c d e f ; do
  cryptsetup luksFormat /dev/sd${x}
  UUID="$(cryptsetup luksUUID /dev/sd${x})"
  echo "luks-${UUID} UUID=${UUID} none" >> /etc/crypttab
done
 
# Rebuild the initial ramdisk with crypt support
echo "dracutmodules+=crypt" >> /etc/dracut.conf.d/crypt.conf
dracut -fv
 
# Verify that it now has my crypttab
lsinitrd /boot/initramfs-$(uname -r).img |grep crypttab
 
# Reboot and verify initramfs prompts to unlock the devices
reboot
 
# After boot, verify devices exist
ls -l /dev/mapper/luks*

OK, so now I have a bunch of encrypted disks, it’s time to put btrfs into action (note the label, btrfs_data):
# Get LUKS UUIDs and create btrfs raid filesystem
for x in b c d e f ; do
  DEVICES="${DEVICES} $(cryptsetup luksUUID /dev/sd${x}\
    |sed 's|^|/dev/mapper/luks-|g')"
done
mkfs.btrfs -L btrfs_data -m raid6 -d raid6 ${DEVICES}'

See all our current btrfs volumes:
btrfs fi show

Get the UUID of the filesystem so that we can create an entry in fstab, using the label we created before:
UUID=$(btrfs fi show btrfs_data |awk '{print $4}')
echo "UUID=${UUID} /mnt/btrfs_data btrfs noatime,subvolid=0 0 0"\
  >> /etc/fstab

Now, let’s create the mountpoint and mount the device:
mkdir /mnt/btrfs_data
mount -a

Check data usage:
btrfs filesystem df /mnt/btrfs_data/

This has mounted the root of the filesystem to /mnt/btrfs_data, however we can also create subvolumes. Let’s create one called “share” for shared network data:
btrfs subvolume create /mnt/btrfs_data/share

You can mount this specific volume directly, let’s add it to fstab:
echo "UUID=${UUID} /mnt/btrfs_share btrfs noatime,subvol=share 0 0"\
  >> /etc/fstab
mkdir /mnt/btrfs_share
mount -a

You can list and delete subvolumes:
btrfs subvolume list -p /mnt/btrfs_data/
btrfs subvolume delete /mnt/btrfs_data/share

Now I plugged in a few backup drives and started rsyncing a few TB across to the device. It seemed to work well!

There are lots of other things you can play with, like snapshots, compression, defragment, scrub (use checksums to repair corrupt data), rebalance (re-allocates blocks across devices) etc. You can even convert existing file systems with btrfs-convert command, and use rebalance to change the RAID level. Neat!

Then I thought I’d try the rebalance command just to see how that works with a RAID device. Given it’s a large device, I kicked it off and went to do something else. I returned to an unwakeable machine… hard-resetting, journalctl -b -1 told me this sad story:

Nov 14 06:03:42 localhost.localdomain kernel: ------------[ cut here ]------------
Nov 14 06:03:42 localhost.localdomain kernel: kernel BUG at fs/btrfs/extent-tree.c:1833!
Nov 14 06:03:42 localhost.localdomain kernel: invalid opcode: 0000 [#1] SMP
Nov 14 06:03:42 localhost.localdomain kernel: Modules linked in: fuse joydev synaptics_usb uas usb_storage rfcomm cmac nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtab
Nov 14 06:03:42 localhost.localdomain kernel: snd_soc_core snd_hda_codec rfkill snd_compress snd_hda_core snd_pcm_dmaengine ac97_bus snd_hwdep snd_seq snd_seq_device snd_pcm mei_me dw_dmac i2c_designware_platform snd_timer snd_soc_sst_a
Nov 14 06:03:42 localhost.localdomain kernel: CPU: 0 PID: 6274 Comm: btrfs Not tainted 4.2.5-300.fc23.x86_64 #1
Nov 14 06:03:42 localhost.localdomain kernel: Hardware name: Gigabyte Technology Co., Ltd. Z97N-WIFI/Z97N-WIFI, BIOS F5 12/08/2014
Nov 14 06:03:42 localhost.localdomain kernel: task: ffff88006fd69d80 ti: ffff88000e344000 task.ti: ffff88000e344000
Nov 14 06:03:42 localhost.localdomain kernel: RIP: 0010:[] [] insert_inline_extent_backref+0xe7/0xf0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: RSP: 0018:ffff88000e3476a8 EFLAGS: 00010293
Nov 14 06:03:42 localhost.localdomain kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: RDX: ffff880000000000 RSI: 0000000000000001 RDI: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: RBP: ffff88000e347728 R08: 0000000000004000 R09: ffff88000e3475a0
Nov 14 06:03:42 localhost.localdomain kernel: R10: 0000000000000000 R11: 0000000000000002 R12: ffff88021522f000
Nov 14 06:03:42 localhost.localdomain kernel: R13: ffff88013f868480 R14: 0000000000000000 R15: 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: FS: 00007f66268a08c0(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 06:03:42 localhost.localdomain kernel: CR2: 000055a79c7e6fd0 CR3: 00000000576ce000 CR4: 00000000001406f0
Nov 14 06:03:42 localhost.localdomain kernel: Stack:
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000000 0000000000000005 0000000000000001 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000000001 ffffffff81200176 0000000000270026 ffffffffa0925d4a
Nov 14 06:03:42 localhost.localdomain kernel: 0000000000002158 00000000a7c0ba4c ffff88021522d800 0000000000000000
Nov 14 06:03:42 localhost.localdomain kernel: Call Trace:
Nov 14 06:03:42 localhost.localdomain kernel: [] ? kmem_cache_alloc+0x1d6/0x210
Nov 14 06:03:42 localhost.localdomain kernel: [] ? btrfs_alloc_path+0x1a/0x20 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] __btrfs_inc_extent_ref.isra.52+0xa9/0x270 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] __btrfs_run_delayed_refs+0xc84/0x1080 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_run_delayed_refs.part.73+0x74/0x270 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] ? btrfs_release_path+0x2b/0xa0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_run_delayed_refs+0x15/0x20 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_commit_transaction+0x56/0xad0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] prepare_to_merge+0x1fe/0x210 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] relocate_block_group+0x25e/0x6b0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_relocate_block_group+0x1ca/0x2c0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_relocate_chunk.isra.39+0x3e/0xb0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_balance+0x9c4/0xf80 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_ioctl_balance+0x3c4/0x3d0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] btrfs_ioctl+0x541/0x2750 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: [] ? lru_cache_add+0x1c/0x50
Nov 14 06:03:42 localhost.localdomain kernel: [] ? lru_cache_add_active_or_unevictable+0x32/0xd0
Nov 14 06:03:42 localhost.localdomain kernel: [] ? handle_mm_fault+0xc8a/0x17d0
Nov 14 06:03:42 localhost.localdomain kernel: [] ? cp_new_stat+0xb3/0x190
Nov 14 06:03:42 localhost.localdomain kernel: [] do_vfs_ioctl+0x295/0x470
Nov 14 06:03:42 localhost.localdomain kernel: [] ? selinux_file_ioctl+0x4d/0xc0
Nov 14 06:03:42 localhost.localdomain kernel: [] SyS_ioctl+0x79/0x90
Nov 14 06:03:42 localhost.localdomain kernel: [] ? do_page_fault+0x2f/0x80
Nov 14 06:03:42 localhost.localdomain kernel: [] entry_SYSCALL_64_fastpath+0x12/0x71
Nov 14 06:03:42 localhost.localdomain kernel: Code: 10 49 89 d9 48 8b 55 c0 4c 89 7c 24 10 4c 89 f1 4c 89 ee 4c 89 e7 89 44 24 08 48 8b 45 20 48 89 04 24 e8 5d d5 ff ff 31 c0 eb ac <0f> 0b e8 92 b7 76 e0 66 90 0f 1f 44 00 00 55 48 89 e5
Nov 14 06:03:42 localhost.localdomain kernel: RIP [] insert_inline_extent_backref+0xe7/0xf0 [btrfs]
Nov 14 06:03:42 localhost.localdomain kernel: RSP
Nov 14 06:03:42 localhost.localdomain kernel: ---[ end trace 63b75c57d2feac56 ]---

Bummer!

Looks like rebalance has a major bug at the moment. I did a search and others have the same problem, looks like I’m hitting this bug. I’ve reported it on Fedora Bugzilla.

Anyway, so I won’t do a rebalance at the moment, but other than that, btrfs seems pretty neat. I will make sure I keep my backups up-to-date though, just in case…

Flashing developer image on Nexus 6P (and maybe 5X)

Normally I just download the developer image tarball, verify the checksum and extract it, boot my phone to the bootloader (volume down and power buttons), install android-tools on Fedora and run “fastboot oem unlock“, then run the “flash-all.sh” script from the image tarball, followed by “fastboot oem lock” once I get back to the bootloader.

With a Nexus 6P this has changed a little. First, the command is now fastboot flashing unlock so you need the latest version of fastboot utility (which Fedora does not have). I did this by downloading the basic Android SDK tools only (android-sdk_r24.4.1-linux.tgz), extracting it and running the SDK Manager (./tools/android binary), and installing latest SDK Platform Tools.

Then I could run fastboot to put boot it to bootloader:
sudo ./platform-tools/fastboot reboot bootloader

I also needed to use the new fastboot to flash the default, and the flash-all.sh script from the developer image uses fastboot from the user’s PATH.

In Fedora fastboot is installed to /usr/bin/fastboot but also /bin/fastboot – a user gets the former, root gets the latter, so I moved both of these out of the way and copied in the fastboot binary from

sudo mv /bin/fastboot{,-fedora}
sudo mv /usr/bin/fastboot{,-fedora}
sudo cp ./platform-tools/fastboot /bin/
sudo cp ./platform-tools/fastboot /usr/bin/

Secondly, once you have that the flash-all.sh script still fails with a cryptic message about being unable to remotely unlock.

You need to also boot the phone to Android, activate developer settings (by browsing to Settings -> About Phone and tapping on build 7 times) and then under Settings -> Developer options turn on the option to allow OEM unlocking.

Now I was finally able to flash the phone.

sudo ./flash-all.sh

Boot back to bootloader and re-lock.

sudo adb reboot bootloader
sudo fastboot flashing lock

Hope this helps someone else out there!

Reset keyboard shortcuts in GNOME

Recently we had a Korora user ask how to reset the keybindings in GNOME, which they had changed.

I don’t think that the shortcuts program has a way to reset them, but you can use dconf-editor.

Open the dconf-editor program and browse to:

org.gnome.desktop.wm.keybindings

Anything that’s been modified should be in bold font. Select it then down the bottom on the right click the “Set to Default” button.

Hope that helps!

Cross-compiling a PowerPC64 LE kernel and hitting a GCC bug

Being new at OzLabs I’m dipping my toes into various projects and having a play with PowerPC and so I thought I’d cross-compile the Linux kernel on Fedora. Traditionally PowerPC has been big endian, however it also supports little endian so I wanted to build all the things.

Fedora uses a single cross toolchain that can build all four variants, whereas Debian/Ubuntu splits this out into two different toolchains (a BE and an LE one).

Install dependencies in Fedora:
$ sudo dnf install gcc make binutils-powerpc64-linux-gnu gcc-powerpc64-linux-gnu gcc-c++-powerpc64-linux-gnu bc ncurses-devel

Get the v4.2 kernel:
$ git clone https://github.com/torvalds/linux.git --branch v4.2 --depth 1 && cd linux

Successful big endian build of the kernel, using the default config for pseries:
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make pseries_defconfig
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make -j$(nproc)
# clean after success
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make clean
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make mrproper

Building a little endian kernel however, resulted in a linker problem:
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make pseries_defconfig
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make menuconfig
# change architecture to little endian:
# Endianness selection (Build big endian kernel) --->
# (X) Build little endian kernel
$ ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- make V=1

Here was the result:
powerpc64-linux-gnu-gcc -mlittle-endian -mno-strict-align -m64 -Wp,-MD,arch/powerpc/kernel/vdso64/.vdso64.so.dbg.d -nostdinc -isystem /usr/lib/gcc/powerpc64-linux-gnu/5.2.1/include -I./arch/powerpc/include -Iarch/powerpc/include/generated/uapi -Iarch/powerpc/include/generated -Iinclude -I./arch/powerpc/include/uapi -Iarch/powerpc/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1 -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -msoft-float -pipe -Iarch/powerpc -mtraceback=no -mabi=elfv2 -mcmodel=medium -mno-pointers-to-nested-functions -mcpu=power7 -mno-altivec -mno-vsx -mno-spe -mspe=no -funit-at-a-time -fno-dwarf2-cfi-asm -mno-string -Wa,-maltivec -fno-delete-null-pointer-checks -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fno-stack-protector -Wno-unused-but-set-variable -fomit-frame-pointer -fno-var-tracking-assignments -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -DCC_HAVE_ASM_GOTO -Werror -shared -fno-common -fno-builtin -nostdlib -Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=sysv -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(vdso64.so)" -D"KBUILD_MODNAME=KBUILD_STR(vdso64.so)" -Wl,-T arch/powerpc/kernel/vdso64/vdso64.lds arch/powerpc/kernel/vdso64/sigtramp.o arch/powerpc/kernel/vdso64/gettimeofday.o arch/powerpc/kernel/vdso64/datapage.o arch/powerpc/kernel/vdso64/cacheflush.o arch/powerpc/kernel/vdso64/note.o arch/powerpc/kernel/vdso64/getcpu.o -o arch/powerpc/kernel/vdso64/vdso64.so.dbg
/usr/bin/powerpc64-linux-gnu-ld: arch/powerpc/kernel/vdso64/sigtramp.o: file class ELFCLASS64 incompatible with ELFCLASS32
/usr/bin/powerpc64-linux-gnu-ld: final link failed: File in wrong format
collect2: error: ld returned 1 exit status
arch/powerpc/kernel/vdso64/Makefile:26: recipe for target 'arch/powerpc/kernel/vdso64/vdso64.so.dbg' failed
make[2]: *** [arch/powerpc/kernel/vdso64/vdso64.so.dbg] Error 1
scripts/Makefile.build:403: recipe for target 'arch/powerpc/kernel/vdso64' failed
make[1]: *** [arch/powerpc/kernel/vdso64] Error 2
Makefile:949: recipe for target 'arch/powerpc/kernel' failed
make: *** [arch/powerpc/kernel] Error 2

All those files were 64bit, however:
arch/powerpc/kernel/vdso64/cacheflush.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
arch/powerpc/kernel/vdso64/datapage.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
arch/powerpc/kernel/vdso64/getcpu.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
arch/powerpc/kernel/vdso64/gettimeofday.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
arch/powerpc/kernel/vdso64/note.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped
arch/powerpc/kernel/vdso64/sigtramp.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped

An strace of the failing powerpc64-linux-gnu-gcc command above showed that collect2 (and ld) were being called with an option setting the format to 32bit:
24904 execve("/usr/libexec/gcc/powerpc64-linux-gnu/5.2.1/collect2", ["/usr/libexec/gcc/powerpc64-linux"..., "-plugin", "/usr/libexec/gcc/powerpc64-linux"..., "-plugin-opt=/usr/libexec/gcc/pow"..., "-plugin-opt=-fresolution=/tmp/cc"..., "--sysroot=/usr/powerpc64-linux-g"..., "--build-id", "--no-add-needed", "--eh-frame-hdr", "--hash-style=gnu", "-shared", "--oformat", "elf32-powerpcle", "-m", "elf64lppc", "-o", ...], [/* 66 vars */]

Alan Modra tracked it down to some 32bit hard-coded entries in GCC sysv4.h and sysv4le.h and submitted a patch to the GCC mailing list (Red Hat bug).

I re-built the Fedora cross-gcc package with his patch and it solved the linker problem for me. Hurrah!

Korora 21 available

It has taken a few weeks longer than we had hoped, but Korora 21 images are now available. I strongly recommend downloading with BitTorrent if you can.

The 21 beta was quite successful and we were able to make some minor changes to help improve the overall experience. Users who are currently on the beta need not re-install, updates are provided via the package manager. Users who are on 20 may consider upgrading, however this is not necessary as version 20 is supported for another 6 months or so.

Creating a DMZ in OpenWRT

In computing, a DMZ (demilitarized zone) is a method for separating untrusted traffic from a trusted network. One of the most common implementations of this would be for supporting a publicly accessible server (such as web) on a local internet connection. The server sits in the DMZ and can be accessed from the Internet, but it cannot access the trusted network.

OpenWRT probably needs no introduction, the brilliant open source and community driven Linux based embedded router stack. I run it on my Netgear WNDR3800.

Netgear WNDR3800 running OpenWRT

Netgear WNDR3800 running OpenWRT

I have an ODRIOD-U3 (little ARM box) running Fedora, which runs a web server. This is what I want to make publicly available in my DMZ.

So, how to create a DMZ in OpenWRT? Some commercial routers have a single button “make a DMZ” and everything is handled behind the scenes for you. Not so with OpenWRT; it’s powerful, transparent, and only does what you tell it to, so we have to create it manually.
Continue reading ‘Creating a DMZ in OpenWRT’

Korora 21 beta images available

Korora 21 beta images are now available! Please leave a comment or ping me on social media with any issues or ideas so we can make it better.

Single emergency mode with systemd

Just to remind myself.. add systemd.unit=emergency.target to the kernel line, or if that fails, try init=/sbin/sh and remove both quiet and rhgb options.

Afterwards, exit or:
exec /sbin/init

Can also enable debug mode to help investigating problems with systemd.log_level=debug

You can get a console early on in the boot process by enabling debug-shell:
systemctl enable debug-shell.service

Creating certs and keys for services using FreeIPA (Dogtag)

The default installation of FreeIPA includes the Dogtag certificate management system, a Certificate Authority for your network. It manages expiration of certificates and can automatically renew them. Any client machines on your network will trust the services you provide (you may need to import the IPA CA cert).

There are a number of ways to make certificates. You can generate a certificate signing request or you can have Dogtag manage the whole process for you. You can also create individual cert and key files or put them into a nss database. My preferred method is to use individual files and have Dogtag do the work for me.

If you so desire, you can join your servers to the realm in just the same manner as a desktop client. However, even if they are not joined to the realm you can still create certs for them! You will need to run a few additional steps though, namely creating DNS records and adding the machine manually.

Let’s create a certificate for a web server on www.test.lan (192.168.0.100) which is has not joined our realm.

SSH onto your IPA server and get a kerberos ticket.
[user@machine ~]# ssh root@ipa-server.test.lan
[root@ipa-server ~]# kinit admin

If the host is not already in the realm, create DNS entries and add the host.
[root@ipa-server ~]# ipa dnsrecord-add test.lan www --a-rec 192.168.0.100
[root@ipa-server ~]# ipa dnsrecord-add 0.168.192.in-addr.arpa. 100 --ptr-rec www.test.lan.
[root@ipa-server ~]# ipa host-add www.test.lan

Add a web service for the www machine.
[root@ipa-server ~]# ipa service-add HTTP/www.test.lan

Only the target machine can create a certificate (IPA uses the host kerberos ticket) by default, so to be able to create the certificate on your IPA server you need to allow it to manage the web service for the www host.
[root@ipa-server ~]# ipa service-add-host --hosts=ipa-server.test.lan HTTP/www.test.lan

Now create the cert and key.
[root@ipa-server ~]# ipa-getcert request -r -f /etc/pki/tls/certs/www.test.lan.crt -k
/etc/pki/tls/private/www.test.lan.key -N CN=www.test.lan -D
www.test.lan -K HTTP/www.test.lan

Now copy that key and certificate to your web server host and configure apache as required.
[root@ipa-server ~]# rsync -P /etc/pki/tls/certs/www.test.lan.crt /etc/pki/tls/private/www.test.lan.key root@www.test.lan:

You can also easily delete keys so that they aren’t tracked and renewed any more, first get the request id.
[root@ipa-server ~]# ipa-getcert list

Take note of the id for the certificate you want to delete.
[root@ipa-server ~]# getcert stop-tracking -i [request id]

A CRL (certificate revocation list) is automatically maintained and published on the IPA server at ​https://ipa-server.test.lan/ipa/crl/MasterCRL.bin