Previously, one would become a maintainer by first submitting a new package and thereby overtime demonstrating an understanding of the Fedora packaging process and guidelines. However recently a new system was approved whereby one could become a co-maintainer of an existing package, instead. This requires being sponsored by an existing maintainer, which Rahul was for me.

It’s quite a big task, but Rahul was very helpful and patient while I learned about the process. There is a lot of information on the Fedora Wiki but I found there wasn’t really a single place which provided a nice overview, broke down each section in detail, and then explained all the steps required. So that (along with my cautious nature) meant it took a dozen or so clarifying emails back and forth between Rahul and myself.

When I got my first understanding and taste of the system though, I’ve gotta say, I was quite impressed. The scale of the system and architecture and how it all works is amazing. The build files (RPM spec files) of all the packages in Fedora are kept under their own git repository. Hooks into this git system and special build tools manage the whole process. All in all, this automation makes it quite seamless.

So after reading and doing everything Rahul sent to me, I started on my journey.

There are two development systems to update now – rawhide (development tree) and F15 (the upcoming version). I wanted to update both of these to the latest upstream version, 17.90. I also wanted to update the stable Fedora 14 package to an upstream bug-fix release, version 16.1.1.

Cloning deja-dup from the git repo put me in the rawhide branch by default. I updated the spec file – increasing the version, adding python-cloudfiles as a new dependency, and putting in a detailed changelog. I then committed that to my local git repo with a comment linking to upstream. I then pulled those changes from rawhide into the f15 branch and updated f14.

commit bb1a5daa9feea74640fd84a0174e74e4e83ad34b
Author: Christopher Smart
Date: Sat Mar 5 21:23:07 2011 +1100
    
     Updated to upstream bugfix version 16.1.1.
     "This release fixes a bug in the just-released 16.1 that caused help
     documentation to not be translated."
     https://launchpad.net/deja-dup/+announcement/7239

Next I needed to test these packages, so I built a source RPM from the spec for each version. I discovered that one can either do a mock build on their local machine, or use Koji to do a scratch build on Fedora’s online build system. I did both, which worked well.

Using Koji meant I needed to first push my changes back to Fedora first though (or so I thought), so I was hesitant to do it without first checking with Rahul. This was the sort of thing that was emailed back and forth – I didn’t fully understand the system and didn’t want to break anything! For example, in order to test my F14 update I had to push back to the git repo – but what if I made a mistake in the process? Would my build on Koji push this broken package onto everyone’s system?

Fortunately, no. Once all the updating of the spec file and building is complete, the maintainer has to formally submit it into the update process before anything will happen. In addition, each package must then pass approval by other maintainers before it is approved and pushed out. You can see why it took some back and forth between Rahul and myself – I wanted to be sure I wasn’t going to do some damage

Along the way I discovered new automated ways of doing things, such as validating a spec file, automatically downloading the source tarball to update the spec file, then uploading it to the Fedora build system.

In the end though, I think that I have a pretty good grasp of the overall process and am quite excited by the prospect of becoming a Fedora maintainer. I’m hoping that deja-dup comes out with a new update soon, so that I can do another update!

I’m still a little hesitant to go and change things – I don’t want to do the wrong thing or step on anyone’s toes. I did however create my own page on the Fedora Wiki, complete with a hackergotchi, of course.

So once again, thanks to Rahul for his patience and guidance

While upgrading via PackageKit is coming, there’s a pretty decent way of doing it with PreUpgrade.

For the impatient, the steps are:

• yum update
• yum install preupgrade
• preupgrade (reboot)
• yum distro-sync
• package-cleanup –orphans (be careful!)
• package-cleanup –leaves (be careful!)

Now the longer version. First, update your current system:
yum update

Install preupgrade:
yum install preupgrade

Run the preupgrade tool, and follow the prompts (remote upgrades over VNC also supported):
preupgrade

PreUpgrade will prompt to reboot your computer and will update your system automatically. All of your packages should be updated and your repositories configured for you (I also had RPMFusion, Chromium and yum-rawhide which all updated as expected). On one of my systems, even the NVIDIA driver was automatically updated and configured. Now you’re booted into your new Fedora 14 system.

OK, this is where things get a little tricky. Some packages might be no-longer supported, there might be removed dependencies and the like. So, there are two neat commands that we will use to identify these packages, so that you can remove them.

package-cleanup --orphans
package-cleanup --leaves

But wait.. I discovered that there’s a more important step to do first, or you’ll cause yourself a headache!

On my machine, this wanted to remove some pretty important packages, like kdelibs.
[chris@shem ~]$ sudo package-cleanup --orphans
[sudo] password for chris:
Loaded plugins: refresh-packagekit, remove-with-leaves
akonadi-1.4.0-3.fc13.x86_64
kdeedu-marble-4.5.2-2.fc13.x86_64
kdeedu-marble-libs-4.5.2-2.fc13.x86_64
kdelibs-4.5.2-8.fc13.x86_64
kdelibs-common-4.5.2-8.fc13.x86_64
schroedinger-1.0.10-1.fc13.x86_64
xorg-x11-drv-wacom-0.10.8-2.fc13.x86_64

Fedora (unlike certain other distros I know) provides major updates to the latest versions of packages (like OpenOffice.org, KDE, GNOME, etc). Yay! It’s possible that your new Fedora system is running the same version of, say KDE, as your old one. This was indeed the case with my Fedora 13 to 14 upgrade (both run KDE 4.5.2 at time of F14 release).

Actually, the build of some important packages on Fedora 13 (like kdelibs-4.5.2-8.fc13.x86_64) were actually more recent than on the newer Fedora 14 system (kdelibs-4.5.2-5.fc14.x86_64).

The package-cleanup tool correctly lists kdelibs-4.5.2-8.fc13.x86_64 as being an orphan and if you were to remove this, you’d break your system badly. In fact, if you’re running yum’s brilliant new autoremove deps feature, as I am, you’ll lose most of your system. It makes sense – you are telling yum to remove kdelibs, so it goes and removes everything that relies on it! Yikes.

So first, we need to fix this by running the yum distro-sync command which recognises that we need to downgrade that kdelibs packages (and not remove it!).

[chris@shem ~]$ sudo yum distro-sync
Loaded plugins: refresh-packagekit, remove-with-leaves
Setting up Distribution Synchronization Process
Resolving Dependencies
--> Running transaction check
---> Package akonadi.x86_64 0:1.4.0-1.fc14 will be a downgrade
---> Package akonadi.x86_64 0:1.4.0-3.fc13 will be erased
---> Package kdeedu-marble.x86_64 0:4.5.2-1.fc14 will be a downgrade
---> Package kdeedu-marble.x86_64 0:4.5.2-2.fc13 will be erased
---> Package kdeedu-marble-libs.x86_64 0:4.5.2-1.fc14 will be a downgrade
---> Package kdeedu-marble-libs.x86_64 0:4.5.2-2.fc13 will be erased
---> Package kdelibs.x86_64 6:4.5.2-5.fc14 will be a downgrade
---> Package kdelibs.x86_64 6:4.5.2-8.fc13 will be erased
---> Package kdelibs-common.x86_64 6:4.5.2-5.fc14 will be a downgrade
---> Package kdelibs-common.x86_64 6:4.5.2-8.fc13 will be erased
---> Package schroedinger.x86_64 0:1.0.9-2.fc14 will be a downgrade
---> Package schroedinger.x86_64 0:1.0.10-1.fc13 will be erased
---> Package xorg-x11-drv-wacom.x86_64 0:0.10.8-1.20100726.fc14 will be a downgrade
---> Package xorg-x11-drv-wacom.x86_64 0:0.10.8-2.fc13 will be erased
--> Finished Dependency Resolution
--> Finding unneeded leftover dependencies
Found and removing 0 unneeded dependencies
----
Dependencies Resolved
=================================================
Package Arch Version Repository Size
=================================================
Downgrading:
akonadi x86_64 1.4.0-1.fc14 fedora 677 k
kdeedu-marble x86_64 4.5.2-1.fc14 fedora 14 M
kdeedu-marble-libs x86_64 4.5.2-1.fc14 fedora 902 k
kdelibs x86_64 6:4.5.2-5.fc14 fedora 12 M
kdelibs-common x86_64 6:4.5.2-5.fc14 fedora 1.7 M
schroedinger x86_64 1.0.9-2.fc14 fedora 276 k
xorg-x11-drv-wacom x86_64 0.10.8-1.20100726.fc14 fedora 73 k
---
Transaction Summary
=============================================================================================================================================
Downgrade 7 Package(s)
---
Total download size: 30 M
Is this ok [y/N]

Once we downgrade these packages, we can then remove any other orphans we might have, with package-cleanup –leaves and package-cleanup –orphans (I didn’t have any). One last thing to note – Fedora will not replace packages from the newer release if they are exactly the same version. For this reason, you will probably have some F13 packages still installed on your computer – don’t worry, that’s correct. They will be upgraded in time (if required).

So, now I think I know how to successfully upgrade the system without breaking it If anyone has some other tips or corrections, please let me know! Hope that’s helpful.

-c

Today I got my hands on a suitable build of yum (add the yum-rawhide repo and set clean_requirements_on_remove=1 under [main] in /etc/yum.conf) and I started to test it out on several packages, all of which introduced dependencies. My initial findings? It’s great.

Single package with dependencies
I started with a single package with several dependencies. When I uninstalled the parent package, yum also removed all of its dependencies which were not needed by any other package (i.e. orphaned dependencies).

Let’s look at an example; installing Ekiga which pulls in 5 dependencies.

yum install ekiga
...
Installed:
ekiga.x86_64 0:3.2.7-4.fc14
Dependency Installed:
evolution-data-server.x86_64 0:2.32.0-3.fc14 libgdata.x86_64 0:0.6.4-4.fc14 libgweather.x86_64 0:2.30.3-1.fc14 opal.x86_64 0:3.6.8-1.fc14
ptlib.x86_64 0:2.6.7-1.fc14


OK, so we have Ekiga, and we have 5 dependencies installed. What happens if I remove Ekiga?

yum erase ekiga
...
Removed:
ekiga.x86_64 0:3.2.7-4.fc14
Dependency Removed:
evolution-data-server.x86_64 0:2.32.0-3.fc14 libgdata.x86_64 0:0.6.4-4.fc14 libgweather.x86_64 0:2.30.3-1.fc14 opal.x86_64 0:3.6.8-1.fc14
ptlib.x86_64 0:2.6.7-1.fc14


As I was hoping, yum removed all 5 dependencies along with Ekiga itself. Ta da!

Result: Tick!

Packages with shared dependencies
I then tested two packages which have shared dependencies. What I would expect is that only the dependencies which are unique to the application being removed, are uninstalled. Other shared dependencies remain, because the other program still requires them and is not being removed.

Let’s look at an example; mplayer and vlc. Both require the libcaca library and installing both packages pulled in this library.
VLC needs libcaca:

yum deplist vlc |grep libcaca
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14


And mplayer needs libcaca:

yum deplist mplayer |grep libcaca
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14


When I want to remove mplayer, yum only removes those mplayer dependencies which vlc does not also require, and leaves the rest (including libcaca).


yum erase mplayer
...
Removed:
mplayer.x86_64 0:1.0-0.119.20100703svn.fc14
Dependency Removed:
libvdpau.x86_64 0:0.4.1-1.fc14.1 lzo.x86_64 0:2.03-3.fc12 mplayer-common.x86_64 0:1.0-0.119.20100703svn.fc14


Note that libcaca was not removed, which is the expected result.

Result: Tick!

Removing a package’s dependency
I then tried to remove a dependency, rather than the parent package. I would expect this to remove all parents which required that package, along with their other dependencies.

Let’s look at an example, gnash-plugin. This package pulls in three dependencies (including the actual gnash package).


yum install gnash-plugin
...
Installed:
gnash-plugin.x86_64 1:0.8.8-4.fc14
Dependency Installed:
agg.x86_64 0:2.5-9.fc13 gnash.x86_64 1:0.8.8-4.fc14 gtkglext-libs.x86_64 0:1.2.0-10.fc12

Now, let’s see what happens if I try and remove one of the dependencies.

yum erase agg
...
Removed:
agg.x86_64 0:2.5-9.fc13
Dependency Removed:
gnash.x86_64 1:0.8.8-4.fc14 gnash-plugin.x86_64 1:0.8.8-4.fc14 gtkglext-libs.x86_64 0:1.2.0-10.fc12


As you can see, it correctly removed the parent package, which could no-longer operate without the dependency.

This is such a great advancement over remove-leaves, where last time I tried to remove gnash-plugin (after testing it out, I didn’t want it) yum wanted to remove Firefox!! Fail.

Result: Tick!

Suggestion – clean up existing system
One thing I would like, is the ability to run a system wide cleanup using this code in yum, rather than other rpm orphan tools. Sort-of like an apt-get autoremove. Theoretically, if you’ve always removed any packages on your system with this new feature then you should be right, but it would be nice to sort-of do a clean up at any point (on any system) and get yum to remove any orphaned deps for you. Maybe this already exists.

If you want to clean up existing orphaned dependencies, then package-cleanup is your friend (from yum-utils).

Take a look at all the packages that are orphaned, and make sure it is sane.

package-cleanup --leaves


If you’re happy with the list, you can just remove them all (after you’ve enabled yum’s new autoremove feature, of course!):


yum erase $(package-cleanup --leaves)


You can instantly see the benefit of this new feature – on my machine this command also removes brand new orphaned dependencies, created by the orphans I’m removing! That’s 59 packages in total.

Without this yum feature, it would only remove 36 packages. You would need to run package-cleanup several times, until you’ve really removed all orphaned packages.

Similarly, this works for package-cleanup –orphans but it would be great it this was built into a single yum cleanup type function – one command to do it all.

Summary
From my initial testing, this works really well, although I’ll continue to test it and see how we go. Hopefully this will find its way into Fedora 12, 13, and 14 rather than just 15.

I’m so grateful for this work – it’s something I’ve been crying out for ever since I made the decision to stick with Fedora. Thank you, thank you, thank you!!

It seems so simple (and Debian has done it since the dawn of time) – if I install package x which pulls in dependencies y and z, then when I remove package x, I want to remove dependencies y and z, if they are not required by any other package.

Yes, there is the remove-leaves plugin for Yum and various RPM orphan checking tools, but in my experience, they are just not reliable.

So, I’m very happy to have discovered that Seth Vidal has merged orphaned dependency cleanup on removal into Yum. Hallelujah!

It’s in rawhide yum-3.2.28-13, and I’ll do some testing soon…

One helpful package that RPMFusion does not package however, is libdvdcss, the free software library which enables Linux to play encrypted DVDs.

Most users keep Livna around for this single purpose, but it is often offline (especially recently). So, here’s another solution – use ATrpms – another third party repository that includes lots of handy software.

The problem is, ATrpms has lots of other software which you might not want to update and which could conflict with the packages from RPMFusion. The solution? Tell yum to only include libdvdcss* packages from that repository. Easy.

To do this, simply add a new repo file (/etc/yum.repos.d/atrpms.repo) for Yum to configure it.
[atrpms]
name=Fedora Core $releasever - $basearch - ATrpms
baseurl=http://dl.atrpms.net/f$releasever-$basearch/atrpms/stable
gpgkey=http://ATrpms.net/RPM-GPG-KEY.atrpms
gpgcheck=1
includepkgs=libdvdcss*

Now, just run the following to update the repo and install libdvdcss (you may wish to remove the old package first, if you have it).
sudo yum check-update
sudo yum install libdvdcss


That’s about it!

-c

In response to my article comparing Yum and Apt (at least I think it was my article, might have been someone else’s I guess), lead developer of Yum, Seth Vidal, wrote his own test script and performed some Yum benchmarks of his own.

He wrote:

Always a fun comparison. It’d be even more fun if any of the numbers seemed accurate.

His ran his test and concluded that Yum is “pretty good” and offers for others to run the test and post their results. So I did, on the same computer I used for the my article. I also compared the results to Ubuntu, as that’s really what my article was talking about

So what did I find?

Scripts
Firstly, here is the script he wrote:
http://skvidal.fedorapeople.org/misc/yum-test.sh.txt

Here is the script I modified for Ubuntu:
http://christophersmart.com/files/apt-test.sh

Preparation
Before running the tests I first deleted all old repo data as I wanted as clean a system as possible. I then ran “yum check-update” and “yum install -y –downloadonly zziplib-utils” on Fedora so that I had an up-to-date package database and the files locally. I did the same thing on Ubuntu (but obviously not with “yum”).

Then I uninstalled “yum-plugin-downloadonly” and “yum-plugin-download-remove-with-leaves” and rebooted, so that it would not negatively affect the speed of Yum having to load extra plugins outside of the default.

I also ran the tests in single user mode so that other system processes such as PackageKit would not interfere.

After I had run both scripts, I also re-ran each of the tests individually (which below I call “freshboot”), rebooting in between each one so that memory caching would not manipulate the results. After all, running all those commands one after another is bound to overlap and I don’t think provides an accurate benchmark of each package management component.

Results
Results for Fedora 11 are here, followed by results on a freshboot:
http://christophersmart.com/files/simple-yum-results.txt
http://christophersmart.com/files/simple-yum-results-freshboot.txt

Ubuntu 9.04 results are here, followed by results on a freshboot:
http://christophersmart.com/files/simple-apt-results.txt
http://christophersmart.com/files/simple-apt-results-freshboot.txt

Analysis

Search by keyword
Yum provided 5 results and took 4.746s
Apt provided 12 results and took 1.366s

Apt was 3.47 times faster.

Fresh boot
Yum took 3.380s
Apt took 1.240s

Apt was 2.73 times faster.
NOTE: I don’t know why this results were faster on a freshboot, they should be about the same as the first test was on a fresh boot anyway!

Search by file provide
Yum took 10.654s
Apt took 2.291s

Apt was 4.65 times faster.
NOTE: I have discovered that running “yum provides” builds a cache and so subsequent searches are much faster. Debian has two tools for this, apt-file and dpkg, but dpkg only works for installed packages while apt-file does the entire repository. Also, apt-file needs to be updated like apt-get does, so perhaps a better comparison is needed.

Fresh boot
Yum took 10.903s
Apt took 2.354s

Apt was 4.63 times faster.

List installed pkgs

Yum listed 1,092 packages and took 1.756s
Apt listed 1,188 packages and took 0.191s

Apt was 9.19 times faster.

Fresh boot
Yum took 3.924s
Apt took 0.187s

Apt was 20.98 times faster.

Install pkg + deps
Yum installed two programs and took 9.775s
Apt installed two programs and took 4.868s

Apt was 2.01 times faster.
NOTE: I didn’t install the same program for my tests, but in my article Apt was about 3 times faster.

Fresh boot
Yum took 14.097s
Apt took 5.671s

Apt was 2.49 times faster.

Removing package

Yum took 3.714s
Apt took 2.337s

Apt was 1.56 times faster.
NOTE: For the tests in my article I used yum with “–remove-leaves”, which checks to see if any other programs requires the dependencies of the package you’re using and if not, removes those too. This was not done here, which is most likely the cause of the big difference between these and the results in my article. In my article I used tomboy for my tests (after I had removed all mono applications and then re-installed tomboy) which had to remove 7 dependencies. The test used in this script is not really testing all that.

Fresh boot
Yum took 10.137s
Apt took 4.916s

Apt was 2.06 times faster.

Conclusion
Seth’s test seem to confirm my own findings, with the exception of installing and removing packages. Naturally I installed different packages than were used here and I also used the remove-with-leaves plugin for Yum to make it remove no-longer needed dependencies.

These two tests probably caused the most eyebrow raising (as in my article it shows Ubuntu as being 3 and 5 times faster, respectively) but they aren’t really addressed in Seth’s script. The results from his script are in Ubuntu’s favour to a factor of 2.5 and 2, respectively so it’s not crazily out of proportion. I’d be interested to see some results on this. Also, a better test for searching by provides would be useful.

Of course it is hard to compare these when they all use different packages and pull in different dependencies. This is why I also had a test to install from a file rather than the database. My test installed Skype via an rpm and deb (ugly I know, but it was the only 3rd party application I could find which provided native packages for both distros), which had zero dependencies and was not in the local database. In my tests Ubuntu was faster than Fedora by a factor of 7. I’d like to see this included in his script too.

It is interesting to see that Apt is faster still, when running from a fresh boot.

Never-the-less, I would invite any others who wish to test to do so also and let me know how it went. Sure, Yum might have bucket loads more metadata and maybe it can do lots more fancy things, but at the end of the day it still comes in at second place here.

And while I’m writing all this up, I want to clarify that I’m not a Yum hater. I do agree with Seth that Yum is “pretty good”, but I just wish it would be better. Actually, I also really like Fedora.

As I concluded in my original article, maybe no-one cares how fast or slow their package manager is. Then again, maybe some people do. Personally, I’d love to see Yum become the best package manager on the planet

What I now want to know are all the tweaks and configurations people do to their Yum systems to get better performance. Tips anyone?

I really like the Fedora project. Really. I like their stance on proprietary drivers and codecs (and of course free software) and these days they seem to be pushing the technological envelope more than others. Sure Red Hat drives the direction of the project somewhat, but I don’t mind Red Hat either.

In fact, I wish I could use Fedora as my main distro! But every time I try it I just get so frustrated with Yum. Sure it’s better than up2date, but it’s so damn slow and annoying. That’s a problem for someone like me who manually updates his package database first thing every morning and checks to see what packages are available and updates the system by hand. Why do I do that? Cause I like to.

But every time I’ve tried to get into Fedora that damn package manager has stopped me. I get frustrated after a day or so. I think the longest I’ve had it on was 2 days before I switched.

Recently I installed Fedora 10 and 11 to see if there was any performance increase. Actually, to tell you the truth I was completely surprised by Yum’s agility and speed. The old Fedora I remember was not to be seen.. or so it felt like anyway.

Hence, I thought it might be good to run some tests to see.

Of course as the article points out, does any of this matter? Do we really need a fast a nimble package manager? Well for me it matters. It matters a great deal. For most users though they probably won’t care, as they just let the package manager do its thing in the background.

Still, it makes for some interesting thoughts. I think.

-c