Update: I’ve tried to post my results back to Seth’s thread but it won’t work, so I’ve emailed him instead.
In response to my article comparing Yum and Apt (at least I think it was my article, might have been someone else’s I guess), lead developer of Yum, Seth Vidal, wrote his own test script and performed some Yum benchmarks of his own.
He wrote:
Always a fun comparison. It’d be even more fun if any of the numbers seemed accurate.
His ran his test and concluded that Yum is “pretty good” and offers for others to run the test and post their results. So I did, on the same computer I used for the my article. I also compared the results to Ubuntu, as that’s really what my article was talking about 🙂
So what did I find?
Scripts
Firstly, here is the script he wrote:
http://skvidal.fedorapeople.org/misc/yum-test.sh.txt
Here is the script I modified for Ubuntu:
http://christophersmart.com/files/apt-test.sh
Preparation
Before running the tests I first deleted all old repo data as I wanted as clean a system as possible. I then ran “yum check-update” and “yum install -y –downloadonly zziplib-utils” on Fedora so that I had an up-to-date package database and the files locally. I did the same thing on Ubuntu (but obviously not with “yum”).
Then I uninstalled “yum-plugin-downloadonly” and “yum-plugin-download-remove-with-leaves” and rebooted, so that it would not negatively affect the speed of Yum having to load extra plugins outside of the default.
I also ran the tests in single user mode so that other system processes such as PackageKit would not interfere.
After I had run both scripts, I also re-ran each of the tests individually (which below I call “freshboot”), rebooting in between each one so that memory caching would not manipulate the results. After all, running all those commands one after another is bound to overlap and I don’t think provides an accurate benchmark of each package management component.
Results
Results for Fedora 11 are here, followed by results on a freshboot:
http://christophersmart.com/files/simple-yum-results.txt
http://christophersmart.com/files/simple-yum-results-freshboot.txt
Ubuntu 9.04 results are here, followed by results on a freshboot:
http://christophersmart.com/files/simple-apt-results.txt
http://christophersmart.com/files/simple-apt-results-freshboot.txt
Analysis
Search by keyword
Yum provided 5 results and took 4.746s
Apt provided 12 results and took 1.366s
Apt was 3.47 times faster.
Fresh boot
Yum took 3.380s
Apt took 1.240s
Apt was 2.73 times faster.
NOTE: I don’t know why this results were faster on a freshboot, they should be about the same as the first test was on a fresh boot anyway!
Search by file provide
Yum took 10.654s
Apt took 2.291s
Apt was 4.65 times faster.
NOTE: I have discovered that running “yum provides” builds a cache and so subsequent searches are much faster. Debian has two tools for this, apt-file and dpkg, but dpkg only works for installed packages while apt-file does the entire repository. Also, apt-file needs to be updated like apt-get does, so perhaps a better comparison is needed.
Fresh boot
Yum took 10.903s
Apt took 2.354s
Apt was 4.63 times faster.
List installed pkgs
Yum listed 1,092 packages and took 1.756s
Apt listed 1,188 packages and took 0.191s
Apt was 9.19 times faster.
Fresh boot
Yum took 3.924s
Apt took 0.187s
Apt was 20.98 times faster.
Install pkg + deps
Yum installed two programs and took 9.775s
Apt installed two programs and took 4.868s
Apt was 2.01 times faster.
NOTE: I didn’t install the same program for my tests, but in my article Apt was about 3 times faster.
Fresh boot
Yum took 14.097s
Apt took 5.671s
Apt was 2.49 times faster.
Removing package
Yum took 3.714s
Apt took 2.337s
Apt was 1.56 times faster.
NOTE: For the tests in my article I used yum with “–remove-leaves”, which checks to see if any other programs requires the dependencies of the package you’re using and if not, removes those too. This was not done here, which is most likely the cause of the big difference between these and the results in my article. In my article I used tomboy for my tests (after I had removed all mono applications and then re-installed tomboy) which had to remove 7 dependencies. The test used in this script is not really testing all that.
Fresh boot
Yum took 10.137s
Apt took 4.916s
Apt was 2.06 times faster.
Conclusion
Seth’s test seem to confirm my own findings, with the exception of installing and removing packages. Naturally I installed different packages than were used here and I also used the remove-with-leaves plugin for Yum to make it remove no-longer needed dependencies.
These two tests probably caused the most eyebrow raising (as in my article it shows Ubuntu as being 3 and 5 times faster, respectively) but they aren’t really addressed in Seth’s script. The results from his script are in Ubuntu’s favour to a factor of 2.5 and 2, respectively so it’s not crazily out of proportion. I’d be interested to see some results on this. Also, a better test for searching by provides would be useful.
Of course it is hard to compare these when they all use different packages and pull in different dependencies. This is why I also had a test to install from a file rather than the database. My test installed Skype via an rpm and deb (ugly I know, but it was the only 3rd party application I could find which provided native packages for both distros), which had zero dependencies and was not in the local database. In my tests Ubuntu was faster than Fedora by a factor of 7. I’d like to see this included in his script too.
It is interesting to see that Apt is faster still, when running from a fresh boot.
Never-the-less, I would invite any others who wish to test to do so also and let me know how it went. Sure, Yum might have bucket loads more metadata and maybe it can do lots more fancy things, but at the end of the day it still comes in at second place here.
And while I’m writing all this up, I want to clarify that I’m not a Yum hater. I do agree with Seth that Yum is “pretty good”, but I just wish it would be better. Actually, I also really like Fedora.
As I concluded in my original article, maybe no-one cares how fast or slow their package manager is. Then again, maybe some people do. Personally, I’d love to see Yum become the best package manager on the planet 🙂
What I now want to know are all the tweaks and configurations people do to their Yum systems to get better performance. Tips anyone?
