Monthly Archive for November, 2010

Miracle patch out-performed by four lines of bash

Well it’s nothing if not interesting. The ~200 line “miracle” patch promises a huge improvement in desktop responsiveness, but Lennart Poettering (from Red Hat) has replied to Linus’ praising comments with 4 lines of bash which out-perform it, on the current vanilla kernel. (And by the way, Lennart seems to agree with Con Kolivas about this patch, in that make -j is not a valid desktop use case.)

Lennart said:

Binding something like this to TTYs is just backwards. No graphical
session has a TTY attached anymore. And there might be multiple TTYs
used in the same session.

I really wonder why logic like this should live in kernel space at all,
since a) the kernel has no real notion of a session, except audit and b)
this is policy and as soon as people have this kind of group then they
probably want other kind of autogrouping as well for the other
controllers, which hence means userspace is a better, and configurable
place for this.

To which Linus then replied:

Numbers talk, bullshit walks.

The numbers have been quoted. The clear interactive behavior has been seen.

And you’re just full of bullshit.

Come back when you have something working and with numbers and better
interactive performance. Until then, nobody cares.

So, Lennart replied with his something better, 4 lines of bash:

Here’s my super-complex patch btw, to achieve exactly the same thing
from userspace without involving any kernel or systemd patching and
kernel-side logic. Simply edit your own ~/.bashrc and add this to the end:

if [ "$PS1" ] ; then
mkdir -m 0700 /sys/fs/cgroup/cpu/user/$$
echo $$ > /sys/fs/cgroup/cpu/user/$$/tasks
fi

Then, as the superuser do this:

mount -t cgroup cgroup /sys/fs/cgroup/cpu -o cpu
mkdir -m 0777 /sys/fs/cgroup/cpu/user

Done. Same effect. However: not crazy.

Of course the debate raged on because Linus thinks it’s stupid to make people have to set this up in userspace, and Lennart disagrees. Despite all this, finally everyone is at least acknowledging these issues and looking into them! Excellent.

-c

Using the serial port on an Apple Xserve

Some have found my posts on running Linux on the (EFI only) Xserves useful, and one such person is Eddie, who emailed me with some details on how he got the serial port working. Hopefully it will be useful to someone else, so with his permission, the following is what he sent me:

Believe it or not, Apple screwed up the pin-outs on the back of the DB-9 port for the Xserve (it might have been a logic board design error and perhaps they didn’t want to spend the money to re-design it). I made my own null modem cable and got it to work with OS X Server (taking the other end of the null modem cable over to a USB-to-DB9 adapter and then plugged it into my PC running Windows 7). As for how Ubuntu handles serial ports including on Xserves, I have some good news. Today, after much digging around, I found out how this works. First of all the GRUB load / configuration that you helped put together and document for the Xserve recognizes the built-in serial port (hurray!). This built in serial port is recognized as /dev/ttyS0. For example, from my Ubuntu 9.10 kern.log file:

$ cat /var/log/kern.log | grep ttyS
Nov 16 09:34:38 ubuntu kernel: [ 1.140244] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Nov 16 09:34:38 ubuntu kernel: [ 1.140343] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Nov 16 09:34:38 ubuntu kernel: [ 1.140720] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

I’m not sure, however, what ttyS1 is on the Xserve (as you can see in the output above).

Another way to look at this:

root@ubuntu:~# dmesg | grep serial
[ 1.140244] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 1.140343] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

Ubuntu specifically made changes to how one goes about configuring from one major version to another (so this is going to also probably differ among Linux distros such as Red Hat) but for what its worth, under 9.10 you have to set up a ttyS0 file and place it in /etc/init/ and the contents should be the following:

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L 115200 ttyS0 vt102

Once this /dev/ttyS0 file has been created and in place, you can load it like this:

# start ttyS0

which, if successful, should output a line like this:

ttyS0 start/running, process 4478

You can also read the device like this:

# stty -F /dev/ttyS0 -a

and expect output as follows:

speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke

I had no problem firing up PuTTY on a laptop running Windows 7 and connecting to the Xserve running Ubuntu. I could log in just as I would be able to at any console whether via KVM or SSH.

Great stuff! Another option to keep our Xserves going into perpetuity especially since Apple is dropping them in a few months!

Con Kolivas posts his thoughts on the new ~200 line kernel patch

Con Kolivas (of SD/BFS fame) has posted his thoughts on the new ~200 line “miracle” kernel patch. It’s an interesting read.

In short, he had already implemented something like this in a 10 line patch to his BFS scheduler, but he dropped them because it introduced regressions.

Those following the development of the patches for interactivity at massive load, I have COMPLETELY DROPPED them as they introduce regressions at normal workloads, and I cannot under any circumstances approve changes to improve behaviour at ridiculous workloads which affect regular ones. I still see precisely zero point at optimising for absurd workloads. Proving how many un-niced jobs you can throw at your kernel compiles is not a measure of one’s prowess. It is just a mindless test.

He goes on to say:

Again, I can’t for the life of me see why you’d optimise for make -j64 on a quad core machine. It is one workload, unique to people who compile all the time, but done in a way you wouldn’t normally do it anyway. It is not going to magically make anything else better. If for some god-forsaken reason you wanted to do that, you could already do that with nice, or even better, by running it SCHED_IDLEPRIO.

Perhaps this patch really helps the existing CFS implementation, but it’s still lacking when compared to BFS. Maybe it is a step in the wrong direction, but perhaps some improvement like this is better than none at all? It might be the catalyst needed to improve the kernel further. I never could understand why we couldn’t have more than one CPU scheduler in the kernel (like we do for block I/O).

Anyway, this is all quite interesting!

-c

~200 line miracle Linux patch

There’s been lots of criticism from certain camps that Linus and the kernel developers don’t care enough about the desktop and that it has too much focus on big iron. There certainly have been (and continue to be) issues with Linux on the desktop, in terms of performance. Lately there has been a push to improve it, perhaps driven by the need to have Linux running well on small devices like Android and MeeGo.

Or perhaps it stems from much further back. We all remember anaesthetist Con Kolivas’s kernel patches (Kororaa used -ck, back in the day) and schedulers which improved desktop performance a whole lot, but in the end he was shunned, and then quit development. Linus wasn’t convinced, and Ingo Molnár (the CPU scheduler maintainer) then wrote a new scheduler which was more like Con’s (ironically named, Completely Fair Scheduler). Although he quit, two years later Con was still using Linux and couldn’t stand the desktop responsiveness any more, so he created a new scheduler, BFS (with no intention if getting it into the mainstream kernel).

Why is this important? Well now someone else is having a go. The small 200 line patch by Mike Galbrait has such a tremendous impact that even Linus can’t argue with the result.

If you want to see it in action, Phoronix has an article about the patch along with some videos showing the dramatic difference.

“Tests done by Mike show the maximum latency dropping by over ten times and the average latency of the desktop by about 60 times.”

With this patch, the desktop (running GNOME) can smoothly play the Ogg 1080p version of Big Buck Bunny and glxgears at the same time, while scrolling up and down a Firefox window and.. wait for it.. performing a kernel compile with make -j64.

xkcd Supported Features
xkcd, “Supported Features”

I don’t think that this will be merged until 2.6.38 as the 2.6.37 window has already closed.

-c

How to upgrade Fedora (using PreUpgrade)

Coming from a Debian background, upgrading to new releases is second nature. Unfortunately, Fedora doesn’t seem to have it quite as down pat. Nevertheless, it’s possible and works quite well (but there are pitfalls, so read on).

While upgrading via PackageKit is coming, there’s a pretty decent way of doing it with PreUpgrade.

For the impatient, the steps are:

  • yum update
  • yum install preupgrade
  • preupgrade (reboot)
  • yum distro-sync
  • package-cleanup –orphans (be careful!)
  • package-cleanup –leaves (be careful!)

Now the longer version. First, update your current system:
yum update

Install preupgrade:
yum install preupgrade

Run the preupgrade tool, and follow the prompts (remote upgrades over VNC also supported):
preupgrade

PreUpgrade will prompt to reboot your computer and will update your system automatically. All of your packages should be updated and your repositories configured for you (I also had RPMFusion, Chromium and yum-rawhide which all updated as expected). On one of my systems, even the NVIDIA driver was automatically updated and configured. Now you’re booted into your new Fedora 14 system.

OK, this is where things get a little tricky. Some packages might be no-longer supported, there might be removed dependencies and the like. So, there are two neat commands that we will use to identify these packages, so that you can remove them.

package-cleanup --orphans
package-cleanup --leaves

But wait.. I discovered that there’s a more important step to do first, or you’ll cause yourself a headache!

On my machine, this wanted to remove some pretty important packages, like kdelibs.
[chris@shem ~]$ sudo package-cleanup --orphans
[sudo] password for chris:
Loaded plugins: refresh-packagekit, remove-with-leaves
akonadi-1.4.0-3.fc13.x86_64
kdeedu-marble-4.5.2-2.fc13.x86_64
kdeedu-marble-libs-4.5.2-2.fc13.x86_64
kdelibs-4.5.2-8.fc13.x86_64
kdelibs-common-4.5.2-8.fc13.x86_64
schroedinger-1.0.10-1.fc13.x86_64
xorg-x11-drv-wacom-0.10.8-2.fc13.x86_64

Fedora (unlike certain other distros I know) provides major updates to the latest versions of packages (like OpenOffice.org, KDE, GNOME, etc). Yay! It’s possible that your new Fedora system is running the same version of, say KDE, as your old one. This was indeed the case with my Fedora 13 to 14 upgrade (both run KDE 4.5.2 at time of F14 release).

Actually, the build of some important packages on Fedora 13 (like kdelibs-4.5.2-8.fc13.x86_64) were actually more recent than on the newer Fedora 14 system (kdelibs-4.5.2-5.fc14.x86_64).

The package-cleanup tool correctly lists kdelibs-4.5.2-8.fc13.x86_64 as being an orphan and if you were to remove this, you’d break your system badly. In fact, if you’re running yum’s brilliant new autoremove deps feature, as I am, you’ll lose most of your system. It makes sense – you are telling yum to remove kdelibs, so it goes and removes everything that relies on it! Yikes.

So first, we need to fix this by running the yum distro-sync command which recognises that we need to downgrade that kdelibs packages (and not remove it!).

[chris@shem ~]$ sudo yum distro-sync
Loaded plugins: refresh-packagekit, remove-with-leaves
Setting up Distribution Synchronization Process
Resolving Dependencies
--> Running transaction check
---> Package akonadi.x86_64 0:1.4.0-1.fc14 will be a downgrade
---> Package akonadi.x86_64 0:1.4.0-3.fc13 will be erased
---> Package kdeedu-marble.x86_64 0:4.5.2-1.fc14 will be a downgrade
---> Package kdeedu-marble.x86_64 0:4.5.2-2.fc13 will be erased
---> Package kdeedu-marble-libs.x86_64 0:4.5.2-1.fc14 will be a downgrade
---> Package kdeedu-marble-libs.x86_64 0:4.5.2-2.fc13 will be erased
---> Package kdelibs.x86_64 6:4.5.2-5.fc14 will be a downgrade
---> Package kdelibs.x86_64 6:4.5.2-8.fc13 will be erased
---> Package kdelibs-common.x86_64 6:4.5.2-5.fc14 will be a downgrade
---> Package kdelibs-common.x86_64 6:4.5.2-8.fc13 will be erased
---> Package schroedinger.x86_64 0:1.0.9-2.fc14 will be a downgrade
---> Package schroedinger.x86_64 0:1.0.10-1.fc13 will be erased
---> Package xorg-x11-drv-wacom.x86_64 0:0.10.8-1.20100726.fc14 will be a downgrade
---> Package xorg-x11-drv-wacom.x86_64 0:0.10.8-2.fc13 will be erased
--> Finished Dependency Resolution
--> Finding unneeded leftover dependencies
Found and removing 0 unneeded dependencies
----
Dependencies Resolved
=================================================
Package Arch Version Repository Size
=================================================
Downgrading:
akonadi x86_64 1.4.0-1.fc14 fedora 677 k
kdeedu-marble x86_64 4.5.2-1.fc14 fedora 14 M
kdeedu-marble-libs x86_64 4.5.2-1.fc14 fedora 902 k
kdelibs x86_64 6:4.5.2-5.fc14 fedora 12 M
kdelibs-common x86_64 6:4.5.2-5.fc14 fedora 1.7 M
schroedinger x86_64 1.0.9-2.fc14 fedora 276 k
xorg-x11-drv-wacom x86_64 0.10.8-1.20100726.fc14 fedora 73 k
---
Transaction Summary
=============================================================================================================================================
Downgrade 7 Package(s)
---
Total download size: 30 M
Is this ok [y/N]

Once we downgrade these packages, we can then remove any other orphans we might have, with package-cleanup –leaves and package-cleanup –orphans (I didn’t have any). One last thing to note – Fedora will not replace packages from the newer release if they are exactly the same version. For this reason, you will probably have some F13 packages still installed on your computer – don’t worry, that’s correct. They will be upgraded in time (if required).

So, now I think I know how to successfully upgrade the system without breaking it :-) If anyone has some other tips or corrections, please let me know! Hope that’s helpful.

-c

Testing yum’s autoremove orphaned deps feature

Yesterday I wrote about yum’s new feature which can automatically remove unused dependencies when a package is uninstalled.

Today I got my hands on a suitable build of yum (add the yum-rawhide repo and set clean_requirements_on_remove=1 under [main] in /etc/yum.conf) and I started to test it out on several packages, all of which introduced dependencies. My initial findings? It’s great.

Single package with dependencies
I started with a single package with several dependencies. When I uninstalled the parent package, yum also removed all of its dependencies which were not needed by any other package (i.e. orphaned dependencies).

Let’s look at an example; installing Ekiga which pulls in 5 dependencies.

yum install ekiga
...
Installed:
ekiga.x86_64 0:3.2.7-4.fc14
Dependency Installed:
evolution-data-server.x86_64 0:2.32.0-3.fc14 libgdata.x86_64 0:0.6.4-4.fc14 libgweather.x86_64 0:2.30.3-1.fc14 opal.x86_64 0:3.6.8-1.fc14
ptlib.x86_64 0:2.6.7-1.fc14

OK, so we have Ekiga, and we have 5 dependencies installed. What happens if I remove Ekiga?

yum erase ekiga
...
Removed:
ekiga.x86_64 0:3.2.7-4.fc14
Dependency Removed:
evolution-data-server.x86_64 0:2.32.0-3.fc14 libgdata.x86_64 0:0.6.4-4.fc14 libgweather.x86_64 0:2.30.3-1.fc14 opal.x86_64 0:3.6.8-1.fc14
ptlib.x86_64 0:2.6.7-1.fc14

As I was hoping, yum removed all 5 dependencies along with Ekiga itself. Ta da!

Result: Tick!

Packages with shared dependencies
I then tested two packages which have shared dependencies. What I would expect is that only the dependencies which are unique to the application being removed, are uninstalled. Other shared dependencies remain, because the other program still requires them and is not being removed.

Let’s look at an example; mplayer and vlc. Both require the libcaca library and installing both packages pulled in this library.
VLC needs libcaca:

yum deplist vlc |grep libcaca
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14

And mplayer needs libcaca:

yum deplist mplayer |grep libcaca
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14
dependency: libcaca.so.0()(64bit)
provider: libcaca.x86_64 0.99-0.10.beta17.fc14

When I want to remove mplayer, yum only removes those mplayer dependencies which vlc does not also require, and leaves the rest (including libcaca).


yum erase mplayer
...
Removed:
mplayer.x86_64 0:1.0-0.119.20100703svn.fc14
Dependency Removed:
libvdpau.x86_64 0:0.4.1-1.fc14.1 lzo.x86_64 0:2.03-3.fc12 mplayer-common.x86_64 0:1.0-0.119.20100703svn.fc14

Note that libcaca was not removed, which is the expected result.

Result: Tick!

Removing a package’s dependency
I then tried to remove a dependency, rather than the parent package. I would expect this to remove all parents which required that package, along with their other dependencies.

Let’s look at an example, gnash-plugin. This package pulls in three dependencies (including the actual gnash package).


yum install gnash-plugin
...
Installed:
gnash-plugin.x86_64 1:0.8.8-4.fc14
Dependency Installed:
agg.x86_64 0:2.5-9.fc13 gnash.x86_64 1:0.8.8-4.fc14 gtkglext-libs.x86_64 0:1.2.0-10.fc12

Now, let’s see what happens if I try and remove one of the dependencies.

yum erase agg
...
Removed:
agg.x86_64 0:2.5-9.fc13
Dependency Removed:
gnash.x86_64 1:0.8.8-4.fc14 gnash-plugin.x86_64 1:0.8.8-4.fc14 gtkglext-libs.x86_64 0:1.2.0-10.fc12

As you can see, it correctly removed the parent package, which could no-longer operate without the dependency.

This is such a great advancement over remove-leaves, where last time I tried to remove gnash-plugin (after testing it out, I didn’t want it) yum wanted to remove Firefox!! Fail.

Result: Tick!

Suggestion – clean up existing system
One thing I would like, is the ability to run a system wide cleanup using this code in yum, rather than other rpm orphan tools. Sort-of like an apt-get autoremove. Theoretically, if you’ve always removed any packages on your system with this new feature then you should be right, but it would be nice to sort-of do a clean up at any point (on any system) and get yum to remove any orphaned deps for you. Maybe this already exists.

If you want to clean up existing orphaned dependencies, then package-cleanup is your friend (from yum-utils).

Take a look at all the packages that are orphaned, and make sure it is sane.

package-cleanup --leaves

If you’re happy with the list, you can just remove them all (after you’ve enabled yum’s new autoremove feature, of course!):


yum erase $(package-cleanup --leaves)

You can instantly see the benefit of this new feature – on my machine this command also removes brand new orphaned dependencies, created by the orphans I’m removing! That’s 59 packages in total.

Without this yum feature, it would only remove 36 packages. You would need to run package-cleanup several times, until you’ve really removed all orphaned packages.

Similarly, this works for package-cleanup –orphans but it would be great it this was built into a single yum cleanup type function – one command to do it all.

Summary
From my initial testing, this works really well, although I’ll continue to test it and see how we go. Hopefully this will find its way into Fedora 12, 13, and 14 rather than just 15.

I’m so grateful for this work – it’s something I’ve been crying out for ever since I made the decision to stick with Fedora. Thank you, thank you, thank you!!

Yum gets autoremove dependency feature

Coming from a Gentoo/Debian background, one thing that has constantly bugged me on RPM based platforms like Fedora is the lack of decent, reliable dependency removal.

It seems so simple (and Debian has done it since the dawn of time) – if I install package x which pulls in dependencies y and z, then when I remove package x, I want to remove dependencies y and z, if they are not required by any other package.

Yes, there is the remove-leaves plugin for Yum and various RPM orphan checking tools, but in my experience, they are just not reliable.

So, I’m very happy to have discovered that Seth Vidal has merged orphaned dependency cleanup on removal into Yum. Hallelujah!

It’s in rawhide yum-3.2.28-13, and I’ll do some testing soon…

Anandtech takes a look at AMD’s new “Atom killing platform”

Anandtech takes a hands-on look at AMD’s new CPU+GPU combination platform, Brazos. It’s designed to compete in the low performance power, and mobile spaces.

How well will it perform, especially in comparison to next-gen Atom SOC, Sandy Bridge, or NVIDIA’s ION platform? Not sure yet, but one thing I know, it will be practically useless for Linux without decent driver support.

Update: Looks like some open source drivers are on the way..