udevd problems in 6.2.X? (kernel 2.6.32.6)

[Capricorny]

I have had a couple of instances of udevd losing control over itself lately. I'm not sure whether it is due to bugs or read/write errors, but it gets confused, uses all the CPU and memory, kills programs to free up memory and makes the system almost unusable. I have killed it, and restarted the daemon by issuing sudo udevd --daemon &. Everything seems to work just fine. But I'm curious as to what this is? I'm not able to see anything leading up to the situation in dmesg output. Anyone experiencing the same or similar?

[krishna.murphy]

I have had a couple of instances of udevd losing control over itself lately. I'm not sure whether it is due to bugs or read/write errors, but it gets confused, uses all the CPU and memory, kills programs to free up memory and makes the system almost unusable. I have killed it, and restarted the daemon by issuing sudo udevd --daemon &. Everything seems to work just fine. But I'm curious as to what this is? I'm not able to see anything leading up to the situation in dmesg output. Anyone experiencing the same or similar?

Sounds pretty bad. Any chance this could be hardware? I'm particularly suspecting memory. There were problems awhile back with "cosmic rays" (high energy gamma particles, actually) causing memory cells to change state, leading to symptoms that could be compared to these (but more random.) If there's one bad bit that will be switched under circumstances you can't nail down, it might produce this effect. Anyway, you've found the trick to get it back under control again - good work, and thanks for sharing.

Cheers!
Krishna :mrgreen:
p.s. I'm sure you know already, but for the benefit of others reading this, there is a memory test available at boot-time by typing at the prompt:
memtest

[klaus2008]

I never experienced this problem. If the udevd problem occurs periodically one could restart udevd as a cron job. Instead of killing and starting udevd manually, I would use the shell script /etc/init.d/udev.
sudo /etc/init.d/udev restart In the configuration file /etc/udev/udev.conf there is a line setting the log priority. Using the value debug instead of err could help to find an answer. The program udevadm lets you watch and control the activity of the udev daemon:
sudo udevadm monitor Since udevadm is a very complex program it seems useful to read its man page.

[Capricorny]

Thank you very much!
I can now rule out my initial guess, KNOPPIX file corruption, as I have checked the md5sums from the image I use and a freshly downloaded, loop-mounted ISO image.
The results of memtest are somewhat confusing, as it aborts when run with 2x2GB modules, but finds no errors with either of them (ASUS UL30VT.
I still suspect there is some udevd bug involved, as I had similar behavior on a Dell Vostro.
Can conflicts between /etc/fstab and /etc/mtab confuse udevd? I have one, /media/sdaX is mounted as /store/local. In any case, I think restarting udevd when system is fully set up may be a good idea.

[Capricorny]

Editing /etc/fstab and restarting does not seem to work. The simplest workaround seems to be to start and stop udevd as needed.

[kl522]

Some of these problems are really nasty. A quick check in the internet revealed that there are quite a few of this same problem reported. Some says it's due to the CDROM, but there is probably no single cause for such misbehavior, so I guess 'udevadm' is probably the best choice for checking root cause. Upgrade the udev version might be an easier thing to try.

Fortunately I have never have to deal with this problem. However, I do encounter some other nasty problems though. They are widely reported problems, so have nothing to do with knoppix per se.

[Capricorny]

It's definitely not the CDROM here, the UL30VT hasn't got a optical drive at all. It seems to crop up in different ways, last time udevd went crazy it happened while downloading to an ordinarily mounted HD partition with a perfect ext3 filsystem on it. Just had to kill it before it started killing everything else. And the drive was correctly listed in /etc/fstab.

I'm not terribly interested in getting to the roots of this, as they seem to be multiple, and the basic architecture may have some flaws - I interpret some kernel dev discussions that way.
Basically, we only need the hotplug manager when things are plugged in and out, otherwise it can rest. Therefore, a safe and simple workaround may be to let it autostart, but stop the daemon when system is set up fine, and only start it when changes are made. And if udevd takes 1% or more of CPU in top's listing, it may be an indication of malfunctioning. Then, it may be safest to stop it (sudo /etc/init.d udev stop) and only restart it when needed.

A program like udevadm is surely useful for debugging, and for system administration when a lot of complex hotplugging is going on. But it should not be necessary to use when no hotplugging is going on at all! ;)

[Capricorny]

Update: Leaving udevd turned off for an extended period of time may lead to losing contact with peripherals, even if no configuration changes are made. So, even though 1-2% of CPU certainly shouldn't be necessary to monitor a system that is not changing, it may be better to have udevd running, and only stop it when it runs wild. (Which happens quite often to me...)

[krishna.murphy]

Have you tested for hardware faults, i.e. bad memory?This is "passing strange" - maybe it's time to search bug reports for udev.

[Capricorny]

As kl522 relates, there are lots of reports about udevd problems. Maybe hardware related, too. I wrote about the conflicting memtest results above. But hardware problems should show up elsewhere too, I think. And here, it seems that everything is working fine, except for udevd. I start wondering if there is some kind of slight incompatibility between Knoppix and udevd in the kernel version 2.6.1 uses.

[krishna.murphy]

As kl522 relates, there are lots of reports about udevd problems. Maybe hardware related, too. I wrote about the conflicting memtest results above. But hardware problems should show up elsewhere too, I think. And here, it seems that everything is working fine, except for udevd. I start wondering if there is some kind of slight incompatibility between Knoppix and udevd in the kernel version 2.6.1 uses.

What is 2.6.1 above? If it's the kernel, that conflicts with the starting premise embodied in the title. Perhaps you can down/up-grade udev, anyway, and see if it changes things.

Cheers!
Krishna :mrgreen:

[Capricorny]

knoppix@Microknoppix:~$ uname -a
Linux Microknoppix 2.6.32.6 #8 SMP PREEMPT Thu Jan 28 10:51:16 CET 2010 i686 GNU/Linux
Meant of course 6.2.1 with the standard kernel.

Haven't done any modifications at all - I might try to upgrade the kernel to see if the problems persist.
But, basically, I don't want to know about such things.. :)

[krishna.murphy]

knoppix@Microknoppix:~$ uname -a
Linux Microknoppix 2.6.32.6 #8 SMP PREEMPT Thu Jan 28 10:51:16 CET 2010 i686 GNU/Linux
Meant of course 6.2.1 with the standard kernel.

Haven't done any modifications at all - I might try to upgrade the kernel to see if the problems persist.
But, basically, I don't want to know about such things.. :)

Okay. I try not to make a lot of assumptions, and I was just confused.

If it was my system, I'd probably just ignore it and restart as needed - but if I had a less technical person in the place who needed to be able to deal with it, it would be better if I made a script to do it and hooked that to something on the toolbar.

Cheers!
Krishna :mrgreen: