[PATCH] Faster hardware detection!

[Repzilon]

I downloaded and booted PCLinuxOS 2004 preview 3 live CD and its hardware detection system is very fast! It is faster than my port of Knoppix auto-configuration on my main Mandrake system (For your information: I still work on it, but the project stalled for a month because of data destruction brought to you by Microsoft Windows 98).

I was curious. I needed to know why it is so fast. I found.

Auto-configuration in Knoppix and PCLinuxOS
Knoppix relies on a custom written Bash boot script, knoppix-autoconfig, which calls many other Bash scripts, like scanpartitions, rebuildfstab, hotplug-knoppix, mkxf86config, alsa_autoconfig and a libkudzu powered program, hwsetup. On the other hand, PCLinuxOS uses a Perl script, called hwdetect, placed in the initrd. It uses the HardDrake library Perl bindings, and does not call other scripts. Yes I know, there is also a custom linuxrc in the Knoppix initrd, but only to probe SCSI devices.

Why PCLinuxOS detection is fast?
Part of the answer is in Perl. Most of the scripts called by knoppix-autoconfig are used to generate configuration files. Reading and writing files is not Bash speciality, but Perl's. Also, Bash is a slow interpreter, it is written in its manual page. Moreover, Knoppix scripts often uses regular expressions via external programs (grep, awk) and the few support Bash provides built-in (case conditions). Regular expressions are built-in in Perl (and probably one of its most powerful and frightening features), hence faster.

Also, external calls cause performance hits and increase memory usage. hwdetect does not have this problem as it does not call external programs or scripts.

HardDrake? I'm not sure. I admit that I am not a big fan of it, even if I still use Mandrake 9.0 as my main Linux system. It's because of my disappointment about HardDrake that I started to port Knoppix auto-configuration after all.

Suggested ideas
I suggest to proceed by two parallel ways: a incremental rewrite of the current scripts in Perl and a complete rewrite. There is also the idea of using hwdetect, but it raises two problems. Is HardDrake available for Debian? Would Klaus Knopper be furious knowing his efforts going to garbage?

I prefer to work on the incremental rewrite path, because of the maintenance of the port. I would

1. Release Knoppix autoconfig port 0.2.5 .
2. merge scanpartions and rebuildfstab into rebuildfstab.pl
3. edit rebuildfstab.pl to detect removable drives (Installed distros port specific. At this moment, I use updfstab utility from Kudzu)
4. merge mkxf86config and ktbl2xkbd.sh into mkxf86config.pl (Port specific. ktbl2xkbd.sh converts keytable code in /etc/sysconfig/keyboard into a XFree86 Keyboard code, because lang= does not make sense for an installed distro)
5. convert alsa_autoconfig in Perl
6. convert my xfrepzi.sh to xfrepzi.pl (this is my XFree86 configuration utility)
7. move hardware detection handling parts from knoppix-autoconfig to a Perl script.
8. as I move along more code to the Perl script, merge the other Perl scripts in it.

Configuration files should be written to a ramdisk first, then copied to their final locations.


I need that Camel book fast!

[Repzilon]

1. Release Knoppix autoconfig port 0.2.5 .
2. merge scanpartions and rebuildfstab into rebuildfstab.pl
3. edit rebuildfstab.pl to detect removable drives (Installed distros port specific. At this moment, I use updfstab utility from Kudzu)
4. merge mkxf86config and ktbl2xkbd.sh into mkxf86config.pl (Port specific. ktbl2xkbd.sh converts keytable code in /etc/sysconfig/keyboard into a XFree86 Keyboard code, because lang= does not make sense for an installed distro)
5. convert alsa_autoconfig in Perl
6. convert my xfrepzi.sh to xfrepzi.pl (this is my XFree86 configuration utility)
7. move hardware detection handling parts from knoppix-autoconfig to a Perl script.
8. as I move along more code to the Perl script, merge the other Perl scripts in it.

Task 1 done (late December, Christmas gift), and task 4 partly done (merged ktbl2xkbd.sh, but not converted mkxf86config to Perl).

During the holidays, I tried two other ways to accelerate the auto-configuration. For your information, the startup of my main computer at home with auto-configuration port version 0.2.5-Mandrake is around 55 seconds, from LILO to login screen in run level 2. Without Knoppix auto-configuration port (I provide now an uninstaller for it), bootup takes 30 seconds, from LILO to login screen in run level 2.

One way was running multiple processes that do not require a lot of CPU and that are independent each other at the same time. It required to break the rc.sysinit script into several pieces and launching each part in the background (easily done in bash). The is also a simple mechanism to ensure that all scripts finished their run. I have been able to save 10 seconds but only the half of hardware was configured.

I realized that the parallel execution system was just working around the slow hwsetup. Checking its source, I localized the slow function: probeDevices() from libkudzu. When I went to the kudzu source code, it was too complicated for me (I never touched pointer to functions, for example). Then, I returned to hwsetup sources. updfstab takes only one second to run and its source code contains also a call to probeDevices(). The performance problem is caused by the parameters passed. So I tried a more refined choice than trying to detect the entire world. It did not work. I finally looked at the third parameter, with value PROBE_ALL. I rechecked device.h in kudzu sources and tried PROBE_SAFE. The execution time went from 15s to 0.4s

I implemented the PROBE_SAFE parameter as a command line option. Watch out, any serial device (mouse, modem) and PS/2 mice will NOT work. That's the point of putting it as an option.

--- hwsetup.c_1.0-14	2004-01-04 09:43:52.000000000 -0500
+++ hwsetup.c	2004-01-04 11:55:41.000000000 -0500
@@ -3,6 +3,7 @@
 * loads modules, generates /dev links, no isapnp autoconfiguration (yet)     *
 * needs kudzu-devel (ver. 0.23 and up)                                       *
 * Author: Klaus Knopper <knopper@knopper.net>                                *
+* Modifications: René Rhéaume <rener@mediom.qc.ca>                           *
 \****************************************************************************/
 
 #include <stdio.h>
@@ -22,7 +23,8 @@
 #endif
 
 #define VERSION "HWSETUP 1.0, an automatic hardware configuration tool\n" \
-                "(C) 2002 Klaus Knopper <knoppix@knopper.net>\n\n"
+                "(C) 2002 Klaus Knopper <knoppix@knopper.net>\n" \
+                "Modifications (C) 2004 René Rhéaume <rener@mediom.qc.ca>\n"
 
 #define CARDSDB "/usr/share/hwdata/Cards"
 #define XPATH "/usr/X11R6/bin/"
@@ -51,14 +53,15 @@
 
 int syntax(char *option)
 {
- printf(VERSION);
+ puts(VERSION);
  if(option) fprintf(stderr,"hwsetup: Unknown option '%s'\n\n",option);
- printf("Usage: hwsetup\n"
-         "               -v       be verbose\n"
-         "               -p       print rotating prompt\n"
-         "               -a       ignore audio devices\n"
-         "               -s       ignore scsi controllers\n"
-         "               -n       probe only, don't configure anything.\n");
+ puts("Usage: hwsetup\n"
+         "\t\t-v\tbe verbose\n"
+         "\t\t-p\tprint rotating prompt\n"
+         "\t\t-a\tignore audio devices\n"
+         "\t\t-s\tignore scsi controllers\n"
+         "\t\t-n\tprobe only, don't configure anything\n"
+         "\t\t-f\trun a fast probe.");
  return option?1:0;
 }
 
@@ -336,7 +339,7 @@
 {
  signal(SIGALRM,SIG_IGN);
  fprintf(stderr,"\nWARNING: Autodetection seems to hang,\n"
-                "please check your computers BIOS settings.\n");
+                "please check your computer BIOS settings.\n");
  fflush(stderr);
  if(wpid) { kill(wpid,SIGTERM); usleep(2500000); kill(wpid,SIGKILL); wpid=0; }
  exit(1); /* exit program */
@@ -496,12 +499,13 @@
  return 0;
 }
 
-int hw_setup(enum deviceClass dc, int verbose, int probeonly, int skip)
+int hw_setup(enum deviceClass dc, int verbose, int probeonly, int skip, int fastprobe)
 {
  int i,mouse=0,cdrom=0,modem=0,scanner=0;
+ int probeopt=fastprobe?PROBE_SAFE:PROBE_ALL;
  struct device **currentDevs, *d, *serialmouse=NULL, *usbmouse=NULL;
  if(verbose&VERBOSE_PROMPT) wpid=startwheel();
- if((currentDevs=probeDevices(dc,BUS_UNSPEC,PROBE_ALL))==NULL) return -1;
+ if((currentDevs=probeDevices(dc,BUS_UNSPEC,probeopt))==NULL) return -1;
  if(verbose&VERBOSE_PROMPT&&wpid>0) { kill(wpid,SIGTERM); wpid=0; usleep(160000); write(2,"\033[0m Done.\n",11); }
  for(i=0;(d=currentDevs[i]);i++)
   {
@@ -555,7 +559,7 @@
 
 int main(int argc, char **argv)
 {
- int i, verbose=0, probeonly=0, skip=0;
+ int i, verbose=0, probeonly=0, skip=0, fast=0;
  enum deviceClass dc=CLASS_UNSPEC;
  for(i=1;i<argc;i++)
   {
@@ -564,10 +568,11 @@
    else if(!strcasecmp(argv[i],"-a")) skip|=SKIP_AUDIO;
    else if(!strcasecmp(argv[i],"-s")) skip|=SKIP_SCSI;
    else if(!strcasecmp(argv[i],"-n")) probeonly=1;
+   else if(!strcasecmp(argv[i],"-f")) fast=1;
    else return syntax(argv[i]);
   }
  /* Allow SIGTERM, SIGINT: rmmod depends on this. */
  signal(SIGTERM,SIG_DFL); signal(SIGINT,SIG_DFL);
  signal(SIGALRM,alarm_handler); alarm(MAX_TIME);
- return hw_setup(dc,verbose,probeonly,skip);
+ return hw_setup(dc,verbose,probeonly,skip,fast);
 }