HTC Desire Z custom kernel killed Wi-Fi: version magic and insmod

· 7 min read
Contents

The fix, for anyone in a hurry: rebuild the Wi-Fi module from inside the kernel tree, take one -Werror out of the Broadcom driver's Makefile, copy the new bcm4329.ko over the old one in /system/lib/modules, and reboot. On the way I also added a firmware symlink, and for a while I believed the symlink was the second half of the fix. It was the second half of a mistake I had made a few minutes earlier, and this post is mostly about how I found that out after the fact.

The phone is my HTC Desire Z, a 2010 Android phone whose Wi-Fi turned out to be a separate file from its kernel, running CyanogenMod 7 (Android 2.3) and, as of the previous part, a kernel I compiled myself. Its Wi-Fi chip, a Broadcom BCM4329, is driven by a loadable kernel module rather than code built into the kernel. Twelve minutes after the new kernel booted, I had to report that Wi-Fi did not work. The whole repair took about eleven more.

The first failure: version magic#

Every Linux module carries a string called version magic: the kernel release it was built for, plus a few build flags. The kernel compares it with its own at load time and refuses a module that does not match. The kernel log, read with dmesg, had the refusal in so many words:

$ adb shell "su -c 'dmesg'" 2>&1 | tr -d '\r' | grep -i "bcm4329\|wifi\|wlan\|dhd\|firmware" | tail -15
<6>vision_wifi_init: start
<3>bcm4329: version magic '2.6.35.14-cyanogenmod-g295e82f preempt mod_unload ARMv7 ' should be '2.6.35.14-cyanogenmod-gc5198968-dirty preempt mod_unload ARMv7 '
<3>bcm4329: version magic '2.6.35.14-cyanogenmod-g295e82f preempt mod_unload ARMv7 ' should be '2.6.35.14-cyanogenmod-gc5198968-dirty preempt mod_unload ARMv7 '

The two hashes are git commits. g295e82f is the commit CyanogenMod's kernel was built from, gc5198968-dirty is the Kali- fork I built from, with my uncommitted edits. The module on /system was CyanogenMod's, and my kernel turned it away twice and said exactly why both times. I have had worse error messages from software that was trying much harder to be helpful.

The cause sat in my own build. The kernel I flashed was a zImage and nothing else. The config builds the BCM4329 driver as a module (CONFIG_BCM4329=m), so a new kernel needs a new .ko to go with it, and I had never asked for one.

Building the module, in two wrong ways first#

make modules answered No rule to make target 'modules', the M= variant found no makefile at all, and SUBDIRS= got the same no-rule answer. Make was right. I was running it from /tmp/boot-repack, the scratch folder where I had taken the boot image apart, and no amount of flags turns a scratch folder into a kernel. From inside the tree, the build got as far as the Broadcom driver and stopped:

drivers/net/wireless/bcm4329/wl_iw.c:5164:35: error: array subscript is above array bounds [-Werror=array-bounds]
     bcopy(&pmkid_list.pmkids.pmkid[i+1].PMKID,
                                   ^
drivers/net/wireless/bcm4329/include/linux_osl.h:203:45: note: in definition of macro ‘bcopy’
 #define bcopy(src, dst, len) memcpy((dst), (src), (len))
                                             ^
cc1: all warnings being treated as errors

I had already switched off the kernel-wide warnings-as-errors option for the main build. The BCM4329 driver does not ask the kernel's opinion: its own Makefile adds -Werror to its flags, in case the kernel's was not strict enough. One sed removed it, and the module built: 244,653 bytes, copied over the old one with a backup of the original next to it.

The second failure: firmware download failed#

I loaded the new module by hand with insmod, and this time the kernel accepted the module. The chip did not get its firmware:

=== WiFi ===
<1>DHD: dongle ram size is set to 294912(orig 294912)
<1>_dhdsdio_download_firmware: dongle image file download failed
<1>dhd_bus_start: dhdsdio_probe_download failed. firmware = /system/etc/firmware/fw_bcm4329.bin nvram = /proc/calibration
<1>dhdsdio_probe: failed
<1>dhd_module_init: sdio_register_driver timeout
=== WiFi interface ===
NO_WIFI_IFACE

The BCM4329 runs its own Wi-Fi firmware, which the driver uploads from a file when it starts. That file was not at /system/etc/firmware/fw_bcm4329.bin. This ROM keeps it at /system/vendor/firmware/fw_bcm4329.bin, next to an access-point variant. So I linked both into the place the driver had asked for:

lrwxrwxrwx    1 root     root            38 May 18 09:25 /system/etc/firmware/fw_bcm4329.bin -> /system/vendor/firmware/fw_bcm4329.bin
lrwxrwxrwx    1 root     root            44 May 18 09:25 /system/etc/firmware/fw_bcm4329_apsta.bin -> /system/vendor/firmware/fw_bcm4329_apsta.bin

Reloading did not work either. rmmod refused with errno 16, the module was busy. I stayed calm, the way you stay calm when the only thing on the phone that can be busy is the module you loaded by hand a few minutes earlier. A reboot cleared it, and after the reboot everything came up:

=== WiFi module ===
bcm4329 196888 0 - Live 0xbf000000
=== WiFi dmesg ===
<1>DHD: dongle ram size is set to 294912(orig 294912)
<1>Firmware version = wl0: Dec 20 2010 19:21:39 version 4.218.248.18
<1>dhd_get_dtim_skip bcn_li_dtim=3 DTIM=0 Listen=10
=== WiFi interface ===
eth0: ip 192.0.2.22 mask 255.255.255.0 flags [up broadcast running multicast]

Two problems, two fixes, Wi-Fi up. That was the story I told at the time. Half of it was wrong.

The Wi-Fi HAL already knew the path#

The firmware path in that error comes from my kernel config, CONFIG_BCM4329_FW_PATH="/system/etc/firmware/fw_bcm4329.bin", which the driver copies in as its default. The driver also takes a firmware_path module parameter, and a non-empty one replaces the default. And the CyanogenMod module, the one my kernel had refused, carries exactly the same default; its strings include /system/etc/firmware/fw_bcm4329.bin and /proc/calibration. So the stock setup never relied on that default either.

What finds it is the program that loads the module. On Android 2.3 that is the Wi-Fi HAL, libhardware_legacy.so, the library the framework calls to switch Wi-Fi on. The strings of the copy in the CyanogenMod 7.2 zip contain the module path and, right after it, the arguments it loads the module with:

/system/lib/modules/bcm4329.ko
firmware_path=/vendor/firmware/fw_bcm4329.bin nvram_path=/proc/calibration

The ramdisk's init.rc makes /vendor a symlink to /system/vendor, so that path is the real file. The one load that went without those arguments was mine. I typed insmod with no parameters, the driver fell back to its compiled-in default, and the firmware was not there. The module then stayed loaded and busy, so it never got a second, correct load until the reboot. After the reboot the HAL loaded my new module itself, with its arguments.

The symlink is harmless. It makes the compiled-in default true as well, so I cannot tell from that last log which of the two paths the driver used. Either way, the second failure was self-inflicted: the only thing on this phone that ever loaded the module without its firmware path was me, helping.

What the driver does once it has firmware#

The last line of the working log is the driver's power management announcing itself. dhd_linux.c hooks Android's early-suspend, the event that fires when the screen goes off, and in it switches the chip into its deepest power-save mode, filters incoming packets down to unicast, and sets how many beacons the radio may sleep through:

dhdcdc_set_ioctl(dhd, 0, WLC_SET_PM,
	(char *)&power_mode, sizeof(power_mode));

/* Enable packet filter, only allow unicast packet to send up */
dhd_set_packet_filter(1, dhd);

/* if dtim skip setup as default force it to wake each thrid dtim
 *  for better power saving.
 *  Note that side effect is chance to miss BC/MC packet
*/
bcn_li_dtim = dhd_get_dtim_skip(dhd);

power_mode starts that function as PM_MAX, and on resume it goes back to PM_FAST. None of that depends on the kernel config I had been editing. It lives in the module, which is a fine reason to make sure the module loads.

The symlink is still on the phone, 38 bytes of it. I left it there. It hurts nobody, and it reminds me that the quickest way to break Wi-Fi on this phone is to help it.

Filed under: android, kernel, htc-desire-z, wifi, bcm4329, cyanogenmod

Sources#

  • Kali- kernel for MSM7x30, commit c5198968 -- drivers/net/wireless/bcm4329/dhd_common.c (default fw_path from CONFIG_BCM4329_FW_PATH), dhd_linux.c (firmware_path module parameter, dhd_set_suspend), drivers/net/wireless/bcm4329/Makefile (the driver's own -Werror), arch/arm/mach-msm/board-vision-wifi.c
  • CyanogenMod 7.2.0 for vision (cm-7.2.0-vision.zip) -- system/lib/modules/bcm4329.ko (version magic and default firmware path, via strings), system/lib/libhardware_legacy.so (module load arguments), system/vendor/firmware/fw_bcm4329.bin; the stock boot image's init.rc (symlink /system/vendor /vendor)
  • HTC Desire Z (vision), CyanogenMod 7.2.0, kernel 2.6.35.14 built from the tree above -- direct device testing (dmesg, lsmod, ls -la)