[Development] Bidirectional communication between Linux drivers
===
###### tags: `development`, `C`, `driver`, `linux`, `LKM`, `function pointer`, `kernel module`, `module stacking`, `modules interdependence`
[ToC]

# Preface
When I try to modularize built-in kernel modules, I encounter troubles which are not only module stacking but modules interdependence. The trouble of modules interdependence exists in both built-in module and LKM.

# Resolve modules interdependence through function pointers

Here, we will give an example to show how to resolve this trouble of modules interdependence.

## Scenario
First, there are two modules named mtestA and mtestB, respectively. mtestA will make a function call in mtestB. In general, we can just export symbols in mtestB and install mtestB prior to mtestA on our intuition so that mtestA can make a call to mtestB directly. But if we want to keep the sequence that installing mtestA and then mtestB, we have to utilize function pointers to make the function hooked dynamically.

## Implementation
The whole code are shown in gist.

- Module A: mtestA.ko (from testA.c, testA1.c)
- Module B: mtestB.ko (from testB.c)

{%gist ff2c8a7a770befd7def64fb7aa406150%}

## Experiment
### Goal
In this experiment, our goal is to verify if it is feasible to dynamical hooking function pointers between modules.

### Methodology
We will follow below operating sequence.
1. install mtestA
2. install mtestB
3. uninstall mtestB
4. install mtestB
5. uninstall mtestA

Furthermore, we consider two conditions in Module B, i.e., mtestB:

1. Exit(unlonad) with unreg()

static int __init test_init(void)
{
        reg(&mypfp);
        printk("testB init! ");
        showtime();
        printA();
        return 0;
}

static void __exit test_exit(void)
{
        printk("testB exit!\n");
        unreg(); //This is essential to do.
}
module_init(test_init);
module_exit(test_exit);

2. Exit(unlonad) without unreg()

static int __init test_init(void)
{
        reg(&mypfp);
        printk("testB init! ");
        showtime();
        printA();
        return 0;
}

static void __exit test_exit(void)
{
        printk("testB exit!\n");
        //unreg(); //To see what will happen
}
module_init(test_init);
module_exit(test_exit);

Actually unreg() is just pointing the global function pointer to NULL as shown below.

void unreg(void)
{
        mypf = NULL;
}

### Outcome
We show the outcome in dmesg with our five operation.
1. Exit(unlonad) with unreg()

sudo insmod mtestA.ko

[  744.494516] testA init!  
[  744.494518] TIME: 06:43:09:778125 
[  744.494519] print ModuleA vA = 6
[  756.633434] null pointer 

sudo insmod mtestB.ko

[  764.648120] testB init! 
[  764.648123] TIME: 06:43:29:931729 
[  764.648125] print ModuleA vA = 6
[  768.665090] addr mypf = 0000000010d30cb5
[  768.665097] addr mypf->print = 00000000e9156d49
[  768.665102] This printB in moduleB
[  780.696949] addr mypf = 0000000010d30cb5
[  780.696951] addr mypf->print = 00000000e9156d49
[  780.696952] This printB in moduleB

sudo rmmod mtestB.ko

[  784.484288] testB exit!
[  792.730045] null pointer

sudo insmod mtestB.ko

[  799.291935] testB init! 
[  799.291938] TIME: 06:44:04:575544 
[  799.291939] print ModuleA vA = 6
[  804.760132] addr mypf = 00000000441b6299
[  804.760134] addr mypf->print = 00000000014e515a
[  804.760136] This printB in moduleB

sudo rmmod mtestB.ko

[  812.862133] testB exit!
[  816.792799] null pointer 

sudo rmmod mtestA.ko

[  823.254448] testA exit!

2. Exit(unlonad) without unreg()
In this case, it is obvious to see that we encounter a issue.
**BUG: unable to handle kernel paging request at ffffffffc04a7000**
This is because we do not reset the global pointer to NULL and result in an access of illegal address.

sudo insmod mtestA.ko

[  969.990557] testA init!  
[  969.990560] TIME: 02:38:55:433913 
[  969.990561] print ModuleA vA = 6
[  982.015261] null pointer 
[  994.041583] null pointer 
[ 1006.067057] null pointer 

sudo insmod mtestB.ko

[ 1014.373796] testB init! 
[ 1014.373799] TIME: 02:39:39:839344 
[ 1014.373800] print ModuleA vA = 6
[ 1018.093590] addr mypf = 000000004c8daddf
[ 1018.093592] addr mypf->print = 00000000e1058471
[ 1018.093593] This printB in moduleB
[ 1030.119081] addr mypf = 000000004c8daddf
[ 1030.119085] addr mypf->print = 00000000e1058471
[ 1030.119087] This printB in moduleB
[ 1042.145099] addr mypf = 000000004c8daddf
[ 1042.145105] addr mypf->print = 00000000e1058471
[ 1042.145110] This printB in moduleB
[ 1054.171608] addr mypf = 000000004c8daddf
[ 1054.171611] addr mypf->print = 00000000e1058471
[ 1054.171614] This printB in moduleB
[ 1066.197246] addr mypf = 000000004c8daddf
[ 1066.197252] addr mypf->print = 00000000e1058471
[ 1066.197257] This printB in moduleB

sudo rmmod mtestB.ko

[ 1077.943266] testB exit!
[ 1078.223532] addr mypf = 000000004c8daddf
[ 1078.223541] BUG: unable to handle kernel paging request at ffffffffc04a7000
[ 1078.223969] IP: work_handler+0x28/0x80 [mtestA]
[ 1078.224148] PGD 2a60e067 P4D 2a60e067 PUD 2a610067 PMD 3caf9067 PTE 0
[ 1078.224447] Oops: 0000 [#1] SMP PTI
[ 1078.224643] Modules linked in: mtestA(OE) nls_utf8 isofs vboxsf snd_intel8x0 
snd_ac97_codec ac97_bus snd_pcm snd_seq_midi snd_seq_midi_event intel_powerclamp 
crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel snd_rawmidi snd_seq pcbc 
snd_seq_device aesni_intel aes_x86_64 crypto_simd snd_timer glue_helper cryptd 
intel_rapl_perf input_leds serio_raw vboxvideo(CE) ttm drm_kms_helper 
drm fb_sys_fops syscopyarea sysfillrect sysimgblt snd soundcore vboxguest 
mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 
hid_generic usbhid hid psmouse ahci libahci i2c_piix4 e1000 pata_acpi video 
[last unloaded: mtestB]
[ 1078.251439] CPU: 0 PID: 32 Comm: kworker/0:1 Tainted: GC OE4.15.0-29-generic #31-Ubuntu
[ 1078.256576] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 1078.258534] Workqueue: hello world work_handler [mtestA]
[ 1078.260491] RIP: 0010:work_handler+0x28/0x80 [mtestA]
[ 1078.262264] RSP: 0018:ffffa4f2002cfe78 EFLAGS: 00010286
[ 1078.264049] RAX: ffffffffc04a7000 RBX: ffffffffc04a23a0 RCX: ffffffff86062828
[ 1078.265835] RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000247
[ 1078.267601] RBP: ffffa4f2002cfe78 R08: 000001c2ecd7e5cf R09: 0000000000000244
[ 1078.269376] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89cdbd94d540
[ 1078.271127] R13: ffff89cdbfc22180 R14: ffffc4f1ffc05b00 R15: 0000000000000000
[ 1078.272884] FS:  0000000000000000(0000) GS:ffff89cdbfc00000(0000) knlGS:0000000000000000
[ 1078.284666] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1078.286575] CR2: ffffffffc04a7000 CR3: 000000002a60a002 CR4: 00000000000606f0
[ 1078.288247] Call Trace:
[ 1078.289896]  process_one_work+0x1de/0x410
[ 1078.291469]  worker_thread+0x32/0x410
[ 1078.293097]  kthread+0x121/0x140
[ 1078.294650]  ? process_one_work+0x410/0x410
[ 1078.296352]  ? kthread_create_worker_on_cpu+0x70/0x70
[ 1078.297929]  ret_from_fork+0x35/0x40
[ 1078.299488] Code: <48> 8b 30 48 85 f6 75 2d 48 8b 35 c1 23 00 00 b9 b8 0b 00 00 48 c7 
[ 1078.301267] RIP: work_handler+0x28/0x80 [mtestA] RSP: ffffa4f2002cfe78
[ 1078.302862] CR2: ffffffffc04a7000
[ 1078.304516] fbcon_switch: detected unhandled fb_set_par error, error code -16
[ 1078.308232] fbcon_switch: detected unhandled fb_set_par error, error code -16
[ 1078.312070] ---[ end trace e5f177a48112f26e ]---

sudo rmmod mtestA.ko

[ 1194.346184] testA exit!

Endorphin@Taipei:~/test$ lsmod
Module                  Size  Used by
mtestA                 16384  -1

# Conclusion
We can break through the constraint on module stacking by the applicaion of function pointers. But we have to be more careful when handling these function pointers.