# Dive deeper -- Analyze real mode binaries like a Pro with Qiling Framework This post also has a Chinese version available, see [here](). ## Introduction Analyzing a real mode binary like DOS executables or MBR code is never an easy task. The best approach we have is to utilize Bochs, QEMU or Dosbox to emulate such binaries with some debuggers like gdb, debug.exe and IDA Pro. But that's the situation before [Qiling Framework](https://github.com/qilingframework/qiling). Now we proudly annouce the 16bit emulation support of Qiling Framework, shipped with the ability to analyze the binary in multiple dimensions. This post will serve as an introduction of how real mode emulation is implemented as well as a basic tutorial of the whole Qiling Framework. ## Unicorn Before talking of Qiling, I'd like to introduce the Unicorn Engine briefly since Qiling is based on it. Unicorn is a pure CPU emulator forked from QEMU. Compared to QEMU, Unicorn has some key features: - Platform independent. - Friendly & Straigtforward API design, very very easy to understand and use. - Dynamic hook. - Much faster to build. :D But as stated just now, Unicorn is a PURE CPU emulator without any awareness of operating systems built on it and here comes Qiling. ## Qiling > Sometimes, I also don't know how to explain what Qiling exactly is. ---- Lau Kaijern, the creator of Qiling Framework. Well, indeed, it's really hard to define Qiling in a brief sentence. To have a full understanding of Qiling, I'd like to describe the functions of Qiling Framework in four aspects: emulation, analysis, debugger, extensions. ### Emulation The most basic function of Qiling is emulation but that's is different from Unicorn. As stated just now, Unicorn is a CPU emulator while Qiling is an OS emulator. Below is a simple figure to show the relation between a native application and an emulated application. ``` |-------| |---------| |Program| <-------> | Binary | | OS | <-------> | Qiling | | CPU | <-------> | Unicorn | |-------| |---------| Native app Emulated app ``` Usually, a native application runs in a specific OS environment which provides various API and the underlying hardware is a CPU. For a Qiling emulated application, the architecture is pretty similar: Unicorn serves as a CPU emulator while Qiling does the OS work and executes the target binary. To achieve this, Qiling has three layers. ``` |--------| --- | os | | | loader | Qiling | arch | | |--------| --- ``` - `arch` layer setups many CPU-spefic settings, like endianess, registers etc. - `loader` layer parses the target binary, setups memory layout and loads codes into the memory like the real executable loaders do. - `os` layer is the most important part and provides the implementation of syscalls. #### `HI.COM` example Let's takes DOS and [HI.COM](https://github.com/qilingframework/qiling/blob/bc3de75033cf01652e50d1970d32f10048ea4ed6/examples/rootfs/8086_dos/HI.COM) as an example to illustrate how these layers cooperate to emulate an OS. For `arch` layer, we assume the CPU is `8086` which is quite simple and thus only some basic setup like registers mapping needs to be done. COM files are simply a memory image without any header so the implementation of the `loader` layer is extremely simple: allocate enough memory, write the whole file and set PC and stack. For `os` layer, all the work Qiling needs to do is to implement as many interrupts as possible. Below is the source of `HI.COM`: ```assembly mov ah,9 mov dx, 10d int 21 mov ax, 4c00 int 21 nop db "Hello world!$" ``` When Unicorn executes `INT 21`, it doesn't konw the exact meanings of this interrupt so it will call hooks defined in `os` layer. Below is how `os` layer implement this interrupt: ```python def int21(self): # Handler for INT 21 ah = self.ql.reg.ah if ah == 0x4C: self.ql.uc.emu_stop() # Stop emulation # other interrupts... elif ah == 0x9: s = self.read_dos_string_from_ds_dx() # Read string self.ql.nprint(s) # Print the string to console # other interrupts... ``` That's all things Qiling does to emulate a DOS COM executable. Quite simple, right? #### Petya example In practice, to emulate a 16bit binary, most of my work is to implement interrupts in a proper way. For instance, Petya is a virus which write itself to the Master Boot Record(MBR). It largely depends on BIOS `INT 10` graphics service. The counterpart in modern terminal is `curses` so I spent lots of time translating old-days graphics commands to `curses` calls. Take `INT10, 0` as an example: ```python def int10(self): # BIOS video support # https://en.wikipedia.org/wiki/INT_10H # https://stanislavs.org/helppc/idx_interrupt.html # implemented by curses ah = self.ql.reg.ah al = self.ql.reg.al if ah==0: # time to set up curses # copied from curses.wrapper self.stdscr = curses.initscr() curses.noecho() curses.cbreak() self.stdscr.keypad(1) try: curses.start_color() except: pass if al == 0 or al == 1: curses.resizeterm(25, 40) elif al == 2 or al == 3: curses.resizeterm(25, 80) elif al == 4 or al == 5 or al == 9 or al == 0xD or al == 0x13: curses.resizeterm(200, 320) elif al == 6: curses.resizeterm(200, 640) elif al == 8: curses.resizeterm(200, 160) elif al == 0xA or al == 0xE: curses.resizeterm(200, 640) elif al == 0xF: curses.resizeterm(350, 640) elif al == 0x10: curses.resizeterm(350, 640) elif al == 0x11 or al == 0x12: curses.resizeterm(480, 640) else: self.ql.nprint("Exception: int 10h syscall Not Found, al: %s" % hex(al)) raise NotImplementedError() # Quoted from https://linux.die.net/man/3/resizeterm # # If ncurses is configured to supply its own SIGWINCH handler, # the resizeterm function ungetch's a KEY_RESIZE which will be # read on the next call to getch. ch = self._get_ch_non_blocking() if ch == curses.KEY_RESIZE: self.ql.nprint(f"[!] You term has been resized!") elif ch != -1: curses.ungetch(ch) self.stdscr.scrollok(True) if not curses.has_colors(): self.ql.nprint(f"[!] Warning: your terminal doesn't support colors, content might not be displayed correctly.") # https://en.wikipedia.org/wiki/BIOS_color_attributes # blink support? if curses.has_colors(): for fg in range(16): for bg in range(16): color_pair_index = 16*fg + bg + 1 if fg not in self.color_pairs: self.color_pairs[fg] = {} curses.init_pair(color_pair_index, COLORS_MAPPING[fg], COLORS_MAPPING[bg]) color_pair = curses.color_pair(color_pair_index) self.color_pairs[fg][bg] = color_pair self.revese_color_pairs[color_pair] = (fg, bg) ``` As the saying goes, "Simple is better". Nowadays, terminal has become much more complicated than past screen. As a result, to implement a dummy screen is rather difficult since there are too many corner cases. #### Emulate but differently Maybe some readers have noticed that the emulation here dosen't act the same as a real DOS does. Yes, that's true for Qiling. The perfect and fast emulation is never the primary goal for Qiling. What Qiling focuses on is to provice our users with max convenience for binary analysis. I will talk about that in following sections. Also note that Qiling is designed to platform-independent, which means users can run a Linux ELF on Windows or a Windows exe on Mac OS. It's impossible for QEMU to achieve that because QEMU will forward syscalls to host machine. ### Analysis Qiling is far beyond a simple emulator. It also provides a bunch of powerful API. #### fs_mapper Let's begin with `fs_mapper` (since it was refactored by me a few days ago XD): ```python #!/usr/bin/env python3 # Source: https://github.com/qilingframework/qiling/blob/dev/examples/hello_x86_linux_fake_urandom.py # # Cross Platform and Multi Architecture Advanced Binary Emulation Framework # Built on top of Unicorn emulator (www.unicorn-engine.org) from qiling import * from qiling.os.mapper import QlFsMappedObject class Fake_urandom(QlFsMappedObject): def read(self, size): return b"\x01" # fixed value for reading /dev/urandom def fstat(self): # syscall fstat will ignore it if return -1 return -1 def close(self): return 0 if __name__ == "__main__": ql = Qiling(["rootfs/x86_linux/bin/x86_fetch_urandom"], "rootfs/x86_linux") ql.add_fs_mapper("/dev/urandom", Fake_urandom()) ql.run() ``` The source code of the target binary `x86_fetch_urandom`: ```C // Souce: https://github.com/qilingframework/qiling/blob/dev/examples/src/linux/fetch_urandom.c #include <stdio.h> int main(void) { FILE *fp; int randno; if ((fp = fopen("/dev/urandom", "r")) == NULL) { fprintf(stderr, "Error! Could not open /dev/urandom for read\n"); return -1; } randno = fgetc(fp); printf("randno: %d\n", randno); fclose(fp); return 0; } ``` This program simply fetches 1 byte from `/dev/urandom` which should always return random bytes stream on a real Linux system. However, with `ql.add_fs_mapper("/dev/urandom", Fake_urandom())`, we can hijack such read operations and return a fixed value. Except for device file, `fs_mapper` can also emulate a raw disk. Below is the emulation code for Petya. ```python #!/usr/bin/env python3 # Source: https://github.com/qilingframework/qiling/blob/dev/examples/petya_8086_mbr.py # # Cross Platform and Multi Architecture Advanced Binary Emulation Framework # Built on top of Unicorn emulator (www.unicorn-engine.org) import sys sys.path.append("..") from qiling import * from qiling.os.disk import QlDisk if __name__ == "__main__": ql = Qiling(["rootfs/8086/petya/mbr.bin"], "rootfs/8086", console=False, output="debug", log_dir=".") # Note: # This image is only intended for PoC since the core petya code resides in the # specific sectors of a harddisk. It doesn't contain any data, either encryted # or unencrypted. ql.add_fs_mapper(0x80, QlDisk("rootfs/8086/petya/out_1M.raw", 0x80)) ql.run() ``` `0x80` is the disk index used in real mode. For Linux and Windows, it can be `/dev/sda` or `\\.\PHYSICALDRIVE0`. `QlDisk` is a class which provides some basic support for a raw disk image, like cylinders, heads and sectors. The reason why emulating petya needs a disk mapping is that Petya uses BIOS `INT 13` disk service to read/write disks directly. To implement such interrupts, it's necessary to map a file to a disk in our emulated environment. #### syscall hijacking Another example is syscall hijacking. Below is an example from [docs](https://docs.qiling.io/en/latest/hijack/#posix-qlset_api): ```python from qiling import * def my_puts(ql): addr = ql.os.function_arg[0] print("puts(%s)" % ql.mem.string(addr)) if __name__ == "__main__": ql = Qiling(["rootfs/x8664_linux/bin/x8664_hello"], "rootfs/x8664_linux", output="debug") ql.set_api('puts', my_puts) ql.run() ``` Since Qiling itself serves as the OS environment, it's quite easy to expose any syscall to users for further analysis. #### snapshot & partial executaion Snapshot allows you to save or restore the context of a program including CPU registers, memory content and even file descriptors. With partial execution support from Qiling, snapshots can enable users to fully control a program and test any code with proper context. Below is an example from Petya analysis. ```python def one_round(ql: Qiling, key: bytes, key_address): gkeys = generate_key(key) ql.mem.write(key_address, gkeys) # Partial executaion ql.run(begin=verfication_start_ip, end=verfication_start_ip+6) lba37 = ql.mem.read(ql.reg.sp + 0x220, 0x200) for ch in lba37: if ch != 0x37: return False return True # In this stage, we will crack for the password. def second_stage(ql: Qiling): disk = QlDisk("rootfs/8086/petya/out_1M.raw", 0x80) #nonce = get_nonce(disk) # Prepare stack verfication_data = disk.read_sectors(0x37, 1) nonce_data = disk.read_sectors(0x36, 1) ql.reg.sp -= 0x200 verification_data_address = ql.reg.sp ql.reg.sp -= 0x200 nonce_address = ql.reg.sp + 0x21 ql.reg.sp -= 0x20 key_address = ql.reg.sp ql.mem.write(verification_data_address, verfication_data) ql.mem.write(nonce_address - 0x21, nonce_data) ql.arch.stack_push(0x200) ql.arch.stack_push(verification_data_address) ql.arch.stack_push(0) ql.arch.stack_push(nonce_address) ql.arch.stack_push(key_address) for x in product(list(accepted_chars), repeat=2): ctx = ql.save() # 3xMxjxXxLxoxmxAx key = b"3xMxjxXxLxoxmx" + ("".join(x)).encode("utf-8") print(f"Trying: {key}") if one_round(ql, key, key_address): print(f"Key: {key}") return key else: ql.restore(ctx) return None ``` In the code above, we prepare the stack firstly, and take a snapshot. After that, we write the key as function arguments and call the verfication function directly. If the result suggests the key is wrong, we restore the context and try again. There are also other tons of features that I can't introduce here due to the limit of space. If you are interested, please refer to [documents](https://docs.qiling.io/en/latest/). In a word, all theses powerful API makes Qiling a real dynamic instrumentation framework and distinct from other emulators. ### Debugger As we know, QEMU supports gdb remote debugging by gdbserver. How could Qiling not support that? As the time of writing the post, the Qiling debugger is still under heavy development but the support of 16bit debugging has been done. Simple add `ql.debugger = True` and connect gdb to ``. Below is a screenshot. ![](https://i.imgur.com/7HxjSMj.png) ### Extensions Currently, Qiling is on a move to decouple some logic to standalone extensions. One of our latest work is [IDA plugin](https://docs.qiling.io/en/latest/ida/). Below is a screenshot. ![](https://docs.qiling.io/en/latest/img/ida7.png) Oh, oh, I guess you guys are all experts at command lines and terminal, does that GUI really make sense? Yes! Thanks to Qiling IDA plugin, at least you don't need to buy a Linux IDA license since you can run a Linux ELF on your Windows/Mac OS, does it? There are also some other interesting extensions under [qiling/extensions directory](https://github.com/qilingframework/qiling/tree/dev/qiling/extensions) Please refer to corresponding README or documents for details. ## Summary Qiling Framework is an **ambitious** project. Our final goal is to make a Swiss Knife like analysis framework, lift the restrictions like the boundraries between different operating systems and provide security engineers with the best experience. Nothing could stop us from diving deeper in binary analysis! Nevertheless, Qiling Framework is far from perfect. Give us a star at [here](https://github.com/qilingframework/qiling) and join our development! We are looking forward to merging your first PR! ## PS - `Qiling` is a spelling mistake indeed. It should be `Qilin`. Whatever :P - In fact, it's almost impossible to crack full Petya key by brute force in an acceptable period of time. Here we simply show the usage of Qiling API. - IDA 7.0 doesn't have Python3 support, but Ghidra support is on the way! - TLDR: Star [us](https://github.com/qilingframework/qiling)!