Try   HackMD

ANSY SUBJECTS 2024

slides: https://zarak.fr/resources/ANSY.pdf
Linux kernel mailing lists: https://lkml.org/
Linux sources: https://elixir.bootlin.com/linux/v6.11.1/source

Information about all submissions

The submissions will be done by email at ansy@zarak.fr, tag [ANSY].

You can submit each part individually.
There is no restriction on the language used or the submission architecture, but please make it clear, and readable. The easier it is for me to correct you, the more likely I will be to fully appreaciate your work.

Recommended languages however includes:

  • python
  • bash
  • c

I will also accept "normal" languages for the kind of tasks I'm asking you to do, but I reserve myself the right of refusing a submission if it's an obvious troll like brainfuck or ook!.

Please send files like:

TP<number>-<login>.zip
└── TP<number>-<login>/
    ├── README
    └── ...

I'd need everybody to follow these simple instructions to avoid making me loose time reshaping your submitted files, thanks !

If you have any question, or if you're stuck, please reach out to me !

Note

Please track the time you're spending on each part and provide the information in the submission's README. This will allow me to better estimate the workload of those subjets and possibly adapt it for next year

ANSY TP 1

Rendu

Elle devra contenir :

  • un README avec le résultat de la step 1 et le temps total passé sur le sujet 1
  • le script demandé dans la deuxieme partie, avec l'historique des commandes

La date max de rendu est fixée au début du cours n°2

step 1

  1. Avec strace, trouver le syscall qui est executé plus de 10k fois par le binaire fourni, et me rapporter combien de fois il est appelé
  2. Trouver et rapporter dans le README la phrase la plus importante, philosophie du developpement de linux, de l'échange houleux et très discutable sur la forme entre Linus Torvalds et Mauro Chehab (https://lkml.org/lkml/2012/12/23/75)

step 2

  1. écrire un programme qui permet d'executer le binaire fourni et faire en sorte qu'il fonctionne, c'est à dire qu'il retourne 0 en exit code et qu'il affiche la phrase de fin ("gg!"). Pour ça, il va falloir utiliser strace pour comprendre son fonctionnement, et faire en sorte qu'il puisse s'executer comme il devrait. Il y a 2-3 "pièges", en tout cas du bruit, pour essayer de se mettre dans une condition plus "réelle".
  2. BONUS OPTIONNEL: Atteindre la seconde fin du programme (bonus, plus difficile)

Je vous demanderai aussi de conserver votre bash history (ou zsh history ou équivalent) pour me montrer les commandes et la réflexion que vous avez pu avoir pour ce TP

Le binaire à analyser est disponible sur https://zarak.fr/resources/straceme
Je vous demande de n'utiliser que strace pour l'analyse et pour le comprendre.

Tips and tricks

Vous pouvez modifier l'environment d'execution du binaire, ou bien (et surtout) modifier le comportement des syscalls avec strace -e. Prenez le temps de regarder la man page, de lire un peu les exemples et de voir les possibilités.

ANSY 2 subject

How often is the scheduler waken up ?

Setup the env

For this part, we're going to mess up with the scheduler a bit.

For the moment, let's keep the changes simple, we'll focus on bringing our own linux kernel alive, to explore the linux toolchain and build process.

For this subject, you need to have a Debian 12 VM (this is the recommended way, as I'm going to guide you with such setup. You may do this subject with another setup, but it might be harder for you, and I may not be able to debug you).

You can install a debian 12 VM with virtualbox or libvirt. If you're running AMD64, you can download it here : https://cloud.debian.org/images/release/current-live/amd64/iso-hybrid/ (debian-live-12.7.0-amd64-standard.iso)

Install the VM, install a few tools you might find useful, like vim or a ssh server.

Prepare your linux build setup

You can follow guides like this: https://phoenixnap.com/kb/build-linux-kernel

But here are the things needed for this subject:

$ sudo apt-get install -y git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison dwarves
$ uname -a
Linux ANSY 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux
$ echo Our current kernel is in version 6.1.0. Let's download the Linux source for this version
$ git clone https://github.com/torvalds/linux.git -b v6.1 --depth 1
$ cd linux
$ echo generate your own base kernel config with `make defconfig`
$ make defconfig
$ echo maybe you will need or want to change the configuration. Either edit the .config file newly created, or run make menuconfig
$ echo you can also copy your current linux config, but compilation will take more time
$ #cp /boot/config-6.1.0-12-amd64 .config

Compile linux

You are now ready to compile linux. Simply run make -j$(nproc) to build the kernel.

You will also need the compressed version. You can get it with make bzImage -j$(nproc). You can actually use exclusively this one to compile and build.

The kernel you just built may be slightly incorrect, and report itself as 6.1.0-dirty (uname(2)). This is because you have built from a git repository with dirty changes – non committed changed. To make your new kernel compatible, we'll add another build option:

$ uname -a
Linux ANSY 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux

My current kernel reports itself as 6.1.0-12-amd64. I'll force the kernel I'm building to report to this as well:

$ touch .scmversion && make bzImage -j$(nproc) LOCALVERSION="-12-amd64"

The first build will take quite some time. You may need some amounts of RAM and CPU. Be patient.

Testing your linux

You should have in the end something like this in the output:

  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#3)

You need to copy your newly built kernel in /boot:
$ cp arch/x86/boot/bzImage /boot/vmlinuz-custom

You don't want to replace the real kernel debian-provided youre currently using, as your newly built kernel may be broken. Instead, we're going to add an option in grub, when booting, to choose to boot from this kernel.

Edit the file /boot/grub/grub.cfg

Find the default menu entry. Mine looks like this:

menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e06573ef-48b6-4890-b5c6-622eedf914b0' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_msdos
	insmod ext2
	search --no-floppy --fs-uuid --set=root e06573ef-48b6-4890-b5c6-622eedf914b0
	echo	'Loading Linux 6.1.0-12-amd64 ...'
	linux	/boot/vmlinuz-6.1.0-12-amd64 root=UUID=e06573ef-48b6-4890-b5c6-622eedf914b0 ro  quiet
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-6.1.0-12-amd64
}

Copy your own menu entry (not mine ! it will not work) to have a second one right under the default with a few differences:

1c1
< menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e06573ef-48b6-4890-b5c6-622eedf914b0' {
---
> menuentry 'Debian GNU/Linux with custom kernel' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e06573ef-48b6-4890-b5c6-622eedf914b0' {
9c9
< 	linux	/boot/vmlinuz-6.1.0-12-amd64 root=UUID=e06573ef-48b6-4890-b5c6-622eedf914b0 ro  quiet
---
> 	linux	/boot/vmlinuz-custom root=UUID=e06573ef-48b6-4890-b5c6-622eedf914b0 ro  quiet

Change the name of the entry to show which one is yours and which one is the default one. Change the linux argument path to your own kernel.

Now, on reboot of the VM you should see a new entry in your grub menu to boot from your own kernel

The assignment.

You will have to write a patch for linux. We're going to add a new behavior to the scheduler. We've been wondering in class how many times the scheduler was waken up to make a decision, to schedule.

Let's make our kernel expose this information. We're not going to do anything fancy yet as this is not the point of the exercise.

  1. Setup the developement env
  2. Write a patch to the scheduler so that every time it is called, a counter is increased. Also look at the time. If it's been more than a second since the last report, make a report. To make a report, write the number of times the scheduler has been called (your counter) to kernel logs, and reset the counter.
  3. Test your new patch, and look in the kernel logs at the output of your scheduler

Hints:

  • The scheduler function to schedule a task has a very straightforward name
  • To log, use the function printk
  • To get the time in ns, you can use ktime_to_ns(ktime_get())

Submission

You will provide a .patch that contains the patch you've just written (i.e. the git diffs)
You will also provide a README with information you deem interesting to mention, but with at least:
- the amount of time spent on this
- an output of dmesg to show your scheduler logs

The deadline is for the 27th of october, 23h59.

ANSY 3

Let's observe the CPU/Scheduler in action.

Write a program than can creates situations to prove how the niceness of 2 different processes interferes within each other.

The idea is to be able to provide an answer for such problem:
I have process A with a niceness of X. I have process B with a niceness of Y.
Both processes are started at the same time, and require the same amount of CPU in total. But they're fightining each other for said CPU. How much more real time A (or B) will need compared to the amount of time it needed if it were to run alone ?

Answer can be summed up in an array like:

Niceness of A (X) Niceness of B (Y) Extra time needed for A Extra time needed for B
0 10 0% (as fast as if it was alone) 100% (needed twice more time)
0 0 50% 50%

Of course those values are example.

Create a program that takes as an input X and Y (argv), runs two processes in parallel with given niceness, measure how much time it needed to execute the two and output the % of extra time needed in comparison of a single execution.

Tips

  • You'll need to run a CPU program for this test to make sense
  • You'll need to run in a single core those two processes to see how the scheduler will make decisions. You'll have to force those 2 processes on one core
  • Execution time will vary from a laptop to another, don't hardcode the referential execution time

Extra challenge

This challenge is a bonus, it's quite difficult and may take quite some time to achieve. Don't feel forced to do it if you don't have the time.

Create a program that can create a very high cpu load – let's say twice the amount of cores – without using the CPU (CPU usage % shall remain below ~10%).

Submission

The deadline is set to the 5th of december, 23h59.

As usual, the submission will be done by email, with a .zip described above.
I'm expecting a README with the amount of time spent, an array providing answer for (X=0,Y=0),(X=0,Y=10),(X=0,Y=15), and ofc the program(s)

ANSY BONUS subject

This subject is a bonus, either to help you boost your grade if you believe you did not perform well on other subjects, or to help compensate a missing submission.

The deadline for this exercise will be the 18th of January 23:59.

Manipulating block devices

The point of this exercise will be to write a small program/script to manipulate block devices, kernel modules related to block devices, filesystems and famous userland tools.

The suggested language is bash.

You will have to create a few scenarios and run benchmark to test them out.

The benchmark will be run with fio: fio --name=random-write --rw=randwrite --bs=4k --numjobs=1 --size=1g --iodepth=256 --runtime=60 --time_based --end_fsync=1

As you can read, you'll need a GiB of free space to run this test. It is therefore recommended to create each situation with 5 GiB to begin with, to be sure to have enough room in the end.

Core

The core of your program will:

  1. Create a file that will be used as a block device.
    • It's recommended to use a file rather than a disk partition as it's simpler for theses tests. But keep in mind that once this file has been setup to behave as a block device, things will be the same as if it's a "real" block device (a real hard drive, USB key, …)
  2. Expose this file as a block device via a loop device. Check losetup
    • pro-tip: use -f --show
  3. You now have a block device than can be used. Depending on the arguments of your script, you will apply or not some transformation to it. Refer to the next section to know what transformation are expected here
  4. After the optional transformation, you end up with a block device. This block device to be used needs a filesystem. Create an XFS filesystem on the block device
  5. The filesystem to be accessed needs to be mounted. Mount it to the path specified by --path (defaults to /mnt/final if not specified)
  6. If the user asked for it via --bench, run a benchmark on the final filesystem

Transformations

Here are a list of flags your program can take and that will apply transformation to the block device at the step 3:

  • --loop will take a number X as argument (default: 0) and will repeat the core of the program X times, meaning repeating steps 1 to 4 included, and step 5 with a path of /mnt/loop-$i.
    • If another transformation is asked, this transformation must be done at each loop iteration.
    • The step 1 must be performed on the filesystem created at the previous iteration of course
    • For the last iteration of the loop, the name to be used on step 5 is /mnt/final
    • With --loop 0 (the default), one has one /mnt/final
    • With --loop 1, one has /mnt/loop-1 for the first iteration of the core loop, and /mnt/final for the last iteration
  • --basic-lvm will create a LVM PV, VG and single LV taking all the space possible of the block device.
  • --crypt will create a LUKS2 encrypted block device with cryptsetup. The password will be very secure of course, as it's going to be password
    • you will have to create the encrypted block device, but also open it to use it !
    • The name of the devices is up to you. archlinux doc
  • --dm-linear will create a block device from your whole block device in a linear way
  • --dm-delay will perform as --dm-linear, but will add a delay of 1000 ms
  • --md will create 2 block devices from your block device (split it in half with dmsetup's linear option), and then a RAID 1 disk from the two block devices

Each transformation shall take as an input 1 block device, and provide as an output 1 block device. They can be chained this way.

Pro-tips

It is recommended to start simple. Create the core part of the program, and test it well.

Add a --cleanup flag to not do anything but remove what your program has created, this will help you a lot

Gradually add the transformations support. They may seems complex, but each one taken individually is actually fairly straight forward with the appropriate documentation

Reach me for help if things are unclear, or not working as expected.

You can use argbash.dev to simplify development if you choose bash

Last changed by 
 
zarak
0
687
Published on HackMD