---
tags: interview
---
# Linux
## Core
1. Run levels
* run level 0 - system halt (poweroff.target)
* run level 1 - single user mode, maintenance mode (rescue.target)
* run level 2 - multi-user mode, with no networking (multi-user.target)
* run level 3 - multi-user, with **text only**, **non-graphical**, with networking mode (multi-user.target)
* run level 4 - user defined (multi-user.target)
* run level 5 - mult-user, with graphical mode (graphical.target)
* run level 6 - reboot (reboot.target)
* emergency mode (emergency.target)
2. To set run level

systemctl set-default <target-name>.target

3. Review following topics
* TLB
* Virtual memory
* Virtual memory address space
* MMU
* Page
* Page fault
* Thrashing
* Segmentation fault
* OOM when occurs
### Debugging tools
#### Strace
`strace` is a tool that displays the system calls being executed by a running program.
strace uses `ptrace()` system call to trace all the system calls.

$ strace libreoffice

# To attach an existing process 
$ strace -p firefox
# execve - starts the program
# mmap, mprotect, munmap - calls for memory management by kernel
# brk - allocates memory on the heap
# 

#### ltrace
It is a tool that tracks library calls of a program.

#### xargs
used to parse something from bunch of output

** If the files or text contains special character or space, you mus=t pass **print0** to find and **--null** to xargs.

Important options:
1. place holder: `-I{} {}`
2. ask for confirmation: `-p`
3. pass the command to no. of argument: `-n 1`
4. verbose mode: `-t`
5. read items from file, instead of stdin: `-a`
6. No. of parallel execution: `-P`
Understanding `-n` option

$ ls 
file1 file2 file3
$ ls |xargs -n 1
file1
file2
file3
$ ls |xargs -n 2
file1 file2
file3

# Example
$ ls |xargs -n 1 -I{} chgrp lisa {}

Examples:
1. Find and remove files that starts with 'h' and ends with 'i', only in the current directory, don't remove recursively.

$ find . -maxdepth 1 -name ^h* |grep i$ |xargs rm -rf

2. A file has list of names, create a directory named in the file in /tmp

$ cat file.txt
dir1
dir2
dir3

$ cat file.txt |xargs -I{} mkdir -p /tmp/{}

# -I{} is the place holder
# {} at the end has the value stored in -I{}

#### fmt
Questions
1. If you're unable to create files although enough space
Check permission, ownership and inodes exhaustion.

3. File descriptor
file descriptor is a number that represents open file, Linux references by numbers (file descriptors), as it won't recognize that by name of the file.

4. Find current run-level

$ who -r
run-level 5  2019-10-06 08:58
# OR
$ runlevel
N 5

5. How do you troubleshoot SSH issues

ssh -v user@host

6. A database server is stopping every few mins, at random, but there are no logs in the machine.
How do you troubleshoot?
a. use dmesg
b. Fix a broken syslog
c. Trace the process using strace abd gdb

7. How do you debug strange failure/ crash of programs?
Use `strace`

8. What is `ltrace`?
9. What is context switching?
Switching the CPU to another process requires saving the state of the old process and loading the saved state for the new process. This task is known as context switch.
The context of a process is represented in the PCB of a process.
11. What are the ways to list files in Linux

echo *
find .

11. Kill all the process for the process 'badprocess'

$ ps aux |grep badprocess |cut -c 11-15 |xargs kill -9

12. Find all the directories that starts with `t` and copy those directories and add .backup at the end of the filename to /tmp

$ find . -name t* -maxdepth 1 -type d |xargs -I{} cp -r {} /tmp/{}.backup

13. Find all the log files and backup

$ find . -name *.log |xargs tar backup.tar 

14. List the files in four column

$ ls |xargs -n 4

15. You have a list of files in text file, remove only 4 files from it.

$ cat filelist.txt
file1
file2
file3
file4
file5

$ cat filelist.txt |xargs -n 4 rm -rf

16. How do you grep for a string in all *.txt files recursively?

$ find . -name *.txt |xargs grep string

17. Download files in parallel

$ cat files.txt
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.1.0-amd64-netinst.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.1.0-x86_64-netinst.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.1.0-s91-netinst.iso

$ cat files.txt |xargs -n 1 -P 3 wget

# OR
$ cat files.txt |parallel -j 3 wget 

# OR
$ parallel -j 10 -a files.txt wget

18. Compress all the files in current directory

$ ls |parallel -n 1 -j 5 -I{} tar czf {}.tar {}

#### setuid
Set user ID gives the user to acquire the permissions of the owner during the instance of execution.
example: `passswd`
To set setuid

$ chmod u+s file.txt
# Make ensure the file.txt has execute permission, otherwise the above won't be effective.

#### stickybit
Gives permission to read all of the files written by all of them in a group, but won't allow others to delete a file written by other.

$ chmod +t directory_name

#### Zombie process
Zombie process or **defunct** process is a process which completed its execution and waiting for parent process to read its exit status.

$ ps -elf |grep Z

#### softlinks vs hardlinks

| S.No | Hard link | Soft link |
| -------- | -------- | -------- |
| 1 | Create two exact files without having to duplicate the data | Create a pointer (shortcut) to a directory or a file |
| 2 | **Same inode** number | Different inode number |
| 3 | **Cannot** create a hardlink for a **dir** | Can create a softlink for both dir and file |
| 4 | **Not** allowed to create a hard link **across file systems** | Can link across filesystems |
| 5 | If the file deleted, still the other file can be accessable | If the main file deleted, pointer will be broken link |

#### comm
Prints the common words between two files, **files needs to be sorted before using comm**

# Print common words b/w two files
$ comm -12 file1 file2

# print the uniq words b/w two files
$ comm -3 file1 file2

#### banner
used to create a fancy text

## Process management
### Process states
* Runnable - process that can be executed
* Sleeping - process waiting for an event
* Zombie - process trying to terminate
* Stopped - suspended (not allowed to execute)

Runnable:
* Runnable process are ready to execute whenever CPU time is availab.e

Sleeping:
* Sleeping process are waiting for a specific event to happen.
* E.g. Interactive input from terminal, system daemons, and network connections
* **Uninterruptable sleep** (**D state**):
* A transient state, where it cannot be killed even with **kill -9** or SIGTERM.
* To get rid of this, reboot the server, or wait for the event to be completed
* Example: NFS filesystems mouted with the *hard* option.

Zombie:
* Zombie process are finished its execution, but it is yet to had their status called.
* If thre is a zombie process, check its PPID
### Nice & renice
* Range: -19 to +20, -19 being the highest priority and +20 is the lowest priority.
* A newly created process inherits the value of its parent process (unless user uses nice utility to change)
### LRU cache
* LRU (Least Recently Used) cache is a **cache replacement algorithm**
* It removes least recently used data from the cache, to make room for new data.
### Tools

vmstat 1
pidstat 1

mpstat -P ALL 1
# shows utilization for each CPU core

1. Nice vs PR
Nice value is used to increase the priority of the process in user-space.
-20 is the highest level, 0 is the default and +19 is the lowest level.

To launch a program with default nice value

$ nice -n -20 firefox

To set nice value to an existing program

$ renice -n -20 -p <process-id>

PR is the actual priority level managed in kernel.
Range in PR is 0 to 139, in which 0 to 99 is reserved for real time processes, remainings can be used by user-space programs.

2. Maximum process IDs
The default maximum process that a Linux can run is 32768.
To view the max process that can run

$ cat /proc/sys/kernel/pid_max

Limit the no. of process for a specific user
use `ulimit` or /etc/limits.conf

To get the parent process ID
Use `getppid()` syscall
OR

$ echo ${PPID}

3. Find which process ID is used by which process

lsof -i -n -P 

4. What are system calls?
* System calls provide the **interface between a process and the Operating system**.
* When a system call is executed, it is **treated by the hardware as software interrupt**.
*
5. Fill CPU by 100%

for i in $(seq 1 2); do sha1sum /dev/zero & done

# To kill
pkill sha1sum

## Network
1. Internet is not working
* Is the interface is UP - `ip`
* Is the routing rules are correct - `ip route`
* If DNS is working - `nslookup google.com`
* If `nslookup` not working, that might indicate DNS issue.
* Check name servers config - `cat /etc/resolv.conf`
* Do `nslookup google.com <the nameserver IP you see in resolv.conf>`
* Check `/etc/hosts`
* Check Network configurations
* `cat /etc/sysconfig/network`
* `cat /etc/sysconfig/network-scripts/ifcfg-*`
* Check name service switch configs
* `cat /etc/nsswitch.conf`
* Make ensure `DNS` priority is set appropriately, not `files` or something.
3. How do you calculate network throughput
Network throughput determines how much data can be transferred in a medium.

To calculate avg n/w throughput:

# on server
iperf3 -s -f M
# -f M = in MegaBytes

# on client
iperf3 -c <ip-of-server> -f M

## Memory management
### Tools
#### vmstat
* `vmstat` shows memory usage over a period of time

vmstat -sSM

vmstat -SM 2 10
# vmstat -SM <report-interval> <total-count-of-reports>
# for every 2 seconds, it reports totally for 10 times
# -s = overall report
# -SM = in human readable format
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0  22121    279   4634    0    0    48    51  504  135  7  3 89  1  0
 0  0      0  22129    279   4626    0    0     0   302 2552 2862  4  2 95  0  0

* Explanation on terms
* `procs` - process
* `r` - running or runnable process (process waiting for to run)
* `b` - no. of process in uninterruptable sleep (`D` state)
* `memory`
* `swpd` - amount of virtual memory used
* `free` - amount of ideal memory
* `buff` - amount of memory used as buffers
* `cache` - amount of memory used as cache
* `swap`
* Explanation:
* swapin - pages swapped to disk from memory temporarily
* swapout - pages swapped from disk to memory for execution
* `si` - amount of memory swapped in from disk
* `so` - amount of memory swapped to disk
* `io`
* `bi` - Blocks received from a block device
* `bo` - Blocks sent to a block device
* `System`
* `in` - No. of interrupts per seconds
* `cs` - No. of context switch per second
* `CPU`
* Terms are based on %
* `us` - % of time spent on running userspace code
* `sy` - % of time spent on running kernel space code
* `id` - % of time spent idle
* `wa` - % of time spent waiting for I/O
* `st` - % of time stolen from a virtual memory
### Terminologies
`Page`: the blocks that are used in memory, typically they have a 4K size
`Virtual memory`: total of allocatable memory, also known as the process address space
`Paging`: getting memory from secondary storage in primary storage
`Swap`: emulated memory on disk
`Translation lookaside buffer`: a cache that helps speeding up the translation between virtual memory and RAM ***
`Cache`: fast memory that is close to CPU
`Page cache`: area where recently used memory pages are stored

/proc/meminfo

To remove cache and buffer:

free -h
echo 3 > /proc/sys/vm/drop_caches; free -h

Active/inactive memory:
$ cat /proc/meminfo
active, inactive. active(anon) - anonymous memory used by applications, inactive(anon), active(file) - cache memory, inactive(file)

Inactive Anonymous memory should move to swap.
If not, that means less swap space.
Also, tune the kernel with swappiness parameter.

Swap:

Swap space should be calculated based on inactive anonymous memory. *****
inactive anon memory = swap
Monitor swap usage ==> $ vmstat 3 12 (si = swap in, so = swap out)

a. fdisk -> 82 -> w
b. mkswap /dev/sdb1
c. swapon /dev/sdb1
d. swapon -s (monitor swap usage); also with $ vmstat

Configuring kernel to swapping out the memory: ****
cat /proc/sys/vm/swappiness
modify the value to increase the swappiness
If the inactive anonynous mem very high, do this ^^^^^^ ********

---
Dirty memory:
Dirty memory is memory that has yet to be written to disk
* Writes are optimized by working with dirty memory
If the system crashes in the mean time, it will be gone.
* File system barriers can be enabled to prevent losing data
* /proc tunables are available to manage dirty memory without parameters

Increasing the period of time that the diry memory can stay in the memory:
$ echo 40000 > /proc/sys/vm/dirty_expire_centisecs

---

Monitoring CPU cache:
$ valgrind --tool=cachegrind ls

Memory Leaks:

* A memory leak occurs when a process doesn't give memory back that it doesn't need anymore
* This will result in system running out of resources
* We can find processes with a memory leak using $ valgrind --tool=memcheck memleak
e.g. $ valgrind --tool=memcheck firefox

Find specific process memory utilitzation

cat /proc/<PID>/status
look for Vm*

Find top most process consuming memory

ps -A --sort -rss -o comm,pmem,rss |head -n 11

## Filesystem
1. XFS
* The inode allocation for XFS file system is dynamic, means the inode usage will be increased/decreased based on the demand for the process.
* Ref: https://access.redhat.com/solutions/5687161
2. Backup and restore file system permissions

# to backup perms
$ getfacl -R / > /tmp/perms.txt
# to restore perms
$ setfacl --restore=/tmp/perms.txt

## Systemd

# list all services 
systemctl --type service --all
systemctl --type service --state=stopped
service --status-all

## Spotify
1. What happens when run a command? e.g. cat
a. read and loaded from disk in RAM
b. related libraries must be found and loaded in RAM

all these tasks provided by system calls and library calls

2. Understanding system calls
why system calls? -> process cannot access the kernel directly.
system calls are used as an interface for process to the kernel

what is glibc? -> it provides an interface to use system calls from programs

3. What are the system calls determines a process
fork() and exec() determines how process starts

4. man man -> use options e.g. man 2 intro, etc
man mmap - memory management ***
man 2 syscall
cd /proc/<pid>
cat cmdline (common to all to find which program is it) ****
cat maps **** (memory usage of that process)

5. library functions - man 3 intro

6. User space vs Kernel space
* Hardware access is restricted to the kernel only
* The kernel provides system calls for users and processes to access hardware
* User space is memory that is allocated by the kernel for user processes
* Several elements are running in user space
* Network configuration
* Services like a web server
* Applications
* User interfaces
* Users are created to assign permissions and limits to entities on Linux
* Every process or serviceses running on Linux has an owner

3.6 strace and ltrace ******
* strace - trace system calls
* strace ls
* strace -c ls
* strace ls 2>&1 |grep open
* ltrace ls

3.7 signals
* Signals provides software interrupt
* man 7 signal
* pidof opera
* kill -9 == SIGKILL 9 (kill the process in brutal way)
* kill -s USR1 $(pidof opera) (kills all process of opera) *******

4.1 Boot procedure overview
* UEFI/BIOS -> bootable disk -> boot loader(GRUB2) -> kernel (initramfs) -> init (0r systemd) -> everything

4.2 UEFI vs BIOS (incomplete)
* UEFI has the ability to boot from large disks when a GUID Partition Table (GPT) used
* xxd -l 512 /dev/sda (shows master boot record) **********

4.3 Boot loaders
* Use to load OS kernel
* Boot Loaders -> 1. GRUB, 2. GRUB2, 3. SYSLINUX (common in PXE boot), 4. LILO

4.4 Grub2 config
* /boot/grub2/grub.cfg (stateless)
* /etc/grub2.d/ (<- allow to make changes)
* /etc/default/grub (GRUB interface)
Tip: rhgb - redhat graphical boot, quiet - hide and show logo instead
removing these two string results showing logs instead of logo

After modification of grub2, following must be done
$ grub2-mkconfig -o /boot/grub2/grub.cfg

Booting into single user mode -> boot -> in linux16 line add -> systemd.unit=rescue.target
Generate initramfs -> $ dracut --force
How to add third party drivers in Linux kernel?
a. vim /etc/dracut.conf
add_drivers+="<add-driver-here>" e.g. virtio
b. update -> $ dracut --force; reboot

driver location -> /lib/modules/<kernel-version>/kernel/drivers/

Booting into initramfs -> boot -> in linux16 -> rd.break
Booting from initrafs interface to normal mode -> $ exec init

<=====init================================>
What is the need of init /sbin/init?
Init process starting the all userspace environment.

What are all the userspace services started by init?
low-level services such as udevd, syslog, file system mounts. Then network configuration will load.
Then high-level services such as cron, printing or a web server

Types of loading -> serial and parallel

Find current run-level - $runlevel

What are all run-levels?
0 - halt
1 - single user mode
2 - multi-user, without NFS
3 - full multi-user mode
4 - unused
5 - X11
6 - reboot
<======upstart=============================>
Created by Ubuntu
It is reactionary, it receives events and runs jobs based on these events
It makes ensure that the services are loading only when it needed *****
How to find upstart? -> /etc/init file found
Configuration files -> /etc/init.d/ <- service scripts (shell scripts)
Run-level scripts -> /etc/inittab
<======systemd=============================>
* serial loading, fast boot
* uses target (group of unit files) instead runlevel
* Unit files specify loading of services and more, as well as all dependencies that need to met to load them
* How to find systemd? -> /usr/lib/systemd

Types of unit files
* Service
* Mount
* Timer (replacement for cron)
* Automount
* Target (group of unit files)
* Path
And more

Location of unit files => /usr/lib/systemd/system/ (don't modify here) and /etc/systemd/system/ (system servics, modify here if want)

Systemd dependencies:
* Requires: defines units that must be loaded to load this unit
* Wants: typically seen in targets, defines which units should be loaded but loading continues if this fails
* Requisite: the defined unit must already be active. If it is not, systemd fails loading this unit
* Conflicts: Units that may never be active when this unit is loaded (if the mentioned unit is active, don't load the unit that we're writing)
* Before: the current unit will activate before the listed units
* After: the current unit will activate after the listed units

Reload the systemd after modification -> $ systemctl daemon-reload

List systemd services dependencies -> $ systemctl list-dependencies

Find out the systemd process by process ID

$ systemctl status <PID>

Sample service creation:
$ cp /usr/lib/systemd/system/fstrim.service /etc/systemd/system/buvanesh.service
$ vim /etc/systemd/system/buvanesh.service
Execstart=/usr/bin/dd if=/dev/zero of=/dev/null

$ systemctl daemon-reload
$ systemctl start buvanesh

**Note: Do $ systemctl show buvanesh
set Restart=yes, it will restart the service automatically even if the service fails

Sockets:
* Used to make a listen on an IP address and port.

Targets:
* Group of units
* Targets are only allowed as service if AllowIsloate=yes defined.

Systemd performance analyser: ****
$ systemd-analyze critical-chain
$ systemd-analyse plot > output.html
browse the html content via browser

Auto-restart service if failed:

systemctl edit svc.service
# under service section
Restart=on-failure
RestartSec=30s

<==============Logging=============>

* stderr
* own log e.g. apache log /var/log/access_log
* rsyslog ***** for centralized logging
* systemd-journald