---
tags: interview
---
# Linux questions
## Linux networking
1. How do you troubleshoot DNS resolution issues?
2. Types of network bonding in Linux
3. View ARP table/ ARP table cache
```bash=
$ arp -n
? (10.76.99.253) at c8:e7:f0:b3:2f:00 [ether] on enp0s31f6
? (10.76.99.254) at 00:00:5e:00:01:01 [ether] on enp0s31f6
? (10.76.99.252) at c8:e7:f0:b3:25:00 [ether] on enp0s31f6
# OR
$ ip neigh
```
3. How do you troubleshoot packet drops?
* To find the packet drops `$ netstat -i` or `ethtool -S eth0`
* Check in `/sys/class/net/eth0/statistics/`
* Check the **MTU values** of both adapters. The MTU values must be compatible to allow communication. A machine drops packets if the incoming packet (including headers) is greater than the adapter's MTU values. For instance, a 1500-byte packet sent over the bridge collects an 8-byte LLC header, making the total packet size 1508. If the receiving machine MTU is set to 1500, a packet of 1508 bytes is dropped.
* Increase RX ring buffer `$ ethtool -G eth0 rx 4080`
4. How do you troubleshoot DNS?
```bash=
# check network configuration
$ cat /etc/resolv.conf
# deep
$ dig google.com +trace
```
5. Force `dig` to use custom nameserver
```bash=
$ dig @8.8.8.8 google.com
```
6. Ensure the machine is always reachable
* Make ensure you have two network interface
* Bind the two interface to single via teaming as below:
```bash
# consider you have eth1 and eth2 interfaces
# goal is to make ensure if one interface is down,
# the traffic should route to another interface
## Create an interface for teaming
nmcli con add type team con-name Team0 ifname team0
## Set the ip address for the team interface
nmcli con mod Team0 ipv4.address 10.0.1.15/24 ipv4.gateway 10.0.1.1 ipv4.method manual
## Add eth1 and eth2 interfaces to Team0 interface
nmcli con add type team-slave ifname eth1 con-name Slave1 master
nmcli con add type team-slave ifname eth2 con-name Slave2 master team0
nmcli con up Team0
ping 10.0.1.15
```
5. How to copy files between two machines?
```bash=
$ scp -3 user@server1:/file1 user@server2:/dest/
```
6. How do you troubleshoot network issues?
Layer by layer approach.
Layer 1: Physical layer
a. Check if the interface is UP (cable connected?)
```bash=
$ ip a
# UP the interface
$ ip link set eth0 up
# see if there is a packet loss
$ netstat -i
# OR for detailed
$ ip -s link show eth0
# OR
$ ethtool -S eth0
# for advanced troubleshooting
$ ethtool -S eth0
```
Layer 2: Data Link layer
A common problem you might encounter is an ARP entry that won’t populate, particularly for your host’s default gateway. If your localhost can’t successfully resolve its gateway’s Layer 2 MAC address, then it won’t be able to send any traffic to remote networks. This problem might be caused by having the wrong IP address configured for the gateway, or it may be another issue, such as a misconfigured switch port.
```bash=
# ARP entries
$ ip neighbor show
# If there is an issue
$ ip neighbor show
192.168.122.1 dev eth0 FAILED
# To fix, flush arp table
$ ip -s -s neigh flush all
```
Layer 3: Network layer
IP configuration in local system, router config etc.
```bash=
$ ping google.com
$ route -n
# OR
$ ip route
$ traceroute google.com
$ mtr google.com
$ nslookup google.com
```
Layer 4: Transport layer
```bash=
# check listening ports
$ ss -tunlp4
# check if the port is open (TCP)
$ telnet google.com 22
# check if the port is open (UDP)
$ nc google.com -u 22
```
7. Use of /etc/nsswitch.conf
It is used to prioritice the order of services (network, dns, protocols etc.)
Refer: http://linuxconfig.net/manuals/howto/unix-file-etc-hosts-and-etc-nsswitch-conf.html
8. How to allow only specific IPs for a specific services
OR
How to block specific user from a specific system
Use **tcp wrappers*
/etc/hosts.allow
/etc/hosts.deny
9. How do you troubleshoot network issues?
1. Check if others can access
2. Check if the port is open
```
nmap -p 80 mysite.com
telnet mysite.com 80
```
3. Check if you're able to resovle the address
```
nslookup mysite.com
```
4. If you're able to ping with IP, then the issue is with DNS.
1. Check `/etc/resolv.conf`
2. Try dig with that nameserver
```
dig @10.0.4.5 mysite.com
```
3. If it responds, then the issue might be with `/etc/nsswitch.conf` file
4. Check `/etc/nsswitch.conf` and make ensure dns is set for address resolution
5. Check for `/etc/hosts` to confirm there is no misconfiguration found.
10. [IMP] Find which network interface you're using when you're connecting to internet
```bash
$ ip route
default via 192.168.43.1 dev wlp0s20f3 proto dhcp metric 600
192.168.43.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.43.132 metric 600
192.168.56.0/24 dev vboxnet0 proto kernel scope link src 192.168.56.1
# As per above, default interface is wlp0s20f3
```
11. tcpdump
```bash
# monitor port 80 traffic of 192.168.56.129 host
tcpdump -i vboxnet0 -n src host 192.168.56.129 and src port 80
```
12. nc
```bash
# nc, also called netcat is useful utility to troubleshoot ports
# On server
nc -l 8080
# here, 8080 is the port you want to check
# On client
nc server.example.com 8080
```
13. Refer the important commands
https://wizardzines.com/networking-tools-poster/
## Core
1. Interesting questions: http://lifepluslinux.blogspot.com/2017/01/my-solutions-to-cmdchallenge.html
1. Generate random password
```bash
$ echo $RANDOM
# OR
openssl rand -base64 12
```
2. Print currently used kernel arguments - `cat /proc/cmdline`
2. Check harware details - `dmidecode -t memory`
4. Find what shell currently using
```bash
echo $SHELL
```
2. How to know which distro you're using? `cat /etc/os-release` or `hostnamectl`
2. Create a file in which even a root use cannot delete (immutable file)
```bash
chattr +i immutable.txt
# to revert
chattr -i immutable.txt
```
3. Use of `/etc/profile` and `/etc/profile.d/`
* `/etc/profile` - system wide environment variables set
* `/etc/profile.d/` - custom scripts that can be added to run at the time of system startup
5. Troubleshooting - kernel panic - hardware failure, kdum, broken fs
6. How to print the kernel parameters which are being in use?
```
sysctl -a
```
7. **fork()** and **exec()** system calls
* fork() **creates a new process by duplicating existing process**.
when the the new process si created, it receives a PID, using getpid() system call.
when the child is terminated, the parent gets back the child PID.
* exec() **creates a mew process by replacing the parent process**.
when the child is exits, the child sends an exit() to the parent, who can next cleanup.
5. https://mycyberuniverse.com/linux/find-and-delete-the-zero-size-files-and-empty-directories.html
6. Run a command for a specified time
```
$ timeout 5 date
```
7. Print all the available commands
```compgen```
8. Unable to delete a file/dir
```
#check attributes of the file
lsattr
#remove if immutable (i) or append (a) ttributes found
chattr -i filename
chattr -a filename
#check if permission bits are correct
#check if the file system mounted readonly
```
9. Difference between bashrc and profile
bashrc - for lon-login shell, specific to bash, it gets **executed after system boot**
profile - will be executed **after user login**, NOT specific to bash, for login shell users.
10. How do you check runlevel?
```bash=
$ runlevel
$ who -r
```
## Filesystem
1..`journalctl --disk-usage`
1. Find the most biggest files/directory in current folder
```bash=
du -sh * |sort -rh |head
```
2. Create N size of file
```bash=
fallocate -l 10G gentoo_root.img
```
3. What are inodes?
* inodes are data structures in Linux file system
* when a filesystem is created, a set of inodes also created for the filesystem.
* usually about 1% of file system disk space is allocated to inode table.
inodes are Linux data structures, that contains information about files.
4. Diff b/w `superblock` and `inodes`
* `superblock` - Metadata of the filesystem, it contains the filesystem type, size, status
* `inodes` - Metadata of the files stored in the filesystem, it contains the ownership (user, group), access mode (read, write, execute permissions), file type, and the data blocks with the file's content.
5. Do `inodes` contains the data of the file - No
4. Your filesystem is full, you have deleted most of the log files, but still the system reports the filesystem is full.
* Check if `inodes` are full
```bash
df -ih
```
5. How do you decide between XFS and ext4?
* XFS performs better when:
* you have the large server and storage (and large size of files)
* XFS has limitations:
* Cannot shrik filesystem
* XFS consumes about twice the CPU-per-metadata operation compared to ext4.
* ext4 performs better when:
* the system has limited I/O capability
* the system has limited bandwidth (less than 200MB/s) and upto 1000 IOPS capability. (anything above, XFS is the choice)
6. Do inode contains filenames?
* No, inode contains only the metadata of the file.
* Filename is stored under the directory which it presents.
* The directory also has an inode entry for the filename.
7. What are the details included in inode metadata?
* inode metadata contains:
* inode number
* UID & GID of the owner
* Size of the file
* The number of blocks the file uses
* The last time of modification/creation/acccess and changes to the file
## Process management
1. What is a proc filesystem?
3. Common ports
DNS: 53, SMTP: 25
1. Difference between `process` and `thread`
https://www.linkedin.com/posts/alex-xu-a8131b11_systemdesign-coding-interviewtips-activity-6924379939022528513-sXVc/?utm_source=linkedin_share&utm_medium=member_desktop_web
1. Find the process ID of which using the specific port
```bash
fuser 80/tcp
# This will return the process ID which is using port 80
```
2. Find parent process ID of a process
```bash
ps -o ppid= <process ID>
pstree -aps <process ID>
```
3. Kill a service with systemd
```bash
systemctl kill -s SIGKILL servicename.service
```
4. Stop a service forcefully
```bash
systemctl stop servicename
systemctl mask servicename
```
5. Run the process in background: `command &`
5. Find which process is using which CPU
6. How `kill -9 <pid>` works?
* 9 is the signal number for SIGKILL, which terminates the process. It will not be a graceful shutdown, unsaved data may get lost or corrupted.
7. SIGKILL vs SIGTERM
* SIGKILL: Terminates the process without waiting for application's response. This signal will not be ignored by the application.
* SIGTERM: Terminates the process, graceful shutdown.
8. Zombie vs Orphan process
* Orphan: A process which don't have parent process, the init process will be the parent process.
* Zombie:
7. Move process from one CPU to another CPU
```bash
$ taskset -pc 3 $(pidof dd)
# p = process, c= CPU, moves process to 3rd CPU
```
3. How do you disable specific CPU?
```bash
$ echo 0 > /sys/bus/cpu/devices/cpu1/online
$ lscpu
```
3. List processes and the users accessing /var/www/html
```bash=
fuser -av /var/www/html
# If malicious user/process found, kill the user/process
fuser -k /var/www/html
```
4. Find first 5 highest file opener processes
```bash
yum install pcp-system-tools -y
pmrep proc.fd.count -J 5 -1 -g
[ 1] - proc.fd.count["002241 dbus-broker"] - count
[ 2] - proc.fd.count["002341 /usr/bin/gnome-shell"] - count
[ 3] - proc.fd.count["003084 /usr/lib64/firefox/firefox"] - count
[ 4] - proc.fd.count["007603 /usr/lib/slack/slack"] - count
[ 5] - proc.fd.count["007738 /usr/lib64/firefox/firefox"] - count
1 2 3 4 5
99 113 307 128 79
99 113 307 128 79
```
6. Process states
$ ps aux
In the STAT column, you'll see lots of values. A linux process can be in a number of different states. The most common state codes you'll see are described below:
**R**: running or runnable, it is just waiting for the CPU to process it
**S**: Interruptible sleep, waiting for an event to complete, such as input from the terminal
**D**: Uninterruptible sleep, processes that cannot be killed or interrupted with a signal, usually to make them go away you have to reboot or fix the issue.
State D means uninterruptible sleep, which usually means IO. When the IO completes the process will change state. If you think a specific process is hung you can truss or strace it to see what it is doing. i.e.
`strace -p <pid of process>`
How to clear D-state process ?
D-state process is nothing but uninterruptible sleep. When doing IO, the process is locked as uninterruptible so the data will not be corrupted. When IO finished, the process will not be D-sate
But for some reason the IO could not be finished or hung, then the process may stick in D. in this cases we need to reboot the server to clear D-state process.
**Z**: Zombie, we discussed in a previous lesson that zombies are terminated processes that are waiting to have their statuses collected
**T**: Stopped, a process that has been suspended/stopped
5. How do you kill zombie process?
Zombie process cannot be killed, as it is already dead.
7. Find how many CPUs in the system
```bash=
nproc
lscpu
top
cat /proc/cpuinfo
```
6. Get environment variables used by a specific process
```bash=
cat /proc/<pid>/environ
cat /proc/*/environ
# ^ for all processes
```
7. List all of the pseduo file systems: `cat /proc/self/mounts` or `/etc/mtab`
9. kill all process of a specific user
```bash=
killall -u username
```
8. How do you limit the no. of process for a specific user?
```
cat /etc/security/limits.conf
ramesh hard nproc
# nprc = max. Number of processes
```
10. What is procfs?
* Profs is a virtual (pseduo) file system mounted in `/proc`
* It contains the current running state of the Linux system
9. How to find the total number of process? `ps -e |wc -l`
9. Difference between RSS (Resident Set Size) and VS (Virtual Size)
RSS:
* Indicates the memory it is consumed = it includes both main memory (RAM) and Swap
10. Difference between `dmesg` and `/var/log/messages`
`dmesg` - stores contents of kernel ring buffer, a ring buffer is a *fixed size* (512K bydefault), it stores logs data as soon as the system passes bootloader phase. since it is a fixed size, it cleans up the log as soon as the ring buffer fills up. It also sends the log to /var/log/messages
`/var/log/messages` - `dmesg` only takes care of kernel logs, and it sends only kernel logs to `/var/log/messages`, `/var/log/messages` also stores logs about the system, which is achieved by using `rsyslog` service.
11. How do you list list of open files by a process? IMP
```bash=
# By listing file descriptors of the process
ls -l /proc/<pid>/fd/
# Using lsof
lsof -p <pid>
# Using strace, on the fly
strace -e open ls
## for existing process
strace -e open -p $(pgrep -o top)
```
12. Find files which are used by a specific port
```bash
# first find the process ID of that port
$ netstat -tulpn |grep 80
tcp6 0 0 :::80 LISTEN 44138/httpd
# use 11th method to list the files for the process id
```
12. How do you troubleshoot SSH with strace? [IMP]
```bash
strace -e open -ff -p $(pgrep -o sshd)
## -ff - follows forked processes
strace -e accept -ff -p $(pgrep -o sshd)
# On another machine, login and notice the log
# Do ls on the another machine, notice the log
## Useful options
# -ff = follow forked processes
# -e = show or exclude (using !) specific system calls.
# -o file = save output to a file
# -t = print timestampt
# -c = statics on how many times a call is executed. press CTRL+C to see the statics.
```
14. When you do `pgrep chrome` you'll get multiple process IDs, how do you figureout the parent process ID?
```
pgrep -o chrome
```
15. What are cgroups?
cgroups has a set of controllers, that limits how much resources a process can consume.
17. Find who deleted the files
* Install and configure `auditd`
* Specify the path which you want to monitor in auditd config
`-F dir=/opt/test/kubernetes.io~aws-ebs`
* Search with `ausearch -k <filename>` to find out
* check if any process producing log files
17. You're trying to start a service, but the service failed to start and not giving any useful output as well. How do you debug?
```
systemctl cat service-name.service
# Find the execution command
# Execute it manually with strace
strace <command>
```
18. How do you troubleshoot cronjob?
https://stackoverflow.com/a/34872041/9403545
19. How do you monitor cronjob?
* Use an `exit` code in cronjob script
* check https://phrye.com/code/periodic-monitoring/
## Memory management
1. What is page fault?
Page fault occurs when a program access a page that is mapped in virtual address, but not in main memory.
2. What is tranparent hugepages
## Package management
1. How do you figure out the package name by a binary or a directory?
```bash=
rpm -qf /etc/ansible
ansible-2.9.2-1.fc30.noarch
rpm -qf /bin/vncpasswd
tigervnc-server-minimal-1.10.1-1.fc30.x86_64
```
2. Find if the configuration of installed application is modified or a fresh software
```bash=
$ rpm -V bluejeans-1.37.22-1.x86_64
.M....... /opt/bluejeans/package.nw.json
.M....... /opt/bluejeans/packages
.M....... /opt/bluejeans/packages/bundle.json
.M....... /opt/bluejeans/packages/carmelrivet_1.37_r_car1.37_ef7c509_1.37.8.zip
.M....... /opt/bluejeans/packages/com.bluejeans.meetings.json
.M....... /opt/bluejeans/packages/rivet.json
.M....... /opt/bluejeans/packages/skinny_plugin.json
.M....... /opt/bluejeans/packages/skinny_plugin_linux_beta_beta-3._3.0.6.5.zip
# M means those files are modified by the user/process after the installation.
```
3. Find what package provides the executable/configuration/any file ***
```bash
rpm -q --whatprovides /etc/httpd
httpd
rpm -q --whatprovides /var/adm
filesystem
rpm -q --whatprovides /bin/ls
coreutils
```
4. Find executables which are not provided by any package managers
```bash
find /usr/bin/ -exec rpm -q --whatprovides {} \; |grep "not owned"
# OR
find /usr/bin |xargs rpm -q --whatprovides |grep "not owned"
```
5. Find the dependencies for a package
```bash=
rpm -Qpr
```
6. List currently instaled RPMs
```bash
yum list installed
# OR
rpm -qa
```
7. List dependencies for a package
```bash
yum deplist <pkg>
```
8. Find packages which are installed recently.
```bash
rpm -qa --last
```
9. How do you lock an RPM, so it cannot be uninstalled by users
```bash
Add the packages in /etc/yum/protected.d/
```
## Security
1. Top 10 important security hardening for Linux
1. To see SELinux security context
```bash=
ls -lZ
```
2. How do you troubleshoot SELinux errors
```bash
audit2allow
# generates SELinux policy allow/don't audit rules from logs of denied operations
audit2why
# translates SELinux audit messages into a description of why the access was denied
```
## Scenario based
1. The user is unable to start the process
* Check the user's **process limit**: `/etc/security/limits.conf`
* Check the pidmax: `cat /proc/sys/kernel/pidmax` compare it with the currrent usage `ps -e |wc -l`
* Check the max file usage: `cat /proc/sys/fs/file-max`
* Check the **inode usage**: `df -i`
* Try `strace`
Note: If the filesystem is XFS, the inode usage is dynamic, means it is based on the available free disk space.
2. The user unable to login to the system
* SSH service up?
* The port number for the SSH service?
* Firewall blocked?
* Enable verbose mode: `ssh -v`
* Check the listening on the server: `ss -tulpn`
* Configuration issue
3. The server is slow, how do you troubleshoot?
https://scoutapm.com/blog/slow_server_flow_chart
### Important
Review Linux interview questions in
* https://www.thegeekdiary.com/?s=%22linux%22+%22interview%22
* https://github.com/myllynen/rhel-troubleshooting-guide
* https://github.com/myllynen/rhel-performance-guide
Sign in with Wallet
Connect another wallet