# Kernel Same-Page Merging (KSM)

###### tags: `linux` `kernel` `ksm`

The Kernel same-page merging (KSM) is used for KVM hypervisor to merge memory pages (share identical memory pages) occupied by many running virtual machines. It is useful especially memory over commit in your compute node.

I did a basic test about this feature when I was in developing the CAKE system in III.

### Test Environment

- CAKE 2.0 – 1.2.19 (Center+Node)
- Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (4 cores, 8 threads (HT enabled))
- 32 GB RAM (Kingston, 1600MHz, DDR3, 8192 MB * 4)
- 16 GB Swap
- Memory Page Size: 4096 bytes (4K) (echo $(getconf PAGESIZE))
- VM Specification: 1 Core, 4 GB RAM
- Guest VM OS:
  - Ubuntu  10.04 – 64bit
  - CentOS  6.4 – 64bit
  - WinXP – 32bit
- Memory Test: Using PRNG – generating 1 ~ 99999999 random numbers (output is temporary cached in memory to simulate memory read/write, and then release memory after finished)
- “ksmtuned” service is stopped to stop affect ksm 

!!! I used 3 different kind of guest OS to test the merging efficiency. The ubuntu and centOS should have similar memory pages compare to the WinXP.

### Test Flow

1. Stop KSM service
    "service ksm stop"

2. Release merged page
    "echo 2 > /sys/kernel/mm/ksm/run"

3. Clear memory cache
    "echo 3 > /proc/sys/vm/drop_caches”"

4. Get memory usage
    "free -m”"

5. Start KSM service and wait until “pages_sharing” stable
    "service ksm start”"

6. Monitor KSM status and memory usage
    "free -m”, “./checkKSM_pages_info.sh"

7. Running PRNG in all VMs and wait until all PRNG completed. *To test the page merging when running heavy application in VMs.

8. Monitor KSM status and memory usage.
    "free -m”, “./checkKSM_pages_info.sh"

### Test Result Matrix - 1 (KSM disabled)

### Test Result Matrix - 2 (KSM enabled)

!!! We can see that KSM takes effect to merge same page memory.

### KSM status on the CAKE system
- The CAKE has “ksm” and “ksmtuned” services enabled by default. (runlevel 3,4,5)
- The “ksmtuned” is a service to control “ksm”. It will start or stop the “ksm” and also configure the “ksm” setting, such as “sleep_millisecs” and “pages_to_scan”
- The default configure of “ksmtuned” is set to monitor system every 60 seconds to see whether or not to start/stop “ksm”.
- CAKE’s memory usage from WebUI is matched to memory usage after KSM compressed memory.
- The KSM could merge more same pages if the Guest OS is similar or the same, but it still depends on actual status of memory pages. The KVM could request that identical guest virtual machine memory regions be shared (according to KSM manual).

### Operation of "ksmtuned"
- Configuration file is “/etc/ksmtuned.conf”
- KSM_THRES_COEF is a percentage of system memory that used to calculate “thres”. It is set to 20 by default (20 percent of system memory).
- The “thres” value is an activation threshold which is used to decide whether to start “ksm”.
- If the “thres” value plus sum of all “qemu-kvm” (VM process) processes RSZ size (or RSS column, in ps command) is greater than entire system memory, it will start the “ksm”.
- KSM_NPAGES_MIN and KSM_NPAGES_MAX are boundary of npages value(number of pages to scan).
- Different system memory size has different default configuration.

### When the "ksmtuned" start or stop "ksm"?
- Test on CAKE 2.0 – 1.2.19, 32GB RAM System.
- “ksmtuned” start the “ksm” when free memory is less than 7 GB around.
- “ksmtuned” stop the “ksm” when free memory is more than 7 GB around.
- This result somehow match to the “thres” value to trigger “ksm”. (20% of system memory (6565MB)+ qemu-kvm memory usage)
- This test is not very accurate to indicate the actual memory size to trigger ksm running. Because ksmtuned is running every 60 seconds by default.

Starting ksmtuned:                                         [  OK  ]
2014-07-01 17:38:17: ksmtuned start
Starting ksm:                                              [  OK  ]
2014-07-01 17:38:17: ksm start
2014-07-01 17:38:17: KSM is running. 8783 MB used. 23273 MB free.
2014-07-01 17:39:17: KSM is not running. 6959 MB used. 25097 MB free.
2014-07-01 18:10:18: KSM is running. 25742 MB used. 6315 MB free.
2014-07-01 18:14:18: KSM is not running. 22219 MB used. 9837 MB free.
2014-07-01 18:18:18: KSM is running. 25337 MB used. 6719 MB free.
2014-07-02 09:26:14: KSM is not running. 5536 MB used. 26521 MB free.
2014-07-02 10:36:15: KSM is running. 26088 MB used. 5969 MB free.
2014-07-02 10:40:15: KSM is not running. 21113 MB used. 10944 MB free.
2014-07-02 11:04:16: KSM is running. 26019 MB used. 6038 MB free.
2014-07-02 11:05:16: KSM is not running. 25890 MB used. 6166 MB free.
2014-07-02 11:06:15: KSM is running. 26069 MB used. 5987 MB free.
2014-07-02 11:08:16: KSM is not running. 21428 MB used. 10628 MB free.
2014-07-02 11:15:16: KSM is running. 29210 MB used. 2847 MB free.
2014-07-02 11:16:16: KSM is not running. 19230 MB used. 12827 MB free.

### Scripts for the testing
#### checkKSM_pages_info.sh

#!/bin/bash

pages_shared=`cat /sys/kernel/mm/ksm/pages_shared`
pages_sharing=`cat /sys/kernel/mm/ksm/pages_sharing`
pages_unshared=`cat /sys/kernel/mm/ksm/pages_unshared`
pages_volatile=`cat /sys/kernel/mm/ksm/pages_volatile`
pages_to_scan=`cat /sys/kernel/mm/ksm/pages_to_scan`
sleep_millisecs=`cat /sys/kernel/mm/ksm/sleep_millisecs`
full_scans=`cat /sys/kernel/mm/ksm/full_scans`
run=`cat /sys/kernel/mm/ksm/run`

ksm_rate=$(echo $pages_sharing $pages_shared | awk '{printf($1/$2)}')

echo "KSM saved:    $(( $pages_sharing * $(getconf PAGESIZE) /1024/1024 )) MB"
echo "Sharing_Rate: $ksm_rate"
echo "---------------------------------"
echo "Run:             $run"
echo "Sleep_Millisecs: $sleep_millisecs"
echo "Full_Scans:      $full_scans"
echo "Pages_Sharing:   $pages_sharing"
echo "Pages_Shared:    $pages_shared"
echo "Pages_Unshared:  $pages_unshared"
echo "Pages_Volatile:  $pages_volatile"
echo "Pages_To_Scan:   $pages_to_scan"

#### checkKSM_start_time.sh

#!/bin/bash

echo "$(date +"%Y-%m-%d %T"): Test Start"
echo "stop ksm and ksmtuned. clear cache and merged page."
service ksm stop
service ksmtuned stop
echo 2 > /sys/kernel/mm/ksm/run
echo 3 > /proc/sys/vm/drop_caches

sleep 3

echo "*******************************************"
free -m
echo "*******************************************"

service ksmtuned start
echo "$(date +"%Y-%m-%d %T"): ksmtuned start"
service ksm start
echo "$(date +"%Y-%m-%d %T"): ksm start"

run=0
while [ 1 ]
do
  ksmStatus=`service ksm status | grep 'is running'`
  mem=`free -m | grep Mem:`
  used=`p1=${mem//'Mem:'/''}; arr=($p1); echo ${arr[1]}`
  free=`p1=${mem//'Mem:'/''}; arr=($p1); echo ${arr[2]}`
  if [[ $ksmStatus != '' ]]
  then
    if [[ $run == 0 ]]
    then
      line1="$(date +"%Y-%m-%d %T"): KSM is running."
      line2=" $used MB used."
      line3=" $free MB free."
      echo $line1 $line2 $line3
      run=1
    fi
  else
    if [[ $run == 1 ]]
    then
      line1="$(date +"%Y-%m-%d %T"): KSM is not running."
      line2=" $used MB used."
      line3=" $free MB free."
      echo $line1 $line2 $line3
      run=0
    fi
  fi
  sleep 1
done