LSA 番外篇 - 從入門到放棄 - Docker/Container

--- tags: 1082, lsa --- - Book mode https://hackmd.io/@ncnu-opensource/book LSA 番外篇 - 從入門到放棄 - Docker/Container === ## Slide - [從入門到放棄 - Docker/Container ](https://docs.google.com/presentation/d/1_yO4bGYioqd503ub4J2XLO3vK4_WFuryYpQvRYU_I1Y/edit?usp=sharing) ## docker - Docker components/docker daemon - https://docs.docker.com/get-started/overview/ ``` $ curl -s --unix-socket /var/run/docker.sock http:/localhost/info | jq . ``` - You can also access other [docker APIs](https://docs.docker.com/engine/api/v1.24/ ) - [docker lifecycle](https://docs.docker.com/get-started/overview/) - How the docker terminate the running container. Please refer [docker stop](https://docs.docker.com/engine/reference/commandline/stop/). > The main process inside the container will receive SIGTERM, and after a grace period, SIGKILL. - `docker stop` - `docker ps -a` - Exitcodes: http://tldp.org/LDP/abs/html/exitcodes.html - nginx - sigterm : https://github.com/nginxinc/docker-nginx/blob/594ce7a8bc26c85af88495ac94d5cd0096b306f7/mainline/buster/Dockerfile#L101 ## namespace - namespace and cgroup - Linux capabilities - How many type namespace does docker use? Ans: [Docker overview](https://docs.docker.com/get-started/overview/#namespaces) > The pid namespace: Process isolation (PID: Process ID). The net namespace: Managing network interfaces (NET: Networking). The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication). The mnt namespace: Managing filesystem mount points (MNT: Mount). The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System). - `docker inspect <YOUR_DOCKER>` - cgroup: https://docs.docker.com/config/containers/runmetrics/ - https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt - [OOM](https://www.kernel.org/doc/gorman/html/understand/understand016.html) - Please verify your Ubuntu enable the cgroup swap limit capabilities(`cgroup_enable=memory` ). - How to find - `pstree` - `ps` - `/proc/<pid>/cgroup` - How to mapping the host pid and the container process. Thanks for `yiyu` assistance. ![](https://i.imgur.com/w8DB79s.png) ## logging - log driver ``` $ docker inspect ff7c3b30abd4 | jq .[].HostConfig.LogConfig { "Type": "json-file", "Config": {} } ``` - How the docker logs - `docker logs` - https://docs.docker.com/config/containers/logging/ ``` $ docker inspect ff7c3b30abd4 | jq .[].LogPath "/var/lib/docker/containers/ff7c3b30abd46c2d4c138e9a59bd9f1e0bc367da5db81e384d32d2659e5cfc7b/ff7c3b30abd46c2d4c138e9a59bd9f1e0bc367da5db81e384d32d2659e5cfc7b-json.log" ``` - [Nginx Dockerfile](https://github.com/nginxinc/docker-nginx/blob/master/stable/alpine/Dockerfile#L104-L106) ## storage - What is docker image layer - https://docs.docker.com/storage/storagedriver/ ## Network - docker network - iptable: how the `docker run -d -p 80:8080` works? If you want to check the network namespace, we can use `nsenter` command to run the command. ``` $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c51caebc6999 test-oom:latest "bash" 10 minutes ago Up 10 minutes my-test-oom $ for id in `docker ps -n 1 -q`; do pid=`docker inspect $id -f '{{.State.Pid}}'`; echo container $id; sudo nsenter -t $pid -n ip link show; done container c51caebc6999 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 ``` ## 延伸閱讀有興趣可以在 `1082 - LSA` Telegram group 討論： - Why containers can be accessed with the services name in `docker-compose.yml` - k8s: why k8s pods can asscess with `localhost`? ## Test environment ``` $ sudo lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.4 LTS Release: 18.04 Codename: bionic ``` ``` $ docker info Running: 0 Server Version: 19.03.8 [0/53] Storage Driver: overlay2 Backing Filesystem: <unknown> Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 5.3.0-1017-aws Operating System: Ubuntu 18.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 978.8MiB Name: ip-172-31-32-58 ID: DXL4:POV4:6DLW:IBTW:2ZHX:NX6S:3ULN:4ME5:QQUY:4LYC:ZSYX:FRME Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support ``` - By default, Ubuntu did not enable the `cgroup_enable=memory`, You may see messages similar to the following when working with an image. - Please follow the [Post-installation steps for Linux - Your kernel does not support cgroup swap limit capabilities](https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities) section. ``` WARNING: Your kernel does not support swap limit capabilities. Limitation discarded. ``` ### Test OOM via stress-ng Add a Dcokerfile. ```Dockerfile= From ubuntu:20.04 RUN apt update && apt install stress-ng -y ``` Built a image with the `stress-ng` tool. ``` $ docker build . -t test-OOM invalid argument "test-OOM" for "-t, --tag" flag: invalid reference format: repository name must be lowercase See 'docker build --help'. 125 ec2-user@ip-172-31-35-188:~$ docker build . -t test-oom Sending build context to Docker daemon 4.755MB Step 1/2 : From ubuntu:20.04 ---> 1d622ef86b13 Step 2/2 : RUN apt update && apt install stress-ng -y ---> Running in bd3213d5bc74 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB] Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB] Get:3 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [1077 B] Get:4 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [13.2 kB] Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [134 kB] Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [42.3 kB] Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [107 kB] Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB] Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB] Get:10 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB] Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB] Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB] Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [13.2 kB] Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [251 kB] Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [136 kB] Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [1077 B] Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [2900 B] Fetched 14.0 MB in 5s (2712 kB/s) Reading package lists... Building dependency tree... Reading state information... 5 packages can be upgraded. Run 'apt list --upgradable' to see them. WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: libaio1 libapparmor1 libbsd0 libipsec-mb0 libjudydebian1 libsctp1 Suggested packages: lksctp-tools The following NEW packages will be installed: libaio1 libapparmor1 libbsd0 libipsec-mb0 libjudydebian1 libsctp1 stress-ng 0 upgraded, 7 newly installed, 0 to remove and 5 not upgraded. Need to get 2377 kB of archives. After this operation, 20.2 MB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libapparmor1 amd64 2.13.3-7ubuntu5.1 [34.1 kB] Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libbsd0 amd64 0.10.0-1 [45.4 kB] Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio1 amd64 0.3.112-5 [7184 B] Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 libipsec-mb0 amd64 0.53-1 [491 kB] Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjudydebian1 amd64 1.0.5-5 [94.6 kB] Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libsctp1 amd64 1.0.18+dfsg-1 [7876 B] Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 stress-ng amd64 0.11.07-1 [1697 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 2377 kB in 2s (1077 kB/s) Selecting previously unselected package libapparmor1:amd64. (Reading database ... 4122 files and directories currently installed.) Preparing to unpack .../0-libapparmor1_2.13.3-7ubuntu5.1_amd64.deb ... Unpacking libapparmor1:amd64 (2.13.3-7ubuntu5.1) ... Selecting previously unselected package libbsd0:amd64. Preparing to unpack .../1-libbsd0_0.10.0-1_amd64.deb ... Unpacking libbsd0:amd64 (0.10.0-1) ... Selecting previously unselected package libaio1:amd64. Preparing to unpack .../2-libaio1_0.3.112-5_amd64.deb ... Unpacking libaio1:amd64 (0.3.112-5) ... Selecting previously unselected package libipsec-mb0. Preparing to unpack .../3-libipsec-mb0_0.53-1_amd64.deb ... Unpacking libipsec-mb0 (0.53-1) ... Selecting previously unselected package libjudydebian1. Preparing to unpack .../4-libjudydebian1_1.0.5-5_amd64.deb ... Unpacking libjudydebian1 (1.0.5-5) ... Selecting previously unselected package libsctp1:amd64. Preparing to unpack .../5-libsctp1_1.0.18+dfsg-1_amd64.deb ... Unpacking libsctp1:amd64 (1.0.18+dfsg-1) ... Selecting previously unselected package stress-ng. Preparing to unpack .../6-stress-ng_0.11.07-1_amd64.deb ... Unpacking stress-ng (0.11.07-1) ... Setting up libapparmor1:amd64 (2.13.3-7ubuntu5.1) ... Setting up libjudydebian1 (1.0.5-5) ... Setting up libipsec-mb0 (0.53-1) ... Setting up libsctp1:amd64 (1.0.18+dfsg-1) ... Setting up libaio1:amd64 (0.3.112-5) ... Setting up libbsd0:amd64 (0.10.0-1) ... Setting up stress-ng (0.11.07-1) ... Processing triggers for libc-bin (2.31-0ubuntu9) ... Removing intermediate container bd3213d5bc74 ---> 4b3493a0fd76 Successfully built 4b3493a0fd76 Successfully tagged test-oom:latest ``` Verify the the image you built with `test-oom` tag. ``` $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE test-oom latest 4b3493a0fd76 35 seconds ago 116MB ``` Launch and attach into the image. ``` $ docker run --rm --name my-test-oom -m 200m -it test-oom:latest bash ``` Open another terminal to view `docker stats`. ``` $ docker stats CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS c51caebc6999 my-test-oom 0.00% 1.082MiB / 200MiB 0.54% 906B / 0B 0B / 0B 1 ``` Run the following `stress-ng` command in the container. More information for the [stress-ng](https://wiki.ubuntu.com/Kernel/Reference/stress-ng). ``` root@c51caebc6999:/# stress-ng --brk 2 --stack 2 --bigheap 2 ``` We can view the OOM message in your syslog(`/var/log/syslog`). ```syslog Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25511 (stress-ng-brk) score 1275 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25511 (stress-ng-brk) total-vm:124528kB, anon-rss:56264kB, file-rss:804kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25511 (stress-ng-brk), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25516 (stress-ng-stack) score 1244 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25516 (stress-ng-stack) total-vm:118184kB, anon-rss:49912kB, file-rss:868kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25516 (stress-ng-stack), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25515 (stress-ng-bighe) score 1294 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25515 (stress-ng-bighe) total-vm:128540kB, anon-rss:60076kB, file-rss:864kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25515 (stress-ng-bighe), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25517 (stress-ng-stack) score 1323 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25517 (stress-ng-stack) total-vm:134212kB, anon-rss:66008kB, file-rss:868kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25517 (stress-ng-stack), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25514 (stress-ng-brk) score 1328 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25514 (stress-ng-brk) total-vm:135320kB, anon-rss:67064kB, file-rss:804kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25514 (stress-ng-brk), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25535 (stress-ng-stack) score 1248 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25535 (stress-ng-stack) total-vm:119088kB, anon-rss:50708kB, file-rss:868kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25535 (stress-ng-stack), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25521 (stress-ng-bighe) score 1295 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25521 (stress-ng-bighe) total-vm:128668kB, anon-rss:60344kB, file-rss:864kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25521 (stress-ng-bighe), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25536 (stress-ng-bighe) score 1279 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25536 (stress-ng-bighe) total-vm:125212kB, anon-rss:56912kB, file-rss:864kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25536 (stress-ng-bighe), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25537 (stress-ng-stack) score 1311 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25537 (stress-ng-stack) total-vm:131972kB, anon-rss:63636kB, file-rss:868kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25537 (stress-ng-stack), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25534 (stress-ng-brk) score 1353 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25534 (stress-ng-brk) total-vm:140428kB, anon-rss:72176kB, file-rss:804kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25534 (stress-ng-brk), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25540 (stress-ng-bighe) score 1250 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25540 (stress-ng-bighe) total-vm:119452kB, anon-rss:51104kB, file-rss:864kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25540 (stress-ng-bighe), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25539 (stress-ng-stack) score 1291 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25539 (stress-ng-stack) total-vm:127784kB, anon-rss:59404kB, file-rss:868kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25539 (stress-ng-stack), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25541 (stress-ng-bighe) score 1304 or sacrifice child Jun 17 01:05:28 ip-172-31-35-188 kernel: Killed process 25541 (stress-ng-bighe) total-vm:130460kB, anon-rss:62188kB, file-rss:864kB, shmem-rss:4kB Jun 17 01:05:28 ip-172-31-35-188 kernel: oom_reaper: reaped process 25541 (stress-ng-bighe), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:29 ip-172-31-35-188 kernel: oom_kill_process: 28 callbacks suppressed Jun 17 01:05:29 ip-172-31-35-188 kernel: stress-ng-bighe invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=1000 Jun 17 01:05:29 ip-172-31-35-188 kernel: stress-ng-bighe cpuset=c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 mems_allowed=0 Jun 17 01:05:29 ip-172-31-35-188 kernel: CPU: 0 PID: 25547 Comm: stress-ng-bighe Not tainted 4.14.177-139.253.amzn2.x86_64 #1 Jun 17 01:05:29 ip-172-31-35-188 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 Jun 17 01:05:29 ip-172-31-35-188 kernel: Call Trace: Jun 17 01:05:29 ip-172-31-35-188 kernel: dump_stack+0x66/0x82 Jun 17 01:05:29 ip-172-31-35-188 kernel: dump_header+0x94/0x229 Jun 17 01:05:29 ip-172-31-35-188 kernel: oom_kill_process+0x223/0x420 Jun 17 01:05:29 ip-172-31-35-188 kernel: out_of_memory+0x112/0x4d0 Jun 17 01:05:29 ip-172-31-35-188 kernel: mem_cgroup_out_of_memory+0x49/0x80 Jun 17 01:05:29 ip-172-31-35-188 kernel: mem_cgroup_oom_synchronize+0x2ed/0x330 Jun 17 01:05:29 ip-172-31-35-188 kernel: ? mem_cgroup_css_reset+0xd0/0xd0 Jun 17 01:05:29 ip-172-31-35-188 kernel: pagefault_out_of_memory+0x32/0x77 Jun 17 01:05:29 ip-172-31-35-188 kernel: __do_page_fault+0x4b4/0x4c0 Jun 17 01:05:29 ip-172-31-35-188 kernel: ? page_fault+0x2f/0x50 Jun 17 01:05:29 ip-172-31-35-188 kernel: page_fault+0x45/0x50 Jun 17 01:05:29 ip-172-31-35-188 kernel: RIP: 6309a0f0:0x10000 Jun 17 01:05:29 ip-172-31-35-188 kernel: RSP: ad423010:00007ffd6309a0e0 EFLAGS: 00001000 Jun 17 01:05:29 ip-172-31-35-188 kernel: Task in /docker/c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 killed as a result of limit of /docker/c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 Jun 17 01:05:29 ip-172-31-35-188 kernel: memory: usage 204800kB, limit 204800kB, failcnt 49204 Jun 17 01:05:29 ip-172-31-35-188 kernel: memory+swap: usage 204800kB, limit 409600kB, failcnt 0 Jun 17 01:05:29 ip-172-31-35-188 kernel: kmem: usage 2700kB, limit 9007199254740988kB, failcnt 0 Jun 17 01:05:29 ip-172-31-35-188 kernel: Memory cgroup stats for /docker/c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573: cache:2184KB rss:199916KB rss_huge:0KB shmem:2164KB mapped_file:2184KB dirty:0KB writeback:0KB swap:0KB inactive_anon:2152KB active_anon:199912KB inactive_file:0KB active_file:0KB unevictable:12KB Jun 17 01:05:29 ip-172-31-35-188 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 17 01:05:29 ip-172-31-35-188 kernel: [22120] 0 22120 1028 250 7 3 0 0 bash Jun 17 01:05:29 ip-172-31-35-188 kernel: [23961] 0 23961 17158 1175 13 3 0 1000 stress-ng Jun 17 01:05:29 ip-172-31-35-188 kernel: [23965] 0 23965 17158 378 13 3 0 1000 stress-ng-brk Jun 17 01:05:29 ip-172-31-35-188 kernel: [23966] 0 23966 17158 333 13 3 0 1000 stress-ng-stack Jun 17 01:05:29 ip-172-31-35-188 kernel: [23967] 0 23967 17158 374 13 3 0 1000 stress-ng-bighe Jun 17 01:05:29 ip-172-31-35-188 kernel: [23968] 0 23968 17158 345 13 3 0 1000 stress-ng-brk Jun 17 01:05:29 ip-172-31-35-188 kernel: [23969] 0 23969 17158 322 13 3 0 1000 stress-ng-stack Jun 17 01:05:29 ip-172-31-35-188 kernel: [23970] 0 23970 17158 291 13 3 0 1000 stress-ng-bighe Jun 17 01:05:29 ip-172-31-35-188 kernel: [25538] 0 25538 33440 16578 45 3 0 1000 stress-ng-brk Jun 17 01:05:29 ip-172-31-35-188 kernel: [25542] 0 25542 31002 14146 40 3 0 1000 stress-ng-stack Jun 17 01:05:29 ip-172-31-35-188 kernel: [25543] 0 25543 21217 4356 21 3 0 1000 stress-ng-brk Jun 17 01:05:29 ip-172-31-35-188 kernel: [25544] 0 25544 25143 8301 29 3 0 1000 stress-ng-bighe Jun 17 01:05:29 ip-172-31-35-188 kernel: [25546] 0 25546 23040 6163 25 3 0 1000 stress-ng-stack Jun 17 01:05:29 ip-172-31-35-188 kernel: [25547] 0 25547 18615 1737 16 3 0 1000 stress-ng-bighe Jun 17 01:05:29 ip-172-31-35-188 kernel: Memory cgroup out of memory: Kill process 25538 (stress-ng-brk) score 1320 or sacrifice child Jun 17 01:05:29 ip-172-31-35-188 kernel: Killed process 25538 (stress-ng-brk) total-vm:133760kB, anon-rss:65504kB, file-rss:804kB, shmem-rss:4kB Jun 17 01:05:29 ip-172-31-35-188 kernel: oom_reaper: reaped process 25538 (stress-ng-brk), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB Jun 17 01:05:29 ip-172-31-35-188 kernel: stress-ng-stack invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=1000 Jun 17 01:05:29 ip-172-31-35-188 kernel: stress-ng-stack cpuset=c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 mems_allowed=0 Jun 17 01:05:29 ip-172-31-35-188 kernel: CPU: 0 PID: 25542 Comm: stress-ng-stack Not tainted 4.14.177-139.253.amzn2.x86_64 #1 Jun 17 01:05:29 ip-172-31-35-188 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 Jun 17 01:05:29 ip-172-31-35-188 kernel: Call Trace: Jun 17 01:05:29 ip-172-31-35-188 kernel: dump_stack+0x66/0x82 Jun 17 01:05:29 ip-172-31-35-188 kernel: dump_header+0x94/0x229 Jun 17 01:05:29 ip-172-31-35-188 kernel: oom_kill_process+0x223/0x420 Jun 17 01:05:29 ip-172-31-35-188 kernel: out_of_memory+0x112/0x4d0 Jun 17 01:05:29 ip-172-31-35-188 kernel: mem_cgroup_out_of_memory+0x49/0x80 Jun 17 01:05:29 ip-172-31-35-188 kernel: mem_cgroup_oom_synchronize+0x2ed/0x330 Jun 17 01:05:29 ip-172-31-35-188 kernel: ? mem_cgroup_css_reset+0xd0/0xd0 Jun 17 01:05:29 ip-172-31-35-188 kernel: pagefault_out_of_memory+0x32/0x77 Jun 17 01:05:29 ip-172-31-35-188 kernel: __do_page_fault+0x4b4/0x4c0 Jun 17 01:05:29 ip-172-31-35-188 kernel: ? page_fault+0x2f/0x50 Jun 17 01:05:29 ip-172-31-35-188 kernel: page_fault+0x45/0x50 Jun 17 01:05:29 ip-172-31-35-188 kernel: RIP: 6309a0e0:0x1000 Jun 17 01:05:29 ip-172-31-35-188 kernel: RSP: 40000:00007ffd5ef54d80 EFLAGS: 7ffd5ef54d90 Jun 17 01:05:29 ip-172-31-35-188 kernel: Task in /docker/c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 killed as a result of limit of /docker/c51caebc69999ea567d7acb8b1d7a30eb7f7b3d00b5e7cad7c80ad519e066573 Jun 17 01:05:29 ip-172-31-35-188 kernel: memory: usage 204800kB, limit 204800kB, failcnt 49236 Jun 17 01:05:29 ip-172-31-35-188 kernel: memory+swap: usage 204800kB, limit 409600kB, failcnt 0 Jun 17 01:05:29 ip-172-31-35-188 kernel: kmem: usage 2700kB, limit 9007199254740988kB, failcnt 0 ```