About Me

My photo
Rohit leads the Pivotal Labs App Modernization Practice in engineering, delivery training & cross-functional enablement, tooling, scoping, selling, recruiting, marketing, blog posts, webinars and conference sessions. Rohit has led multiple enterprise engagements including ones featured in the Wall Street Journal. Rohit focuses on designing, implementing and consulting with enterprise software solutions for Fortune 500 companies on application migration and modernization.

Monday, October 12, 2015

Heapdumps and CoreDumps on Cloud Foundry containers

SSH into the warden container following instructions here and follow the instructions below ...

Heapdumps:
su vcap
PID=` ps -ef | grep java | grep -v "bash\|grep" | awk '{print $2}'`
jmap -dump:format=b,file=/home/vcap/app/test.hprof $PID

Coredumps:
I played with this some more and the process below worked for generating core dumps:

ulimit -c unlimited
gdb --pid=33
Attaching to process 33
Reading symbols from /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java...(no debugging symbols found)...done.
Reading symbols from /home/vcap/app/.java-buildpack/open_jdk_jre/bin/../lib/amd64/jli/libjli.so...(no debugging symbols found)...done.
Loaded symbols for /home/vcap/app/.java-buildpack/open_jdk_jre/bin/../lib/amd64/jli/libjli.so
...
Reading symbols from /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java...(no debugging symbols found)...done.
Reading symbols from /home/vcap/app/.java-buildpack/open_jdk_jre/bin/../lib/amd64/jli/libjli.so...(no debugging symbols found)...done.
Loaded symbols for /home/vcap/app/.java-buildpack/open_jdk_jre/bin/../lib/amd64/jli/libjli.so
...
(gdb) gcore


(gdb) quit
A debugging session is active.
Inferior 1 [process 33] will be detached.
Quit anyway? (y or n) y
Detaching from program: /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java, process 33

root@18tlhc59f7e:~# ls -al
total 1236312
drwx------  2 root root       4096 Oct 12 14:19 .
drwxr-xr-x 33 root root       4096 Oct 12 14:19 ..
-rw-------  1 root root        985 Oct  6 04:47 .bash_history
-rw-r--r--  1 root root       3106 Feb 20  2014 .bashrc
-rw-r--r--  1 root root 1265958640 Oct 12 14:20 core.33
-rwxr-xr-x  1 root root        213 Jul 24 21:08 firstboot.sh
-rw-r--r--  1 root root        140 Feb 20  2014 .profile

Note kill -SIGABRT 33 did not work. 

In miscellaneous notes everyone should read the following article on memory management in CF containers http://fabiokung.com/2014/03/13/memory-inside-linux-containers/
Most of the Linux tools providing system resource metrics were created before cgroups even existed (e.g.: free and top, both from procps). They usually read memory metrics from the proc filesystem: /proc/meminfo,/proc/vmstat/proc/PID/smaps and others. Unfortunately /proc/meminfo/proc/vmstat and friends are not containerized
Most container specific metrics are available at thecgroup filesystem via /path/to/cgroup/memory.stat,/path/to/cgroup/memory.usage_in_bytes,/path/to/cgroup/memory.limit_in_bytes and others.