About Me

My photo
Rohit leads the Pivotal Labs App Modernization Practice in engineering, delivery training & cross-functional enablement, tooling, scoping, selling, recruiting, marketing, blog posts, webinars and conference sessions. Rohit has led multiple enterprise engagements including ones featured in the Wall Street Journal. Rohit focuses on designing, implementing and consulting with enterprise software solutions for Fortune 500 companies on application migration and modernization.

Monday, September 28, 2015

Debugging DEA Issues on Cloud Foundry - Halloween Edition

Ever wondered why Cloud Foundry does not support ssh'ing into the warden container. Such a feature could be useful in so many situations - TechOps, Troubleshooting, Debugging etc. cf-ssh is coming to Cloud Foundry via Diego; however till you deploy Diego in production you will need to live with the process below to ssh into a particular application instance warden container.

# Step 1 - Find app guid

C:\Users\Administrator\workspace\FizzBuzz>cf app --guid fizzbuzz
90291bd7-ce52-43ee-aaa1-ed0405863c4a

# Step 2 - Find DEA IP address and port


C:\Users\Administrator\workspace\FizzBuzz>cf curl /v2/apps/90291bd7-ce52-43ee-aaa1-ed0405863c4a/stats
{
   "0": {
      "state": "RUNNING",
      "stats": {
         "name": "fizzbuzz",
         "uris": [
            "fizzbuzz.kelapure.cloud.pivotal.io"
         ],
         "host": "192.168.200.27",
         "port": 61015,
         "uptime": 1489673,
         "mem_quota": 1073741824,
         "disk_quota": 1073741824,
         "fds_quota": 16384,
         "usage": {
            "time": "2015-09-29 02:36:12 +0000",
            "cpu": 0.003104336638753874,
            "mem": 546877440,
            "disk": 187445248
         }
      }
   }
}


# Step 3 - Locate DEA Job that maps to the DEA IP from previous step

ubuntu@pivotal-ops-manager:~$ bosh vms
Acting as user 'director' on 'microbosh-38a3a7433db69fa7d159'
Deployment `cf-938e3d9bec67dbffeacc'

Director task 178

Task 178 done


ubuntu@pivotal-ops-manager:~$ bosh vms --details
Acting as user 'director' on 'microbosh-38a3a7433db69fa7d159'
Deployment `cf-938e3d9bec67dbffeacc'

Director task 179

Task 179 done

+----------------------------------------------------------------+---------+--------------------------------------------------------------+----------------+-----------------------------------------+--------------------------------------+--------------+
| Job/index                                                      | State   | Resource Pool                                                | IPs            | CID                                     | Agent ID                             | Resurrection |
+----------------------------------------------------------------+---------+--------------------------------------------------------------+----------------+-----------------------------------------+--------------------------------------+--------------+
| ccdb-partition-0d7a243620d08147fd3a/0                          | running | ccdb-partition-0d7a243620d08147fd3a                          | 192.168.200.15 | vm-df650acf-88e9-4b1a-b68e-a2ff11b47a65 | 1e315067-aef5-4f5d-ad00-50b492f98085 | active       |
| clock_global-partition-0d7a243620d08147fd3a/0                  | running | clock_global-partition-0d7a243620d08147fd3a                  | 192.168.200.22 | vm-f905d0b1-d4f4-4646-826b-6255e19fc0f0 | 992e57f7-003b-42f2-9818-3db5d4a64402 | active       |
| cloud_controller-partition-0d7a243620d08147fd3a/0              | running | cloud_controller-partition-0d7a243620d08147fd3a              | 192.168.200.18 | vm-29aeeadb-b9c3-4ca2-ab32-49573e957697 | 413a5a5f-06f4-4359-9074-13d3bbd11e35 | active       |
| cloud_controller_worker-partition-0d7a243620d08147fd3a/0       | running | cloud_controller_worker-partition-0d7a243620d08147fd3a       | 192.168.200.23 | vm-ded90340-de9f-4d15-8ac4-540669395d8a | 300814cb-6f23-4068-86ba-b6569bedf259 | active       |
| consoledb-partition-0d7a243620d08147fd3a/0                     | running | consoledb-partition-0d7a243620d08147fd3a                     | 192.168.200.17 | vm-9ef7f4fe-f6a9-4bab-89cb-0942d499e40c | bef26f14-cb91-4bbe-bc1d-3d2d27368ba6 | active       |
| consul_server-partition-0d7a243620d08147fd3a/0                 | running | consul_server-partition-0d7a243620d08147fd3a                 | 192.168.200.12 | vm-e57e77dd-25c6-4d3f-a440-4dd0b31d28f2 | af13ba63-068f-4ee8-bd8a-48b02438767e | active       |
| dea-partition-0d7a243620d08147fd3a/0                           | running | dea-partition-0d7a243620d08147fd3a                           | 192.168.200.27 | vm-699ed235-f7f4-4094-a92e-963ef726b1d6 | 437bbc76-7d1c-4ce9-916d-4e6a6355537f | active       |
| dea-partition-ee97ca1101e7cc2c048a/0                           | running | dea-partition-ee97ca1101e7cc2c048a                           | 192.168.200.28 | vm-91378b2c-7592-4b14-9f73-07c57043fe75 | 561899ad-00d4-4eec-a7f3-a1e8e9c84e10 | active       |
| doppler-partition-0d7a243620d08147fd3a/0                       | running | doppler-partition-0d7a243620d08147fd3a                       | 192.168.200.29 | vm-7ca4b286-13d9-4d84-aec0-8a1fcdf81cae | 4f243e01-2c89-4f77-8a37-31b8b0c5f1c8 | active       |
| doppler-partition-ee97ca1101e7cc2c048a/0                       | running | doppler-partition-ee97ca1101e7cc2c048a                       | 192.168.200.30 | vm-150f7c12-2bf9-4da3-bfec-8e342051a203 | f454702e-db36-4e37-a02d-3c44e1c57822 | active       |
| etcd_server-partition-0d7a243620d08147fd3a/0                   | running | etcd_server-partition-0d7a243620d08147fd3a                   | 192.168.200.13 | vm-fe166f4f-e060-407d-8ba1-74d50f17e22e | 1ea07f41-aa95-4ef0-9a3a-8a7a90886b1c | active       |
| ha_proxy-partition-0d7a243620d08147fd3a/0                      | running | ha_proxy-partition-0d7a243620d08147fd3a                      | 192.168.200.20 | vm-1fb87e81-d1cd-4abe-82fc-91bddf9e99dc | 2d4a5048-9a9d-4a8f-bf34-0967b8f0bbf5 | active       |
| health_manager-partition-0d7a243620d08147fd3a/0                | running | health_manager-partition-0d7a243620d08147fd3a                | 192.168.200.21 | vm-9e84e893-22a2-45c4-8f49-bcd65013494a | 7e4ce9e1-d5e9-4ec8-9ba0-2b9885a20fd5 | active       |
| loggregator_trafficcontroller-partition-0d7a243620d08147fd3a/0 | running | loggregator_trafficcontroller-partition-0d7a243620d08147fd3a | 192.168.200.31 | vm-f3d7075b-f3b6-4ea0-8307-590ee63e7bf3 | 7bf8256d-903f-4ce0-b56e-8f88c00c80ea | active       |
| loggregator_trafficcontroller-partition-ee97ca1101e7cc2c048a/0 | running | loggregator_trafficcontroller-partition-ee97ca1101e7cc2c048a | 192.168.200.32 | vm-f4ed6afa-2073-47b0-8ae3-d9a08e85a209 | 0df8b5e8-e539-471f-ac0d-4e6d68c79c09 | active       |
| mysql-partition-0d7a243620d08147fd3a/0                         | running | mysql-partition-0d7a243620d08147fd3a                         | 192.168.200.26 | vm-a73374e9-ffd8-4cd3-8bb4-519d35778143 | c4da718c-0964-4004-82b8-58c3341e8cb9 | active       |
| mysql_proxy-partition-0d7a243620d08147fd3a/0                   | running | mysql_proxy-partition-0d7a243620d08147fd3a                   | 192.168.200.25 | vm-e94670b3-3754-40f6-b86b-3dcf88d8542e | 06a3c9ae-3e64-4eec-82e7-0c91be97e81c | active       |
| nats-partition-0d7a243620d08147fd3a/0                          | running | nats-partition-0d7a243620d08147fd3a                          | 192.168.200.11 | vm-9b9855f5-3e5f-4de8-872b-347fbc56c984 | 4e621e1a-718a-4c55-931f-2c7b68504d80 | active       |
| nfs_server-partition-0d7a243620d08147fd3a/0                    | running | nfs_server-partition-0d7a243620d08147fd3a                    | 192.168.200.14 | vm-a28bad65-4fac-40ac-a383-4b30725966e1 | f02359a8-5c03-4942-ab6c-5ac32c536d6a | active       |
| router-partition-0d7a243620d08147fd3a/0                        | running | router-partition-0d7a243620d08147fd3a                        | 192.168.200.19 | vm-5b00e6db-669d-4115-b786-5c0b6c5b6c78 | 82111c58-b944-4099-8f56-5ec6b195e54a | active       |
| uaa-partition-0d7a243620d08147fd3a/0                           | running | uaa-partition-0d7a243620d08147fd3a                           | 192.168.200.24 | vm-aab6ab86-3eca-490d-aa7a-9d720b057721 | 5e95d2f3-f871-4a0c-ae81-ee1effb5fa1a | active       |
| uaadb-partition-0d7a243620d08147fd3a/0                         | running | uaadb-partition-0d7a243620d08147fd3a                         | 192.168.200.16 | vm-a472fe4e-40d8-4c3f-a95a-e6e8e521cf86 | 336af737-8bc1-4f62-bd40-ae3b63f85637 | active       |
+----------------------------------------------------------------+---------+--------------------------------------------------------------+----------------+-----------------------------------------+--------------------------------------+--------------+

VMs total: 22

In our case the DEA Job is  *dea-partition-0d7a243620d08147fd3a/0*


# Step 4 Login into the DEA VM

ubuntu@pivotal-ops-manager:~$ bosh ssh dea-partition-0d7a243620d08147fd3a/0 --public_key y.pub
Acting as user 'director' on deployment 'cf-938e3d9bec67dbffeacc' on 'microbosh-38a3a7433db69fa7d159'
Enter password (use it to sudo on remote host): ********
Target deployment is `cf-938e3d9bec67dbffeacc'

see https://github.com/cloudfoundry/bosh-lite/issues/134  if stuck on bosh ssh


# Step 5 Locate the warden container path


If you grep for `fizzbuzz`and the instance port gleaned from step-2 in the list below you will find the following 

"warden_container_path": "/var/vcap/data/warden/depot/18tlhc59f3v",

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot$ sudo cat /var/vcap/data/dea_next/db/instances.json

[sudo] password for bosh_yms06qjkj:
{
  "time": 1443494275.6732097,
  "instances": [

    {
      "cc_partition": "default",
      "instance_id": "07a4a39d7ec14249863303246c73dfa2",
      "instance_index": 0,
      "private_instance_id": "5e15175c18644561828e52534b7c7a71fad5773f13b045058eaed76793618b1f",
      "warden_handle": "18tlhc59erj",
      "limits": {
        "mem": 512,
        "disk": 1024,
        "fds": 16384
      },
      "health_check_timeout": null,
      "environment": {
        "CF_PROCESS_TYPE": "web"
      },
      "services": [

      ],
      "application_id": "093a9e11-0b06-45f1-b3c4-e801ad0aca81",
      "application_version": "2febead7-c0fe-43a5-b4a4-724c5581c037",
      "application_name": "tmfnodetest",
      "application_uris": [
        "tmfnodetest.kelapure.cloud.pivotal.io"
      ],
      "droplet_sha1": "fcba91c3099ba1218febfcb043d3792ccc9bba97",
      "droplet_uri": null,
      "start_command": null,
      "state": "RUNNING",
      "warden_job_id": 461,
      "warden_container_path": "/var/vcap/data/warden/depot/18tlhc59erj",
      "warden_host_ip": "10.254.2.41",
      "warden_container_ip": "10.254.2.42",
      "instance_host_port": 61127,
      "instance_container_port": 61127,
      "syslog_drain_urls": [

      ],
      "state_starting_timestamp": 1442435859.1693246
    },

    {
      "cc_partition": "default",
      "instance_id": "9383f43f0e3549ad9c29c511c5f4211e",
      "instance_index": 0,
      "private_instance_id": "23065b9faddb43d2ae296802c1db8cbdbf2a2d00dd2343849c8825bcfc7eb044",
      "warden_handle": "18tlhc59enu",
      "limits": {
        "mem": 1024,
        "disk": 1024,
        "fds": 16384
      },
      "health_check_timeout": null,
      "environment": {
        "CF_PROCESS_TYPE": "web"
      },
      "services": [
        {
          "credentials": {
            "agent-name": "nginx-hello",
            "host-name": "ca-apm.springapps.io",
            "port": "5001"
          },
          "options": {

          },
          "syslog_drain_url": "",
          "label": "user-provided",
          "name": "ca_apm_10",
          "tags": [

          ]
        }
      ],
      "application_id": "90291bd7-ce52-43ee-aaa1-ed0405863c4a",
      "application_version": "09f093ae-c9e8-46ba-89c8-e56d7b84b671",
      "application_name": "fizzbuzz",
      "application_uris": [
        "fizzbuzz.kelapure.cloud.pivotal.io"
      ],
      "droplet_sha1": "e5fc94cc79eb72489e94c6b620887c5b72244b89",
      "droplet_uri": "http://staging_upload_user:c601ae7f5ae3745d40ee@192.168.200.18:9022/staging/droplets/90291bd7-ce52-43ee-aaa1-ed0405863c4a/download",
      "start_command": null,
      "state": "RUNNING",
      "warden_job_id": 92,
      "warden_container_path": "/var/vcap/data/warden/depot/18tlhc59enu",
      "warden_host_ip": "10.254.0.85",
      "warden_container_ip": "10.254.0.86",
      "instance_host_port": 61015,
      "instance_container_port": 61015,
      "syslog_drain_urls": [
        ""
      ],
      "state_starting_timestamp": 1442004499.0728624

    },

    {
      "cc_partition": "default",
      "instance_id": "fa057152a7094ee6bfd0cd28c8cb76dc",
      "instance_index": 0,
      "private_instance_id": "18b2515c046742ae979fbfa58428120608960bc04f7444759983d1c91621e7da",
      "warden_handle": "18tlhc59f3v",
      "limits": {
        "mem": 1024,
        "disk": 1024,
        "fds": 16384
      },
      "health_check_timeout": null,
      "environment": {
        "CF_PROCESS_TYPE": "web"
      },
      "services": [

      ],
      "application_id": "d9969088-1f7b-40b3-a048-c71814d172c4",
      "application_version": "431e7f3d-774b-4c24-ad4e-c6fda884dab1",
      "application_name": "spring-music",
      "application_uris": [
        "spring-music.kelapure.cloud.pivotal.io"
      ],
      "droplet_sha1": "9e35ddfd97a7c1a32e2a69d5e1c5f90c6d4b7e06",
      "droplet_uri": "http://staging_upload_user:c601ae7f5ae3745d40ee@192.168.200.18:9022/staging/droplets/d9969088-1f7b-40b3-a048-c71814d172c4/download",
      "start_command": null,
      "state": "CRASHED",
      "warden_job_id": 1265,
      "warden_container_path": "/var/vcap/data/warden/depot/18tlhc59f3v",
      "warden_host_ip": "10.254.2.113",
      "warden_container_ip": "10.254.2.114",
      "instance_host_port": 61395,
      "instance_container_port": 61395,
      "syslog_drain_urls": [

      ],
      "state_starting_timestamp": 1443494262.4416354
    }
  ],
  "staging_tasks": [

  ]
}

# Step 6 wsh into the warden container

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot$ cd 18tlhc59enu

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ ls
bin  destroy.sh  etc  jobs  lib  mnt  net_rate.sh  net.sh  run  setup.sh  snapshot.json  start.sh  stop.sh  tmp

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ sudo ./bin/wsh
wsh   wshd  

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ sudo ./bin/wsh
root@18tlhc59enu:~# ls
firstboot.sh
root@18tlhc59enu:~# cd /home
root@18tlhc59enu:/home# cd vcap/
root@18tlhc59enu:/home/vcap# ls
app  logs  run.pid  staging_info.yml  tmp

root@18tlhc59enu:/home/vcap# ll
total 40
drwxr-xr-x 5 vcap vcap 4096 Sep 11 20:48 ./
drwxr-xr-x 3 root root 4096 Sep 11 20:48 ../
drwxr--r-- 7 vcap vcap 4096 Sep 11 20:47 app/
-rw-r--r-- 1 vcap vcap  220 Apr  9  2014 .bash_logout
-rw-r--r-- 1 vcap vcap 3637 Apr  9  2014 .bashrc
drwxr-xr-x 2 vcap vcap 4096 Sep 11 20:47 logs/
-rw-r--r-- 1 vcap vcap  675 Apr  9  2014 .profile
-rw------- 1 vcap vcap    3 Sep 11 20:48 run.pid
-rw-r--r-- 1 vcap vcap 2000 Sep 11 20:47 staging_info.yml
drwxr-xr-x 3 vcap vcap 4096 Sep 11 20:48 tmp/
root@18tlhc59enu:/home/vcap#



# Step 7 - Now you are free to ftp files out or take heap dumps or thread-dumps

For instance to take a heapdump
root@18tlhc59enu:/home/vcap/app/.java-buildpack/open_jdk_jre/bin# ll
total 384
drwxr-xr-x 2 vcap vcap   4096 Sep 11 20:47 ./
drwxr-xr-x 5 vcap vcap   4096 Sep 11 20:47 ../
-rwxr-xr-x 1 vcap vcap   8798 Jul 16 09:29 java*
-rwxr-xr-x 1 vcap vcap   8909 Jul 16 09:29 jcmd*
-rwxr-xr-x 1 vcap vcap   8909 Jul 16 09:29 jjs*
-rwxr-xr-x 1 vcap vcap   8973 Jul 16 09:29 jmap*
-rwxr-xr-x 1 vcap vcap   8981 Jul 16 09:29 jstack*
-rwxr-xr-x 1 vcap vcap   8917 Jul 16 09:29 keytool*
-rwxr-xr-x 1 vcap vcap   1146 Sep 11 20:47 killjava.sh*
-rwxr-xr-x 1 vcap vcap   8981 Jul 16 09:29 orbd*
-rwxr-xr-x 1 vcap vcap   8917 Jul 16 09:29 pack200*
-rwxr-xr-x 1 vcap vcap   8917 Jul 16 09:29 policytool*
-rwxr-xr-x 1 vcap vcap   8909 Jul 16 09:29 rmid*
-rwxr-xr-x 1 vcap vcap   8917 Jul 16 09:29 rmiregistry*
-rwxr-xr-x 1 vcap vcap   8917 Jul 16 09:29 servertool*
-rwxr-xr-x 1 vcap vcap   8989 Jul 16 09:29 tnameserv*
-rwxr-xr-x 1 vcap vcap 217462 Jul 16 09:29 unpack200*

root@18tlhc59enu:/home/vcap/app/.java-buildpack/open_jdk_jre/bin# su vcap

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ PID=` ps -ef | grep java | grep -v "bash\|grep" | awk '{print $2}'`
vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ echo $PID
29

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ ./jmap -dump:format=b,file=/home/vcap/app/test.hprof $PID
Dumping heap to /home/vcap/app/test.hprof ...
Heap dump file created

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ ls -al /home/vcap/app
total 230616
drwxr--r-- 7 vcap vcap      4096 Sep 29 03:06 .
drwxr-xr-x 5 vcap vcap      4096 Sep 11 20:48 ..
drwxr-xr-x 5 vcap vcap      4096 Sep 11 20:47 .java-buildpack
-rw-r--r-- 1 vcap vcap     82155 Sep 11 20:47 .java-buildpack.log
drwxr--r-- 3 vcap vcap      4096 Sep 11 20:46 META-INF
drwxr--r-- 3 vcap vcap      4096 Sep 11 20:46 my-resources
drwxr--r-- 3 vcap vcap      4096 Sep 11 20:46 org
-rw------- 1 vcap vcap 236032777 Sep 29 03:06 test.hprof
drwxr--r-- 5 vcap vcap      4096 Sep 11 20:46 WEB-INF

No comments:

Post a Comment