About Me

My photo
Rohit leads the Pivotal Labs App Modernization Practice in engineering, delivery training & cross-functional enablement, tooling, scoping, selling, recruiting, marketing, blog posts, webinars and conference sessions. Rohit has led multiple enterprise engagements including ones featured in the Wall Street Journal. Rohit focuses on designing, implementing and consulting with enterprise software solutions for Fortune 500 companies on application migration and modernization.

Tuesday, December 30, 2014

Debugging DEA Issues in Cloud Foundry

Scenario 0: Failure of the app at staging or runtime and the container still exists
cloudfoundry-community/cf-ssh

Scenario 1: Tracking down app by CPU usage, Accessing and Preserving a warden container, Monitoring the NATs message bus

Troubleshooting Applications | Cloud Foundry Docs
Scenario 2: Failures in Staging with no access to BOSH
1. Do a push with 'CF_TRACE=true' and look for a HTTP header called 'X-App-Staging-Log'. This will contain a route to the staging DEA's directory server. 
2. Then ssh onto a router VM and curl http://XXXX:XXXX@ROUTER_IP:PORT/routes. The username, password, and port can be found in /var/vcap/jobs/gorouter/config/gorouter.yml.
3. Look for the route to the directory server in the router's routes, which will tell you the IP of the staging DEA.
4. When you get on the DEA, you can look for your app's name in the dea_next/dea_next.log file.
5. Look for a warden_handle_id in the same log line as your app's name within the dea_next.log.
6. Once you get the warden_handle cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

Scenario 3: How can i determine where an app deployed is actually running ?  I need to debug something between an app and an external service
1. Find the GUID of the app using the following command CF_TRACE=true cf app APP_NAME
2. Use cf curl /v2/apps/GUID/stats to find the DEA VM host/port
3. If its a connection issue from the dea to the service .. get into the container and try to connect ... also look at the security groups
4. look for the app GUID in a  file called /var/vcap/data/dea_next/db/instances.json on the DEA VM. instances.json lists the state of the container, the app guid, and the warden_container_path.
5. Once you get the warden_handle then cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh