Debugging Node.js high CPU, crash and memory issues in Cloud Foundry - part 1

These are the scenarios that keep a Devops engineer awake in the night - an instance of the node.js app is running at 100% CPU in Cloud Foundry with NO logging or resiliency protection like circuit breakers. This is the third occurrence of the problem in the last month. Management is dragging you into these long drawn day long meetings where each side blames each other - the app is restarted - the problem goes away and the cycle repeats. 

With apps increasingly being wrapped in russian doll layers of abstraction  [baremetal => VM => Container => Node process] getting to the root cause has become increasingly difficult.

So if you are a Java guy that has no experience in debugging a production node.js app. After drinking an appropriate amount of coffee and fruity alcoholic drinks you have landed on http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html

The recommendations from Yunong Xiao in the presentation essentially boils down to:

1. Using node-restify as a REST framework in the node app to log and observe app performance

2. Use Linux Perf Events to statistically sample stack traces  to construct CPU flame-graphs. This technique can reliably find the proverbial needle in the haystack.

3. Take Coredumps to dump the app state for post-mortem debugging with the mdb debugger. Unfortunately the mdb debugger can only run on Solaris- therefore you have to transfer the coredump to a debug solaris VM instance [1] [2]. 

Unfortunately both options 1 and 2 are not possible. 1 because the app-dev team will need to get involved and they don't look upon kindly on a devops guy advising a REST framework. 2 because CF does not bake perf events into the OS stemcell. Yikes!!! Looks like good old gcore is the only way to go  ... 

ok so with the introduction of Diego, finding your way into the container should be as easy as doing a cf-ssh and then issuing the gcore command rite ? - NOPE Wrong - Once ssh'ed into the container there is no way to elevate the privilege and attach gdb to your node.js process. You will need to find another way to elevate privilege and take a core dump.

This other way is using BOSH and Veritas. The gist of the method is that you bosh ssh into the correct Diego cell by first examining the app metadata. Thereafter on the Diego cell install veritas the Garden explorer tool. Use Veritas to map from the app instance to the container instance. Thereafter ./wsh into the garden container. You can now elevate your privilege with a sudo su - and then issue a  gdb, gcore of the node.js server process.

Once the dump has been generated sftp it to a debug solaris instance and run diagnostics with mdb commands like jsstack etc ... see this video from Joyent that explains how to walk the dump.

Part 2 of the blog goes into the gory detail of discovery and  generation of the core dump.

Supplying an external configuration to the Java Buildpack

I wanted to highlight a very important commit recently made to the Java Buildpack i.e.You can now configure the Java Buildpack with a custom server.xml via a repository.

Extension point for supplying an external Tomcat configuration

This allows a CF Platform Operator to provide a vetted server.xml that contains custom Tomcat configuration to the Java Buildpack without forking the buildpack. This feature is OFF by default. In order to enable it set the following env variable in the manifest:

Example in a manifest.yml
env: JBP_CONFIG_TOMCAT: "{ tomcat: { external_configuration_enabled: true }, external_configuration: { repository_root: \"http://repository...\" } }"

The artifacts that the repository provides must be in TAR format and must follow the Tomcat archive structure:

tomcat
|__conf
   |__context.xml
   |__server.xml
   |__web.xml
   |...

You can expect this feature to show up in the next release of the JavaBuildpack and its downstream clones like TomEE.

Deigo BBS, ETCD and Consul - 3 is a crowd

PCF 1.6 has now been completely redesigned on the Diego runtime. In case you are wondering all this all works now checkout the picture below -

Diego internally uses a couple of registries to store internal state primarily Consul for NAT-less service registry and ETCD as an ephemeral data store for the Diego bulletin board service. Once there are corruption issues due to connectivity or disks, these distributed registries  start arguing. They cannot reconcile the state of the world with the CloudController due to quorum or issues with leader election or distributed key lockout. Basically the instances need to get the same key.  You can specify the key in the newer versions, but there is no enforcement of state in the case of corruption.

I am sure there is a proper technical reason why these distributed K,V stores behave they way they do in these situations; however my intent here is to teach you how to get out of this situation if somehow you end up here.

So in situations where Consul and Etcd and Diego Brain BBS jobs act like errant brothers and fight with another to reconstitute the state of the world from the Cloud Controller, leverage the following recipe to keep sanity in the house.

1. Restart all the VMs of the Consul, ETCD and Diego BBS nodes in that order.

2. Wipe out member DB files i.e. nuke the data dir. on the ETCD VMs. monit restart all services on the ETCD nodes.To delete the contents of the etcd data store see:

1. Run bosh vms to identify the etcd nodes in your deployment.
2. Run bosh ssh to open a secure shell into each etcd node.
3. Run monit stop etcd on each etcd node.
4. Delete or move the etcd storage directory, /var/vcap/store, on each etcd node.
5. Run monit start etcd on each etcd node.

3. Reduce the number of Consul instances to 1 and  deploy the BOSH release again. If this fails, follow the same process for the other errant jobs. In the end you may end up with just one instance of these 3 jobs. Once you have a healthy deployment bump up the # of Job instances in the Resources page of the OpsManager.

4. Etcd may sometimes experience orphaned processes that leads to the Diego BBS job failing. This can be remedied by killall etcd on the Diego BBS VMs and subsequent redeployment.

5.  Nuke and Pave ... Take backup of your PCF deployment with cfops and do a brand new deployment followed by a restore.

State of JavaEE

There is a lot of chatter on the internet about Java EE being dead. As someone who spent a lot of time implementing some of the core specs like CDI and also helping customers adopt JavaEE I was interested in determining if this is true.

Unfortunately the anecdotal evidence points to JavaEE dying slowly. This is the evidence now points towards JavaEE being dead:

1. Many prominent Java EE evangelists - Folks who in some cases are friends and I admire deeply have left their respective Java EE shepherd companies. Reza Rahman from Oracle, Kenny Bastani from Oracle and Arun Gupta from Redhat to name just a few ...

2. Oracle, Redhat and IBM - the three most prominent supporters of JavaEE are now spending their money and developers innovating elsewhere. Oracle cloud,OpenShift and Bluemix respectively.

3. The advent of microservices has rendered the monolithic app-server dead. Gone are the days that you have the pack the kitchen sink in your app-server runtime. Nowadays developers ONLY want to use the runtime they want - case in point the rise of the micro-servers - Spring Boot, Dropwizard, Vert.x, Ratpack etc. The next evolution is the disappearance of these micro-servers and the move to serverless computing i.e. the developers are coding chained reactive lambda functions in the cloud. Not sure how I feel about that....

4. The pace of JavaEE specification implementation by vendors and the evolution of the specification itself has slowed to a glacial pace. Where is Java EE8 or Java EE9 ? Where is the Java EE equivalent of all the resiliency patterns like Circuit Breaker, Bulkhead and microservice patterns like service discovery, logging everything to stdout etc. CDI is the only major hope left for JavaEE. They need to rebase the entire platform on CDI and start innovating like crazy. Unfortunately this cannot be done by committee.

There is still a LOT of code running on application servers. WebSphere, WebLogic and JBOSS were the goto runtime and middleware OS in the 90s and early 2Ks.  I care about these apps and if you do too start decomposing them and running them on lightweight runtimes like Spring Boot, Jetty whatever blows your horn. Then move these services to the cloud using Docker or Buildpacks.  If the apps don't add any business value then move them to lighter runtimes within the JavaEE stack i.e. move to WebSphere Liberty or JBOSS Wildfly.

This should serve as a cautionary tale. I am sure a lot of people with disagree with what I have written. But hey this my opinion you should form one too ....

Au Revoir

Update: There is a new organization called Java EE Gaurdians https://dzone.com/articles/who-are-the-java-ee-guardians-and-why-you-should-c that seems to be JavaEEs best shot at survival.

Backup and Restore Pivotal Cloud Foundry

Customers often ask us "What is the proper way to backup and restore Pivotal Cloud Foundry ?"

I want to go over all the options you have to successfully and expediently backup and restore Pivotal Cloud Foundry. The advice here is generally applicable to open source cloud foundry as well to a certain extent.

First a bit of a background. Pivotal Cloud Foundry is comprised of a number of tiles which roughly map to a BOSH release in open source. Each of these tiles need to backed up and restored individually. The OpsManager Tile is special in that it provides the inception VM and is responsible for deploying the BOSH Director. 

Manual Backup and Restore:

This is the recommendation from the product team itself. See 

Backup => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/backup-pcf.html and Restore => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/restore-pcf.html.  So what exactly happens during a backup and restore is explained by the pictures below.

Backup

Restore

Automated Backup and Restore: 

Manually backing up a foundation every night is not practical. Therefore the Pivotal Cloud Foundry Solutions team has an open source tool that automates the manual process for taking a backup and restoring it. This tool is called cfops => https://github.com/pivotalservices/cfops

Documentation on how to run backup/restore with cfops* can be found Here http://www.cfops.io/docs/  The basic process here is  Backup all Tiles and place the backup on a reliable file system. During a restore, restore each tile, apply changes and rinse and repeat for all tiles. 

cfops as of today does not support backing up or restoring data service tiles like MySQL, RabbitMQ etc., The tile settings are restored; however the data is not imported. So you will need an independent backup/restore strategy for data stores. This is where a data backup a tool like Shield => https://github.com/starkandwayne/shield from Stark and Wayne looks attractive. The project is in its infancy. Another option for backing up  the data services is leveraging Concourse.

Concourse can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off tasks, either locally or in another CI system. Concourse encourages practices that keep your project loosely coupled to the details of your continuous integration infrastructure.

If for some reason your BOSH Director disappears or the Director VM gets nuked you can bring it back using the procedure described here : https://blog.pivotal.io/pivotal-cloud-foundry/features/restoring-pivotal-cloud-foundry-after-disaster

Please note that the Backup and Restore process can be done with zero downtime if done right particularly because of the canary support provided by BOSH.

Canary instances are first instances updated within an instance group. Any update error in a canary instance causes the deployment to stop. Since only canaries are affected before an update stops, problem jobs and packages are prevented from taking over all instances.

At some point Pivotal Cloud Foundry will provide an official tool and automation strategy for backup & restore. Till then we can leverage cfops backup and restore function- automated with cron or in a CI pipeline with (Wercker/Concourse/Jenkins) to ensure that our sysadmins can sleep safe and sound.

* Please note cfops is not officially supported by the Pivotal Product Team; however we as Pivotal make our best effort to support this tool and keep it functioning across product upgrades.

13 Factor apps for .NET - Cloud Foundry on Azure

Why bother with Cloud Foundry on Azure ? 

Quick Answer: To optimize for the cloud programming model and build apps that can scale agnostically across cloud platforms. Microsoft haas published 13 recommended patterns from MS for cloud development in BuildingCloudAppsMsAzure. Cloud Foundry through it platform contracts, partner tiles and programming model constraints enables the best way to develop, design and code cloud apps on Azure.

• Automate everything

   • Use scripts to maximize efficiency and minimize errors in repetitive processes. aka Use Cloud Foundry

• Source control

   • Set up branching structures in source control to facilitate a DevOps workflow.

   • Example: add scripts to source control.

   • Example: keep sensitive data out of source control.

   • Example: use Git in Visual Studio.

• Continuous integration and delivery

   • Automate build and deployment with each source control check-in.

• Web development best practices

   • Keep web tier stateless

   • Example: scaling and autoscaling in Cloud Foundry.

   • Avoid session state.

   • Use a Content Delivery Network (CDN).

   • Use an asynchronous programming model.

   • Example: async in ASP.NET MVC and Entity Framework.

• Single sign-on

   • Introduction to Azure Active Directory.

   • Example: create an ASP.NET app that uses Azure Active Directory.

• Data storage options

   • Types of data stores.

   • How to choose the right data store.

   • Example: Azure SQL Database.

• Data partitioning strategies

   • Partition data vertically, horizontally, or both to facilitate scaling a relational database.

• Unstructured blob storage

   • Store files in the cloud by using the Blob service.

   • Example: using blob storage

• Design to survive failures

   • Types of failures.

   • Failure scope.

   • Understanding SLAs.

• Monitoring and telemetry

   • Why you should both buy a telemetry app and write your own code to instrument your app.

   • Example: New Relic for Azure

   • Example: logging code in the Fix It app.

   • Example: built-in logging support in Cloud Foundry.
   • Example: built-in Telemetry for websites and systems in Cloud Foundry

• Transient fault handling

   • Use smart retry/back-off logic to mitigate the effect of transient failures.

   • Example retry/back-off in Entity Framework 6.

• Distributed caching

   • Improve scalability and reduce database transaction costs by using distributed caching.
   • Example: Using Gemfire in ASP.NET app

• Queue-centric work pattern

   • Enable high availability and improve scalability by loosely coupling web and worker tiers.

   • Example: Using RabbitMQ 

Write sample Java apps using the User Authentication Authorization (UAA) API on Cloud Foundry

Do you want to use UAA as an OAuth2 authorization server (eg SSO, token creation / verification)? 

Take a look at github.com/pivo...ity-sample-apps for basic examples of integrating with Pivotal SSO which is UAA with subdomain multi-tenancy, and comes free with PCF 1.5, or look at github.com/will...ervice-security and spring.io/blog/...-cloud-security for a more involved example of using straight UAA to secure microservices.

For PCF customers and interested in using Pivotal SSO and Spring Cloud Services to manage a secure microservice architecture, github.com/will...ran/freddys-bbq is the the microservice-security example and goes the extra mile and integrates with all those products.

Do you want use UAA APIs to dynamically create OAuth2 clients and manage user roles?

There aren't many examples in the OSS world for that, but we do some of that in both Pivotal SSO and Spring Cloud Services. There are API docs however: github.com/clou...cs/UAA-APIs.rst

THANK YOU Will Tran

Cloud Foundry in a Box

You may be wondering how do I hitch a ride on the PaaS train. You would like to spin up a Cloud Foundry on your local machine in less than 10 minutes ? Let me present to you Micro Cloud Foundry aka cloud in a box brought to you by the Cloud Foundry Foundation:

MicroPCF is the simplest way to get a complete Cloud Foundry on a single machine

You gotta verify this. So head on over to https://github.com/pivotal-cf/micropcf and spin up Cloud Foundry in a VM using vagrant. The Future of MicroPCF is bright. From the release notes

In addition to the standard Cloud Foundry components, the MicroPCF distro will include the default PCF service brokers (Redis, MySQL, and RabbitMQ), as well as other PCF-specific applications. The first service available is the Redis service broker. This allows Redis instances to be created and bound to apps running on MicroPCF. More information can be found on Redis for Pivotal Cloud Foundry.

WOA! Services and a full featured platform all in one VM.

Mendix loves Cloud Foundry

In June 2014 Mendix app platform (aPaas) married Cloud Foundry (PaaS). In case you are wondering if this announcement was just vaporware please rest your concerns. Mendix apps work on Cloud Foundry. Kaushik Bhattacharya has done all the legwork to flush this out. Read below ... 

Mendix buildpack documentation describes the steps to push the application. The short version is:

1> cf push mendix_app -b https://github.com/mendix/cf-mendix-buildpack -p <mendix_app.mda> --no-start

2> Create and bind PCF MySQL service to the app in the usual way.

3> Create and bind PCF RiakCS service to the app in the usual way.

4> The app expects values for two variables ADMIN_PASSWORD and DATABASE_URL which can be set as follows:

cf set-env my-app ADMIN_PASSWORD password

cf set-env my-app DATABASE_URL ‘mysql://username:password@hostname:port/database’

Note: You can copy-paste the database URL from the VCAP_SERVICES variable but all parameters such as “?reconnect=true” have to be removed.

5> Restage/start app:
cf restage/start my-app

That's it! 

Cloud Nativeness of an Application

Questions to determine the cloud readiness of your  application to move to the cloud

Q0 - Pick an appropriate path to the cloud:
    1. Java Buildpack
    2. TomEE Buildpack
    3. JBOSS Buildpack
    4. Liberty Buildpack
    5. WebLogic Buildpack
    6. Fat jar (Bundle app server with app)
    7. Docker

Q1 - Does the application rely on persistent file system ? Do you expect the file system to persist across restarts.
     - Disk Caches
     - Transaction Logs
     - Harcoded File Paths
     - Any java.io.* use needs to be looked at

Q2 - Does the app rely on non HTTP inbound protocol routing.
    - TCP support coming in the near future

Q3 - Does the app take a long time to start or stop
    - any dependencies or semantic ordering between apps

Q4 - Does the app use any native code libs  
    - Those libs need to be recompiled for the PCF Stemcell

Q5 - Does the app require distributed XA 2-phase commit transaction ?

Q6 - Is the app packaged as an ear/war/jar file ?
     - JBOSS buildpack and java buildpack do not support ear files

Q7 - Configuration - How is the app configured.
     - Hardcoded property files
     - Generate one artifact configured based on the environment
     - Where is the config coming from

Q8 - All logging to stdout/stderr
    - No log persistence
    - Use Console appenders

Q9 - What do you use for monitoring today
  - Any agents installed on the OS wont work
  - Agents have to bundled with the app or the app server or the buildpack

Q10 - Size of app should be ideally less than 250MB
    - More than that then you have issues with scaling and cf push

Q11 - Favor external caching instead of in-jvm Caches
      - all caching should be done in a distributed cache outside the app JVM
      - in-JVM caches are transient
      - all state to be managed outside the app via backing services
        - punch firewall holes for all backing services

Q12 - Libraries and Frameworks
    - Use cloud native / cloud friendly frameworks like Spring Cloud etc.?

Q13 - Does your app need specialized hardware in terms of specific CPU/Memory/Disk resources
    - Dependency on specific JVMS/JDKs ?

Q14 - No app server specific HA/Clustering or Administration will work in CF

Q15 - OS based registry security will NOT work on the cloud
    - Rely on  3rd party trust protocols like OAuth or SAML or Authentication and Authorization ?