Day 2 Application Operations

What are the set of practices you need to inculcate and internalize to keep your app alive in production ? Here is a checklist that can help ...

1/ Use spring boot actuators in production … AppManager has secret sauce integrating with actuator allowing for easier management  and visibility of your app of PCF. all your actuator endpoints

2/ When it makes sense add custom health indicators that add to the default health indicators

3/ Add spring cloud sleuth to your classpath dependencies so that you can visualize the call flow across microservices threads with zipkin in PCF Metrics http://docs.pivotal.io/pcf-metrics/1-4/using.html

4/ Plug a syslog drain to the end of the Firehose using the appropriate nozzle like https://docs.pivotal.io/partners/splunk/index.html

5/ Emit application or domain level metrics using Micrometer https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html

6/ Starting PCF 2.2 you can configure alerting and autoscaling on any custom app metric. Custom rules for autoscaling.

7/ Use VisualVM and sister tools from the JVM to get deeper insight particularly for performance issues. Here is a good article on setting up VisualVM with an app running on Cloud Foundry. Furthermore the ability to take threaddumps and heapdumps is critical in analyzing performance issues specifically How to generate Java Application thread dump from Cloud Foundry container and How to generate and download Java Application heap dump from Cloud Foundry container and How to know if an app is responsible for high CPU in a Diego Cell and how to find it

8/ Leverage canaries to understand the quality of the app and then scale up with Blue/Green. concourse-pipeline.

9/ Create a SLI/SLO dashboard to get immediate visibility into your error budgets and uptime.

10/ Practice an outage handling event and run through debugging common scenarios like OOM, hangs, deadlocks using cf-ssh, and boot actuator endpoints. Create a program around incident response management.

Tools for scanning apps for cloud readiness

Spring Migration Analyzer
  https://github.com/spring-projects/spring-migration-analyzer

Ephemerol
  https://github.com/Pivotal-Field-Engineering/ephemerol

Migration Toolkit for App Binaries from IBM
  https://developer.ibm.com/wasdev/docs/migration-toolkit-application-binaries-tech/#evaluate

Application Migration Toolkit from Redhat
  https://developers.redhat.com/products/rhamt/hello-world/

Onboarding applications onto Pivotal Cloud Foundry

We often talk about reducing the barrier of entry for developers onto a new platform. Remember your platform - PCF is a product. One of the core advantages of PCF is how it catalyzes developer productivity. This productivity often comes from having an opinionated prescriptive approach to developing apps. With the appropriate gaurdrails and productivity practices in play enterprises can truly harness the power of the platform. In this blog post I put forth the developer workflow for developing and onboarding apps at the highest level of abstraction.

What should be the onboarding process for an app on PCF ?

1. App Developer logs in into the App On boarding portal.  Operators need to automate access to the PCF platform for developers in an enterprise using cf-mgmt with the following commands. 

2. 

   
   

   Customized version of start.spring.io for Java and .NET apps that generates a running app bound to the relevant PCF service - Redis, Rabbit, MySQL, …. Fork this spring intializr repository for customizing start.spring.io.
3. Scaffolding: Portal generates a corresponding CI pipeline that deploys said app to dev space. Reuse spring-cloud-pipelines. Minimal refactoring to comply with basic Spring Cloud Pipelines requirements. At the end of this stage, each app will have a corresponding pipeline on Concourse. The pipelines will successfully build the apps, store the artifacts in Bintray, tag the GitHub repositories, and deploy the apps to Test, Stage, and Prod spaces in Cloud Foundry.
   

   
4. The app generated by the start.spring.io has appropriate tie-ins into credhub and a config server for managing secrets and config respectively.
5. Security integration with the app putting it in the right authentication/authorization flows with Pivotal SSO or custom services
6. Iterate on app features tracking the objectives and key results of the application. 
7. Testing: Add/organize tests to comply with Spring Cloud Pipelines recommendations. Incorporate flyway for database schema versioning and initial data loading. At the end of this stage, the pipelines will trigger unit and integration tests during the Build stage, smoke tests in the Test environment, and end-to-end tests in the Stage environment. The pipelines will also ensure backward compatibility for the database, such that you can safely roll back the backend service app, even after the database schema has been updated. Release workflow pipeline starts to promote the app into test and staging => Blue/Green deploy with tagging in artifactory. Integrate with ESB and other down stream integrations. Insert release mgmt. gate here.
   

   
8. Contracts: Incorporate Spring Cloud Contract to define the API between the UI and service apps and auto-generate tests and stubs.At the end of this stage, the pipelines will catch breaking API changes during the Build stage and ensure backward compatibility for the API, such that you can safely roll back the backend service (producer) app, even after an API change.
9. Pipeline to promote to prod.. This may deploy to multiple foundations. This has the capability to do a canary and dark deploys and rollback. Insert release mgmt. gate here
   

   

References:

• Continuous Deployment to the Cloud - Marcin Grzejszczak, Cora Iberkleid  https://content.pivotal.io/springone-platform-2017/continuous-deployment-to-the-cloud-marcin-grzejszczak-cora-iberkleid
• Migrate Apps to Spring Cloud Pipelines https://cloud.spring.io/spring-cloud-pipelines/multi/multi__step_by_step_cloud_foundry_migration.html

Generating Load and chaos on Pivotal Cloud Foundry

If you wanted to generate load on the apps running on PCF and wanted some load scripts here are some repos that stress test Spring Music and other apps

https://github.com/pivotal-cf-experimental/spring-music-on-cf-benchmarking

https://github.com/xchapter7x/cf-app-attack

https://github.com/cloudfoundry/cf-acceptance-tests/tree/master/assets/dora

https://github.com/bijukunjummen/boot2-load-demo

https://github.com/cloudfoundry-incubator/pat

https://content.pivotal.io/blog/250k-containers-in-production-a-real-test-for-the-real-world

https://github.com/dsyer/spring-boot-startup-bench

https://github.com/jamlo/jmeter-bosh-release

https://github.com/chaostoolkit and https://github.com/chaostoolkit/chaostoolkit-cloud-foundry

https://github.com/ragstorooks/testrr

Chaos Loris which is chaos monkey for application instances which will exercise your applications resiliency.

Chaos Lemur which is chaos monkey for BOSH releases and jobs, so it exercises the resiliency of the platform

Generic Script to create chaos ...

apt-get install apache2-utils

# ab -n <num_requests> -c <concurrency> <addr>:<port><path>po

ab -n 1000 -c 100 http://localhost:4567/

Connecting to Oracle Databases from apps in Cloud Foundry

A common use case when replatforming apps is connecting to oracle databases via a user provided service instance. There are three ways these CUPs can defined as specified in the spring cloud connectors #oracle doc. There are some gotchas when defining oracle CUPs. This post provides examples of when and which form of Oracle CUPs to define ...

The connector will check for:

1. uri using the scheme oracle
2. jdbcUrl using the subprotocol oracle in the JDBC URL
3. oracleUri, oracleuri, oracleUrl, or oracleurl fields in credentials

Any of the three approaches will work ...  however;

- If you are using oracle subprotocol , then the CUPs has to be a uri. 

- If your DB url contains the ldap:// prefix before the host name in the URL then use the jdbcUrl credentials property instead of url. 

cf create-user-provided-service catalog-db -p '{"jdbcUrl":"jdbc:oracle:thin:TOMHANKS/<pwd>$2011@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(FAILOVER=on)(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555)))(CONNECT_DATA=(SERVICE_NAME=cd_app)(FAILOVER_MODE=(TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5))))"}'


cf cups ulrDataSource -p “dbURL” oracle://ulr_wsph:facebook123#@rwev10exp.abc.facebook.com:1576/e_facebooklog_rgwd.nbc.us

The Spring cloud connectors library provides ORACLE connectors that automatically wire up your ORCL cups to your Spring datasource bean

http://cloud.spring.io/spring-cloud-connectors/spring-cloud-connectors.html#_extending_spring_cloud_connectors


Significant contributions to this post by Anwar Chirakkattil , Oleg Iavorskyi and Brad Starkenberg

A View on Chargeback and Billing with Pivotal Cloud Foundry

We see that when enterprises install a Platform like Pivotal Cloud Foundry one of the primary motivations is to reduce the time to market. Enterprises want to fulfill their customer needs and leverage the platform and app dial tone provided by Pivotal Cloud Foundry to enable developers to push apps faster than before and rapidly experiment. There is however a real temptation to institute the same practices of yore to manage and onboard apps to the platform. Establish fine grained chargeback models that penalize innovation and count every nickel and dime when it comes to memory or network traffic or app or service instances on the PaaS. This defeats the purpose of PCF to encourage creativity and rapid genesis of ideas to production.

# So how does one think of chargeback in a new world. How do we get the profusion of apps on the platform and gain optics into cost and charge reasonably ?

Keep it simple. Observe, orient and act.

First and foremost remember PCF is a  PRODUCT and when you’re launching a new product whether it’s ice cream cones or a PaaS, if you don’t price it right based upon the market conditions, then no customers will come and you’ll be left with a bunch of melted ice cream or an empty PaaS. If  the market is undefined i.e. the platform engineers are not already doing chargeback for other infrastructure, any attempt to price PCF now would be a shot in the dark and more likely to over/under estimate the value of the product than to be hit the mark. And over/under pricing it both come with big risks in the both the short and long term. Put in place all the facility you need to understand consumption, do market research on what people are willing to pay.  Customers don’t even know the value yet until they start using it!.

Promote usage internally first, by observing usage amount/usage patterns/costs with earlier adopters before start charging other teams, otherwise adoption rate by dev teams may be low if the platform costs too much.

Build a model based on quotas as a starting point.  Show what the costs are for their entitlement NOT what they are actually using i.e. similar to the AWS model. Organization Quotas force accountability and make it way easier to do capacity planning. Pretty easy to get started.  There’s a cost to buying a lot and the Platform Engineers should not care what the app teams do with that lot.  If the app teams decide to deploy nothing platform engineers have still reserved whatever is in your quota.  Doesn’t mean that platform team can’t over subscribe resources.

# Resources

1. Automated Chargeback CI pipeline https://github.com/pivotalservices/concourse-pcf-usage-report/tree/master/presentations
2. apptio provides apptio-usage-service to build chargeback

# Credits

PCFs Team - Caleb, Parker, Luciano, Zac, Clayton, David, Ian and Eamon 

Pushing WebSphere Application Sever Applications to Cloud Foundry with Open Liberty

Thanks to a  friendly nudge and significant contributions from Michael Wright, I have had the opportunity to reacquaint myself with an old friend - the WebSphere Liberty Profile Buildpack. In case you missed it, in a bold move IBM open sourced the WebSphere Liberty Profile their lightweight application server.  Open Liberty as it is called now is an excellent landing spot for ALL current WebSphere Application Server classic apps. 

To get them running with the least amount of changes WebSphere apps should typically be pushed with the WebSphere Liberty Profile Buildpack. The Liberty buildpack has myriad ways of configuring it with lots of environment variables to control the set of features and WebSphere server configuration and so on ... Here are some best practices to follow a repeatable and simple process of migrating your apps using Open Liberty to Cloud Foundry: 

1. It’s OK to develop on STS or Eclipse, but don’t rely on Eclipse to package the application inside the server configuration.

2. Develop your application with Open Liberty's maven plugin. If you have an existing project convert it to Open Liberty by cloning and copying the build pom.xml from one of the guides.  This is a very similar to a spring boot like experience for JavaEE/EE4J.  An alternate way of developing is to configure the server.xml for your server and then keep copying the app to the dropins folder. 

3. After your application runs locally package a Liberty server with your application in the dropins folder and the right server configuration in the usr directory. This is akin to packaging a fat jar for your WebSphere app. Creating a server package avoids the headache of configuring the buildpack with the right set of environment variables when pushing the app. 

server package defaultServer --include=usr 

4. When the application does run in Cloud Foundry play special attention to this message 2017-10-12T22:59:27.80-0400 [APP/PROC/WEB/0] OUT [AUDIT] CWWKT0016I: Web application available (default_host): http://9e1d0c1c-a318-46eb-7760-0302:8080/war-with-resource/

This informational message reveals your context-root for your application. Context root determination for an app is a multi-step evaluation based multiple files.

The context-root attribute specifies the entry point of the deployed application. The entry point of a deployed application is determined in the following precedence:

- context-root in the server.xml file

- application.xml, if an EAR application

- ibm-web-ext.xml, if a web application

- name of the application in the server.xml file, if a web application

- Directory name or the file name relative to the drop-ins directory of Liberty

My rule of the thumb is NEVER to set a context root and manage the public URL via a Cloud  Foundry route. By default the buildpack set context root for an ear file is "/".

Notes of Caution 

1. Your application does not need to be cloud native to run on Cloud Foundry. You can push zero factor apps to CF no problem. If the application does not function correctly look under the hood by cf ssh'ing into the container and examine the final server.xml. If your EE resources are not being served or recognized by the container check your list of features and corresponding configuration in server.xml
2. WebSphere Liberty Profile Buildpack provisions the DEV licensed WebSphere Liberty Profile runtime and NOT the Open Liberty runtime. To avoid licensing headaches you should fork the Liberty Profile Buildpack to provision the Open Liberty runtime instead of the DEV only licensed WebSphere Liberty Profile runtime.

In Closing

If you want to completely avoid the IBM apple cart, an excellent alternative is the TomEE buildpack which also runs JavaEE7 Full , Web and micro-profiles  ear, war,  fat and skinny jar apps. For more on this topic read a previous post yes-we-can.

Replatforming  is just the first phase of your multi-step journey to Cloud Native.  Your ideal end-state is to make the app cloud native and therefore vendor and platform agnostic leveraging the app  dialtone based on the contracts established by the  platform API. 

The Cost Curve of Application Replatforming

PCF Is the best place to run Spring Apps

Forgot to mention the auto patching of vulnerabilities by the buildpack. Instead of the developer managing all the middle layers the CF buildpack  curates and maintains this list. 

Emergent Systems and the need for Chaos Architecture

The ideas below are an amalgamation of key signals from Adrian Cockcroft, Neal Ford, Matt Stine, Russ Miles, Michael Nygard and the rockstar engineers of Netflix who have pioneered Chaos Engineering.

For the long term survival of your microservices system some key concepts have now come together chiefly: anti-fragility, continuos partial failure  and evolutionary architecture. 

Since all of us seem to building networks of distributed microservices there is no way to test the emergent behavior of these systems in test. We HAVE to run controlled experiments in production. Chaos engineering and a chaos friendly architecture is critical for enterprises to maintain availability of their applications and survive breaches.  Adrian Cockcroft in his recent Cloud Native London keynote espoused four layers, two teams and an attitude.

Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.

- http://principlesofchaos.org/

Chaos engineering is the best continuously holistic function to evaluate a distributed system to withstand impact and external perturbation. So what are the different levels at which chaos engineering

Layer-1: Infrastructure: Lay out the infra. so that there is no SPOF. Multiple zones, regions. app is distributed enough times and enough ways - diversity in the infrastructure

Layer-2: Switching and interconnect. Strategy for interconnecting. NO SPOF => data in more than one place. This will require data to be replicated in a different side of the world. Data needs be more than one disk - in  a different building. Routing needs to transparently handle failover across datacenters. Unfortunately DR routing/failover is the Least well tested set of components in the system. Usually all error handling code explodes at impact. Once the fallen datacenter comes up there is a need to Re-route and re-synchronize - introduce anti-entropy people back into the system. It is critical to regularly test Failover regularly to the backup datacenter. Test HA/DR across data-centers  properly instead of availability theater.

Layer-3: Application Layer: What does app do when it  experiences data loss , n/w connectivity failures, timeouts, error returns, slow responses, network partitions - app hangup - goes 100% busy
Single functions and microservices can be tested to one thing. Lambda is easy to develop for. Unit of testing and deployment. Monoliths lead to combination testing. Lot of variations

Layer 4 People: When machines  mis-behave people really screw it up. Usually folks made it worse. There are countless stories of systems that were thrashed by the operators due to a comedy of errors. It is super-important to practice gamedays similar to how children in kindergarten practice a fire drill.  Chernobyl.  Reboot may be the wrong thing to do when you have services. Play out fire-drills so that when there is an actual fire people take the right action - disaster preparedness.  Practice, Practice, Practice. ... Fire drills

Tools that attack the different layers.

1. Game Days - exercise outage. Right way everyone is to behave. folks on a call. how to find the dashboards etc. as well digging into details
2. Simian army- Tests once a month
3. FIT - Deep injection of failures. CHAP. Chaos automation platform
4. Gremlin Inc. -  Auto mates chaos engineering scenarios and gamedays.  Undo button. Safer with automation.
5. There is an excellent catalog tools at the end of the chaos engineering eBook

Two Teams

1. Security Blue Team/Security Red Team(Break into your site)
2. Chaos Engineering Team/SRE Team

Companies offer services to make System secure and resilient.

• AttackIQ
• Safebreach
• Spear fishing

Attitude - Improve your chaos posture
- OReilly book - Chap and the Chaos Maturity Model gives a roadmap to improve your chaos game.
- Chaos engineering community day coming up - London this is becoming a thing

If you want to create a system with 99999/99999999 availability it is important to establish  a Chaos Engineering Practice that keeps the team safe, and whole stack reliable.

Attitude

Go run a gameday. People experienced in simulating outages. easy cleanup.
Start at top and work your way down.