SNAP Not Analysis and Paralysis

SNAP is a technique we employ extensively in Pivotal App Transformation to triage an application and determine technical feasibility. SNAP can be done for an individual application as well as for a bucket of applications that groups like apps. We typically grade each answer along S, M, L, XL t-shirt sizes and determine an overall technical feasibility of the application aka its cloud readiness score. 

SNAP Analysis of Apps :½ hour per app:

Questions to ask:

0. Versions and APIs in-use of Java, JavaEE, J2EE & Spring Framework
1. External Integrations - Databases, Caching, Messaging Queues
4. In-house frameworks
5. Logging
6. Configuration
7. SLAs
8. Packaging and Build (ear/war/jar)
9. CI/CD in place ?
10. Use of Distributed 2PC XA Transactions
11. Persistent File System Access like NFS or SMB/CIFS... ?
12. People location
13. non-HTTP inbound networking protocols like RMI-IIOP, etc ..
14. Total LOC
15. App Server ? and appserver specific code
15. Security Requirements (Siteminder, Ping, SSO)
16. Batch/ETL
17. Front-end tech
18. Runs on PCF
19. Statefulness - session state
20. Data Access - JDBC, ORM, JMS
21. Startup/shudown times
22. 3rd party frameworks/libraries- 32/64 bit References - bit.ly/app-replat & bit.ly/migrate-jee


Typical Outcomes of a SNAP include: (0. Characteristics, 1. User Stories, 2. Metrics, 3. Risks, 4. Score aka Summary))

Outcome for each app is a list of stories to get the app to the degree of cloud nativeness desired by customer. Each App maps to multiple MVPs. The first MVP is always getting the app running on PCF. Thereafter there can be multiple MVPs gradually moving the app on the modernization scale. The scorecard of each app is used to map all the apps on a quadrant of Business Value/ Technical Risk. Apps of low tech. Risk and high business value are chosen first

Auto Generating cloud pipelines for existing applications - THANKS TO Marcin Grzejszczak

This is a guest post where I copy - paste complete nuggets of wisdom from Spring and Pipelines guru Marcin Grzejszczak

The question I posed Marcin was how do we mass generate CI pipelines for existing enterprise applications ?

Marcin indicated that this functionality already exists in spring-cloud-pipelines via project crawler. However this functionality is currently only restricted to Jenkins because there’s no Java API to work with Concourse YAML or Concourse as such.

Project Crawler -  In Jenkins, you can generate the deployment pipelines by passing an environment variable with a comma separated list of repositories. This however doesn’t scale. We would like to automatically fetch a list of all repositories from a given organization / team.

Documentation for Crawler support 

Cloud Pipelines Concourse comes with a crawler.groovy file that allows to go through the repositories in an organization in an SCM server (e.g. Github, Gitlab, Bitbucket) and for each repo it will create a pipeline in Concourse.

How does this work with Jenkins ?

This is the logic for Jenkins that does the creation of pipelines -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy

This is the crawler logic that creates the crawler -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy#L34-L41

This is the logic that fetches the repos -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy#L43-L45


Here you have the logic that iterates over repos and does some stuff (generates pipelines)

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/src/main/groovy/org/springframework/cloud/pipelines/common/PipelineFactory.groovy#L40-L73


What can we do for Concourse ?

The following Groovy script to mass - generate pipelines:
// crawl the org
ProjectCrawler crawler = new ProjectCrawler(OptionsBuilder
    .builder().rootUrl(urlRoot)
    .username(gitUsername)
    .password(gitPassword)
    .token(gitToken)
    .exclude(repoProjectsExcludePattern)
    .repository(repoType).build())

// get the repos from the org
ListRepository repositories = crawler.repositories(org)
MapString, Exception errors = [:]

repositories.each { Repository repo ->
            try {
                // fetch the descriptor
                String descriptor = projectCrawler.fileContent(org, repo.name, repo.requestedBranch, "sc-pipelines.yml")
                // parse it
                PipelineDescriptor pipeline = PipelineDescriptor.from(descriptor)
                if (pipeline.hasMonoRepoProjects()) {
                    // for monorepos treat the single repo as multiple ones
                    pipeline.pipeline.project_names.each { String monoRepo ->
                        Repository monoRepository = new Repository(monoRepo, repo.ssh_url, repo.clone_url, repo.requestedBranch)
                        // generate a concourse pipeline for monoRepository
                    }
                } else {
                        // generate a concourse pipeline for pipeline
                }
            } catch (Exception e) {
                errors.put(repo.name, e)
            }
        }

if (generatedJobs.hasErrors()) {
    println "\n\n\nWARNING, THERE WERE ERRORS WHILE TRYING TO BUILD PROJECTS\n\n\n"
    generatedJobs.errors.each { String key, Exception error ->
        println "Exception for project [${key}], [${error}]"
        println "Stacktrace:"
        error.printStackTrace()
    }
}


In the future the spring cloud pipelines developer workflows shall be simplified such that

As a developer, you will clone one of our sc-pipelines prepared sample apps, alter it to suit your needs (and maintain the conventions that are already there). You’re done.

As an ops, ensure that the Jenkins seed job is running every now or then (or is triggered when a new repo is added to your Github org). That way a pipeline is created out of the box.

Thank you Marcin!!!

Are PHP Microservices a Thing ?

So cloud native architecture does not  apply only to .NET or Java . Cloud native is a secular term that applies to applications in any language including PHP. The concepts and 15 factors of cloud native can apply to any language.

So PHP Microservices are indeed a thing. There is even a book published Mar 17 https://www.packtpub.com/mapt/book/application_development/9781787125377/1

Not only can you write cloud native apps in PHP you can also practice domain driven design in PHP. The book Domain-Driven Design in PHP is an excellent resource to get started.

Please also checkout True Tales of publishing PHP Microservices.

So with a combination of cloud native 15 factor implementation techniques and monolith decomposition driven by DDD you too can achieve nirvana with PHP microservices.

PHP microservices can be pushed to PCF using the PHP buildpack.  For practice take a look at the getting-starter guide to easily push PHP apps to Pivotal Cloud Foundry in under 5 minutes.

Migrating WebLogic workloads to the cloud

# CaaS

0. Containerize the app - PAS ... zero changes to the app - research - Weblogic standalone. Single inbound route. Stateful workload => volume services on PCF. Docker images.
1. Containerize the app - PKS ... zero changes to the app - Use existing Helm charts or operators to run on PKS - Weblogic clustered https://github.com/oracle/weblogic-kubernetes-operator/blob/master/site/design.md

#PaaS

2. Using a buildpack - Weblogic buildpack https://github.com/pivotal-cf/weblogic-buildpack
3. Using a buildpack - TomEE buildpack - Runs JavaEE. YMMW if there is a lot of ORCL specific deployment descriptors and API usage
4. Using the Java buildpack - Java Buildpack - Bootify the app and replace all EE framework usage with corresponding Spring components and frameworks - some changes - very well understood - and recipes exist for ALL mods EJB Migration
5. Using the Java buildpack - Package up the app as a fat app in WebLogic and then run it simply as a jar with the JBP. - experimental but could yield replatforming with 0 changes. same restrictions as PAS apply.

# FaaS

- Does not apply

#VM 

7. BOSH to deploy Weblogic as a release. BOSH release for weblogic
8. Ansible, chef recipes ...

# Bare Metal

9. Run as-is today on servers with NO virtualization.

GKE-on prem, Azure on-prem , AWS on-prem

All the major clouds are coming to a datacenter near you*. You can get Google, or Azure megascale in your own house datacenter. You can start taking advantage of the cloud platforms to improve and shift your workloads to the cloud and get the benefits of being cloud native. The cloud will be a force function for innovation and change in your organizations allowing you to finally throw away the yolk of agilefall or wagile or DAD or Scrum whatever has kept you behind. Your software can eat the world. You will no longer be threatened by the millennial's of Silicon Valley. 

wait a minute ... there is something missing here !! ... 

Humans. Where is the human cloud ? Where is the Cloud Native Transformation as a service. Where is the Platform as a Product Service. Where is the Domain Driven Design as a Service. Where is the SRE as a service. Where is the Outside-in product thinking as a service. Where is the  Training as a service. Where is the Incident Management as a Service. Where is the Busting Canonical Domain Model as a service. Where is decomposing monoliths as a service. Where is the Refactoring databases as a service. Where is Decomposing UIs as a service ? Where is the Outcomes as a service ? Where is the Human as a Service. 

Remember that the human aspect of the cloud is the biggest force function enabler that will enforce the razor focus on outcomes rather than technology envy. As I sit here reflecting on all the technology choices that are passing through > microservices, serverless, kubernetes, cloud foundry, the million choices from CNCF,  please remember the cloud revolution needs to be customized to your needs and not the other way around. 

Pick and chose the boring technology choices available that get the job done and allow you to reach your objectives and key results. Do not forget that the ultimate goal is increased frequency software releases, reduced change failure rate, reduced lead and cycle time to iterate fast and converge on the right outcome. 

So yeah please install PCF, gke, aws, openstack, vmware , azure or whatever but remember software is a means to the end and it is the END that has always to be kept in mind unless your core business is the business of software itself. 

* I have money on AWS-on-prem being announced this ReInvent. 

Day 2 Application Operations

What are the set of practices you need to inculcate and internalize to keep your app alive in production ? Here is a checklist that can help ...

1/ Use spring boot actuators in production … AppManager has secret sauce integrating with actuator allowing for easier management  and visibility of your app of PCF. all your actuator endpoints

2/ When it makes sense add custom health indicators that add to the default health indicators

3/ Add spring cloud sleuth to your classpath dependencies so that you can visualize the call flow across microservices threads with zipkin in PCF Metrics http://docs.pivotal.io/pcf-metrics/1-4/using.html

4/ Plug a syslog drain to the end of the Firehose using the appropriate nozzle like https://docs.pivotal.io/partners/splunk/index.html

5/ Emit application or domain level metrics using Micrometer https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html

6/ Starting PCF 2.2 you can configure alerting and autoscaling on any custom app metric. Custom rules for autoscaling.

7/ Use VisualVM and sister tools from the JVM to get deeper insight particularly for performance issues. Here is a good article on setting up VisualVM with an app running on Cloud Foundry. Furthermore the ability to take threaddumps and heapdumps is critical in analyzing performance issues specifically How to generate Java Application thread dump from Cloud Foundry container and How to generate and download Java Application heap dump from Cloud Foundry container and How to know if an app is responsible for high CPU in a Diego Cell and how to find it

8/ Leverage canaries to understand the quality of the app and then scale up with Blue/Green. concourse-pipeline.

9/ Create a SLI/SLO dashboard to get immediate visibility into your error budgets and uptime.

10/ Practice an outage handling event and run through debugging common scenarios like OOM, hangs, deadlocks using cf-ssh, and boot actuator endpoints. Create a program around incident response management.

Tools for scanning apps for cloud readiness

Spring Migration Analyzer
  https://github.com/spring-projects/spring-migration-analyzer

Ephemerol
  https://github.com/Pivotal-Field-Engineering/ephemerol

Migration Toolkit for App Binaries from IBM
  https://developer.ibm.com/wasdev/docs/migration-toolkit-application-binaries-tech/#evaluate

Application Migration Toolkit from Redhat
  https://developers.redhat.com/products/rhamt/hello-world/

Onboarding applications onto Pivotal Cloud Foundry

We often talk about reducing the barrier of entry for developers onto a new platform. Remember your platform - PCF is a product. One of the core advantages of PCF is how it catalyzes developer productivity. This productivity often comes from having an opinionated prescriptive approach to developing apps. With the appropriate gaurdrails and productivity practices in play enterprises can truly harness the power of the platform. In this blog post I put forth the developer workflow for developing and onboarding apps at the highest level of abstraction.

What should be the onboarding process for an app on PCF ?

1. App Developer logs in into the App On boarding portal.  Operators need to automate access to the PCF platform for developers in an enterprise using cf-mgmt with the following commands. 

2. 

   
   

   Customized version of start.spring.io for Java and .NET apps that generates a running app bound to the relevant PCF service - Redis, Rabbit, MySQL, …. Fork this spring intializr repository for customizing start.spring.io.
3. Scaffolding: Portal generates a corresponding CI pipeline that deploys said app to dev space. Reuse spring-cloud-pipelines. Minimal refactoring to comply with basic Spring Cloud Pipelines requirements. At the end of this stage, each app will have a corresponding pipeline on Concourse. The pipelines will successfully build the apps, store the artifacts in Bintray, tag the GitHub repositories, and deploy the apps to Test, Stage, and Prod spaces in Cloud Foundry.
   

   
4. The app generated by the start.spring.io has appropriate tie-ins into credhub and a config server for managing secrets and config respectively.
5. Security integration with the app putting it in the right authentication/authorization flows with Pivotal SSO or custom services
6. Iterate on app features tracking the objectives and key results of the application. 
7. Testing: Add/organize tests to comply with Spring Cloud Pipelines recommendations. Incorporate flyway for database schema versioning and initial data loading. At the end of this stage, the pipelines will trigger unit and integration tests during the Build stage, smoke tests in the Test environment, and end-to-end tests in the Stage environment. The pipelines will also ensure backward compatibility for the database, such that you can safely roll back the backend service app, even after the database schema has been updated. Release workflow pipeline starts to promote the app into test and staging => Blue/Green deploy with tagging in artifactory. Integrate with ESB and other down stream integrations. Insert release mgmt. gate here.
   

   
8. Contracts: Incorporate Spring Cloud Contract to define the API between the UI and service apps and auto-generate tests and stubs.At the end of this stage, the pipelines will catch breaking API changes during the Build stage and ensure backward compatibility for the API, such that you can safely roll back the backend service (producer) app, even after an API change.
9. Pipeline to promote to prod.. This may deploy to multiple foundations. This has the capability to do a canary and dark deploys and rollback. Insert release mgmt. gate here
   

   

References:

• Continuous Deployment to the Cloud - Marcin Grzejszczak, Cora Iberkleid  https://content.pivotal.io/springone-platform-2017/continuous-deployment-to-the-cloud-marcin-grzejszczak-cora-iberkleid
• Migrate Apps to Spring Cloud Pipelines https://cloud.spring.io/spring-cloud-pipelines/multi/multi__step_by_step_cloud_foundry_migration.html

Generating Load and chaos on Pivotal Cloud Foundry

If you wanted to generate load on the apps running on PCF and wanted some load scripts here are some repos that stress test Spring Music and other apps

https://github.com/pivotal-cf-experimental/spring-music-on-cf-benchmarking

https://github.com/xchapter7x/cf-app-attack

https://github.com/cloudfoundry/cf-acceptance-tests/tree/master/assets/dora

https://github.com/bijukunjummen/boot2-load-demo

https://github.com/cloudfoundry-incubator/pat

https://content.pivotal.io/blog/250k-containers-in-production-a-real-test-for-the-real-world

https://github.com/dsyer/spring-boot-startup-bench

https://github.com/jamlo/jmeter-bosh-release

https://github.com/chaostoolkit and https://github.com/chaostoolkit/chaostoolkit-cloud-foundry

https://github.com/ragstorooks/testrr

Chaos Loris which is chaos monkey for application instances which will exercise your applications resiliency.

Chaos Lemur which is chaos monkey for BOSH releases and jobs, so it exercises the resiliency of the platform

Generic Script to create chaos ...

apt-get install apache2-utils

# ab -n <num_requests> -c <concurrency> <addr>:<port><path>po

ab -n 1000 -c 100 http://localhost:4567/

Connecting to Oracle Databases from apps in Cloud Foundry

A common use case when replatforming apps is connecting to oracle databases via a user provided service instance. There are three ways these CUPs can defined as specified in the spring cloud connectors #oracle doc. There are some gotchas when defining oracle CUPs. This post provides examples of when and which form of Oracle CUPs to define ...

The connector will check for:

1. uri using the scheme oracle
2. jdbcUrl using the subprotocol oracle in the JDBC URL
3. oracleUri, oracleuri, oracleUrl, or oracleurl fields in credentials

Any of the three approaches will work ...  however;

- If you are using oracle subprotocol , then the CUPs has to be a uri. 

- If your DB url contains the ldap:// prefix before the host name in the URL then use the jdbcUrl credentials property instead of url. 

cf create-user-provided-service catalog-db -p '{"jdbcUrl":"jdbc:oracle:thin:TOMHANKS/<pwd>$2011@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(FAILOVER=on)(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=espndb.net)(PORT=1555)))(CONNECT_DATA=(SERVICE_NAME=cd_app)(FAILOVER_MODE=(TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5))))"}'


cf cups ulrDataSource -p “dbURL” oracle://ulr_wsph:facebook123#@rwev10exp.abc.facebook.com:1576/e_facebooklog_rgwd.nbc.us

The Spring cloud connectors library provides ORACLE connectors that automatically wire up your ORCL cups to your Spring datasource bean

http://cloud.spring.io/spring-cloud-connectors/spring-cloud-connectors.html#_extending_spring_cloud_connectors


Significant contributions to this post by Anwar Chirakkattil , Oleg Iavorskyi and Brad Starkenberg