THOUGHTS ON SCALING MICROSERVICESMuch of this is a rehash of Susan Fowler's excellent book
Production-Ready Microservices Published by O'Reilly Media, Inc., 2016 http://shop.oreilly.com/product/0636920053675.do
# Qualitative and Quantitative growth scales of a Microservice
## QuantitativeQualitative growth scales allow the scalability of a service to tie in with higher-level business metrics: a microservice may, for example, scale with the number of users, with the number of people who open a phone application (“eyeballs”), or with the number of orders (for a food delivery service). These metrics, these qualitative growth scales, aren’t tied to an individual microservice but to the overall system or product(s).
- Business Metrics
- Number of Health Care Claims Adjudicated
- Number of Insurance claims processed
## QualitativeIf the qualitative growth scale of our microservice is measured in “eyeballs”, and each “eyeball” results in two requests to our microservice and one database transaction, then our quantitative growth scale is measured in terms of requests and transactions, resulting in requests per second and transactions per second as the two key quantities determining our scalability.
- HTTP Throughput
- CPU Utilization
- (negative - scaling) Threadpool saturation
- (negative - scaling) Number of open database connections _is it near the conn limit_
## What To Monitor For Each Microservice Availability
### Infrastructure Metrics
- CPU utilized by the microservice across all containers
- RAM utilized by the microservice across all containers,
- The available threads,
- The microservice’s open file descriptors (FD)
- The number of database connections that the microservice has to any databases it uses.
### Monitor the availability of the service
- Service-level agreement (SLA) of the service,
- Latency (of both the service as a whole and its API endpoints), -
- Success of API endpoints, Responses
- Average response times of API endpoints, the services (clients) from which API requests originate (along with which endpoints they send requests to),
- Errors and exceptions (both handled and unhandled), and the health and status of dependencies.
# Monitoring ADVICE
- CUSTOM DASHBOARD FOR EACH MICROSERVICE that ALERT FOR EACH MICROSERVICE FOR KEY METRICS
- Normal, Warning and Critical Alerts
- On call Runbook procedure for remediating all alerts
- Low level Remediations should be automated
A microservice should never experience the same exact problem twice.