Monitoring Microservices with Spring Boot Actuator and Aspectj

What makes modern applications so different? Modern applications adhere to several essential architectural tenets: They are cloud-native, containerized, auto-scalable, microservice-based, and multi-tenant and supported by DevOps-driven deployment automation. At BMC, our application development environment solution comprises 100-plus microservices running on a containerized cloud platform (AWS), scaled using the Kubernetes platform.

Each microservice can be deployed as Kubernetes Pods (from three to 16 depending on horizontal scaling and replication requirements) spanning multiple availability zones (AZ). The services expose representational state transfer (REST) interfaces for external consumption and remote procedure calls (gRPC) for internal microservice-level communication. As more services are introduced, monitoring and managing modern applications becomes increasingly challenging.

Site reliability engineering (SRE) for 24×7 uptime

Adopting appropriate agile and DevOps practices is critical to successfully running enterprise-class solutions on a software-as-a-service (SaaS) model and keeping deployments highly available and responsive. BMC has adopted SRE practices to keep our solutions up and running 24×7. For self-observability of stacks, our SRE team uses our in-house monitoring solution. When it comes to self-observability, SRE and development teams require a lot of metrics to keep an eye on the health of the overall stack. Examples of data we monitor include:

• Specific tenants in multi-tenant deployments that generate large numbers of events at particular times of day
• The rate of publishing metrics to Kafka topics for further processing by artificial intelligence for IT operations (AIOps)
• Log data ingestion rates and abnormally large log entries

For microservices built using the Spring Boot framework, you can enable the Spring Boot Actuator to provide many different types of metrics. For example, the Actuator exposes resource utilization metrics such as CPU and memory utilization; network and disk space utilization; and more. It also exposes other standard Java Management Extensions (JMX) metrics, including garbage collection (GC) overhead, GC time, heap utilization, thread usage/blocked threads, etc. But standard data points are not enough. Depending on the responsibility of a given microservice, you might need additional custom metrics, such as:

• Message processing rate
• Message drops due to communication errors
• Message drops due to size limits
• Average message size received
• Communication failures with infra-services
• Create, read, update, delete (CRUD) operations latency

Application telemetry and microservices

For both SRE and development teams, capturing telemetry is the key to seeing what’s going on inside an application. You need to build a telemetry infrastructure to capture and process the data. The two main functions of a telemetry processing infrastructure are:

• Collecting and exposing telemetry data—BMC microservices need to be instrumented to expose standard and custom metrics
• Processing and storing telemetry data—the BMC application development environment can ingest huge volumes of metrics data (including support for the Prometheus format) while enriching and storing it in Time Series format so it can be processed and further visualized

Below is a high-level diagram of the telemetry pipeline. For our purposes here, we will focus on how to expose standard and custom metrics for microservices built using the Spring Boot framework.