Introduction
Observability is a concept involved in assessing a running system. It gives the means to measure the relevant metrics and evaluate how the software works internally. Spring Boot provides Micrometer and Micrometer Tracing packages to support metrics, logging, and tracing. External software provides an overall set of tools to deal with observability. This article will discuss a stack based on Spring Boot 3, Micrometer, Micrometer Tracing, Grafana, Prometheus, Tempo, and Loki.
Observability with Micrometer
In a Spring Boot application, we can enforce observability by the Micrometer library. To define an observation we must use an ObservationRegistry and register an ObservationHandler in it. An ObservationHandler defines a set of specific methods that are called to start and close observations, observation events, and scopes:
- onStart: an observation starts
- onStop: an observation ends
- onEvent: an event that happens during an observation
- onScopeOpened: an observation scope starts. Thread local objects can be passed in and closed when the scope ends
- onScopeClosed: an observation scope closes
- onError: an error occurs during an observation
These methods are called passing in an observation context as an argument.
Observability Server-Side Implementation
To illustrate how observability works we can start with an example. We implement two simple Spring Boot applications: a Book client exposing a getAuthor REST service that calls a remote Author service to get an author information by its name.
The Author’s application will expose a service like this:
@GetMapping("/author/{name}")
public Optional<Author> findByName(@PathVariable("name") String name) {
...
}
The Book service will have a getAuthor endpoint with the following implementation:
@GetMapping(value = "/getAuthor", params = { "authorName" })
public Optional<Author> getAuthor(@RequestParam("authorName") String authorName) {
...
}
Configuring the Author‘s server application
The Author service plays the role of the server. To implement observability we have to provide several dependencies. To handle the Micrometer metrics we use Prometheus with the following dependency:
<!-- Prometheus -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
To support Micrometer Tracing in the correlation of requests into single spans we add Zipking Brave:
<!-- Zipkin Brave -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
We must provide spans to a server with a specific format to obtain latency visualization. To do this we need a zipkin-reporter-brave library:
<!-- Zipkin Latency Visualization -->
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>
To send the logs, which are correlated by a tracer ID, to Graphana, we need the loki-logback-appender package:
<!-- Grafana Loki Appender-->
<dependency>
<groupId>com.github.loki4j</groupId>
<artifactId>loki-logback-appender</artifactId>
<version>1.4.2</version>
</dependency>
To make Micrometer support OpenFeign we add also the following:
<!-- Openfeign Micrometer Support -->
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-micrometer</artifactId>
</dependency>
We then need some configuration in the application.xml file. We set the actuator and metrics to include a Prometheus endpoint, the sampling probability property to 1.0 to use all traces for latency analysis, and enable percentiles histograms:
management:
tracing:
sampling:
probability: 1.0
endpoints:
web:
exposure:
include: prometheus
metrics:
distribution:
percentiles-histogram:
http:
server:
requests: 'true'
logging:
pattern:
level: '%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]'
We also provide a configuration for Logback to send logs to Loki:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml" />
<springProperty scope="context" name="appName" source="spring.application.name"/>
<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
<http>
<url>http://localhost:3100/loki/api/v1/push</url>
</http>
<format>
<label>
<pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern>
</label>
<message>
<pattern>${FILE_LOG_PATTERN}</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<root level="INFO">
<appender-ref ref="LOKI"/>
</root>
</configuration>
Configuring the Book‘s client application
The configuration for the client side is almost the same, the only difference is the lack of a histogram configuration fragment:
management:
tracing:
sampling:
probability: 1.0
endpoints:
web:
exposure:
include: prometheus
logging:
pattern:
level: '%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]'
Server Application Code
On the server side, we first define an ObservationHandler to capture the onStart and onStop events of the observation:
@Component
public class ObsHandler implements ObservationHandler<Observation.Context> {
private static final Logger log = LoggerFactory.getLogger(ObsHandler.class);
private String getAuthorTypeFromContext(Observation.Context context) {
return StreamSupport.stream(context.getLowCardinalityKeyValues().spliterator(), false)
.filter(keyValue -> "authorType".equals(keyValue.getKey()))
.map(KeyValue::getValue)
.findFirst()
.orElse("UNKNOWN");
}
@Override
public void onStart(Observation.Context context) {
log.info("Before running the observation for context [{}], userType [{}]", context.getName(), getAuthorTypeFromContext(context));
}
@Override
public void onStop(Observation.Context context) {
log.info("After running the observation for context [{}], userType [{}]", context.getName(), getAuthorTypeFromContext(context));
}
@Override
public boolean supportsContext(Observation.Context context) {
return true;
}
}
We also configure an ObservedAspect bean to use the @Observed annotation:
@Configuration(proxyBeanMethods = false)
class ObsConfiguration {
@Bean
ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
return new ObservedAspect(observationRegistry);
}
}
We finally annotate the method that the client application will cal with @Observed. This way, we are creating an observation on that method:
@Observed(name = "authorInfo", contextualName = "getting-author-info", lowCardinalityKeyValues = { "authorType", "Poet" })
public Optional<Author> findByName(String name) {
return authorRepository.findById(name);
}
Client Application Code
On the client side, we define an ObservationHandler as in the server side and we manually create an observation in the method that calls the server side, with a low cardinality key named “authorType” and high cardinality key named “authorName”:
@GetMapping(value = "/getAuthor", params = { "authorName" })
public Optional<Author> getAuthor(@RequestParam("authorName") String authorName) {
ObservationRegistry registry = ObservationRegistry.create();
registry.observationConfig()
.observationHandler(new ObsHandler());
ThreadLocal<Optional<Author>> thLocal = new ThreadLocal<Optional<Author>>();
Observation.createNotStarted("my.observation", registry)
.lowCardinalityKeyValue("authorType", "Poet")
.highCardinalityKeyValue("authorName", authorName)
.contextualName("getAuthorInfo")
.observe(() -> {
logger.info("request to the server");
Optional<Author> response = bookService.getAuthor(authorName);
thLocal.set(response);
logger.info("response [{}]", response);
});
return thLocal.get();
}
Running the Sample
You can run all the components required to observe our couple of demo applications using a docker-compose yaml file. You have to save the file in a directory of your choice and execute “docker compose up” on the command line. You can access Grafana and go to http://localhost:3000 and Prometheus at http://localhost:9000.
You then run the client and server applications and you are all set. On Graphana you can open the dashboard related to logs, traces, and metrics. There you can pick a traceid, and see all logs and traces related to it. You will also see the latency metrics provided by default by Spring Boot with the support of Micrometer.
You can also see the metrics related to the method annotated with @Observed on the server side and the manually created observation on the client side by opening the Prometheus view.
Conclusion
Assessing a system once in production is crucial. Spring provides solid support to this with its Micrometer and Micrometer Tracing features. You can find the source code of the example of this article on GitHub.