Hello! I'm 
id:RyuGoo, and I am the Sub-Director of the Mackerel team. I joined the Mackerel team in April 2024, and I have been working primarily in introducing and implementing product management systems and concepts.
Mackerel is actively working to enhance its support for observability based on the OpenTelemetry standard. Since the acquisition of Vaxila, announced in June 2024 (written in Japanese), we have continued our developmental efforts to deliver the tracing experience, an essential signal in observability, to our users as swiftly as possible. On August 26th, 2024, we were pleased to announce that we were able to offer Vaxila which offers tracing services, as a Mackerel feature for free (available as a trial edition of tracing services).
Due to the particular nature of the positioning of this service, quality standards and product support for Vaxila differ from regular Mackerel services. Please note the following:
- Services may be shut down without prior notice due to maintenance.
- Responses to inquiries may require some time, including the initial replies.
- Direct support for instructions or configuring Vaxila, etc., will not be provided.
With this release, the Vaxila service currently available via Vaxila Labs (https://vaxila-labs.com/) is scheduled to be discontinued. Current users will be notified of service discontinuation dates, etc., at a later date.
This post will introduce the Vaxila service and information on how to use it.
What is Vaxila?
Vaxila is an OpenTelemetry-compatible distributed tracing service. When you send telemetry data to Vaxila via an OpenTelemtry-compatible application or system, you will be able to visualize the information as a tree within the Vaxila UI.
What is distributed tracing?
Distributed tracing is a technology which enables the tracing and visualization of a single data request, executed by an application or system, to show how it flows through several services (for example, a web server or database, or microservice such as an authentication service).
For example, consider an HTTP request made to a specific endpoint, in which said request involves authentication and authorization, the retrieval of information from a database, the processing of that information, etc., and then the return of a response to the user. In such an instance, the flow of distributed services may look like the following sequence of events.
- The request is received by the web server.
- Authentication and authorization of the user who made the request.
- Upon successful authorization, access to the database is granted, and the information request is executed.
- When the information is retrieved from the database, it is processed by the application server.
- The information is then returned to the web server from the application server.
- The response is sent back to the user.
In the process of such a flow, if you got a report indicating that the time required from the user request to the response was excessive, how should this report be investigated? Comprehending the system flow and investigating each and every possibility would be a time-constraining task, and a more complicated service could make the investigation itself even more difficult.
Distributed tracing is a technology which makes it possible to comprehend this flow, and will assist in such investigations. It tracks the flow of a request between one service to another, and will visually represent which service generated the delay, as well as what processes were implemented. Visualization also makes it possible to better identify which services require monitoring, and which metrics to keep an eye on.
Using Vaxila
Several steps must be performed in order to use Vaxila.
1. Submit a usage request
Anyone using Mackerel can utilize Vaxila, but a usage request is required. Furthermore, any usage request must contain a valid email address as well as the name of the Mackerel organization which you wish to use.
Submit the usage request by using the entry form which you can access via the Vaxila menu, which will be displayed in the side menu of the organization which you wish to use after signing in to Mackerel1. After submitting the usage request, once Vaxila becomes available for use, you will receive an email at the address you provided which contains information required for sending data to Vaxila2.
2. Access Vaxila
Upon receiving the email which contains information about sending data to Vaxila, once you reaccess the Vaxila menu of the organization for which you made the submission, you will be able to log in to Vaxila. Your access to Vaxila will be implemented as a currently signed-in user of Mackerel. Please note that if your Mackerel organization permission is Viewer, then you will not be able to access Vaxila.
3. Send data to Vaxila
To send data to Vaxila, you will be required to instrument your application in order to send the telemetry data. Because Vaxila utilizes the OpenTelemetry architecture, please use either the SDK provided for various programming languages or OpenTelemetry Collector to send the telemetry data. Here, we will describe an example of sending data to Vaxila using OpenTelemetry Collector.
The OpenTelemetry Collector configuration is written in a YAML file. By default, it is located at /etc/<otel-directory>/config.yaml with <otel-directory> consisting of names such as otelcol or otelcol-contrib.
A configuration example is displayed as follows:
receivers: otlp: protocols: grpc: http: processors: memory_limiter: check_interval: 1s limit_mib: 500 spike_limit_mib: 100 batch: # Vaxila does not accept requests exceeding 6 MB. # Therefore, set the maximum number of spans per request as appropriate. # A span represents a unit of work or operation for a trace. # A span may represent the execution of a database query or a portion of an application process, etc. # https://opentelemetry.io/docs/concepts/signals/traces/#spans # Up to 5,000,000 spans per month may be sent to Vaxila per Mackerel organization. send_batch_size: 5000 send_batch_max_size: 5000 exporters: otlphttp/vaxila: endpoint: "https://otlp-vaxila.mackerelio.com" headers: Accept: "*/*" "Mackerel-Api-Key": ${env:MACKEREL_APIKEY} extensions: health_check: service: extensions: [health_check] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp/vaxila]
The Vaxila API token can be displayed by clicking the "Reveal" button in the Token row within the Settings page. Set this environmental variable to MACKEREL_APIKEY.
Furthermore, the endpoint for sending telemetry data is as follows.
https://otlp-vaxila.mackerelio.com
When you send your telemetry data to Vaxila, you will be able to verify the data in visualized form as Traces and/or Issues, etc.
Future outlook
To date, Mackerel has been structured primarily around the world of metrics. Metrics can be evaluated by lining up the values sequentially. However, in order to optimally utilize them, you need to predetermine which metrics need to be observed. In contrast, by using this release of Vaxila to implement tracing, you can detect where bottlenecks are occurring through the visualized telemetry data, and gain insight into which metrics should be observed.
Currently, Mackerel and Vaxila exist as two separate products, requiring users to go back and forth between them in utilization. But we are working toward integrating them together in the future.
Furthermore, strengthening our presence in the observability domain necessitates not only integrating Vaxila's features into Mackerel, but also further refining it into a product which can effectively solve the issues that concern our users. We hope you will take this opportunity to experience Vaxila, and provide your concerns—for example, "I wish it could be like ..." or "if it could be like ... it would make operations more enjoyable and simple for us"—to us as feedback!
