Mackerel uses the OpenTelemetry mechanism (instrumentation) to collect data.
This page explains how to send data from Go to Mackerel.
OpenTelemetry for Go
OpenTelemetry provides an SDK for Go.
In addition to this SDK, you can use SDKs for Gin and Gorm to instrument various areas.
Is it appropriate to use a Collector?
When sending data to Mackerel, you can either send it directly from the SDK or use a Collector.
Refer to the following page when deciding whether to use a Collector.
Deciding whether to use Collector
Implementation
There are several web frameworks available for Go, but this page explains how to implement the most basic net/http
. If you are using another framework, you can implement instrumentation in much the same way as described below.
You can implement Mackerel's tracing feature by following the steps below.
- go get
- Initial setup
- Inserting middleware
- Adding error handling
- Adding custom instrumentation (optional)
1. go get
Install the following modules.
go get go.opentelemetry.io/otel \ go.opentelemetry.io/otel/trace \ go.opentelemetry.io/otel/exporters/otlp/otlptrace
This command installs only the basic modules.
However, OpenTelemetry has modules for various purposes, such as Gin, Gorm, and AWS, and you can add them according to your application to enable more detailed instrumentation.
Each module can be found on the OpenTelemetry page.
2. Initial setup
To send OpenTelemetry data to Mackerel, you need to set the following items in TracerProvider.
- Exporter
- Resource
For example, the following will allow you to send data directly from the SDK to Mackerel.
import ( ... "go.opentelemetry.io/otel/exporters/otlp/otlptrace" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/exporters/stdout/stdouttrace" "go.opentelemetry.io/otel/sdk/trace" "go.opentelemetry.io/otel/attribute" semconv "go.opentelemetry.io/otel/semconv/v1.20.0" ) func initTracerProvider(ctx context.Context) (func(context.Context) error, error) { client := otlptracehttp.NewClient( otlptracehttp.WithEndpoint("otlp-vaxila.mackerelio.com"), otlptracehttp.WithHeaders(map[string]string{ "Accept": "*/*", "Mackerel-Api-Key": os.Getenv("MACKEREL_API_KEY"), }), otlptracehttp.WithCompression(otlptracehttp.GzipCompression), ) exporter, err := otlptrace.New(ctx, client) // stdouttrace is useful for debugging // exporter, err := stdouttrace.New() if err != nil { return nil, err } resources, err := resource.New( ctx, resource.WithProcessPID(), resource.WithHost(), resource.WithAttributes( semconv.ServiceName("acme_service"), semconv.ServiceVersion("vX.Y.Z"), semconv.DeploymentEnvironment("production"), ), ) if err != nil { return nil, err } tp := trace.NewTracerProvider( trace.WithBatcher(exporter), trace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.TraceContext{}) return tp.Shutdown, nil }
This example sets the following items.
- Exporter
- Endpoint & Path
- Setting the endpoint to
otlp-vaxila.mackerelio.com
sends data to Mackerel. - If using a Collector, set the Collector endpoint (such as localhost:4318).
- Setting the endpoint to
- Headers
- Setting the
Mackerel-Api-Key
andAccept
headers enables communication with Mackerel. - Set the
Mackerel-Api-Key
to the write-enabled API key issued by Mackerel. It may take about a minute for changes to take effect after modifying the API key permissions. - When using Collector, headers are not necessary.
- Setting the
- Compression
- You can specify the data compression method when sending data. If not specified, data will not be compressed.
- Endpoint & Path
- Resource
- By setting the Resource of TracerProvider, you can identify the source of the data.
3. Inserting middleware
If you are running an HTTP server, you need to insert dedicated middleware to collect information for each request.
If you are using net/http
, you can use otelhttp
.
For example, using the initTracerProvider
function created in the previous section, you can do the following.
import ( ... "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" ) func main() { ctx := context.Background() shutdown, err := initTracerProvider(ctx) if err != nil { panic(err) } defer shutdown(ctx) otelHandler := otelhttp.NewHandler( http.HandlerFunc(awesomeActionHandler), "awesome_span_name", ) http.Handle("/awesome_path/", otelHandler) http.ListenAndServe(":80", nil) } func awesomeActionHandler(w http.ResponseWriter, r *http.Request) { ... }
⚠️ If you are using another framework
This page explains how to instrument net/http
, but even if you are using Gin or Echo, the interface is similar, so it is basically the same as net/http
.
In other words, you can instrument it with the following steps.
- Get the module corresponding to your framework using go get
- Perform initial setup
- Insert middleware and start tracing
In addition, since routing is well established, instrumentation is generally easier than with net/http
.
You can search for modules corresponding to each framework on the following page.
4. Catch error information
Due to Go's language constraints, errors cannot be automatically caught, so you need to write code to catch errors and record them to the Span when they occur.
Use the Span.RecordError()
function to add errors. You can also use WithStackTrace()
to record stack traces.
For example, to catch errors that occur in the handler (awesomeActionHandler
) created in the previous section, use the following code.
import ( ... "go.opentelemetry.io/otel/trace" semconv "go.opentelemetry.io/otel/semconv/v1.17.0" ) func awesomeActionHandler(w http.ResponseWriter, r *http.Request) { err := someAction() if err != nil { recordError(r.Context(), err) w.WriteHeader(500) return } ... } func recordError(ctx context.Context, err error) { span := trace.SpanFromContext(ctx) span.RecordError(err, trace.WithStackTrace(true)) }
5. Adding custom instrumentation (optional)
You can add custom spans to instrument any range.
Instrumentation makes it possible to record variable values and processing times.
You can add instrumentation by enclosing it with Tracer.Start()
and Span.End()
, as shown below.
const ( instrumentationName = "my-instrumentron" instrumentationVersion = "v0.1.0" ) var ( tracer = otel.GetTracerProvider().Tracer( instrumentationName, trace.WithInstrumentationVersion(instrumentationVersion), trace.WithSchemaURL(semconv.SchemaURL), ) ) func awesomeActionHandler(w http.ResponseWriter, r *http.Request) { ctx, span := tracer.Start(r.Context(), "awesome_action") defer span.End() ... }
Other instrumentation approaches are also available. For details, refer to the OpenTelemetry documentation.