Trace Requests Across Go APIs With OpenTelemetry Spans

A request to an API often touches several services before it returns. When it is slow or fails, logs from each service tell separate stories, and joining them by a request id is manual work. A trace is the joined story: one id for the whole request, a tree of spans, each span a unit of work with a start, a duration, attributes and a parent. You can see that 600 ms of a 700 ms request went to one call, in one service, below one handler.

OpenTelemetry is the vendor-neutral standard for producing that data, and its Go library is solid. This article sets up tracing for two cooperating Go services, shows the span tree it produces, and covers the choices that matter for production: what to name spans, what to put in attributes, how errors are recorded and how much to sample. It sits beside request ID middleware, API logging without leaking secrets and timeouts across service calls. Everything ran on Go 1.25.6 with OpenTelemetry Go v1.38.0.

The moving parts

Tracing has four pieces that are easy to blur together.

1. The API (go.opentelemetry.io/otel) is what your code calls: get a tracer, start spans, set attributes. It does nothing until configured.
2. The SDK (.../otel/sdk/trace) implements the API: it creates spans, applies sampling and hands finished spans to exporters.
3. Exporters send spans out: to the OpenTelemetry Collector over OTLP, directly to a backend, or to stdout in development.
4. Propagators move trace context between processes, usually in the W3C traceparent HTTP header. Without propagation you get disconnected single-service traces.

The instrumentation libraries, otelhttp here, sit on top of the API and create the standard HTTP spans so you do not write them yourself.

The experiment

The lab builds two services in one process, each on its own httptest server so the calls cross a real HTTP connection. The orders service handles POST /orders: it records a database insert as a child span, calls the inventory service through an instrumented client, and optionally fails a payment step. The inventory service does a bit of work in its own span. Spans go to an in-memory exporter so the program can print the tree.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"sort"
	"strings"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/sdk/trace/tracetest"
)

func main() {
	exporter := tracetest.NewInMemoryExporter()
	tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(exporter))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
	tracer := otel.Tracer("shop")

	// Service B (inventory): a downstream HTTP service with its own span.
	inventory := httptest.NewServer(otelhttp.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		_, span := tracer.Start(r.Context(), "inventory.reserve")
		defer span.End()
		span.SetAttributes(attribute.Int("sku.count", 2))
		time.Sleep(2 * time.Millisecond)
		fmt.Fprint(w, "reserved")
	}), "inventory"))
	defer inventory.Close()

	// Service A (orders): the public API. It calls a "database" and then service B using an instrumented client.
	client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
	orders := httptest.NewServer(otelhttp.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()

		_, db := tracer.Start(ctx, "db.insert_order")
		db.SetAttributes(attribute.String("db.system", "postgresql"), attribute.String("db.operation", "INSERT"))
		time.Sleep(time.Millisecond)
		db.End()

		req, _ := http.NewRequestWithContext(ctx, http.MethodPost, inventory.URL+"/reserve", nil)
		resp, err := client.Do(req) // the transport injects the traceparent header from ctx
		if err != nil {
			http.Error(w, err.Error(), 502)
			return
		}
		io.Copy(io.Discard, resp.Body)
		resp.Body.Close()

		if r.URL.Query().Get("fail") == "1" {
			_, s := tracer.Start(ctx, "payment.charge")
			s.RecordError(fmt.Errorf("card declined"))
			s.SetStatus(codes.Error, "card declined")
			s.End()
			http.Error(w, "payment failed", http.StatusPaymentRequired)
			return
		}
		fmt.Fprint(w, "created")
	}), "POST /orders"))
	defer orders.Close()

	for _, q := range []string{"", "?fail=1"} {
		resp, _ := http.Post(orders.URL+"/orders"+q, "application/json", strings.NewReader("{}"))
		resp.Body.Close()
	}
	tp.ForceFlush(context.Background())

	spans := exporter.GetSpans()
	byID := map[string]string{}
	for _, s := range spans {
		byID[s.SpanContext.SpanID().String()] = s.Name
	}
	traces := map[string][]tracetest.SpanStub{}
	var order []string
	for _, s := range spans {
		id := s.SpanContext.TraceID().String()
		if _, ok := traces[id]; !ok {
			order = append(order, id)
		}
		traces[id] = append(traces[id], s)
	}
	for i, id := range order {
		fmt.Printf("trace %d (%d spans, one trace id shared by both services)\n", i+1, len(traces[id]))
		ss := traces[id]
		sort.SliceStable(ss, func(a, b int) bool { return ss[a].StartTime.Before(ss[b].StartTime) })
		for _, s := range ss {
			parent := "(root)"
			if s.Parent.IsValid() {
				parent = byID[s.Parent.SpanID().String()]
			}
			var notes []string
			for _, a := range s.Attributes {
				switch string(a.Key) {
				case "http.response.status_code", "http.request.method", "db.system", "db.operation", "sku.count":
					notes = append(notes, fmt.Sprintf("%s=%v", a.Key, a.Value.Emit()))
				}
			}
			if s.Status.Code == codes.Error {
				notes = append(notes, "status=error("+s.Status.Description+")")
			}
			fmt.Printf("  %-28s kind=%-8s parent=%-26s %s\n", s.Name, s.SpanKind, parent, strings.Join(notes, " "))
		}
	}
}

trace 1 (5 spans, one trace id shared by both services)
  POST /orders                 kind=server   parent=(root)                     http.request.method=POST http.response.status_code=200
  db.insert_order              kind=internal parent=POST /orders               db.system=postgresql db.operation=INSERT
  HTTP POST                    kind=client   parent=POST /orders               http.request.method=POST http.response.status_code=200
  inventory                    kind=server   parent=HTTP POST                  http.request.method=POST http.response.status_code=200
  inventory.reserve            kind=internal parent=inventory                  sku.count=2
trace 2 (6 spans, one trace id shared by both services)
  POST /orders                 kind=server   parent=(root)                     http.request.method=POST http.response.status_code=402
  db.insert_order              kind=internal parent=POST /orders               db.system=postgresql db.operation=INSERT
  HTTP POST                    kind=client   parent=POST /orders               http.request.method=POST http.response.status_code=200
  inventory                    kind=server   parent=HTTP POST                  http.request.method=POST http.response.status_code=200
  inventory.reserve            kind=internal parent=inventory                  sku.count=2
  payment.charge               kind=internal parent=POST /orders               status=error(card declined)

The setup is the first six lines of main: a tracer provider with an exporter, registered globally, and a propagator that handles traceparent and baggage. After that, each server is wrapped with otelhttp.NewHandler(handler, "name") and the client's transport with otelhttp.NewTransport(...). The handler code does not mention tracing apart from the spans the author chose to add.

Reading the trace

The first trace has five spans, all with one trace id, and the printed tree shows each span's parent:

• POST /orders is the server span for the orders service. It is the root, since the test client sent no traceparent. It carries the HTTP method and the response status code, which otelhttp records automatically.
• db.insert_order is a child span created with tracer.Start(ctx, ...). Its parent is the server span because the context passed in came from the request. It has db.system and db.operation attributes, following the OpenTelemetry semantic conventions for databases.
• HTTP POST is the client span created by the instrumented transport when the orders service calls inventory. It is a child of the server span.
• inventory is the server span in the other service. Its parent is the client span from the orders service. That one fact is the whole point of propagation: the transport wrote a traceparent header containing the client span's id, and the inventory handler extracted it and made its own span a child. Nothing in the inventory code mentions the orders service.
• inventory.reserve is a custom child span inside the inventory handler, with a sku.count attribute.

The second trace repeats that and adds a sixth span. payment.charge has status error(card declined), set with span.RecordError(err) and span.SetStatus(codes.Error, ...). The server span for that request shows a response status of 402. In a tracing UI, that request would be flagged red with the failing span marked, and the error event attached to it.

What to put in spans

Names. A span name should be low-cardinality: POST /orders, db.insert_order, payment.charge. Do not put ids or user input in names; every unique name is a separate series in many backends. otelhttp lets you pass a name formatter, and the route pattern from your mux is the right basis. Go 1.22's ServeMux exposes it as r.Pattern.

Attributes. Put the identifiers you will search by in attributes: order id, tenant id, a hash of the user id. Follow the semantic conventions for standard keys (http.request.method, db.system, url.path), so dashboards work without translation. Keep attribute values small.

Privacy. Spans are exported to a third party's storage. Do not attach emails, tokens, request bodies or raw SQL with literal values. The same rule as logging applies, and more strongly, since trace retention is often longer. Record query shapes, not parameters.

Errors. Use RecordError to attach the error as an event and SetStatus(codes.Error, msg) to mark the span failed. Marking a span as an error is a decision: a 404 from a lookup is usually not an error for the client span, and a 500 from the downstream service is. The convention is that server spans are errors for 5xx and client spans for 4xx and 5xx, and otelhttp applies it for you.

Events and links. Use span events for notable moments inside a span (a cache miss, a retry attempt) rather than creating tiny spans. Use links to connect a span to causally related traces that are not parents, such as a message consumer linking to the trace that produced the message.

Propagation beyond HTTP

traceparent carries a version, a trace id, the parent span id and flags, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The format is the W3C Trace Context standard. Any client that sends it and any server that reads it will join traces regardless of language.

For other transports you need a carrier. gRPC has otelgrpc interceptors (gRPC and OpenTelemetry). For message queues, inject the context into message headers on publish and extract it on consume, using propagation.MapCarrier or a carrier for your client's header type. For background jobs started from a request, pass the context's span context through the job payload and start a new span linked to it.

Be careful at the edge of your system. An external caller can send any traceparent; honouring it lets them stitch their traces into yours (usually harmless and sometimes useful), while a public service may prefer to start a fresh trace and link to the incoming one.

Exporting for real

The in-memory exporter is for tests. In production, use the OTLP exporter and send to an OpenTelemetry Collector running next to your service:

exp, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint("otel-collector:4318"), otlptracehttp.WithInsecure())
tp := sdktrace.NewTracerProvider(
	sdktrace.WithBatcher(exp),
	sdktrace.WithResource(resource.NewSchemaless(attribute.String("service.name", "orders"))),
	sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
)
defer tp.Shutdown(ctx)

Three details matter. WithBatcher sends spans in groups asynchronously; the lab used WithSyncer to get predictable output, which blocks on every span end and does not belong in production. service.name is what distinguishes your services in the backend; set it. And Shutdown flushes buffered spans, so call it during graceful shutdown, or the last seconds of traces are lost.

I did not run the OTLP exporter or a collector for this article. The snippet follows the library documentation.

Sampling

Recording every trace is affordable at small scale and ruinous at large. The sampler decides at the root span. TraceIDRatioBased(0.1) keeps 10 percent, and ParentBased makes services honour the decision of the caller, so a trace is either complete or absent, never half-recorded. Head sampling like this cannot know whether a trace will turn out slow or failed. If you want all the errors and slow requests, sample at the Collector instead (tail sampling), which waits for the whole trace before deciding, at the cost of memory in the Collector.

Costs and pitfalls

Instrumentation costs a little CPU and allocation per span. For typical APIs it is a few microseconds per span, and a handler that makes thousands of tiny spans in a loop is a mistake. Create spans for operations a human would want to see on a timeline.

Context must be passed along. The most common reason for missing child spans is a function that takes no context.Context or a goroutine that is started with context.Background(). Fix the signature; tracing will tell you where the chain broke. Start spans with the context from the request, and pass the span's context to what you call.

End every span, usually with defer span.End() right after Start. A span that never ends is never exported.

The documentation for the Go API and SDK is at opentelemetry.io, and the context propagation format is in the W3C Trace Context specification.