コンテンツにスキップ

Application Tracing With Golang, OpenTelemetry, and Grafana Tempo

チェック

  • [ ] 本文を確認した
  • [ ] 概要を確認した
  • [ ] タグを確認した
  • [ ] inbox/ 直下へ移行した

概要

Application Tracing With Golang, OpenTelemetry, and Grafana Tempo のWebクリップ。本文からGo、AWS、Observability、設計、キャリア評価などの学習材料として使えそうな内容を保存した。関連タグ: go, go/context, observability, opentelemetry, web-clip。

本文

DEV Community

Posted on Aug 28, 2025

Application Tracing With Golang, OpenTelemetry, and Grafana Tempo

Trace

Trace is a record or history from a request. Trace give us big picture abour what is happening from application accepts request to return a response. This record is build above span.

Span

hello

json

{ "name": "hello", "context": { "trace_id": "5b8aa5a2d2c872e8321cf37308d69df2", "span_id": "051581bf3cb55c13" }, "parent_id": null, "start_time": "2022-04-29T18:52:58.114201Z", "end_time": "2022-04-29T18:52:58.114687Z", "attributes": { "http.route": "some_route1" }, "events": [ { "name": "Guten Tag!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } } ] }

Span above contains informations about

start_time

end_time

http.route

event

parent_id

null

hello_greetings

{ "name": "hello-greetings", "context": { "trace_id": "5b8aa5a2d2c872e8321cf37308d69df2", "span_id": "5fb397be34d26b51" }, "parent_id": "051581bf3cb55c13", "start_time": "2022-04-29T18:52:58.114304Z", "end_time": "2022-04-29T22:52:58.114561Z", "attributes": { "http.route": "some_route2" }, "events": [ { "name": "hey there!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } }, { "name": "bye now!", "timestamp": "2022-04-29T18:52:58.114585Z", "attributes": { "event_attributes": 1 } } ] }

greetings

span_id

hello-salutation

{ "name": "hello-salutations", "context": { "trace_id": "5b8aa5a2d2c872e8321cf37308d69df2", "span_id": "93564f51e1abe1c2" }, "parent_id": "051581bf3cb55c13", "start_time": "2022-04-29T18:52:58.114492Z", "end_time": "2022-04-29T18:52:58.114631Z", "attributes": { "http.route": "some_route3" }, "events": [ { "name": "hey there!", "timestamp": "2022-04-29T18:52:58.114561Z", "attributes": { "event_attributes": 1 } } ] }

OpenTelemetry Collector

Collector

  • Receiver

Receiver focus on collecting data from various sources and formats. Here is an example of receiver.

receivers: # Data sources: logs fluentforward: endpoint: 0.0.0.0:8006 # Data sources: metrics hostmetrics: scrapers: cpu: disk: filesystem: load: memory: network: process: processes: paging: # Data sources: traces jaeger: protocols: grpc: endpoint: 0.0.0.0:4317 thrift_binary: thrift_compact: thrift_http: # Data sources: traces, metrics, logs kafka: protocol_version: 2.0.0 # Data sources: traces, metrics opencensus: # Data sources: traces, metrics, logs otlp: protocols: grpc: endpoint: 0.0.0.0:4317 tls: cert_file: cert.pem key_file: cert-key.pem http: endpoint: 0.0.0.0:4318 # Data sources: metrics prometheus: config: scrape_configs: - job_name: otel-collector scrape_interval: 5s static_configs: - targets: [localhost:8888] # Data sources: traces zipkin:
  • Processor

Processor is use to process data from receiver. Processed data on this component cosists of filtering, renaming, or recalculating. We could not only define processor once, but also multiple of them, depending on our needs. Outcome of telemetry data depends on sequence or order of data processing on this component. Here is example of processor definition

processors: # Data sources: traces attributes: actions: - key: environment value: production action: insert - key: db.statement action: delete - key: email action: hash # Data sources: traces, metrics, logs batch: # Data sources: metrics, metrics, logs filter: error_mode: ignore traces: span: - 'attributes["container.name"] == "app_container_1"' - 'resource.attributes["host.name"] == "localhost"' - 'name == "app_3"' spanevent: - 'attributes["grpc"] == true' - 'IsMatch(name, ".*grpc.*")' metrics: metric: - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"' - 'type == METRIC_DATA_TYPE_HISTOGRAM' datapoint: - 'metric.type == METRIC_DATA_TYPE_SUMMARY' - 'resource.attributes["service.name"] == "my_service_name"' logs: log_record: - 'IsMatch(body, ".*password.*")' - 'severity_number < SEVERITY_NUMBER_WARN' # Data sources: traces, metrics, logs memory_limiter: check_interval: 5s limit_mib: 4000 spike_limit_mib: 500 # Data sources: traces resource: attributes: - key: cloud.zone value: zone-1 action: upsert - key: k8s.cluster.name from_attribute: k8s-cluster action: insert - key: redundant-attribute action: delete # Data sources: traces probabilistic_sampler: hash_seed: 22 sampling_percentage: 15 # Data sources: traces span: name: to_attributes: rules: - ^\/api\/v1\/document\/(?P<documentId>.*)\/update$ from_attributes: [db.svc, operation] separator: '::'
  • Exporter

type/name

type

name

exporters: # Data sources: traces, metrics, logs file: path: ./filename.json # Data sources: traces otlp/jaeger: endpoint: jaeger-server:4317 tls: cert_file: cert.pem key_file: cert-key.pem # Data sources: traces, metrics, logs kafka: protocol_version: 2.0.0 # Data sources: traces, metrics, logs # NOTE: Prior to v0.86.0 use `logging` instead of `debug` debug: verbosity: detailed # Data sources: traces, metrics opencensus: endpoint: otelcol2:55678 # Data sources: traces, metrics, logs otlp: endpoint: otelcol2:4317 tls: cert_file: cert.pem key_file: cert-key.pem # Data sources: traces, metrics otlphttp: endpoint: https://otlp.example.com:4318 # Data sources: metrics prometheus: endpoint: 0.0.0.0:8889 namespace: default # Data sources: metrics prometheusremotewrite: endpoint: http://prometheus.example.com:9411/api/prom/push # When using the official Prometheus (running via Docker) # endpoint: 'http://prometheus:9090/api/v1/write', add: # tls: # insecure: true # Data sources: traces zipkin: endpoint: http://zipkin.example.com:9411/api/v2/spans

Instrumenting Go Application With OpenTelemetry Tracing

Spin Up Collector

Trace and span data emitted by server will be collected, stored, filtered, and exported by OpenTelemetry collector. The rules of collector defined by a configuration file as shown below.

receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlphttp/trace: endpoint: http://tempo:4318 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp/trace]

receivers.otlp.http.endpoint

0.0.0.0:4318

0.0.0.0

processor

exporters

service

Defining Exporter

NewTracer

package trace import ( "context" "fmt" "net/http" "runtime" "time" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/trace" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlptrace" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" traceSdk "go.opentelemetry.io/otel/sdk/trace" ) var globalServiceName string func NewTracer( ctx context.Context, serviceName string, ) error { propagator := propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{}, ) otel.SetTextMapPropagator(propagator) // Create OpenTelemetry tracing client that exports // trace data to OpenTelemetry HTTP receiver traceClientHTTP := otlptracehttp.NewClient( otlptracehttp.WithInsecure(), ) traceExporter, err := otlptrace.New( ctx, traceClientHTTP, ) if err != nil { return err } res, err := resource.New( ctx, resource.WithContainer(), resource.WithFromEnv(), ) if err != nil { return err } tracerProvider := traceSdk.NewTracerProvider( traceSdk.WithBatcher( traceExporter, traceSdk.WithBatchTimeout(time.Second), ), traceSdk.WithResource( res, ), ) globalServiceName = serviceName otel.SetTracerProvider(tracerProvider) return nil }

Define Tracing Components

HTTPLevelSpan

ServiceLevelSpan

package trace import ( "context" "fmt" "net/http" "runtime" "time" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/trace" "go.opentelemetry.io/otel/attribute" ) func HTTPLevelSpan( ctx context.Context, r *http.Request, ) (context.Context, trace.Span) { spanName := fmt.Sprintf("%s %s", r.Method, r.URL.String()) ctx, span := otel.Tracer(globalServiceName).Start(ctx, spanName) if span.IsRecording() { span.SetAttributes(attribute.String("http.request.method", r.Method)) span.SetAttributes(attribute.String("server.address", r.RemoteAddr)) span.SetAttributes(attribute.String("url.full", r.URL.String())) } return ctx, span } func ServiceLevelSpan( ctx context.Context, ) (context.Context, trace.Span) { pc := make([]uintptr, 10) runtime.Callers(2, pc) f := runtime.FuncForPC(pc[0]) ctx, span := otel.Tracer(globalServiceName).Start(ctx, f.Name()) if span.IsRecording() { span.SetAttributes(attribute.String("code.function.name", f.Name())) } return ctx, span }

code.function

server.address

url.full

Instrumenting Application

package server import ( "log" "net/http" ) func RunServer() { http.HandleFunc("/", home()) if err := http.ListenAndServe(":9320", nil); err != nil { log.Fatalf("could not start server: %v\n", err) } }

package server import ( "fmt" "log" "net/http" "strconv" "time" "go-trace-demo/service" "go-trace-demo/utils/metrics" utilsTrace "go-trace-demo/utils/trace" ) func home() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ctx := r.Context() ctx, span := utilsTrace.HTTPLevelSpan(ctx, r) defer span.End() respCode := 200 failQuery := r.URL.Query().Get("fail") if failQuery != "" { respCode = 500 isFail, err := strconv.ParseBool(failQuery) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte("error parse query")) return } if isFail { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte("encounter fail")) return } } err := service.GetUserByID(ctx) if err != nil { return } err = service.ListStores(ctx) if err != nil { return } w.Write([]byte("Hello, World!")) } }

package service import ( "context" utilsTrace "go-trace-demo/utils/trace" ) func GetUserByID( ctx context.Context, ) error { ctx, span := utilsTrace.ServiceLevelSpan(ctx) defer span.End() return nil } func ListStores( ctx context.Context, ) error { ctx, span := utilsTrace.ServiceLevelSpan(ctx) defer span.End() return nil }

main

func main() { ctx := context.Background() serviceName := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") err := utilsTrace.NewTracer(ctx, serviceName) if err != nil { log.Fatalln("fail setup tracer") } server.RunServer() }

Glue Everything

We'll run application, OpenTelemetry Collector, and Grafana Tempo using docker compose file below

version: "3" networks: observability: services: tempo: image: grafana/tempo:main-c5323bf command: [ "-config.file=/etc/tempo.yaml" ] volumes: - ./config/tempo-config.yaml:/etc/tempo.yaml - ./config/tempo-data:/var/tempo ports: - "3200:3200" # tempo - "4317" # open this port for OpenTelemetry gRPC exporter - "4318" # open this port for OpenTelemetry HTTP exporter networks: - observability otel-collector: image: otel/opentelemetry-collector:latest volumes: - ./config/otel-config.yaml:/etc/otel/config.yaml command: - '--config=/etc/otel/config.yaml' ports: - "4317:4317" #grpc - "4318:4318" #http depends_on: - loki networks: - observability app: container_name: app build: context: ./ dockerfile: ./Dockerfile ports: - 9320:9320 networks: - observability depends_on: - otel-collector environment: - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 - OTEL_SERVICE_NAME=go-loki-app-demo grafana: environment: - GF_PATHS_PROVISIONING=/etc/grafana/provisioning - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin entrypoint: - sh - -euc - | mkdir -p /etc/grafana/provisioning/datasources cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy orgId: 1 url: http://loki:3100 basicAuth: false isDefault: true version: 1 editable: false - name: Tempo type: tempo access: proxy orgId: 1 url: http://tempo:3200 basicAuth: false version: 1 editable: false EOF /run.sh image: grafana/grafana:latest ports: - "3000:3000" networks: - observability

service.app

OTEL_EXPORTER_OTLP_ENDPOINT

WithEndpointURL

otlptracehttp.NewClient

OTEL_SERVICE_NAME

unknown_service

Visualizing Traces Using Grafana Tempo

localhost:9320/

localhost:3000

Exlpore

Search

TraceQL

Service Name

Run Query

Trace

Additionally, explore span attribute by click one of span. We will see span attribute and resource attribute we defined when we are creating OpenTelemetry trace instance.

References

  • OpenTelemetry Collector

  • OpenTelemetry Collector Configuration

  • Semantic Convention

  • Go OpenTelemetry HTTP Exporter

Top comments (0)

Templates let you quickly answer FAQs or store snippets for re-use.

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.

Hide child comments as well

Confirm

For further actions, you may consider blocking this person and/or reporting abuse

  • Joined Oct 31, 2022

More from Syukur

We're a place where coders share, stay up-to-date and grow their careers.

要点

  • Goの実装・設計・標準的な書き方を面接や実務の深掘り材料にする。
  • トレーシング、メトリクス、可観測性の説明材料として使う。
  • 元URL: https://dev.to/sykrabadi/application-tracing-with-golang-opentelemetry-and-grafana-tempo-lfg

タグ

go #go/context #observability #opentelemetry #web-clip