domains / devops / distributed-tracing-setup / distributed-tracing

distributed-tracing

Skill card

combined

The first copy is a concrete, code-heavy tutorial for deploying Jaeger and instrumenting Python, Node.js, and Go applications. The second copy is a high-level, risk-aware procedure that emphasizes security, privacy, and verification steps, and it references external files (playbooks and references) that are not present in the first copy. The first copy lacks the security/privacy constraints and verification rigor of the second, while the second lacks the specific implementation details (K8s manifests, code snippets) of the first. Combining them provides a complete skill that covers both the 'how-to' implementation and the 'best-practice' safety/verification checks.

Repositories

Other copies

Combined skill

---
name: distributed-tracing
description: Implement distributed tracing with Jaeger and Tempo to track requests across microservices, ensuring security and performance.
---

# Distributed Tracing

Implement distributed tracing to track requests across microservices and identify performance bottlenecks.

## When to Use
- Debug latency issues or error propagation.
- Understand service dependencies.
- Implement observability for new integrations.

## Prerequisites
- Record installed SDK and backend versions.
- Identify permitted collector endpoints and service graph.
- Inspect installed version documentation before selecting exporter APIs.

## Procedure

1. **Define Scope**: Identify one user journey to trace. Clarify goals and constraints.
2. **Configure Backend**:
   - **Jaeger (Kubernetes)**:
     ```bash
     kubectl create namespace observability
     kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability
     # Apply Jaeger CRD with production strategy and Elasticsearch storage
     ```
   - **Jaeger (Docker Compose)**:
     ```yaml
     services:
       jaeger:
         image: jaegertracing/all-in-one:latest
         ports:
           - "16686:16686" # UI
           - "14268:14268" # Collector
     ```
   - **Security Check**: Review listeners, authentication, transport protection, and storage retention before deployment.
3. **Instrument Application (OpenTelemetry)**:
   - **Python (Flask)**:
     ```python
     from opentelemetry import trace
     from opentelemetry.exporter.jaeger.thrift import JaegerExporter
     from opentelemetry.sdk.resources import SERVICE_NAME, Resource
     from opentelemetry.sdk.trace import TracerProvider
     from opentelemetry.sdk.trace.export import BatchSpanProcessor
     from opentelemetry.instrumentation.flask import FlaskInstrumentor
     
     resource = Resource(attributes={SERVICE_NAME: "my-service"})
     provider = TracerProvider(resource=resource)
     processor = BatchSpanProcessor(JaegerExporter(agent_host_name="jaeger", agent_port=6831))
     provider.add_span_processor(processor)
     trace.set_tracer_provider(provider)
     
     app = Flask(__name__)
     FlaskInstrumentor().instrument_app(app)
     ```
   - **Node.js (Express)**:
     ```javascript
     const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
     const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
     const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
     const { registerInstrumentations } = require('@opentelemetry/instrumentation');
     const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
     
     const provider = new NodeTracerProvider({ resource: { attributes: { 'service.name': 'my-service' } } });
     const exporter = new JaegerExporter({ endpoint: 'http://jaeger:14268/api/traces' });
     provider.addSpanProcessor(new BatchSpanProcessor(exporter));
     provider.register();
     
     registerInstrumentations({ instrumentations: [new HttpInstrumentation()] });
     ```
   - **Security & Privacy**:
     - Propagate context over HTTP and async messages.
     - Use stable operation names and allowlisted attributes.
     - **Do not** log credentials, raw request bodies, or sensitive query values.
     - Ensure bounded attribute cardinality.
4. **Verify Implementation**:
   - Send successful and failing staging requests.
   - Verify connected spans in the backend (UI or API).
   - Check service identity, parentage, and duration units.
   - Test exporter failure behavior (request handling must retain defined behavior).
   - Verify shutdown flush behavior.
5. **Optimize**:
   - Measure queue loss and overhead.
   - Select sampling strategy based on observations (head vs. tail).

## Verification Checklist
- [ ] Actual trace evidence for each exercised boundary.
- [ ] No sensitive data in spans.
- [ ] Shutdown flush and exporter failure behavior observed.
- [ ] Access, storage, and retention checked independently from UI availability.

## Limitations
- Head sampling can discard errors before tail sampling sees them.
- In-memory demos do not prove durable storage.
- Context propagation and cross-service clocks need actual integration tests.