OpenTelemetry: automatic vs manual instrumentation

Introduction to OpenTelemetry instrumentation #

Instrumentation is the process of adding code to an application to collect telemetry data (traces, metrics, logs). OpenTelemetry offers two main approaches to instrument your applications: automatic instrumentation and manual instrumentation. The choice between these two methods depends on your specific needs, the level of control you want and the complexity of your application.

1. Automatic instrumentation (auto-instrumentation) #

Automatic instrumentation collects telemetry data without modifying your application's source code. It usually relies on agents or libraries that inject themselves into the application runtime and automatically instrument popular libraries and frameworks.

1.1. How does it work? #

Language-specific agents (for example a Java agent or a Python hook) monitor your application's execution. When they detect the use of a supported library (such as an HTTP client, a database or a web framework), they automatically inject instrumentation points to generate spans, metrics or logs.

1.2. Advantages and drawbacks #

Advantages:
- Easy to implement: often a single command line or environment variable is enough.
- Time saving: no source code changes, fast deployment.
- Broad coverage: automatically instruments many common libraries and frameworks.
- Easy updates: the agent can be updated independently of the application.
Drawbacks:
- Less control: hard to instrument specific business logic or finely customize spans.
- Potential overhead: can introduce slight overhead if the agent instruments too many irrelevant things.
- Agent dependency: requires keeping the agent up to date and compatible with your environment.

1.3. Configuration examples #

Automatic instrumentation is usually enabled through environment variables or command-line arguments.

Java agent

For a Java application, you download the OpenTelemetry agent and launch it with your JVM:

java -javaagent:/path/to/opentelemetry-javaagent.jar \n     -jar my-application.jar

You can configure the exporter and other settings through environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-java-app"
java -javaagent:/path/to/opentelemetry-javaagent.jar -jar my-application.jar

Python auto-instrumentation

For Python, you use the opentelemetry-instrument script:

opentelemetry-instrument python my_app.py

Or by setting environment variables for the exporter:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-python-app"
opentelemetry-instrument python my_app.py

Node.js auto-instrumentation

For a Node.js application, you can preload the OpenTelemetry SDK using Node's -r (or --require) option:

node -r @opentelemetry/sdk-node/register my-app.js

Configuration is also done through environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-nodejs-app"
node -r @opentelemetry/sdk-node/register my-app.js

Go auto-instrumentation (through eBPF)
Auto-instrumentation for Go is less common because Go is compiled, but it is possible thanks to eBPF. It lets you instrument Go binaries without any code change. It is a powerful technology, but still under development.
Once the Go instrumentation agent is installed, you launch it pointing to your application binary:
```
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="my-go-app"

# L'agent Go prend en charge le binaire de votre application
/path/to/opentelemetry-go-instrumentation-launcher --service_name my-go-app /path/to/your/app-binary
```
Note: This method is still considered experimental and may not cover all libraries. The hybrid approach (auto + manual) often remains the most reliable for Go.

2. Manual instrumentation (SDK) #

Manual instrumentation means adding calls to the OpenTelemetry API and SDK directly in your application's source code. This gives you full control over the telemetry data collected.

2.1. How does it work? #

The developer uses the OpenTelemetry libraries to explicitly create spans, record metrics, add custom attributes and propagate the trace context across process boundaries.

2.2. Advantages and drawbacks #

Advantages:
- Full control: precisely instrument business logic, add specific attributes and define complex span relationships.
- Granularity: capture exactly the information you need for diagnosis.
- Flexibility: adapt instrumentation to very specific use cases not covered by auto-instrumentation.
Drawbacks:
- Code changes: requires changes to the application's source code.
- Development effort: more initial work and maintenance.
- Risk of errors: incorrect instrumentation can lead to incomplete traces or performance problems.

2.3. Configuration examples #

Manual instrumentation requires initializing the OpenTelemetry SDK in your application.

Java SDK

Example of initializing the SDK and creating a span in Java:

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;

public class ManualInstrumentationExample {

    private static OpenTelemetry openTelemetry;

    public static void initOpenTelemetry() {
        // Configure l'exportateur OTLP (gRPC par défaut)
        OtlpGrpcSpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
            .setEndpoint("http://localhost:4317")
            .build();

        // Configure le processeur de span pour envoyer les spans à l'exportateur
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(SimpleSpanProcessor.create(spanExporter))
            .build();

        // Initialise l'instance globale d'OpenTelemetry
        openTelemetry = OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .buildAndRegisterGlobal();

        // Assurez-vous de fermer le provider à l'arrêt de l'application
        Runtime.getRuntime().addShutdownHook(new Thread(tracerProvider::close));
    }

    public static void main(String[] args) {
        initOpenTelemetry();
        Tracer tracer = openTelemetry.getTracer("my-manual-instrumentation-library");

        Span span = tracer.spanBuilder("myOperation").startSpan();
        try (var scope = span.makeCurrent()) {
            span.setAttribute("my.custom.attribute", "value");
            System.out.println("Executing my operation...");
            // ... votre logique métier ...
        } finally {
            span.end();
        }
    }
}

Python SDK

Example of initializing the SDK and creating a span in Python:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OtlpSpanExporter

# Configure le Resource pour le service
resource = Resource.create({
    "service.name": "my-python-manual-app",
    "service.version": "1.0.0"
})

# Configure le TracerProvider
provider = TracerProvider(resource=resource)

# Configure l'exportateur OTLP
otlp_exporter = OtlpSpanExporter(endpoint="http://localhost:4317")

# Ajoute le processeur de span au provider
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))

# Définit le provider comme global
trace.set_tracer_provider(provider)

# Obtient un tracer
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("myOperation") as span:
    span.set_attribute("my.custom.attribute", "another_value")
    print("Executing my Python operation...")
    # ... votre logique métier ...

3. The importance of resource attributes with OTEL_RESOURCE_ATTRIBUTES #

Resource attributes describe the entity that produces the telemetry data (for example the service, application, host, container or cloud). They are crucial to correlate data and provide a global context to all the traces, metrics and logs emitted by that resource.

3.1. What is OTEL_RESOURCE_ATTRIBUTES? #

OTEL_RESOURCE_ATTRIBUTES is an environment variable standardized by OpenTelemetry that lets you define resource attributes for your application. These attributes are added to all the telemetry data generated by the application, whether it is instrumented automatically or manually.

It takes a comma-separated list of key=value pairs. Using the semantic conventions (SemConv) for these attributes is strongly recommended.

3.2. Why use it? #

Global context: provides essential information about the runtime environment (service name, version, deployment environment, pod/container IDs, cloud region, etc.).
Easier correlation: lets observability tools group and filter data by service, environment or infrastructure.
Consistency: ensures that all the telemetry data of the same service instance shares the same resource context.
Independent of instrumentation: can be configured without changing the application's code, even with manual instrumentation, to add generic resource attributes.

3.3. Usage example #

Set this environment variable before starting your application:

export OTEL_RESOURCE_ATTRIBUTES="service.name=mon-service-web,service.version=1.2.3,deployment.environment=production,host.name=web-server-01"

# Pour une application Java avec auto-instrumentation
java -javaagent:/path/to/opentelemetry-javaagent.jar -jar my-application.jar

# Pour une application Python avec auto-instrumentation
opentelemetry-instrument python my_app.py

# Pour une application avec instrumentation manuelle (les attributs seront fusionnés avec ceux définis dans le code)
python my_manual_app.py

Note that if resource attributes are defined both through OTEL_RESOURCE_ATTRIBUTES and in the code (SDK), the ones defined in the code may override or complement those of the environment variable, depending on the SDK implementation.

Conclusion: when to choose which approach? #

Use automatic instrumentation for a quick start and baseline coverage of standard libraries and frameworks. It is excellent to get fast visibility without development effort.
Adopt manual instrumentation when you need deep visibility into your specific business logic, critical custom attributes or complex span relationships.
In most cases, a hybrid approach is the most effective: use auto-instrumentation for baseline coverage and complement it with manual instrumentation for your application's critical points.
Always use OTEL_RESOURCE_ATTRIBUTES to provide a consistent, essential resource context to all your telemetry data.