OpenTelemetry¶

Canopy can export traces, metrics and logs with OpenTelemetry. Traces show one request or one Celery task end to end. Metrics show request rate, request duration, the number of requests in progress, and the number of browser errors as canopy.frontend.errors. Logs are the same records that Canopy writes to the console, but with the structured fields kept separate from the message.

The canopy.frontend.errors metric has no attributes, because the values come from the browser. Group the browser errors in the traces or in the logs instead.

All three are off by default. Set OTEL_ENABLED in canopy.ini to switch them on. Canopy also reads all of the options below from the environment, which lets you keep credentials out of the configuration file.

Canopy sends the data to an OpenTelemetry Collector or to any backend that accepts OTLP over HTTP. Use the HTTP port of the collector, which is 4318 by default. Canopy cannot send to the gRPC port 4317.

Basic configuration¶

One line is sufficient for a collector on the same host, because the endpoint defaults to http://localhost:4318:

OTEL_ENABLED=true

Set the endpoint only when the collector is on a different host:

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.example.com:4318

Restart the canopy and the canopy-celery services after a change. You must restart both.

Configuration options¶

Configuration	Description	Default
OTEL_ENABLED	Enable the export of traces, metrics and logs	false
OTEL_EXPORTER_OTLP_ENDPOINT	Base URL of the collector. Canopy appends `/v1/traces`, `/v1/metrics` and `/v1/logs`	http://localhost:4318
OTEL_EXPORTER_OTLP_HEADERS	Headers for the collector, as `key=value,key2=value2`. Use this for an API key
OTEL_EXPORTER_OTLP_CERTIFICATE	Path to the CA certificate of the collector
OTEL_SERVICE_NAME	Name of the service in the backend	canopy
OTEL_RESOURCE_ATTRIBUTES	More attributes on every trace and metric, as `key=value,key2=value2`
OTEL_TRACES_SAMPLER	Sampler name. Use `parentbased_traceidratio` to record a part of the traffic	parentbased_always_on
OTEL_TRACES_SAMPLER_ARG	Sample rate, from 0.0 to 1.0, for a ratio sampler
OTEL_METRICS_ENABLED	Export metrics as well as traces	true
OTEL_LOGS_ENABLED	Export log records as well as traces	true
OTEL_INSTRUMENT_DATABASE	Record one span for each database query. PostgreSQL and Oracle only	false
OTEL_PYTHON_DJANGO_EXCLUDED_URLS	Comma-separated URL patterns to leave out of the traces. Canopy records nothing for a request to one of these URLs	health/

Two more options control which request headers Canopy records. Their names are too long for the table above:

OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST: Request headers to record on the request span. The default is x-forwarded-for,user-agent. Read the warning below before you make this list longer.
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS: Headers whose value becomes [REDACTED]. The default is authorization,proxy-authorization,cookie,set-cookie,x-csrftoken,x-api-key.

What Canopy records¶

One span for each HTTP request, with the URL, the method and the response status.
One span for each Celery task, both where the task is scheduled and where a worker runs it. The two spans are part of the same trace.
One span for each canopy-manage command.
One span for each JavaScript error in the browser. The user interface sends these errors to the server.
One span for each request that Canopy makes to another service, for example the document server, a ticket tracker or object storage.
One span for each database query, if you set OTEL_INSTRUMENT_DATABASE.
One log record for each log line at LOG_LEVEL or above, if you keep OTEL_LOGS_ENABLED on.

Each request span also carries the user.id of the signed-in user, and the request headers that the capture option selects, as http.request.header.<name>.

Database spans¶

Database spans are off by default. One request can make tens of queries, so they multiply the data that Canopy sends. Each span holds the text of the statement, but not the values that Canopy binds to it.

PostgreSQL and Oracle support database spans. MS SQL Server does not.

Warning: a captured header goes to the collector. Do not add authorization or cookie to the capture list. The default redaction list replaces the value of such a header with [REDACTED], but the safest list is the one that does not capture the header at all.

Browser errors¶

The user interface sends each JavaScript error to the server. Canopy records one span with the name frontend error for each report. The span is a child of the request that reports the error.

The span holds an event with the name exception. The event holds the type of the error, its message and its stack. An observability backend shows these events in its exceptions view. The span also holds:

error.type, the class of the error, for example TypeError. A backend groups the errors by this attribute.
url.full, the page in the browser.
user_agent.original, the browser of the user.
user.id, if the user is signed in.
canopy.frontend.component_stack, the React components above the error.
canopy.frontend.schema_issues, the fields that did not match the schema, if the error is a schema mismatch. Each entry names the path of the field and the reason, for example objects.0.parent: invalid_type expected undefined, received null. Canopy records no more than 10 of these. It records the expected and the received value only where both are the name of a type, because the other reasons hold values from the response.

Canopy records no more than 15 stack frames, and no more than 200 characters for each line. These are the only values on the span. The span holds no part of the reported payload.

The log record is different. Its message holds the full report, which Canopy cuts to 10 kB for a user who is not signed in, and to 100 kB for a user who is. Canopy cuts the text at that length, so a report above the limit is no longer valid JSON.

Warning: the report can hold the data of the request that failed, and the log record goes to the collector while OTEL_LOGS_ENABLED is on. Set OTEL_LOGS_ENABLED = false to keep the reports on the host. The user interface removes the value of a field whose name shows a credential, for example password or token, before it sends the report. It cannot remove data that no field name identifies, for example the text of a report or the value of a search filter.

Canopy accepts 5 reports each minute from one IP address for the users who are not signed in, and 15 reports each minute for each user who is. It discards the reports above that rate. Many browsers behind one address share the first limit.

A ratio sampler removes the span, because the span is a child of the request. The log record and the canopy.frontend.errors metric stay. If you add this URL to OTEL_PYTHON_DJANGO_EXCLUDED_URLS, Canopy records no span and no metric, but keeps the log record.

Sampling¶

Canopy records every request by default. A large installation can send too much data to the collector. Set a ratio sampler to record only a part of the traffic:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

This records 10% of the traces. Each trace stays complete: a Celery task keeps the decision that Canopy made for the request that scheduled it.

Logs¶

OTEL_LOGS_ENABLED sends each log record to the collector as well. The export is an addition to the console output, not a replacement for it. The console handler keeps writing to standard error, so journalctl and the existing log files stay as they are.

Each exported record keeps its structured fields, for example request_id, user_id and task_id. This makes a query such as “all records of request X” possible without a text search. A FrontendError record also carries exception.type, exception.message, url.full and user_agent.original. Canopy also sets the trace and the span of the record, so the backend can show the log records of a trace next to its spans.

Set LOG_STYLE=json if you collect the console output as well. The two outputs then contain the same fields.

Log correlation¶

Canopy adds the trace_id and the span_id fields to every log record that it writes inside a span. Use these fields to find the log records of a trace, or to find the trace of a log record. The fields are visible in all log formats, but LOG_STYLE=json is the easiest one to query.

Each request span also carries the canopy.request_id attribute. This is the same value as the request_id field in the log records of that request. The user.id attribute matches the user_id log field in the same way.

Warning: a collector that is not available makes the exporter retry and write a warning to the log. Canopy continues to serve requests, but the log becomes noisy. Set OTEL_ENABLED=false if the collector is down for a long time.

The user interface holds the screen in the fragment of the URL, which a browser does not send. A request span therefore cannot name the screen, and one screen makes many API calls. The interface sends two headers on each API call to close this gap:

X-Canopy-Route holds the route pattern of the screen, for example /projects/:id/findings. The interface replaces each record identifier with :id, so the number of different values stays small and no identifier goes to the collector. Canopy records the value as the canopy.route span attribute and as the route log field. Use it to group the traffic by screen, for example to find the screen that makes the slowest query.
X-Canopy-Nav-Id holds one value for each navigation. Canopy records it as the canopy.nav_id span attribute and as the nav_id log field. Use it to find every request and every log record of one page load.

Only the browser knows its screen, so both values come from the client. Canopy accepts a route of no more than 120 characters from a small character set, and a hexadecimal navigation identifier. It drops a value that does not pass this check, and records the request without it.

Do not use canopy.nav_id as a label on a metric. It holds one value for each navigation, so it makes the number of series grow without a limit. The canopy.route attribute is safe for this, because the set of routes is small.