Skip to main content
Skip to main content

tracing

Tracing records the life cycle of a request execution in the system, including the request and its sub-procedure call links, execution time and statistics, which can be used for slow query location, performance bottleneck analysis, etc.

caution

Please note that Tracing is only supported in Doris 1.2 and is no longer avialable since Doris 2.0.

Principle​

doris is responsible for collecting traces and exporting them to a third-party tracing analysis system, which is responsible for the presentation and storage of traces.

Quick Start​

doris currently supports exporting traces directly to zipkin.

Deploy zipkin​

curl -sSL https://zipkin.io/quickstart.sh | bash -s
java -jar zipkin.jar

Configuring and starting Doris​

Add configuration to fe.conf​

enable_tracing = true

# Configure traces to export to zipkin
trace_export_url = http://127.0.0.1:9411/api/v2/spans

Add configuration to be.conf​

enable_tracing = true

# Configure traces to export to zipkin
trace_export_url = http://127.0.0.1:9411/api/v2/spans

# Queue size for caching spans. span export will be triggered once when the number of spans reaches half of the queue capacity. spans arriving in the queue will be discarded when the queue is full.
max_span_queue_size=2048

# The maximum number of spans to export in a single pass.
max_span_export_batch_size=512

# Maximum interval for exporting span
export_span_schedule_delay_millis=500

Start fe and be​

sh fe/bin/start_fe.sh --daemon
sh be/bin/start_be.sh --daemon

Executing a query​

...

View zipkin UI​

The browser opens http://127.0.0.1:9411/zipkin/ to view the query tracing.

Using opentelemetry collector​

Use the opentelemetry collector to export traces to other systems such as zipkin, jaeger, skywalking, or to database systems and files. For more details, refer to collector exporter.

Meanwhile, opentelemetry collector provides a rich set of operators to process traces. For example, filterprocessor , tailsamplingprocessor. For more details, refer to collector processor.

traces export path: doris->collector->zipkin etc.

Deploy opentelemetry collector​

opentelemetry has released collector core and contrib, contrib provides richer features, here is an example of contrib version.

Download collector​

Download otelcol-contrib, available on the official website more precompiled versions for more platforms

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.55.0/otelcol-contrib_0.55.0_linux_amd64.tar.gz

tar -zxvf otelcol-contrib_0.55.0_linux_amd64.tar.gz

Generate configuration file​

The collector configuration file is divided into 5 parts: receivers, processors, exporters, extensions, and service. Among them, receivers, processors and exporters define the way to receive, process and export data respectively; extensions are optional and are used to extend tasks that do not involve processing telemetry data; service specifies which components are used in the collector. See collector configuration.

The following configuration file uses the otlp (OpenTelemetry Protocol) protocol to receive traces data, perform batch processing and filter out traces longer than 50ms, and finally export them to zipkin and file.

cat > otel-collector-config.yaml << EOF
receivers:
otlp:
protocols:
http:

exporters:
zipkin:
endpoint: "http://10.81.85.90:8791/api/v2/spans"
file:
path: ./filename.json

processors:
batch:
tail_sampling:
policies:
{
name: duration_policy,
type: latency,
latency: {threshold_ms: 50}
}

extensions:

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, tail_sampling]
exporters: [zipkin, file]
EOF

Start collector​

nohup ./otelcol-contrib --config=otel-collector-config.yaml &

Configuring and starting Doris​

Add configuration to fe.conf​

enable_tracing = true

# enable opentelemetry collector
trace_exporter = collector

# Configure traces export to collector, 4318 is the default port for collector otlp http
trace_export_url = http://127.0.0.1:4318/v1/traces

Add configuration to be.conf​

enable_tracing = true

# enable opentelemetry collector
trace_exporter = collector

# Configure traces export to collector, 4318 is the default port for collector otlp http
trace_export_url = http://127.0.0.1:4318/v1/traces

# Queue size for caching spans. span export will be triggered once when the number of spans reaches half of the queue capacity. spans arriving in the queue will be discarded when the queue is full.
max_span_queue_size=2048

# The maximum number of spans to export in a single pass.
max_span_export_batch_size=512

# Maximum interval for exporting span
export_span_schedule_delay_millis=500

Start fe and be​

sh fe/bin/start_fe.sh --daemon
sh be/bin/start_be.sh --daemon

Executing a query​

...

View zipkin UI​

...