Troubleshoot Splunk Distribution of OpenTelemetry Collector

There are three official troubleshooting guides afaik. So why am I writing another troubleshooting guide?

Motivation

Kubernetes

Traces

kubectl get pod/<application pod name> -o yaml | grep nodeName
kubectl get pods --field-selector spec.nodeName=<node name>
kubectl logs <splunk otel collector agent pod name>
# access the container in the pod
kubectl exec -it <splunk otel collector agent pod name> -- curl localhost:55679/debug/tracez | lynx -stdin
# or use port forwarding to view on desktop
kubectl port-forward pod/<splunk otel collector agent pod name> 55679:55679
# after which go to http://localhost:55679/debug/tracez
# access the container in the pod
kubectl exec -it <splunk otel collector agent pod name> -- curl localhost:55554/debug/configz/effective | yq e
# or use port forwarding to view on desktop
kubectl port-forward pod/<splunk otel collector agent pod name> 55554:55554
# after which go to http://localhost:55554/debug/configz/effective
kubectl run tmp --image=nginx:alpine
kubectl exec -it tmp -- curl -OL https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/yelp.json

kubectl get pod/tmp -o yaml | grep nodeName

kubectl get node <nodeName> -o wide

kubectl exec -it tmp -- curl -v -X POST http://<node internal ip>:9411/api/v2/spans -H'Content-Type: application/json' -d @yelp.json
kubectl delete pod/tmp
kubectl exec -it <application pod name> -- curl -OL https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/yelp.json
# Edit the service name in the json file to differ from above step tmp pod synthetic trace for differentiation.
kubectl get pod/<application name> -o yaml | grep nodeName
kubectl get node <nodeName> -o wide
kubectl exec -it <application pod name> -- curl -v -X POST http://<node internal ip>:9411/api/v2/spans -H'Content-Type: application/json' -d @yelp.json
  1. the instrumentation of the application,
  2. the application is not triggering any operation that is auto/manual instrumented, or
  3. misconfiguration of the URL endpoint from instrumented application to OpenTelemetry Collector Agent Pod.
  • (3a) How is application Pod communicating with daemonset Pod? Refer to How does Splunk OpenTelemetry Collector (Agent) work as Kubernetes Daemonset. Most importantly, test that the application Pod can reach the daemonset Pod using the communication pattern of your choice.
  • (3b) Verify that the environment variables are configured correctly and aligned accordingly in the Deployment file.

Logs

package main

import (
"context"
"time"

otelattribute "go.opentelemetry.io/otel/attribute"
otlphttpexporter "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/sdk/metric/export"

basicmetriccontroller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
basicmetricprocessor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
simplemetricselector "go.opentelemetry.io/otel/sdk/metric/selector/simple"
otelresource "go.opentelemetry.io/otel/sdk/resource"
)

func main() {
exporter, _ := otlphttpexporter.New(context.Background())
start(exporter)
}

func start(otlpExporter export.Exporter) {
processorFactory := basicmetricprocessor.NewFactory(
simplemetricselector.NewWithInexpensiveDistribution(),
otlpExporter,
)
controller := basicmetriccontroller.New(
processorFactory,
basicmetriccontroller.WithExporter(otlpExporter),
basicmetriccontroller.WithResource(otelresource.NewSchemaless(otelattribute.String("R", "V"))),
)
_ = controller.Start(context.Background())
meter := controller.Meter("my-meter")
syncInt64 := meter.SyncInt64()
counter, _ := syncInt64.Counter("my-counter")
for range time.Tick(time.Second) {
counter.Add(context.Background(), 1)
}
}
create a text file that will be used as our large log messages (fluentd is configured to send a max 2MB, also note logger , the tool we'll be using to send messages, will maxed out to 200KB, so we can create a smaller file if needed)

base64 /dev/urandom | head -c 1900000 | tr -d '\n' > file.tx

Metrics

Disclaimer

--

--

--

Love writing my thoughts, reading biographies, and meeting like-minded friends to talk on B2B software sales, engineering & cloud solution architecture.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Cloud Front in AWS

VWO A/B Testing: A Complete Guide for 2020

CS 373 Fall 2021 Blog #13: Ryan Gahagan

How FLEXable can you be?

Installing Hyperledger Composer Playground

Pandas for UK hoildays

Visualising my music catalog with matplotlib

Build A Finance “Velocity Limits” Real-time data process pipeline in Go with GCP Stacks

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
CHOO Jek Bao

CHOO Jek Bao

Love writing my thoughts, reading biographies, and meeting like-minded friends to talk on B2B software sales, engineering & cloud solution architecture.

More from Medium

Postmodern Considerations of the Intellectual: Prophet or King?

LeoLabs Announces Operational Agreement with OneWeb

Shared Security Unified Consensus Aggregation

Paradigm for Parity® CEO Spotlight Series: Julie Kampf