How does Splunk OpenTelemetry Collector (Agent) work as Kubernetes Daemonset?

How It Works

One of the purposes of OpenTelemetry Collector (Agent) is to receive spans from application.

Source: https://github.com/signalfx/splunk-otel-collector/blob/main/docs/architecture.md
  1. Push: Pods in the DaemonSet are configured to send updates to another service, such as a stats database. They do not have clients.
  2. NodeIP and Known Port: Pods in the DaemonSet can use a hostPort, so that the pods are reachable via the node IPs. Clients know the list of node IPs somehow, and know the port by convention.
  3. DNS: Create a headless service with the same pod selector, and then discover DaemonSets using the endpoints resource or retrieve multiple A records from DNS.
  4. Service: Create a service with the same Pod selector, and use the service to reach a daemon on a random node. (No way to reach a specific node.)

How To Configure

Pattern #2 — add below config to the pod or deployment yaml:

Source: https://github.com/jek-bao-choo/splunk-otel-example/blob/main/apm-cs/k8s-dotnetcore2dot2-auto-instr/dotnetcore2dot2.yaml#L22-L26

FAQ

Q: What configuration allowed deployment pods to reach a daemonset pod using the Node IP address?

Disclaimer

My name is Jek. I am a Sales Engineer specialising in Splunk Observability Cloud. I wrote this to document my learning.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
CHOO Jek Bao

CHOO Jek Bao

55 Followers

Love writing my thoughts, reading biographies, and meeting like-minded friends to talk on B2B software sales, engineering & cloud solution architecture.