FAQ: How to use Splunk OTel Collector Chart?

Q: How to set the memory & CPU usage?

There are two approaches when installing via Helm chart.

A: #1 Imperative — the values.yaml

helm repo update

Update the repository before pulling.

helm pull --untar splunk-otel-collector-chart/splunk-otel-collector

Download (not install) the chart.

cd splunk-otel-collector

Change directory into the downloaded chart. You should see a values.yaml file.

vim values.yaml

Edit the values.yaml by looking for e.g. gateway.resources.limits.cpu to change the CPU limit.

helm install --generate-name splunk-otel-collector-chart/splunk-otel-collector --values values.yaml

Install the chart using the values.yaml

A: #2 Declarative — the set command

helm repo update

Update the repository before installing.

... --set gateway.resources.limits.cpu=2 --set gateway.resources.limits.memory=4Gi ...

The above snippet shows how to set resources for gateway mode.

helm install --set cloudProvider='aws' --set distribution='eks/fargate' --set splunkObservability.accessToken='<redacted>' --set clusterName='jek-eks-fargate-cluster' --set splunkObservability.realm='<redacted>' --set gateway.enabled='true' --set gateway.resources.limits.cpu=2 --set gateway.resources.limits.memory=4Gi --generate-name splunk-otel-collector-chart/splunk-otel-collector

The above example is the command with redacted token and realm. Furthermore, it is important to maintain a ratio of 1 cpu : 2 Gi.

Q: How to scale up or down the Splunk OTel Collector deployment?

It is the same as scaling up or down any deployment.

A: #1 Imperative — the helm values.yaml

helm repo update

Update the repository before pulling.

helm pull --untar splunk-otel-collector-chart/splunk-otel-collector

Download (not install) the chart.

cd splunk-otel-collector

Change directory into the downloaded chart. You should see a values.yaml file.

vim values.yaml

Edit the values.yaml by looking for e.g. gateway.resources.limits.cpu to change the CPU limit.

Change replicaCount to your preferred value

helm install --generate-name splunk-otel-collector-chart/splunk-otel-collector --values values.yaml

Install the chart using the values.yaml

A: #2 Declarative — the helm set command

helm repo update

Update the repository before installing.

... --set gateway.replicaCount=2 ...

The above snippet shows how to set resources for gateway mode.

helm install --set cloudProvider='aws' --set distribution='eks/fargate' --set splunkObservability.accessToken='<redacted>' --set clusterName='jek-eks-fargate-cluster' --set splunkObservability.realm='<redacted>' --set gateway.enabled='true' --set gateway.replicaCount=2 --generate-name splunk-otel-collector-chart/splunk-otel-collector

The above example is the command with redacted token and realm.

A: #3 Declarative — the kubectl edit command

kubectl edit deployment <change to the deployment name>

Q: Does helm upgrade restart deployment and statefulset?

Yes.

helm upgrade [release name] splunk-otel-collector-chart/splunk-otel-collector --values values.yaml

You will see that new pods spin up and old pods are terminating. Helm upgrade will manage the restart; hence, this is automatic. This also means that changes to CPU and Memory limits are reflected in the new pods.

Q: What is the alternatives to using helm upgrade?

You can manually edit the deployment and configmap. However, you will need to restart the deployment yourself. The recommendation is to use helm install or helm upgrade. It will also be a problem if you want to upgrade your helm chart later down the road. So the recommendation is to use helm install and helm upgrade with values.yaml file.

Q: How to add OpenTelemetry’s Processor Filter?

A: Imperative — the values.yaml

The same as above.

filter/exclude_metrics:
  metrics:
    exclude:
      match_type: regexp
      resource_attributes:
        - Key: k8s.namespace.name
          Value: kube-system

Setup filtering rules for the cluster receiver to exclude metrics.

pipelines:
  metrics:
  exporters:
  - signalfx
  - logging
  processors:
  - memory_limiter
  - batch
  - resource
  - resource/k8s_cluster
  - filter/exclude_metrics
  receivers:
  - k8s_cluster

Add to pipeline.

Remember to export a copy of the configmap before upgrade and after upgrade. So you can compare the differences.

Q: Does setting the values.yaml override the existing config?

Yes and no.

A: Yes for (it does override):

service:
  pipelines:
    metrics:
      processors:
      - filter/exclude_metrics

Since the above config will over write the existing config, the existing processors need to be include in the values.yaml’s config. Read this guide.

A: No for (it does NOT override):

processors:
  filter/exclude_metrics:
    metrics:
      exclude:
        match_type: regexp
        resource_attributes:
          - Key: k8s.namespace.name
            Value: "dev"

Hence, it is very important to compare the configmap before and after upgrade. So before upgrade, take a snapshot of the configmap.

Q: Can we mix helm set and values flag?

Yes.

helm install --generate-name splunk-otel-collector-chart/splunk-otel-collector --set gateway.replicaCount=2 --values values.yaml

The above set flag would override the value flag of the replicaCount i.e. whatever replicaCount in values.yaml does NOT matter. The replicaCount will be 2.

Q: Can we reuse values when using helm?

helm upgrade \
YOURCOLLECTORHERE \
--values k3slogs.yaml \
splunk-otel-collector-chart/splunk-otel-collector \
--reuse-values

helm upgrade \
splunk-otel-collector-1661477799 \
--values otel-prometheus.yaml \
splunk-otel-collector-chart/splunk-otel-collector \
--reuse-values

The --reuse-values flag ensures you keep whatever you already set, but if you prefer to specify it again you should specify it all in the values file and command-line arguments.

Disclaimer

My name is Jek. At the time of writing, I am a Sales Engineer specialising in Splunk Observability Cloud. I wrote this to document my learning.

The postings on this site are my own and do not represent the position or opinions of Splunk Inc., or its affiliates.