7 Steps to instrument Splunk Observability Cloud to monitor AWS SAM Lambda Node.js microservice

Capture metrics, traces, and logs from any source, and view them all in one place

CHOO Jek Bao

6 min readOct 17, 2021

Updated: 20 Oct 2021

Step 1

Spin up a Cloud9 instance

Step 2

Enable integration between AWS CloudWatch Log and Splunk Observability Cloud Log Observer by going to Data Setup (in Splunk Observability Cloud) > Amazon Web Services > Add Connection > Authentication Type: select External ID

Remember to select the right region where your lambda function is.

After you deploy your log collector you should see it.

Ensure a function named splunk-aws-logs-collector is created in your AWS Lambda function in the region you are monitoring. This is important.

Step 3

Start Node.js 12.x SAM project in Cloud9 terminal

sam init

Step 4

Add environment variables by going to template.yaml > Resources > Properties > Environment > Variables

Variables:
  SIGNALFX_ACCESS_TOKEN: <your-access-token>
  SIGNALFX_SEND_TIMEOUT: 30000
  SIGNALFX_SERVICE_NAME: <your-microservice-name>
  SIGNALFX_SPAN_TAGS: deployment.environment:<your-name>-sandbox
  SIGNALFX_METRICS_URL: https://ingest.<your-realm>.signalfx.com
  SIGNALFX_ENDPOINT_URL: https://ingest.<your-realm>.signalfx.com/v2/trace

Step 5

Add lambda layer by going to template.yaml > Resources > Properties > Layers

Layers:
        - arn:aws:lambda:ap-southeast-1:254067382080:layer:signalfx-lambda-nodejs-wrapper:23

Get the latest lambda layer for Node.js 12.x version and ensure it is the same as the Lambda region being created.

Step 6

Add SignalFx code for logging (correlate trace and span id to log lines for CloudWatch to Log Observer log correlation) and tracing.

const signalFxLambda = require('signalfx-lambda'); // required for .asyncWrapper or .wrapper
const tracing = require("signalfx-lambda/tracing"); // required to use logs, set our own tags, or invoke other cloud functionslet traceId;
let spanId;exports.lambdaHandler = signalFxLambda.asyncWrapper(async (event, context) => {
    
    const version = '1'; // versioning makes iterative dev and debugging easier
    
    const fullEnvName = process.env ? process.env['SIGNALFX_SPAN_TAGS'] : 'unknown-environment-yet'; // get the full environment name
    const envName = fullEnvName.replace('deployment.environment:', ''); // get the environment name
    const svcName = process.env ? process.env['SIGNALFX_SERVICE_NAME'] : 'unknown-service-yet'; // get the service name
    
    try {
        const contextCarrier = {}; // any writeable object will do
        tracing.inject(contextCarrier); // propagate context to inject x-b3 info
        
        if (contextCarrier) {
            traceId = contextCarrier["x-b3-traceid"]; // get trace id
            spanId = contextCarrier["x-b3-spanid"]; // get span id
            correlateApmToLo(traceId, spanId, envName, svcName, `logging test - linking APM to Log Observer ${version}`); // test logging info
        }
        
        const tracer = tracing.tracer(); // To add any other data into Spans as Tags
        
        if (tracer) {
            const currentSpan = tracer.scope().active(); // get the current span
            currentSpan.setTag('jek_custom_tag.some_tag_key.message', 'some_tag_value') // set custom tag
            // this will appear in APM > Traces > Waterfall > Tags (key value pair)
        }
        
        signalFxLambda.helper.sendGauge('jek-custom-metrics-16oct2021.weeksOn', 6); // send custom metrics
        // this will appear in Dashboard > New Chart > Add Metric or Event field *note: numerical value
        
        // if (event && event['aws.ec2']) {
        //.    // send custom event of cloudwatch event
        //     const dimensions = {source: event['source'], account: event['account']};
        //     const properties = {functionName: context['function-name'], detailType: event['detail-type']};
        //     signalFxLambda.helper.sendCustomEvent('jek-custom-event-17oct2021', dimensions, properties); // this will appear in Dashboard > New Chart > Add Metric or Event field
            
        //     //We can also send CloudWatchEvent using .sendCloudWatchEvent(event)
        //     // forward CloudWatch Event (now called EventBridge) to Splunk O11y Cloud
        //     // Reminder: Lambda function needs to be triggered by EventBridge (former CloudWatchEvent) for .sendCloudWatchEvent(event) to work
        //     signalFxLambda.helper.sendCloudWatchEvent(event); // this will appear in Dashboard > New Chart > Add Metric or Event field   
        // }
        
        const response = {
            'statusCode': 200,
            // 'headers': {
            //     'Access-Control-Allow-Headers' : '*',
            //     'Access-Control-Allow-Origin': '*',
            //     'Access-Control-Allow-Methods': 'OPTIONS,POST,GET',
            //     'Access-Control-Max-Age': 2592000, // 30 days
            //     'Timing-Allow-Origin': '*', 
            //     'Server-Timing': getServerTraceContext(traceId, spanId),
            //     'Access-Control-Expose-Headers': 'Server-Timing'
            // },
            'body': JSON.stringify({
                'message': `jek nodejs12 16 oct 2021 - version ${version}`,
            })
        }
        
        console.log('jek_response', response)
        return response;;
        
    } catch (err) {
        correlateApmToLo(traceId, spanId, envName, svcName, `logging error - linking APM to Log Observer ${version}`);
        return err;
    }
});// connect APM and LO with trace id in this format
// The format is found here https://docs.splunk.com/Observability/get-started/relatedcontent.html#splunk-log-observer
const correlateApmToLo = (traceId, spanId, envName, svcName, message) => {
    const logObj = {
        'trace_id': traceId || '',
        'span_id': spanId || '',
        'deployment': {
            'environment': envName
        },
        'service': {
            'name': svcName
        },
        'message': message || '',
        'severity': 'INFO' || 'ERROR'
    };
    console.log(JSON.stringify(logObj)); // must have JSON.stringify before logging
}

I chose .asyncWrapper over .wrapper because when creating a hello world lambda function using SAM, it uses async (event, context) => {…} by default.

Feel free to explore .wrapper as well.

Step 7

Build and deploy SAM project

sam build && sam deploy --guided

Node.js SignalFx Lambda wrapper and code working — Proof of these steps working #1

Custom tag working in APM — Proof of these steps working #3

From APM to Log Observer with trace id correlation — Proof of these steps working #4

CloudWatch Event — Proof of these steps working #5

Remember to clean up your lambda function and integration when no longer needed. Clean up by reversing the steps above. Refer to the README.md in the sam project for guidance.

FAQ #1

Q: I have used my code to send custom event, where do I find my custom event in Splunk Observability Cloud?

A: From Home > Dashboard > (Your Name / Email Address) > New Chart > Browser > Search.

Enter the name of your custom event > Search

Acknowledgement

This is the main repo that contains the SignalFx Lambda Node.js wrapper. https://github.com/signalfx/lambda-nodejs
Note that Splunk OTel Lambda will be supporting Node.js soon https://github.com/signalfx/splunk-otel-lambda so SignalFx Lambda Node.js wrapper may eventually be deprecated.
Further example on how to use Splunk OTel Lambda can be found here https://github.com/signalfx/lambda-layer-versions/tree/master/splunk-apm
This example code is helpful. Also if you are interested in exploring more than Hello World of AWS Lambda Node.js with Splunk Observability Cloud, then you must check out that example code. https://github.com/kdroukman/splunk_poc_lambda
This other example code is helpful too. https://github.com/bxtp4p/splunk-lambda-examples
This doc talks about the purpose of custom event. It is to add context to metrics. For example, you might send a custom “code push” event each time your development team deploys new code, so that you can correlate it with the resource consumption profiles of your infrastructure before and after the event. https://docs.splunk.com/Observability/alerts-detectors-notifications/view-data-events.html
This API doc shows the API. Essentially, SignalFx Lambda Nodejs wrapper follows this API doc. https://dev.splunk.com/observability/reference/api/ingest_data/latest#endpoint-send-events
This doc talks about how Application Performance Monitoring (APM) connects with Log Observer (LO) using trace_id. https://docs.splunk.com/Observability/get-started/relatedcontent.html
This doc talks about SignalFx CloudWatch Event Forward with code example. https://github.com/signalfx/cloudwatch-event-forwarder

Social Media Policy

My name is Jek. I am a Sales Engineer specialising in Splunk Observability Cloud. I wrote this because I enjoy sharing my knowledge.

The postings on this site are my own and do not represent the position or opinions of Splunk Inc., or its affiliates.