There Was an Error Loading Log Streams Please Try Again by Refreshing This Page Lamda Logs
Centralized AWS Lambda Logs with Kinesis and Serverless
The central to gaining serverless observability is sending all AWS Lambda Logs to a fundamental location where you can later group, filter and make sense of them. Sematext is a full-stack observability solution for your entire software stack. Meaning you can implement centralized logging for AWS Lambda Logs alongside any existing infrastructure, like Kubernetes clusters and containers.
Hither's a scenario. Your APIs are declining and you take absolutely no inkling why. Don't you just hate that!? Now imagine you don't have admission to the VM, cluster or container where your software is running. Want me to continue with this nightmare?
Yes, that's what debugging AWS Lambda functions tends to seem like. A horrid nightmare of non knowing what is happening nor why things are failing. This article will show you a style of logging function invocations, and storing your AWS Lambda Logs in a central location. Letting you rail and monitor failures and errors, while besides giving you a nice structure for logging info and debug logs for when you need to troubleshoot behavior.
Prepare? Let's go started!
Using CloudWatch for AWS Lambda Logs
CloudWatch is the default solution for showing AWS Lambda Logs.
CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing you lot with a unified view of AWS resources, applications and services that run on AWS, and on-premises servers.
— AWS Documentation
In layman'south terms, it's an AWS service for showing your logs across all AWS services. We're interested in knowing how information technology handles AWS Lambda Logs. When an AWS Lambda function executes, any yous write out to the console, a fmt.printf() in Go or panel.log() in Node.js, will be sent to CloudWatch asynchronously in the background. You can notice these AWS Lambda Logs in the AWS CloudWatch panel. Lucky for us, information technology won't add whatsoever overhead to the AWS Lambda function execution time.
Using logging agents in the AWS Lambda function runtime will add overhead to the execution and add together unnecessary latency. We want to avert that, and process the logs afterwards they get added to CloudWatch. Below you can see sample log events that get generated from a generic Hello World function. Accessing AWS CloudWatch logs is as piece of cake as scrolling through the AWS Panel.
Let's take a step back and look at the bigger picture. Every AWS Lambda function will create something called a Log Group in CloudWatch. Click on a particular Log Group.
These log groups will contain Log Streams that are literally equivalent of log events coming from particular function instances.
This is inappreciably a proficient enough solution for system insight and having a proper overview of what your software is doing. Because of its structure, it's incredibly hard to come across and distinguish AWS Lambda Logs. Using centralized logging makes more sense. You tin can use your own Elasticsearch or a hosted setup. Sematext gives you full-stack observability for every office of your infrastructure and exposes an Elasticsearch API. Let me show y'all how easy it is to create CloudWatch log processing of your AWS Lambda Logs and pipe them to a Sematext Logs App.
Creating Centralized Logging with AWS Lambda and Kinesis
By using CloudWatch log group subscriptions and Kinesis you can funnel all of your AWS Lambda Logs to a defended AWS Lambda function that volition send them to Sematext's Elasticsearch API. There y'all have a fundamental location for all your AWS Lambda Logs. You can search and filter logs for all functions and with lilliputian try accept insight into the behavior and health of your AWS Lambda functions.
I'll demohow to build a one-command-deploy solution you tin can use for yourself. It'due south congenital with the Serverless Framework and Node.js. Just, you tin feel free to use AWS SAM or Terraform, and any programming linguistic communication you want. The concept volition stay the same.
Here's what information technology will wait like in the end.
Much prettier than CloudWatch, and you can actually find what you're looking for!
Setting Upwardly The Serverless Project
Kickoff of all install the Serverless Framework, configure your IAM user, and create a new projection. Full guide can exist plant hither.
$ npm install -g serverless $ sls config credentials --provider aws --central xxxxxxxxxxxxxx --hush-hush xxxxxxxxxxxxxx $ sls create --template aws-nodejs --path lambda-cwlogs-to-logsene $ cd lambda-cwlogs-to-logsene $ npm init -y $ npm i logsene-js zlib serverless-iam-roles-per-function
Sweetness! now move on to the serverless.yml.
Configuring AWS Lambda and Kinesis Resource
Open up the lambda-cwlogs-to-logsene directory in a code editor and check out the serverless.yml. Feel free to delete everything and paste this in.
# serverless.yml service: lambda-cwlogs-to-logsene plugins: - serverless-iam-roles-per-function custom: stage: ${opt:stage, self:provider.stage} secrets: ${file(secrets.json)} provider: name: aws runtime: nodejs8.10 stage: dev region: ${cocky:custom.secrets.REGION, 'us-e-1'} versionFunctions: fake functions: shipper: handler: shipper.handler description: Sends CloudWatch logs from Kinesis to Sematext Elastic Search API memorySize: 128 timeout: 3 events: - stream: type: kinesis arn: Fn::GetAtt: - LogsKinesisStream - Arn batchSize: ${self:custom.secrets.BATCH_SIZE} startingPosition: LATEST enabled: truthful environs: LOGS_TOKEN: ${self:custom.secrets.LOGS_TOKEN} LOGS_BULK_SIZE: 100 LOG_INTERVAL: 2000 subscriber: handler: subscriber.handler description: Subscribe all CloudWatch log groups to Kinesis memorySize: 128 timeout: 30 events: - http: path: subscribe method: go - cloudwatchEvent: consequence: source: - aws.logs detail-type: - AWS API Call via CloudTrail detail: eventSource: - logs.amazonaws.com eventName: - CreateLogGroup - schedule: rate: rate(60 minutes) iamRoleStatements: - Effect: "Let" Action: - "iam:PassRole" - "sts:AssumeRole" - "logs:PutSubscriptionFilter" - "logs:DeleteSubscriptionFilter" - "logs:DescribeSubscriptionFilters" - "logs:DescribeLogGroups" - "logs:PutRetentionPolicy" Resource: "*" environment: filterName: ${self:custom.stage}-${cocky:provider.region} region: ${cocky:provider.region} shipperFunctionName: "shipper" subscriberFunctionName: "subscriber" prefix: "/aws/lambda" retentionDays: ${self:custom.secrets.LOG_GROUP_RETENTION_IN_DAYS} kinesisArn: Fn::GetAtt: - LogsKinesisStream - Arn roleArn: Fn::GetAtt: - CloudWatchLogsRole - Arn resources: Resources: LogsKinesisStream: Type: AWS::Kinesis::Stream Properties: Proper name: ${cocky:service}-${self:custom.phase}-logs ShardCount: ${self:custom.secrets.KINESIS_SHARD_COUNT} RetentionPeriodHours: ${self:custom.secrets.KINESIS_RETENTION_IN_HOURS} CloudWatchLogsRole: Type: AWS::IAM::Part Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Event: Let Principal: Service: - logs.amazonaws.com Activeness: - sts:AssumeRole Policies: - PolicyName: root PolicyDocument: Version: "2012-10-17" Argument: - Effect: Permit Activity: - kinesis:PutRecords - kinesis:PutRecord Resource: Fn::GetAtt: - LogsKinesisStream - Arn RoleName: ${self:service}-${self:custom.stage}-cloudwatchrole Let'southward suspension it down piece by piece. The shipper AWS Lambda function will be triggered by a Kinesis stream, and it has some environment variables for configuring Sematext Logs. The Kinesis stream itself is defined at the lesser, in the resources section, and referenced in the AWS Lambda function events by using its ARN.
Moving on to the subscriber role. It tin can be triggered in three ways. It'south upwardly to you lot to choose. If you take a lot of existing Log Groups, you lot may want to hit the HTTP endpoint to initially subscribe them all. Otherwise, having it trigger every once in a while, or only when a new Log Group is created, would be fine.
The LogsKinesisStream is the Kinesis stream to where we're subscribing Log Groups, and CloudWatchLogsRole is the IAM Role which volition let CloudWatch to put records into Kinesis.
Configuring Sematext Logs
With that out of the way, you tin can now meet we're missing a secrets.json file. But, before we continue, jump over to Sematext, log in and create a Logs App. Printing the tiny green push to add together a Logs App.
After calculation the name of the App and some bones info, yous'll see a waiting for data screen pop upwardly. Press the integrations guide and copy your token.
Now you can paste the token in the secrets.json file.
{ "LOGS_TOKEN": "your-token", "REGION": "us-due east-1", "BATCH_SIZE": 1000, "LOG_GROUP_RETENTION_IN_DAYS": 1, "KINESIS_RETENTION_IN_HOURS": 24, "KINESIS_SHARD_COUNT": 1 } Adding the Subscriber AWS Lambda Function
I similar saying Kinesis is a simpler version of Kafka. It'due south basically a pipage. Yous subscribe data to exist sent into it and tell it to trigger an AWS Lambda function as an event, once it satisfies a certain batch size.
The purpose of having a subscriber function is to subscribe all the AWS Lambda Log Groups to a Kinesis stream. Ideally they should be subscribed upon cosmos, and of course, initially when you lot desire to subscribe all existing Log Groups to a new Kinesis stream. As a fallback, I also like to have an HTTP endpoint for when I want to manually trigger the subscriber.
In your code editor, create a new file and name information technology subscriber.js. Paste this snippet in.
// subscriber.js const AWS = require('aws-sdk') AWS.config.region = process.env.region const cloudWatchLogs = new AWS.CloudWatchLogs() const prefix = procedure.env.prefix const kinesisArn = process.env.kinesisArn const roleArn = process.env.roleArn const filterName = process.env.filterName const retentionDays = process.env.retentionDays const shipperFunctionName = procedure.env.shipperFunctionName const filterPattern = '' const setRetentionPolicy = async (logGroupName) => { const params = { logGroupName: logGroupName, retentionInDays: retentionDays } await cloudWatchLogs.putRetentionPolicy(params).promise() } const listLogGroups = async (acc, nextToken) => { const req = { limit: 50, logGroupNamePrefix: prefix, nextToken: nextToken } const res = await cloudWatchLogs.describeLogGroups(req).promise() const newAcc = acc.concat(res.logGroups.map(logGroup => logGroup.logGroupName)) if (res.nextToken) { return listLogGroups(newAcc, res.nextToken) } else { return newAcc } } const upsertSubscriptionFilter = async (options) => { console.log('UPSERTING...') const { subscriptionFilters } = await cloudWatchLogs.describeSubscriptionFilters({ logGroupName: options.logGroupName }).promise() const { filterName, filterPattern } = subscriptionFilters[0] if (filterName !== options.filterName || filterPattern !== options.filterPattern) { await cloudWatchLogs.deleteSubscriptionFilter({ filterName: filterName, logGroupName: options.logGroupName }).promise() look cloudWatchLogs.putSubscriptionFilter(options).hope() } } const subscribe = async (logGroupName) => { const options = { destinationArn: kinesisArn, logGroupName: logGroupName, filterName: filterName, filterPattern: filterPattern, roleArn: roleArn, distribution: 'ByLogStream' } try { await cloudWatchLogs.putSubscriptionFilter(options).promise() } catch (err) { console.log(`FAILED TO SUBSCRIBE [${logGroupName}]`) console.error(JSON.stringify(err)) expect upsertSubscriptionFilter(options) } } const subscribeAll = async (logGroups) => { await Promise.all( logGroups.map(async logGroupName => { if (logGroupName.endsWith(shipperFunctionName)) { console.log(`SKIPPING [${logGroupName}] Considering IT WILL CREATE Cyclic EVENTS FROM IT'S Own LOGS`) return } console.log(`SUBSCRIBING [${logGroupName}]`) look subscribe(logGroupName) console.log(`UPDATING Memory POLICY TO [${retentionDays} DAYS] FOR [${logGroupName}]`) await setRetentionPolicy(logGroupName) }) ) } const processAll = async () => { const logGroups = expect listLogGroups([]) await subscribeAll(logGroups) } exports.handler = async () => { console.log('subscriber start') await processAll() console.log('subscriber done') return { statusCode: 200, trunk: JSON.stringify({ message: `Subscription successful!` }) } } Bank check out the processAll() function. Information technology'll grab all Log Groups from CloudWatch which match the prefix, and put them in an easily accessible assortment. Yous'll and so laissez passer them to a subscribeAll() function, which volition map through them while subscribing them to the Kinesis stream you defined in the serverless.yml.
Some other cool thing is setting the retentiveness policy to 7 days. Y'all'll rarely need more than that and it'll cut the cost of keeping your Lambda logs in your AWS account.
Continue in listen you lot can too edit the filterPattern past which AWS Lambda Logs volition get ingested. For now, I've chosen to keep it blank and not filter out annihilation. But, based on your needs yous tin match it with what kind of pattern your logger of choice creates.
Sweet, with that done, let's move on to shipping some logs!
Adding the Shipper AWS Lambda Part
Later the Kinesis stream receives Lambda logs from CloudWatch, information technology'll trigger an AWS Lambda function defended to sending the logs to an Elasticsearch endpoint. For this example, we'll use LogseneJS as the log shipper. It'south rather simple if you break it downwardly. A batch of records will be sent in the event parameter to the shipper function. You lot parse the AWS Lambda Logs, giving them your desired structure, and ship them to Sematext. Here'south what it looks like. Create a new file, name it shipper.js and paste this lawmaking in.
// shipper.js const Zlib = require('zlib') const Logsene = require('logsene-js') const logger = new Logsene(process.env.LOGS_TOKEN) const errorPatterns = [ 'error' ] const configurationErrorPatterns = [ 'module initialization fault', 'unable to import module' ] const timeoutErrorPatterns = [ 'task timed out', 'process exited before completing' ] /** * Sample of a structured log * *************************************************************************** * Timestamp RequestId Message * 2019-03-08T15:58:45.736Z 53499d7f-60f1-476a-adc8-1e6c6125a67c Howdy Globe! * *************************************************************************** */ const structuredLogPattern = '[0-9]{four}-(0[i-nine]|1[0-2])-(0[1-9]|[1-ii][0-9]|3[0-one])T(2[0-3]|[01][0-ix]):[0-v][0-9]:[0-v][0-9].[0-nine][0-9][0-9]Z([ \t])[a-zA-Z0-ix]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-ix]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-nine]{12}([ \t])(.*)' const regexError = new RegExp(errorPatterns.join('|'), 'gi') const regexConfigurationError = new RegExp(configurationErrorPatterns.join('|'), 'gi') const regexTimeoutError = new RegExp(timeoutErrorPatterns.bring together('|'), 'gi') const regexStructuredLog = new RegExp(structuredLogPattern) const lambdaVersion = (logStream) => logStream.substring(logStream.indexOf('[') + one, logStream.indexOf(']')) const lambdaName = (logGroup) => logGroup.split up('/').opposite()[0] const checkLogError = (log) => { if (log.bulletin.match(regexError)) { log.severity = 'error' log.fault = { type: 'runtime' } } else if (log.message.match(regexConfigurationError)) { log.severity = 'error' log.error = { type: 'configuration' } } else if (log.message.match(regexTimeoutError)) { log.severity = 'error' log.error = { type: 'timeout' } } render log } const splitStructuredLog = (message) => { const parts = message.separate('\t', three) return { timestamp: parts[0], requestId: parts[1], msg: parts[2] } } /** * Create payload for Logsene API */ const parseLog = (functionName, functionVersion, message, awsRegion) => { if ( message.startsWith('Commencement RequestId') || bulletin.startsWith('End RequestId') || message.startsWith('Study RequestId') ) { return } // if log is structured if (message.match(regexStructuredLog)) { const { timestamp, requestId, msg } = splitStructuredLog(message) return checkLogError({ bulletin: msg, function: functionName, version: functionVersion, region: awsRegion, type: 'lambda', severity: 'debug', timestamp: timestamp, requestId: requestId }) } else { // when log is NOT structured return checkLogError({ message: message, office: functionName, version: functionVersion, region: awsRegion, type: 'lambda', severity: 'debug' }) } } const parseLogs = (event) => { const logs = [] result.Records.forEach(record => { const payload = Buffer.from(tape.kinesis.data, 'base64') const json = (Zlib.gunzipSync(payload)).toString('utf8') const data = JSON.parse(json) if (data.messageType === 'CONTROL_MESSAGE') { return } const functionName = lambdaName(information.logGroup) const functionVersion = lambdaVersion(data.logStream) const awsRegion = tape.awsRegion data.logEvents.forEach(logEvent => { const log = parseLog(functionName, functionVersion, logEvent.message, awsRegion) if (!log) { return } logs.push(log) }) }) render logs } const shipLogs = async (logs) => { return new Promise((resolve) => { if (!logs.length) { return resolve('No logs to ship.') } logs.forEach(log => logger.log(log.severity, 'LogseneJS', log)) logger.transport(() => resolve('Logs shipped successfully!')) }) } exports.handler = async (event) => { try { const res = await shipLogs(parseLogs(outcome)) console.log(res) } take hold of (err) { console.log(err) return err } return 'shipper done' } The centre of the shipper Lambda lies in the parseLogs() and shipLogs() functions. The erstwhile will take the event parameter, extract all log events, parse them, add them to an assortment, and return that assortment. While the latter will take that same logs array, add every single log event to the LogseneJS buffer, and send them all in one go. The location is the Logs App yous created above.
Do you remember the image from the start of the article where you lot saw log events of a typical function invocation? There you can see information technology generates 4 different types of log events.
Beginning RequestId ... Terminate RequestId Study RequestId
They can offset with any of these iii patterns, where the ellipsis represents any type of string that is printed to stdout in the function runtime (console.log() in Node.js).
The parseLog() function will skip the START, END, and Study log events entirely, and only return user-defined log events as either debug or error based on if they're user-defined stdout or any blazon of error in the function runtime, configuration or duration.
The log message itself can exist structured by default, but not always. By default in the Node.js runtime it has a structure that looks like this.
Timestamp RequestId Message 2019-03-08T15:58:45.736Z 53499d7f-60f1-476a-adc8-1e6c6125a67c Hello World!
The code in the shipper Lambda is configured to piece of work with the structure above or with a construction that merely has the bulletin role. If yous're using some other runtime, I'd advise you to use structured logging to have a common structure for your AWS Lambda Logs.
With the coding role done, you lot're set up to deploy and test your custom log shipper.
Deploy and Exam Your Centralized AWS Lambda Logs
The dazzler of using infrastructure every bit code tools like the Serverless Framework is how uncomplicated deployments are. You lot can push everything to the deject with one command. Leap back to your terminal and in the directory of your project run:
$ sls deploy
Y'all'll see output get printed to the console.
[output] Serverless: Packaging service... Serverless: Excluding development dependencies... Serverless: Uploading CloudFormation file to S3... Serverless: Uploading artifacts... Serverless: Uploading service .nothing file to S3 (2.15 MB)... Serverless: Validating template... Serverless: Updating Stack... Serverless: Checking Stack update progress... ............ Serverless: Stack update finished... Service Information service: lambda-cwlogs-to-logsene stage: dev region: the states-due east-ane stack: lambda-cwlogs-to-logsene-dev api keys: None endpoints: Go - https://.execute-api.us-east-1.amazonaws.com/dev/subscribe functions: shipper: lambda-cwlogs-to-logsene-dev-shipper subscriber: lambda-cwlogs-to-logsene-dev-subscriber layers: None Serverless: Removing sometime service artifacts from S3…
That's it. You now take a setup for shipping all logs from your Lambda functions into Sematext Cloud. Make certain to trigger the subscriber Lambda function to subscribe the Log Groups to the Kinesis stream. Subsequently triggering the subscriber you'll see the generated AWS Lambda Logs in Sematext, and you lot can remainder assured information technology works.
Higher up y'all can see how I added severity filtering. You lot can hands choose which value to filter by, giving you an easy mode to track errors, timeouts and debug AWS Lambda Logs.
What nearly AWS Kinesis costs?
The cost of having a setup like this in your AWS account is rather cheap. The flat cost of a single shard Kinesis stream is roughly $14/month with additional costs for the amount of data streamed. The unmarried shard has an ingest capacity of 1MB/sec or one thousand records/sec, which is fine for nigh users.
The Kinesis cost is split into shard hours and PUT payload units the size of 25KB. 1 shard costs $0.36 per solar day, while one million PUT Payload Units cost $0.014. Hypothetically, if y'all take one shard and 100 PUT payload units per second that'll end up costing you lot $10.viii for the shard and $3.6288 for the payload units during a 30 day period.
What about AWS Lambda costs?
The AWS Lambda functions are configured to use the minimum corporeality of retention possible, 128MB, meaning the costs will oft stay in the free tier during moderate use. That'southward the least of your worries.
Concluding Centralized AWS Lambda Logs with Sematext
Having a key location for your AWS Lambda Logs is crucial. Even though CloudWatch is useful in its own way, it lacks a sense of overview. By using centralized logging you don't need to switch contexts for debugging dissimilar types of applications. Sematext tin monitor your whole software stack. Having your Kubernetes logs, container logs and AWS Lambda logs in Sematext Logs where you tin easily go along runway of everything is a major benefit.
If you need to cheque out the code one time over again, here'southward the repo, give it a star if you want more people to see it on GitHub. You can also clone the repo and deploy it correct away. Don't forget to add y'all Logs App token first.
If yous need an observability solution for your software stack, cheque out Sematext. We're pushing to open source our products and brand an touch.
Hope you lot guys and girls enjoyed reading this equally much as I enjoyed writing it. If you lot liked it, slap that tiny share push button so more people will see this tutorial. Until adjacent time, be curious and have fun.
Source: https://sematext.com/blog/centralized-aws-lambda-logs-kinesis-serverless/
0 Response to "There Was an Error Loading Log Streams Please Try Again by Refreshing This Page Lamda Logs"
Post a Comment