Recent Posts


Recent Comments


Archives


Categories


Meta



“Alerting” is one of the essential systems in managing applications and infrastructures. But as new applications are added to your environment, it is difficult to unify the alert processes because each has its own method. I’d like to introduce an integrated alert system that provides the main backbone where new applications’ alert processes can be added with a few simple steps.

Let’s create one before diving into the details. Please click the button below to create a stack using a CloudFormation:

Launch Stack

  • In the ‘Select Template’ page, click ‘Next’.
  • In the ‘Specify Details’ page, leave the default values, fill ‘SlackChannel’ and ‘SlackWebHookUrl’, and click ‘Next’.
    • SlackChannel: channel name starting with ‘#’
    • SlackWebHookUrl:  Incoming WebHook URL without ‘https://’

(see “Incoming WebHooks’ page in Slack App Portal of your team to find how to get Slack WebHook URL of target channel)

  • In the ‘Options’ page, click ‘Next’.
  • In the ‘Review’ page, click all checkboxes in ‘Capabilities’, click ‘Create Change Set’ button to transform SAM (Serverless Application Model) to a normal template, and click ‘Execute’.

Once the stack is successfully created, choose the newly created stack, ‘sungardas-aws-services-alerts’ and click the ‘Outputs’ tab below. Write down the value of “KinesisStreamArn,” which will be used when we create a stack in a different region.

Now let’s test to see if the newly created alert system works as expected. To make the test simple, the stack created a test Lambda function. Please go to the ‘Lambda’ console and click “SungardAS-Alerts-Sample-Alert-us-east-1-xxxx” function. In the function page, choose ‘Configure test events’ in a dropdown box (upper right of page) with ‘Select a test event..’ on the top left. In ‘Configure test event’ window, leave ‘Event template’, set ‘Event name’ with any value and enter input JSON value as ‘{}’ like below.

After clicking ‘Create’, click ‘Test’ button to generate alert data for a test.

Once the test is successfully completed, you should receive an alert message in the assigned Slack channel as below. If a new item is added in ‘sungardas-aws-services-alerts-AlertTable-xxxx’ DynamoDB table with the same information (‘Account Id’ and ‘Region’), you should receive a Slack alert message.

Now let’s see the detailed architecture. This is an architecture diagram of the newly created alert system:

To send data to the Alert, each Alert Trigger just needs to store that in its designated AWS CloudWatch Log. Then the data will be automatically fed into Kinesis Stream through the linked CloudWatch Log Destination. Finally, the Kinesis Stream invokes Lambda functions that store the alert data in DynamoDB table, and send it to a Slack channel.

Under this infrastructure, you can add a new Alert Trigger by just making it store data to alert in a CloudWatch Log that is linked to the CloudWatch Log Destination in the same region. And in the other side, if you want to add other communication platforms like Twillo/Microsoft Teams, you can just add a new Lambda function that sends what it receives from the Kinesis Stream to the target platform. Don’t forget to add the Kinesis Stream as a trigger of the newly created Lambda function.

As you see in the architecture diagram, the Alert Triggers can be in different regions. By setting up another CloudWatch Log Destination in the same region with a Trigger, this system can let the Trigger save the data to alert in a CloudWatch Log in the same region, which will be eventually fed into the Kinesis Stream.

To set up a new Destination in a different region (‘us-east-2’ here), please click this button and be sure you’re in a different region (us-east-2) in CloudFormation console:

Launch Stack

Follow the same steps with the previous stack, except filling the value of ‘KinesisStreamArn’ with ‘KinesisStreamArn’ value you captured after the previous stack was successfully created. Once this stack is successfully created, go to the Lambda console and run the test function, ‘SungardAS-Alerts-Sample-Alert-us-east-2-xxxx’ like you did in the previous test. You should receive another Slack message like below when the test successfully completes. Check the same DynamoDB table if a new item is created. This time, the account is the same but the region should be ‘us-east-2’.

What if the Alert Triggers are in different accounts? In this case, it is not necessary to set up another CloudWatch Log Destination. Just link the CloudWatch Log of the target Alert Trigger to a CloudWatch Log Destination in the same region.

Before creating a new stack in a different account, we need to add a permission for the new account to access the Destination in the main account. Go to the Lambda console of the main account and ‘us-east-1’ region, find a function called ‘SungardAS-Alerts- Permission-xxxx’ and configure the test event as below:

  • region: The region where your new Alert Trigger in a different account. Here, enter ‘us-east-2’ to test a case with both different account & region.
  • account : The account where your new Alert Trigger exists
  • destination: “alertDestination”

Once the test event is configured, run this function by clicking the ‘Test’ button and confirm that it completes with success.

Now, we need to get the ARN of the target Destination in the same region, which will be entered during creating a stack in a different account. Go to the Cloudformation console of ‘us-east-2’ region in the main account, because we will send alert data from ‘us-east-2’ region of a new account. Choose ‘sungardas-aws-services-alerts-destination’ stack and write down the value of ‘CloudWatchDestinationArn’ in ‘Output’. Once again, be sure you’re in the ‘us-east-2’ region.

After adding the permission and making a note of ‘CloudWatchDestinationArn’ value, we can now create a new stack in a different account. Copy the link of the button below and use it to create a new stack in ‘us-east-2’ region of another account (you may use a different session in the same browser or use a different browser to login to the different AWS account):

Launch Stack

Please follow the same steps with the previous stacks, except filling the value of ‘CloudWatchDestinationArn’ with the ‘CloudWatchDestinationArn’ value you captured. Once this stack is successfully created, go to the Lambda console and run the test function, ‘SungardAS-Alerts-Sample-Alert-us-east-2-xxxx’ like you did in the previous tests. You should receive another Slack message like the one below when the test completes with success. Check the same DynamoDB table if a new item is created. This time, the account should be another account you choose and the region should be ‘us-east-2’.

So far, we’ve built an integrated alert system that enables to manage all alert messages from various applications in a unified way. We also learned how easily it can be extended with simple steps, even when the applications are in different regions and/or accounts.

Check out our open source project site, https://github.com/SungardAS/aws-services-alerts/tree/blog to get the all resources like SAM templates and Lambda function codes.

Alex is a CTO Architect in the CTO Architecture Team, which is responsible for researching and evaluating innovative technologies and processes. He has been working at Sungard Availability Services for more than 14 years.
Before joining this team, he worked on developing public/private cloud orchestration platform and integrating various applications to automate the processes in managed hosting service companies including Sungard Availability Services. Prior to the managed hosting companies, he worked at Samsung for 9 years in developing a reporting tool and RDBMS engine.


Comments are closed.