Recent Posts


Recent Comments


Archives


Categories


Meta



It’s been awhile since Lambda was released, yet it is still increasing in popularity, and it is even being used by Amazon to support AWS services themselves. Right after the GA release in April 2015, I started to implement a few microservices to help manage AWS accounts using the Lambda.

In the development process, you still need a server to implement codes and test them. However, once completed, Lambda saves a great deal of work in deploying and managing codes, because you don’t need additional servers for staging and/or production environments. Lambda also makes it easier for you to implement microservices because it is not possible to implement complex modules by nature.

For example, one of the microservice projects implemented as open source of Sungard Availability Services | Labs, called ‘awsconfig,’ provides convenient ways to manage AWSConfig services. Using these microservices, you can turn the AWSConfig service on/off as necessary in any region–and even in any AWS account–without logging into the AWS console for different accounts. In this blog, I’ll cover how the microservices work and how to deploy them with CloudFormation. To demonstrate the features step by step, we’ll create three different CloudFormation stacks, which are ‘Basic’, ‘Custom’ and ‘Federated.’

Before explaining about the service in detail, let’s start with creating a stack to deploy the microservices:

First ‘Basic’ Stack

In the ‘CloudFormation’ service page after logging in to the AWS console:

In the  ‘Specify Details’ page:

  • Enter ‘AWSConfigBasic’ for ‘Stack name’
  • Enter ‘basic’ for ‘Environment’ and click ‘Next’

In the ‘Options’ page,

  • Just click ‘Next’

In the ‘Review’ page:

  • Click to approve ‘I acknowledge that this template might cause AWS CloudFormation to create IAM resources. and click ‘Create’

After being redirected to the main page, please wait for the stack to complete and, once completed, click the ‘Outputs’ tab of the stack, which lists information about created Lambda functions, as seen below:

basic.cloudformation.outputs

We see there are three Lambda functions created, along with a role assigned to the functions. The function names will be slightly different in the trailing alpha-numerics because they are randomly created.

AWSConfigCheckerFunction

This is a function to check the state of the AWSConfig service, whether the service is currently turned on or off. As you see below, it checks if recorders and delivery channels are setup and running and returns ‘true’ only when all of them are running. Otherwise, it returns ‘false.’ You can see the completed code here.

var flows = [
{func:aws_sts.assumeRoles, success:aws_config.findRecorders},
{func:aws_config.findRecorders, success:aws_config.findRecordersStatus},
{func:aws_config.findRecordersStatus, success:aws_config.findChannels},
{func:aws_config.findChannels, success:aws_config.findChannelsStatus},
{func:aws_config.findChannelsStatus, success:succeeded}
];

You may wonder why ‘assumeRoles’ is called in the first step in ‘flows’, but let’s put that aside and get back to it later.

AWSConfigEnablerFunction

This is a function to enable the AWSConfig service by creating necessary resources, such as recorders and delivery channels, like S3 bucket and SNS topics, along with a necessary role. You can see the completed code here

var flows = [
  {func:aws_sts.assumeRoles, success:aws_role.findRoleByPrefix},
  {func:aws_role.findRoleByPrefix, success:aws_role.findInlinePolicy, failure:aws_role.createRole},
  {func:aws_role.createRole, success:aws_role.findInlinePolicy},
  {func:aws_role.findInlinePolicy, success:aws_bucket.findBucket, failure:aws_role.createInlinePolicy},
  {func:aws_role.createInlinePolicy, success:aws_role.wait},
  {func:aws_role.wait, success:aws_bucket.findBucket},
  {func:aws_bucket.findBucket, success:aws_topic.findTopic, failure:aws_bucket.createBucket},
  {func:aws_bucket.createBucket, success:aws_topic.findTopic},
  {func:aws_topic.findTopic, success:aws_config.findRecorders, failure:aws_topic.createTopic},
  {func:aws_topic.createTopic, success:aws_config.findRecorders},
  {func:aws_config.findRecorders, success:aws_config.setRoleInRecorder, failure:aws_config.setRoleInRecorder},
  {func:aws_config.setRoleInRecorder, success:aws_config.findChannels},
  {func:aws_config.findChannels, success:aws_config.findRecordersStatus},
  {func:aws_config.setChannel, success:aws_config.findRecordersStatus},
  {func:aws_config.findRecordersStatus, success:succeeded, failure:aws_config.startRecorder},
  {func:aws_config.startRecorder, success:succeeded}
];

AWSConfigRemoverFunction

This is a function that disables the currently running AWSConfig service by stopping the records and removing the delivery channels along with their role. You can see the completed code here.

var flows = [
  {func:aws_sts.assumeRoles, success:aws_config.findRecorders},
  {func:aws_config.findRecorders, success:aws_config.findRecordersStatus, failure:aws_config.findChannels},
  {func:aws_config.findRecordersStatus, success:aws_config.stopRecorder, failure:aws_config.findChannels},
  {func:aws_config.stopRecorder, success:aws_config.findChannels},
  {func:aws_config.findChannels, success:aws_config.deleteChannel, failure:aws_topic.findTopic},
  {func:aws_config.deleteChannel, success:aws_topic.findTopic},
  {func:aws_topic.findTopic, success:aws_topic.deleteTopic, failure:aws_role.findRoleByPrefix},
  {func:aws_topic.deleteTopic, success:aws_role.findRoleByPrefix},
  {func:aws_role.findRoleByPrefix, success:aws_role.findInlinePolicy,failure:succeeded},
  {func:aws_role.findInlinePolicy, success:aws_role.deleteInlinePolicy, failure:aws_role.findRole},
  {func:aws_role.deleteInlinePolicy, success:aws_role.deleteRole},
  {func:aws_role.deleteRole, success:succeeded}
];

AWSConfigFunctionRole

This is a role that enables the Lambda functions to access necessary resources in managing the AWSConfig service.

Now, let’s check the resources specified in the CloudFormation template. One of them is AWSConfigFunctionChecker to create a Lambda function to check the status, and the other is AWSConfigRole to create a necessary role to run the functions. As you see in AWSConfigFunctionChecker, its resource type is AWS::Lambda::Function and it is given a role, AWSConfigRole.

"AWSConfigFunctionChecker": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Code": {
      "S3Bucket": {"Fn::Join": [".", ["sgas.particles-awsconfig.blog",...] ] },
      "S3Key": "particles/assets/awsconfig.zip"
    },
    "Handler": "awsconfig/index_checker.handler",
    "Runtime": "nodejs",
    "Timeout": "60",
    "MemorySize": "128",
    "Role": {"Fn::GetAtt": ["AWSConfigRole", "Arn"]}
  }
}
"AWSConfigRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [ {
            "Sid": "",
            "Effect": "Allow",
            "Principal": { "Service": "lambda.amazonaws.com"},
            "Action": "sts:AssumeRole"
          } ]
        },
        "Path": "/",
        "Policies": [ {
          "PolicyName": "InlinePolicy",
          "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
               ....
            ]
          }
        } ]
      }
    }

Didn’t we create Lambda functions? Yes, we did. Then, why don’t we actually run them to see if they work as intended? Let’s go to ‘Lambda’ service page in the AWS console, and you’ll see the new functions, as below:

basic.lambda.list

To run them, please choose the checker function, ‘AWSConfigBasic-AWSConfigFunctionChecker-XXXXX’ and click ‘Test’ button. Because this is the first test, you’ll see a popup window of ‘Input test event’ where you can enter the input data. Please enter the json below and click ‘Save and test.’

{
  "account": "<aws-account-id-you-currently-logged-in>",
  "region": "<any-region>"
}

test.input

If you get ‘true’, it means that this account has the AWSConfig service currently running. Otherwise, the service is off or not set up.

execution.result

Please test two other functions, enabler & remover, and see if the service is correctly configured in the AWSConfig service page.

Here, we found that the function names are auto-generated and the AWS::Lambda::Function does not have the name in its ‘Properties.’ When the functions are integrated with other AWS services like API Gateway, randomly generated names will be okay. But, if you want to call them directly from outside applications, you may want to have the names you assign. So, let’s see how we can give the names we want in the next stack.

 

Second “Custom” Stack

Because the current CloudFormation does not support naming, we’ll use this custom resource feature. More details here.

Like the first stack, let’s again create a stack using the below url:

https://s3.amazonaws.com/sgas.particles-awsconfig.blog.custom.us-east-1/particles/cftemplates/template.json

In the ‘Specify Details’ page:

  • Enter ‘AWSConfigCustom’ for ‘Stack name’
  • Enter ‘custom’ for ‘Environment’

Once completed, click the ‘Outputs’ tab to see the output of the newly created stack. As you see in the “Value” Column, the function names are what we specified, instead of random names:

custom.cloudformation.outputs

We used a different template for this stack, so let’s see what the differences are. First, we added a new custom resource, CustomLambdaDeployerFunction, that is a new Lambda function to deploy our three main functions to manage the AWSConfig service. We’re using this custom resource function to deploy the main functions with names you want to assign.

"CustomLambdaDeployerFunction": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Code": {
      "S3Bucket": {"Fn::Join": [".", ["sgas.particles-awsconfig.blog",...},
      "S3Key": "particles/assets/cloudformation_builder.zip"
    },
    "Handler": "cloudformation/index_lambda_deployer.handler",
    "Runtime": "nodejs",
    "Timeout": "60",
    "Role": {"Fn::GetAtt": ["CustomLambdaDeployerRole", "Arn"]}
  }
}

And AWSConfigFunctionCheckerhas these changes.

  • ‘Type’ is changed to Custom::AWSConfigFunction from AWS::Lambda::Function
  • 3 new properties are added
    • ServiceToken : This is the id of custom resource that will be used to deploy
    • Region : AWS region where the function will be deployed
    • FunctionName : Function name we want to assign
"AWSConfigFunctionChecker": {
  "Type": "Custom::AWSConfigFunction",
  "Properties": {
    "ServiceToken": {"Fn::GetAtt": ["CustomLambdaDeployerFunction","Arn"]},
    "Region": {"Ref": "AWS::Region"},
    "FunctionName": "awsconfig-checker",
    "Code": {
      "S3Bucket": {"Fn::Join": [".", ["sgas.particles-awsconfig.blog",...] ] },
      "S3Key": "particles/assets/awsconfig.zip"
    },
    "Handler": "awsconfig/index_checker.handler",
    "Runtime": "nodejs",
    "Timeout": "60",
    "MemorySize": "128",
    "Role": {"Fn::GetAtt": ["AWSConfigRole", "Arn"]}
  }
}

Please go to the Lambda service page and run the newly created functions using the same input format.

custom.lambda.list

I hope everything went as expected for you. If so, let’s think about how to run this Lambda functions in different accounts without deploying the same functions in the accounts. If you want to manage multiple AWS accounts, you need to deploy the Lambda functions in all of the accounts, but there is a convenient way to do this. So let’s create another stack that covers this requirement.

 

Last Stack, “Federated”

The AWS Security Token Service (STS) that provides temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users also supports federating roles through assuming the roles, which will be used in this stack. More information related to STS can be found here.

Basically, we’ll create a federate role (FederationMainRole) that will be assumed by the role of the Lambda functions (AWSConfigRole) in the same account and will create account roles (FederationAccountRole) both in the account where the federate role is created and in a different account whose AWSConfig service is managed remotely. The account roles (FederationAccountRole) will be assumed by the federate role (FederationMainRole), so that the AWSConfig Lambda functions can access the resources in the remote account.

We may let AWSConfigRole directly assume FederationAccountRole in different accounts without going through FederationMainRole, but if we add more microservices with different roles in more accounts, FederationMainRole will make it easy to manage all the federations.

As mentioned above, we’ll create two stacks in two different accounts. The first stack will create the AWSConfig Lambda functions, along with FederationMainRole and FederationAccountRole with trust relationships in the main account. The second stack will create only FederationAccountRole with trust relationships in the remote account.

Main account

Create a stack using the below url

https://s3.amazonaws.com/sgas.particles-awsconfig.blog.federated.us-east-1/particles/cftemplates/template.json

And in the ‘Specify Details’ page,

  • Enter ‘AWSConfigFederated’ for ‘Stack name’
  • Enter ‘federated’ for ‘Environment’
  • Leave blank for ‘FederationMainRoleArn’

Once completed, click the ‘Outputs’ tab to see the output of the newly created stack. We can see two different roles created, (FederationMainRole and FederationAccountRole),  along with the Lambda functions.

federated.cloudformation.outputs.1

Now, let’s check if the newly created roles have a??correct trust relationship. Please go to the IAM service page and click ‘Roles’ in the left menu. First, find ‘AWSConfigFederated-FederationMainRole-XXXX’ and click it. In the ‘Summary’ page, please click the ‘Trust Relationships’ tab. AWSConfigRole should be in the ‘Trusted Entities’ as below:

federation.main.role

To check another role, go back to the IAM role list page and choose: ‘AWSConfigFederated-FederationAccountRole-XXXX’ and click it. In the ‘Summary’ page, please click the ‘Trust Relationships’ tab.  ‘AWSConfigFederated-FederationMainRole-XXXX’  should be in the ‘Trusted Entities’ as below:

federation.account.role.1

Remote account

Using another browser session or a different browser, log into the AWS console of a remote account and create a stack using the same url, with the one in main account in the CloudFormation service page:

https://s3.amazonaws.com/sgas.particles-awsconfig.blog.federated.us-east-1/particles/cftemplates/template.json

In ‘Specify Details’ page:

  • Enter ‘AWSConfigFederated’ for ‘Stack name’
  • Enter ‘federated’ for ‘Environment’
  • Enter value of ‘FederationMainRoleArn’ in outputs of the previous stack in the main account for ‘FederationMainRoleArn’

federation.main.role.arn

Once completed, click ‘Outputs’ tab to see the output of the newly created stack. We can see only FederationAccountRole is created and the value FederationMainRole is the same as the value of the stack in the main account.

federated.cloudformation.outputs.2

Now, let’s check if the newly created roles have correct trust relationship. Please go to the IAM service page and choose ‘AWSConfigFederated-FederationAccountRole-XXXX.’ In the ‘Summary’ page, please click the ‘Trust Relationships’ tab.  ‘AWSConfigFederated-FederationMainRole-XXXX’ of the main account should be in the ‘Trusted Entities,’ as below:

federation.account.role.2

Before actually running the Lambda functions to check if they work as intended, let’s quickly review the new and changed resources in the CloudFormation template.

The Lambda functions and the federate main role should only be created in the main account.  So, we’re using a condition called, IsFederationAccount that will have a value of ‘true’ only when the input parameter of FederationMainRoleArn is not given.

"Conditions": {
  "IsFederationAccount": {"Fn::Equals": ["Ref": "FederationMainRoleArn"},""]}
}

There are only two changes in AWSConfigFunctionChecker compared to the one in the stack of ‘Custom’

  • A condition, IsFederationAccount, is added because it should be created only in the main account
  • The function name is changed to awsconfig-checker-with-federation
"AWSConfigFunctionChecker": {
  "Type": "Custom::AWSConfigFunction",
  "Condition": "IsFederationAccount",
  "Properties": {
    "ServiceToken": {"Fn::GetAtt": ["CustomLambdaDeployerFunction","Arn"]},
    "Region": {"Ref": "AWS::Region"},
    "FunctionName": "awsconfig-checker-with-federation",
    "Code": {
      "S3Bucket": {"Fn::Join": [".", ["sgas.particles-awsconfig.blog",...] ] },
      "S3Key": "particles/assets/awsconfig.zip"
    },
    "Handler": "awsconfig/index_checker.handler",
    "Runtime": "nodejs",
    "Timeout": "60",
    "MemorySize": "128",
    "Role": {"Fn::GetAtt": ["AWSConfigRole", "Arn"]}
  }
}

We need another custom resource, CustomLambdaFederationFunction, which is a Lambda function, to add necessary trust relationships in roles.

"CustomLambdaFederationFunction": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Code": {
      "S3Bucket": {"Fn::Join": [".", ["sgas.particles-awsconfig.blog",...] } },
      "S3Key": "particles/assets/cloudformation_builder.zip"
    },
    "Handler": "cloudformation/index_iam_federation.handler",
    "Runtime": "nodejs",
    "Timeout": "60",
    "Role": {"Fn::GetAtt": ["CustomLambdaDeployerRole", "Arn"]}
  }
}

There are two resources that federate roles using the above custom resource, CustomLambdaFederationFunction.

  • RoleFederationAWSConfigToMain adds a trust relationship with AWSConfigRole in FederationMainRole.
  • RoleFederationMainToAccount adds a trust relationship with FederationMainRole in FederationAccountRole.
"RoleFederationAWSConfigToMain": {
  "Type": "Custom::AWSConfigRoleFederation",
  "Condition": "IsFederationAccount",
  "Properties": {
    "ServiceToken": {"Fn::GetAtt":["CustomLambdaFederationFunction","Arn"]},
    "AWSAccountId": {"Ref": "AWS::AccountId"},
    "Region": {"Ref": "AWS::Region"},
    "RoleArn": {"Fn::GetAtt": ["AWSConfigRole", "Arn"]},
    "FederationRoleName": {"Ref": "FederationMainRole"}
  }
}
"RoleFederationMainToAccount": {
  "Type": "Custom::AWSConfigRoleFederation",
  "Properties": {
    "ServiceToken": {"Fn::GetAtt":["CustomLambdaFederationFunction","Arn"]},
    "AWSAccountId": {"Ref": "AWS::AccountId"},
    "Region": {"Ref": "AWS::Region"},
    "RoleArn": {"Fn::If": [
        "IsFederationAccount",
        {"Fn::GetAtt": ["FederationMainRole","Arn"]},
        {"Ref": "FederationMainRoleArn"}
      ]
    },
    "FederationRoleName": {"Ref": "FederationAccountRole"}
  }
}

Now it’s time to test the functions. Go to the Lambda service page and we can see there are three new functions to manage AWSConfig service and two custom Lambda functions to deploy Lambda functions and federate roles.

federated.lambda.list

Choose awsconfig-checker-with-federation and click ‘Test’ button. Again, this is a first test, so you’ll see a popup window of ‘Input test event.’ This time, we need more parameters because we want to manage the service in remote account. Please enter the below json as input data and click ‘Save and test.’

{
  "federateAccount": "<main-account-id>",
  "federateRoleName": "<FederationMainRoleArn-from-main-stack-outputs
-without-arn:aws:iam::<account_id>:role/>",
  "account": "<remote-account-id>",
  "roleName": "<FederationAccountRoleArn-from-remote-stack-outputs
-without-arn:aws:iam::<account_id>:role/>",
  "region": "<any-region>"
}

test.input.federated

Like the previous tests, you should get ‘true,’ when the remote account has the AWSConfig service currently running. Otherwise, it will be false.

execution.result

Now, let’s address what I asked you to put aside while explaining the execution flows of the Lambda functions, which is assumeRoles as the first step. This is to assume necessary roles for the Lambda functions to access resources in the remote accounts.

 

Wrapping it Up

In this blog, we went over the microservices to manage the AWSConfig service, along with deployments using 3 different CloudFormation templates. There are a few more microservices implemented in the open source projects of Sungard Availability Services | Labs as below, so please check the projects if you’re interested.

  1. CloudTrail: This service provides a convenient way to manage a CloudTrail service like the AWSConfig service we covered in this blog. 
  2. Billing Alert: This service provides a notification when estimated billing amount is more than expected. You can get the notifications by just using the current CloudWatch metrics, but it can only tell whether the current estimation exceeds some threshold. This service sends more realistic notifications by comparing the increased rates of the estimations.
  3. Alarm Alert: This service provides a more efficient way to get alerted when highly prioritized notification emails from AWS arrive. Instead of checking the emails manually, you can get the notifications in real time by calling this service. 

Alex is a CTO Architect in the CTO Architecture Team, which is responsible for researching and evaluating innovative technologies and processes. He has been working at Sungard Availability Services for more than 14 years.
Before joining this team, he worked on developing public/private cloud orchestration platform and integrating various applications to automate the processes in managed hosting service companies including Sungard Availability Services. Prior to the managed hosting companies, he worked at Samsung for 9 years in developing a reporting tool and RDBMS engine.


Comments

2
Comments are closed.