Fumbling through a Lambda Deploy with Terraform
What to Automate?
What to Automate?
Last weekend I wrote a simple Lambda function called eventwatch that monitors AWS account activity by maintaining a key-value store in S3 that logs (and can alert) when a new combinations of activity are detected. Up until this weekend I’d only done fairly trivial Terraform resources for EC2 instances and never attempted to use for Lambda functions or CloudWatch Events.
While it was fairly easy to get this built and tested through the AWS Console in a single region, because AWS Events and CloudWatch Logs are regional services, it requires the Lambda function to be deployed (and the events to be created) in each region you want to monitor. So point and click wasn’t an option.
To accomplish this the following resources had to be defined:
CloudWatch
- Event Rules (cloudwatch_event_rule)
- Event Targets (cloudwatch_event_target)
IAM
- Execution Role ( aws_iam_role )
- Policies to be attached to Role ( aws_iam_policy, aws_iam_role, iam_policy_attachment, aws_iam_policy_document )
Lambda
- Function (aws_lambda_function)
- Lambda Permissions (aws_lambda_ permission)
(CAVEATS: This is also just a minimum “get it working” POC and the multi-region deployment works, but definitely needs improvement. This is “manager-grade” code at best and my Terraform style is pretty poor, but it got the job done and I learned a lot. Hopefully others can get something out of this, as I didn’t find any examples of using Terraform to configure Lambda functions that got triggered from CloudWatch events. Lastly, the IAM is way too permissive for anything close to production use.)
CloudWatch Configuration
CloudWatch Rules define the events that will be sent to the Lambda functions for processing. The example below will send events of EC2 resource changes such as instance state changes or changes other resources like security groups but there is a lot more than can be done here and included other event types in this POC.resource "aws_cloudwatch_event_rule" "ec2_events" {
name = "ec2-events"
event_pattern = <<PATTERN
{
"source": [
"aws.ec2"
]
}
PATTERN
}
The corresponding event target would be defined as below and links the event rule with the lambda function.resource "aws_cloudwatch_event_target" "ec2_target" {
rule = "${aws_cloudwatch_event_rule.ec2_events.name}"
arn = "${aws_lambda_function.eventwatch_lambda.arn}"
}
This will result in the following showing up in the UI on the following screens. So you will be part way there after this point.
If you click on the rule, you will then see the target.
IAM Configuration
The IAM configuration was relatively straightforward and simply creates the execution role that defines the permissions available to the Lambda function by defining the role and attaching policies. In this case, the following snippet gives eventwatch full all permissions to S3. Obviously this should be locked down to only the necessary buckets and operations required by the application, but this is just a POC. You would have to attach multiple policies and define multiple aws_iam_role_policy_attachment resources. My inexperience with terraform made me think there had to be a better way than repeating the resources declarations (perhaps using a list?) but this seemed to be the only way I could get it working.resource "aws_iam_role" "eventwatch_exec_role" {
name = "eventwatch_exec_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}data "aws_iam_policy_document" "eventwatch_s3_full_doc" {
statement {
actions = [
"s3:*",
]
resources = [
"arn:aws:s3:::*",
]
}
}resource "aws_iam_policy" "eventwatch_s3_full" {
name = "eventwatch_s3_full"
path = "/"
policy = "${data.aws_iam_policy_document.eventwatch_s3_full_doc.json}"
}resource "aws_iam_role_policy_attachment" "eventwatch_s3_policy_attach" {
role = "${aws_iam_role.eventwatch_exec_role.name}"
policy_arn = "${aws_iam_policy.eventwatch_s3_full.arn}"
}
Lambda Configuration
The basic lambda deployment (uploading the code and configuring the function) was surprisingly easy. It was first thing I got working a few days ago, but getting the Lambda to trigger based on the events proved to be the trickiest and required a bunch of trial and error.
Using aws_lambda_permission was the most difficult for me and the key problem I ran into was not providing a unique statement id. Each target must have a corresponding permission and this wasn’t immediately obvious from the documentation.resource "aws_lambda_function" "eventwatch_lambda" {
function_name = "eventwatch"
handler = "lambda_function.lambda_handler"
runtime = "python2.7"
filename = "../eventwatch.zip"
source_code_hash = "${base64sha256(file("eventwatch.zip"))}"
role = "${aws_iam_role.eventwatch_exec_role.arn}"
timeout = 15
}resource "aws_lambda_permission" "allow_cloudwatch_ec2_events" {
statement_id = "AllowExecutionFromCloudWatch2"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.eventwatch_lambda.function_name}"
principal = "events.amazonaws.com"
source_arn = "${aws_cloudwatch_event_rule.ec2_events.arn}"
}
