A completely unnecessary holiday weekend Bash wrapper for CloudFormation

Matt Franz

08 Jul 2024 — 7 min read

(Five years ago I started the tradition of learning and blogging over the 4th of July weekend, so I figured I'd keep the tradition alive)

Problem & Prior Art

Even with autocompletion, the AWS CLI can be extremely verbose, so I often find myself writing short wrapper scripts for basic CRUD operations to save keystrokes. When you building infrastructure (or learning new IAC constructs) you are creating resources and destroying them, even if you are only deploying from your laptop.

This usually requires writing filters or lots of jq to get the output you want in the format you want, so characters start to add up. Furthermore, I've gotten spoiled by tools like eksctl or CDK scripts provide much better output that deploying CloudFormation templates via the AWS CLI, so I started looking for a wrapper.

There had to be one. I did at least 30 minutes Googling or searching GitHub. I found a few dead shell script on GitHub and iidy but it was a bit heavyweight, especially with the args file, which I'd have to create for each template. So after playing with that for 20 minutes, I decided to write my own.

I still find LLMs unable to create correct infrastructure code (IAM, Terraform, CDK, etc.) they are useful in getting started with boilerplate code. I did use both Gemini and ChatGPT as an assistant, but as usual blogs and documentation provided better examples.

Iterating to Desired Functionality

In addition to saving keystrokes (especially the terribly annoying file:///template.yaml syntax) I wanted to see the resources that were created, just as if I were looking at the AWS Console as well as the stacks in the region. While I didn't need fancy color output, I did need some basic output that looked reasonable and JQ CSV ended up being good enough. Besides CRUD operations, I wanted to see the events after the operation. Although my examples were quite basic it might be nice to see the resources created by a longer template. Lastly, I needed to pass in a service role. So nothing too advanced, but I learned a lot along the way.

Creating a Simple S3 Bucket with Credentials

Because I needed to be able to create IAM Users I needed IAM perms, so decided to use CFN_EXEC_ARN=arn:aws:iam::account:role/CfnExec as an environment variable to specify the service execution role.

The LLM-generated code generated CASE statements (something I don't use a llot for some reason) and usage so I kept them. I was able to have ChatGPT with python-llm (by piping the code to llm -s "add feature x" ) make minor modifications as why went along, since I did most of the development in vi.

% cfr 
Usage: cfr <create|update|delete> <STACK_NAME> <TEMPLATE_FILE> [CAPABILITIES] [ADDITIONAL_ARGS]
or 
       cfr list 
       cfr show <TEMPLATE_FILE>

And I have two basic templates that create S3 Bucket and a User with the permissions to read/write to that bucket.

% ls -al
total 16
drwxr-xr-x  4 mdfranz  staff   128 Jul  7 17:53 .
drwxr-xr-x  3 mdfranz  staff    96 Jul  7 15:53 ..
-rw-r--r--  1 mdfranz  staff  1288 Jul  7 17:53 bucket-ssm.yaml
-rw-r--r--  1 mdfranz  staff  1053 Jul  7 08:09 bucket.yaml

My first example ( bucket.yml which I found on a blog somewhere) outputs the secret key in the Stack itself, which is not great, and not lik what you expect if you are Terraform user, so the -ssm.yaml instead puts it in an SSM Parameter.

The simple stack defines 3 resources and makes use of pseudo-parameters (basically, AWS's name for system variables that you can add to the Names of resources with the !Sub syntax. I've always found string interpolation in CloudFormation a bit nasty, but once I got used it, it was not terrible.

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::StackName}-${AWS::AccountId}
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256

  S3User:
    Type: AWS::IAM::User
    Properties:
      Policies:
        - PolicyName: bucket-access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - s3:*
              Resource:
                - !Sub arn:aws:s3:::${S3Bucket}
                - !Sub arn:aws:s3:::${S3Bucket}/*

  S3UserAccessKey:
    Type: AWS::IAM::AccessKey
    Properties:
      UserName: !Ref S3User

  BasicParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub '${S3Bucket}-${S3UserAccessKey}'
      Value: !GetAtt S3UserAccessKey.SecretAccessKey
      Type: String

NOTE: Single quotes are not required around string interpolation.

The SHOW parameter just lists the resources using yq. See below for more details on how to use yq, which I'd heard of but never really used much.

 % cfr show bucket-ssm.yaml
Showing resources...
- S3Bucket
- S3User
- S3UserAccessKey
- BasicParameter

Let's do a sample run

% cfr create noquote bucket-ssm.yaml CAPABILITY_IAM
{
    "StackId": "arn:aws:cloudformation:ca-central-1:ACCOUNT:stack/noquote/cf3fd3a0-3d1c-11ef-af9a-0e00c3512f6f"
}
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
Loop 1 - You can break at any time, trust me!

"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
Loop 2 - You can break at any time, trust me!

"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
Loop 3 - You can break at any time, trust me!

"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
mdfranz@franz-m1-2020 s3 % cfr list
"2024-07-08T11:25:41.792000+00:00","noquote","CREATE_IN_PROGRESS"
"2024-07-05T16:30:15.555000+00:00","CDKToolkit","CREATE_COMPLETE"
mdfranz@franz-m1-2020 s3 % cfr list
"2024-07-08T11:25:41.792000+00:00","noquote","CREATE_COMPLETE"
"2024-07-05T16:30:15.555000+00:00","CDKToolkit","CREATE_COMPLETE"

Deletion works the same way except it shows all the resources being created (something I need to investigate on the create)

mdfranz@franz-m1-2020 s3 % cfr delete noquote
"2024-07-08T11:34:18.696000+00:00","AWS::CloudFormation::Stack","DELETE_IN_PROGRESS"
"2024-07-08T11:26:38.610000+00:00","AWS::CloudFormation::Stack","CREATE_COMPLETE"
"2024-07-08T11:26:37.822000+00:00","AWS::SSM::Parameter","CREATE_COMPLETE"
"2024-07-08T11:26:37.554000+00:00","AWS::SSM::Parameter","CREATE_IN_PROGRESS"
"2024-07-08T11:26:36.725000+00:00","AWS::SSM::Parameter","CREATE_IN_PROGRESS"
"2024-07-08T11:26:36.363000+00:00","AWS::IAM::AccessKey","CREATE_COMPLETE"
"2024-07-08T11:26:36.091000+00:00","AWS::IAM::AccessKey","CREATE_IN_PROGRESS"
"2024-07-08T11:26:35.736000+00:00","AWS::IAM::AccessKey","CREATE_IN_PROGRESS"
"2024-07-08T11:26:35.395000+00:00","AWS::IAM::User","CREATE_COMPLETE"
"2024-07-08T11:25:58.531000+00:00","AWS::IAM::User","CREATE_IN_PROGRESS"
"2024-07-08T11:25:57.705000+00:00","AWS::IAM::User","CREATE_IN_PROGRESS"
"2024-07-08T11:25:57.336000+00:00","AWS::S3::Bucket","CREATE_COMPLETE"
"2024-07-08T11:25:44.892000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:43.598000+00:00","AWS::S3::Bucket","CREATE_IN_PROGRESS"
"2024-07-08T11:25:41.792000+00:00","AWS::CloudFormation::Stack","CREATE_IN_PROGRESS"
"2024-07-08T11:34:21.984000+00:00","AWS::IAM::User","DELETE_IN_PROGRESS"
"2024-07-08T11:34:21.682000+00:00","AWS::IAM::AccessKey","DELETE_COMPLETE"
"2024-07-08T11:34:21.284000+00:00","AWS::IAM::AccessKey","DELETE_IN_PROGRESS"
"2024-07-08T11:34:20.952000+00:00","AWS::SSM::Parameter","DELETE_COMPLETE"
"2024-07-08T11:34:20.136000+00:00","AWS::SSM::Parameter","DELETE_IN_PROGRESS"
"2024-07-08T11:34:18.696000+00:00","AWS::CloudFormation::Stack","DELETE_IN_PROGRESS"
"2024-07-08T11:26:38.610000+00:00","AWS::CloudFormation::Stack","CREATE_COMPLETE"
"2024-07-08T11:26:37.822000+00:00","AWS::SSM::Parameter","CREATE_COMPLETE"
"2024-07-08T11:26:37.554000+00:00","AWS::SSM::Parameter","CREATE_IN_PROGRESS"
"2024-07-08T11:26:36.725000+00:00","AWS::SSM::Parameter","CREATE_IN_PROGRESS"
"2024-07-08T11:26:36.363000+00:00","AWS::IAM::AccessKey","CREATE_COMPLETE"
"2024-07-08T11:26:36.091000+00:00","AWS::IAM::AccessKey","CREATE_IN_PROGRESS"
"2024-07-08T11:26:35.736000+00:00","AWS::IAM::AccessKey","CREATE_IN_PROGRESS"

And just as before I can list until the stack is destroyed.

 % cfr list          
"2024-07-08T11:25:41.792000+00:00","noquote","DELETE_IN_PROGRESS"
"2024-07-05T16:30:15.555000+00:00","CDKToolkit","CREATE_COMPLETE"
mdfranz@franz-m1-2020 s3 % cfr list
"2024-07-05T16:30:15.555000+00:00","CDKToolkit","CREATE_COMPLETE"
mdfranz@franz-m1-2020 s3 %