Accessing SSM Secure String Parameters from within CloudFormation
For more information on CloudFormation, visit aws.com
Description
CloudFormation is an AWS service that allows administrators to quickly and easily define their application architecture as code via JSON, or YAML file formats. Using CloudFormation, an administrator can easily define parameters, resources, and outputs that will allow user input values, and usage of those values when defining resources that make up an application stack. In some use case's it's desired to have a method that allows an administrator to provide credentials to CloudFormation, in order to do things such as set an RDS master password on a newly defined RDS resource defined within your template. Administrators had the ability to provide passwords through parameter definitions using mechanisms like the NoEcho: true property on a parameter. As this is not the most secure method of providing credentials, in August of 2018, the AWS CloudFormation team has released the ability to utilize Secure Systems Manager (SSM) Secure String Parameter references directly from within a CloudFormation template. This tutorial will walk thought the configuration and usage of that mechanism, allowing us to securely pass credentials from an encrypted location directly to a newly defined RDS resource.
SSM SecureString Parameter Availability
At the time of this writing, SSM SecureString Parameter types are only supported through dynamic references, and are limited to certain CloudFormation resource properties. Dynamic references are inline references. This means that you CAN NOT set up a CloudFormation Parameter and assign the value of the SSM Parameter to the CloudFormation Parameter like you currently can with standard plain text SSM Parameters. Plain text parameters can be referenced both dynamically inline, as well as be assigned to CF parameters. You must reference the SSM SecureString parameter inline with syntax such as the following: '{{resolve:ssm-secure:RDSAdminCreds:1}}', where RDSAdminCreds is the name of the parameter, and 1 is the parameter version. To read more about dynamic references click here
Pre-Requisites
1. Active AWS Account:
You will need to have an active AWS account, as this walk through will cover setting up a Secure Systems Manager (SSM) SecureString Parameter, in the SSM Parameter Store, and then utilizing that parameter from within a CloudFormation template.
2. Multi AZ VPC Configured:
This tutorial will use RDS as an example use case. In order for the embedded CloudFormation template to execute, you must already have an existing VPC with a minimum of 2 usable subnets. RDS utilizes subnet groups, which must consist of at least 2 separate subnets in order to launch successfully.
Navigate to SSM Parameter Store
1. Log into your AWS account:
Open a browser window and visit the AWS Console Page
2. Locate and navigate to SSM:
From the top left side of the navigational menu bar, click on the Services menu, and then choose Systems Manager by either navigating to the 
section of the listed services, or by typing the first few letters of the service name in the search box, and then choosing it from the filtered list.
3. Navigate to the Parameter Store:
Once in the Systems Manager console, from the left side menu, choose the Parameter Store link towards the bottom of the menu. This will get us to the Parameter Store where we will store our credentials for this exercise.
Create SecureString Parameters
The first thing that we will need to do is to create a secure string parameter within SSM. In order to do this, from the SSM Parameter Store console, click on the button titled Create parameter.
1. Create a Secure String Param:
Once the Create parameter button has been selected, you will then need to specify the details about your parameter. First select a name for your parameter (1). This will be the name that is referenced from within CloudFormation when making the call to pull the parameter value into your CloudFormation template, from the CloudFormation service. Next, optionally type a description, and then from the Type select SecureString (2). Once SecureString is selected the KMS options will appear allowing you to use the host default key, or specify a key in another account. For simplicity sake, we will select the My current account option (3), and last leaving the KMS key ID set to its default host key value of alias/aws/ssm, type the password itself into to Value field (4). Once all of your parameter properties are filled in, click the Create parameter button to create the actual parameter in the SSM parameter store.
2. Verify the Parameter Value:
Now that we have created our secure string parameter, lets quickly take a look at the parameter and verify that the parameter value is set properly. In order to do this from the Parameter Store console, click the parameter from the list to view its properties. Once in the parameter properties or detailed view, under the Value section, click the Show link to display the set password. Verify that the password is set to the proper value, and then lets move on to our CloudFormation template.
Sample CloudFormaton Template
Now that our parameter is stored safely in SSM, the next step is to create and run a CloudFormation template that will utilize the new parameter. The following example CloudFormation template creates a subnet group that will contain the subnets that are selected at launch time which the RDS instance will be placed into. It will then create an RDS postgres 10 database instance.
RDS Credentials
This CloudFormation template will set the Master User value to psadmin and set the Master Password to the value set in the SSM Parameter Store.
AWSTemplateFormatVersion: "2010-09-09" # Description of what this CloudFormation Template is going to produce Description: AWS CloudFormation template to test SSM Secure String retrieval from within CloudFormation. # Define parameters that will be used in this template Parameters: AZ1SubnetId: Description: Primary Subnet to be used for this DB instance. Type: AWS::EC2::Subnet::Id AllowedPattern: "[a-z0-9-]*" AZ2SubnetId: Description: Secondary Subnet to be used for this DB instance. (Must be different AZ) Type: AWS::EC2::Subnet::Id AllowedPattern: "[a-z0-9-]*" # Define Resources that will be launched via this template Resources: # Define the Security Group that will be appended to the ENI of the Instance we are creating. DBSubnetGroup: Type: "AWS::RDS::DBSubnetGroup" Properties: DBSubnetGroupDescription: "Group of Subnets that this instance will be launched in." DBSubnetGroupName: "TestDBSubnetGroup" SubnetIds: - !Ref AZ1SubnetId - !Ref AZ2SubnetId RDSDB: Type: AWS::RDS::DBInstance Properties: AllocatedStorage: "100" DBInstanceClass: "db.t2.small" DBSubnetGroupName: !Ref DBSubnetGroup Engine: "postgres" EngineVersion: "10.3" Iops: "1000" MasterUsername: "psadmin" MasterUserPassword: '{{resolve:ssm-secure:RDSAdminCreds:1}}'
Launch the CF Template
Next we should be ready to fire off our CloudFormation template, to test our call to the SSM parameter store. In order to launch our template, first navigate to the CloudFormation by navigating to the top left side of the navigational menu bar, and clicking on the Services menu. From there, choose CloudFormation by either navigating to the section of the listed services, or by typing the first few letters of the service name in the search box, and then choosing it from the filtered list.
1. Create Stack:
From the CloudFormation console, click the Create Stack button to view the various launch options. From the available options click on Design template.
2. Paste Template:
From the design window, click on the Template tab at the bottom of the designer IDE, (1) then paste the template example above into the code window of the editor. (2) Once pasted, click on the Create Stack button to upload the template to CloudFormation. (3)
Select Subnets
In order for this CloudFormation template to launch successfully there must be at least 2 available subnets. RDS uses subnet groups, which must contain a minimum of 2 subnets
3. Create Stack:
Once you click on the Create Stack button, you will be returned to the Select Template screen. Click Next to proceed.
Stack Region
When launching off your CloudFormation stack, ensure that you are launching the stack in the SAME region that was used to create the SSM Parameter Store SecureString Parameter. SSM is region specific, and the CloudFormation template must be in the same region as the region where the secure string parameter was stored or the call to retrieve the secure parameter WILL fail !!!
Next, in the Specify Details page, Type a name for your stack, and choose 2 separate subnets from the drop list.
Next, Specify any options such as tags and click the Next button.
Finally, review the launch parameters and click the Create button.
4. Wait for Stack Launch:
Once the Create button was clicked, you were returned back to your stack list window. From this window watch the stack, and wait until the status changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
Create Stack Time
The process of creating RDS resources can take a little bit of time to complete. Once you launch the stack it could take 5-10 minutes before the status changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
Testing the Stack
CloudFormation should have completed successfully, and instantiated the RDS instance using the psadmin user defined in the CloudFormation template along with the password that we stored in the SSM Parameter Store. In order to verify, lets launch an EC2 instance and verify that our RDS instance is configured with the proper credentials.
1. Navigate to EC2 Console:
In order to test our connection, we need to launch an instance into the primary subnet that we configured our RDS instance to be attached to. To do this, first navigate to the EC2 Console by going to the top left side of the navigational menu bar, and clicking on the Services menu. From there, choose EC2 by either navigating to the 
section of the listed services, or by typing the first few letters of the service name in the search box, and then choosing it from the filtered list.
2. Launch a Jump Host:
From the EC2 Console, click the Launch Instance button. Choose an AMI, such as AmazonLinux 2, choose an instance type, storage, tags, etc.. Ensure that when you select the instances launch parameters, choose the same subnet as the subnet that you chose when launching your RDS instance via CloudFormation. (1) You will also want to ensure that the security group that was assigned to the RDS instance (default-sg (2)), has a rule allowing ingress traffic on port 5432 from the same security group. (3) Launch the instance and then connect to the instance via SSH.
RDS Subnets:
As illustrated in the screenshot below (1), you can double check the subnets that your RDS instance was attached to by going to the Services menu, selecting RDS, then from the RDS Console, click on Instances from the left side menu. Finally, click on the RDS instance that was launched from our CloudFormation Stack. Once the RDS instance has been selected, scroll down to the Details section, and note the subnets attached that are attached to the instance. It will also be helpful to note the Endpoint
Jump Host OS
Although we are using Linux as the jump host in this walk through, if you are more comfortable using Windows, then you can launch a Windows instance in this step as an alternative. If using Windows, then skip the next step, and make sure a Postgres connection tool is installed on your Windows instance so that you can verify database connection credentials.
3. Install PostgreSQL Client:
Now that we have our jump host launched, we will need to install a Postgres client to instantiate a connection to our new RDS instance to validate that our credentials were set properly. Perform the following commands on the Linux host to install the postgres client.
Command:
yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm yum clean all yum update
Command Console Output:
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd pgdg-centos10-10-2.noarch.rpm | 4.6 kB 00:00:00 Examining /var/tmp/yum-root-d2cjYZ/pgdg-centos10-10-2.noarch.rpm: pgdg-centos10-10-2.noarch Marking /var/tmp/yum-root-d2cjYZ/pgdg-centos10-10-2.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package pgdg-centos10.noarch 0:10-2 will be installed --> Finished Dependency Resolution amzn2-core/2/x86_64 | 2.4 kB 00:00:00 Dependencies Resolved ================================================================================================================ Package Arch Version Repository Size ================================================================================================================ Installing: pgdg-centos10 noarch 10-2 /pgdg-centos10-10-2.noarch 2.7 k Transaction Summary ================================================================================================================ Install 1 Package Total size: 2.7 k Installed size: 2.7 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : pgdg-centos10-10-2.noarch 1/1 Verifying : pgdg-centos10-10-2.noarch 1/1 Installed: pgdg-centos10.noarch 0:10-2 Complete!
Yum Error
Because Amazon Linux is slightly different then RHEL or CentOS, we may experience a fail error when attempting to do a yum update. If this occurs, then modify the /etc/yum.repos.d/pgdg-10-centos.repo file and replace the baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-$releasever-$basearch line with baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/. Once modified, save the file, and reperform the yum update.
Example /etc/yum.repos.d/pgdg-10-centos.repo
[pgdg10] name=PostgreSQL 10 $releasever - $basearch # baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-$releasever-$basearch baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/ enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-10
Lastly, install the Postgres client.
Command:
yum install postgresql10
Command Console Output:
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 11 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package postgresql10.x86_64 0:10.5-1PGDG.rhel7 will be installed --> Processing Dependency: postgresql10-libs(x86-64) = 10.5-1PGDG.rhel7 for package: postgresql10-10.5-1PGDG.rhel7.x86_64 --> Processing Dependency: libpq.so.5()(64bit) for package: postgresql10-10.5-1PGDG.rhel7.x86_64 --> Running transaction check ---> Package postgresql10-libs.x86_64 0:10.5-1PGDG.rhel7 will be installed --> Finished Dependency Resolution Dependencies Resolved =============================================================================================================== Package Arch Version Repository Size =============================================================================================================== Installing: postgresql10 x86_64 10.5-1PGDG.rhel7 pgdg10 1.6 M Installing for dependencies: postgresql10-libs x86_64 10.5-1PGDG.rhel7 pgdg10 354 k Transaction Summary =============================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 1.9 M Installed size: 9.6 M Is this ok [y/d/N]: y Downloading packages: (1/2): postgresql10-libs-10.5-1PGDG.rhel7.x86_64.rpm | 354 kB 00:00:01 (2/2): postgresql10-10.5-1PGDG.rhel7.x86_64.rpm | 1.6 MB 00:00:01 --------------------------------------------------------------------------------------------------------------- Total 1.2 MB/s | 1.9 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : postgresql10-libs-10.5-1PGDG.rhel7.x86_64 1/2 Installing : postgresql10-10.5-1PGDG.rhel7.x86_64 2/2 Verifying : postgresql10-10.5-1PGDG.rhel7.x86_64 1/2 Verifying : postgresql10-libs-10.5-1PGDG.rhel7.x86_64 2/2 Installed: postgresql10.x86_64 0:10.5-1PGDG.rhel7 Dependency Installed: postgresql10-libs.x86_64 0:10.5-1PGDG.rhel7 Complete!
4. Attempt to Log in to the RDS Instance:
Now that our jump host is configured and launched, lets instantiate a connection to the RDS Postgres instance to test our credentials. First we will export the PGPASSWORD variable with the value that we have set in our SSM Secure String Parameter, and then use the following connection string to connect to the RDS instance on port 5432 as user psadmin to the Postgres database. If everything worked correctly, we should automatically connect to our Postgres instance.
Command:
export PGPASSWORD="SuperSecurePa55w0rd123" psql -h srmarr6zb74zc9.cpug8ewgtic3.us-east-2.rds.amazonaws.com -p 5432 -U psadmin postgres
Connection String Parameters
In the example command, note that the remote host (-h) URL is set to srmarr6zb74zc9.cpug8ewgtic3.us-east-2.rds.amazonaws.com. This value is the RDS instances Endpoint Value, and can be found in your RDS Console under the Instances section. You must replace this connection endpoint with the endpoint that is assigned to your RDS instance in your RDS console.
Command Console Output:
psql (10.5, server 10.3) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help. postgres=> \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | psadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 | rdsadmin | rdsadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 | rdsadmin=CTc/rdsadmin template0 | rdsadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/rdsadmin + | | | | | rdsadmin=CTc/rdsadmin template1 | psadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/psadmin + | | | | | psadmin=CTc/psadmin (4 rows)
Cleanup
Complete the following steps to roll back and remove resources created during this tutorial:
1. Delete CloudFormation Stack:
The first step that we need to perform is to delete the test RDS instance that we launched by originally running the included CloudFormation template. In order to delete the stack, navigate back to the CloudFormation console, and from the list of stacks, select the Test RDS stack that we launched earlier in this tutorial. Once selected, Click on Actions, and from the list of available actions, choose Delete.
Once you click the Delete action, you will be presented with a confirmation dialog asking if you are sure you want to delete the Stack, click the Yes, Delete button to confirm, and then wait for the stack status to move to DELETE_IN_PROGRESS. During this phase, CloudFormation will remove all resources that were launched off of the original CloudFormation deployment.
2. Delete JumpHost:
Next, we need to remove the Jump Host that we created and used to test the connection to the RDS database. In order to do this, Navigate back to the ECS console, and from the list of available instances, find and select the RDS Jump Host instance, and from the available Actions menu, choose the Instance State and then choose Terminate from the sub menu.
Once you select the Terminate option, you will be presented with an EC2 dialog asking if you are sure you want to terminate the instance and delete the associated EBS volume. Click on the Yes, Terminate button in order to start the instance termination process.
3. Delete SecureString Parameter:
The last thing that we need to clean up is the SSM SecureString parameter that we set in the Systems Manager console. Navigate back to the SSM Parameter Store, and from the list of available parameters, Select the parameter that was created, and click on the Delete button at the top of the Parameters Console.
Again, once you select Delete, an SSM dialog box will appear asking you to confirm removal of the Parameter. Click the Delete parameters button to remove the parameter from SSM.
Conclusion
In this walk through, we configured an SSM Parameter Store Parameter, we then constructed a CloudFormation template to utilize that secure parameter to set the MasterPassword of an RDS instance. We then launched the CF Stack, and instantiated the RDS database. Finally we launched a jump host, and used that jump host to connect to the database to test that the credentials on the RDS Postgres instance were both retrieved from SSM and set in RDS correctly !!
Additional Resources
No Additional Resources.
Site/Information References
Installing PostGreSQL on RHEL, CentOS, AmazonLinux
SSM: Using Dynamic References to Specify Template Values
CloudFormation Parameters