Rotate Secret Manager Credentials

In this section we will review the process to programmatically rotate the secret credentials.

Now we are going to add credential rotation to the secret stored in AWS Secrets Manager. Rotation is a common security best practice.

First, we are going to add another file to the addons directory in which to create the rotation:

cd ~/environment/secretecs
cat << EOF > copilot/todo-app/addons/rotation.yml
---
AWSTemplateFormatVersion: 2010-09-09
Transform:
  - "AWS::Serverless-2016-10-31"
  
Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  SecretRotationTemplate:
      Type: AWS::Serverless::Application
      Properties:
        Location:
          ApplicationId: arn:aws:serverlessrepo:us-east-1:297356227824:applications/SecretsManagerRDSPostgreSQLRotationSingleUser
          SemanticVersion: 1.1.60
        Parameters:
          endpoint: !Sub https://secretsmanager.${AWS::Region}.amazonaws.com
          functionName: !Sub ${AWS::StackName}-func
          vpcSecurityGroupIds: !Ref RotationSecurityGroup
          vpcSubnetIds:
            Fn::Join:
              - ","
              - - !Select [
                    0,
                    !Split [
                      ",",
                      { "Fn::ImportValue": !Sub "${App}-${Env}-PrivateSubnets" },
                    ],
                  ]
                - !Select [
                    1,
                    !Split [
                      ",",
                      { "Fn::ImportValue": !Sub "${App}-${Env}-PrivateSubnets" },
                    ],
                  ]
                  
  SecretRotationSchedule:
      Type: AWS::SecretsManager::RotationSchedule
      Properties:
        SecretId: !Ref AuroraSecret
        RotationLambdaARN: !GetAtt SecretRotationTemplate.Outputs.RotationLambdaARN
        RotationRules:
          AutomaticallyAfterDays: 30
EOF

Here we are adding a RotationTemplate and a RotationSchedule. The RotationTemplate points to a Lambda ARN that is stored in Serverless Application Repository. This will create a new lambda to execute the credential rotation. You will see it as a nested stack in the Cloudformation console. The RotationSchedule sets the duration between credential rotations and attaches itself to the Secret defined in the other stack.

Next, we need to commit the change locally and deploy it to the copilot environment.

git add copilot/todo-app/addons/rotation.yml && git commit -m "Add Credential Rotation"

Then we deploy the change with an arbitrary tag to force a new version. This process take between 5-7 minutes to create the new lambda and attach it to the secret.

copilot svc deploy --tag update-credentials

Now, we will manually rotate the secret to be sure it works.

Start by getting the id of the secret then using it to fetch the actual secret value. Here we grab the copilot application name, service name, and current environment.

APP=$(copilot svc show --json | jq -r .application)
SVC=$(copilot svc show --json | jq -r .service)
CENV=$(copilot svc show --json | jq -r .configurations[].environment)
aws secretsmanager get-secret-value --secret-id $APP/$CENV/$SVC/aurora-pg --query SecretString --output text | jq

to get the current value of the secret stored in Secrets Manager.

{
  "dbClusterIdentifier": "rdsstack-postgresrdsserverless",
  "password": "OLD_PASSWORD",
  "dbname": "tododb",
  "engine": "postgres",
  "port": 5432,
  "host": "rdsstack-xxxxxxxx.us-west-2.rds.amazonaws.com",
  "username": "postgres"
}

Then rotate the secret via the CLI:

aws secretsmanager rotate-secret --secret-id $APP/$CENV/$SVC/aurora-pg | jq

The output will look like:

{
    "VersionId": "5bdcd897-0a60-44e7-aee0-14c44abec425", 
    "Name": "ecsworkshop/test/todo-app/aurora-pg", 
    "ARN": "arn:aws:secretsmanager:us-west-2:xxxxxxxxxx:secret:ecsworkshop/test/todo-app/aurora-pg-jzAIx2"
}

Checking the web app in the browser, the data coming from the database is empty (no todo items will show - just the app scaffolding).

Next, query Secrets Manager again to get the value of the new password to ensure the password has been rotated:

aws secretsmanager get-secret-value --secret-id $APP/$CENV/$SVC/aurora-pg --query SecretString --output text | jq

Output:

{
  "dbClusterIdentifier": "rdsstack-postgresrdsserverless",
  "password": "MYSUPERNEWSUPERSECRETPASSWORD123",
  "dbname": "tododb",
  "engine": "postgres",
  "port": 5432,
  "host": "rdsstack-xxxxxxxx.us-west-2.rds.amazonaws.com",
  "username": "postgres"
}

Now that the secret password has been changed, the ECS service running task is still using the now-stale secret. In order for the service to pick up the new secret, stop the running task and let the ECS Scheduler bring up a new task which will contain the updated secret.

Use the AWS CLI to stop the current task, and then give the service a few mins to launch a new task to get the desired count back to 1.

CLUSTER_ARN=$(aws ecs list-clusters | jq -r .clusterArns[])
TASK_ARN=$(aws ecs list-tasks --cluster $CLUSTER_ARN | jq -r .taskArns[])
aws ecs stop-task --cluster $CLUSTER_ARN --task $TASK_ARN | jq

Once the task is running, go back to the todo app and refresh, you should see a fully functional app once again.

Start by getting the id of the secret then using it to fetch the actual secret value.

SECRET_ID=$(jq -r '.RDSStack.SecretName' result.json)
aws secretsmanager get-secret-value --secret-id $SECRET_ID --query SecretString --output text | jq

to get the current value of the secret stored in Secrets Manager.

{
  "dbClusterIdentifier": "rdsstack-postgresrdsserverless",
  "password": "OLD_PASSWORD",
  "dbname": "tododb",
  "engine": "postgres",
  "port": 5432,
  "host": "rdsstack-xxxxxxxx.us-west-2.rds.amazonaws.com",
  "username": "postgres"
}

Then rotate the secret:

aws secretsmanager rotate-secret --secret-id $SECRET_ID | jq

The output will look like:

{
    "VersionId": "5bdcd897-0a60-44e7-aee0-14c44abec425", 
    "Name": "ecsworkshop/test/todo-app/aurora-pg", 
    "ARN": "arn:aws:secretsmanager:us-west-2:xxxxxxxxxx:secret:ecsworkshop/test/todo-app/aurora-pg-jzAIx2"
}

This will result in a JSON object returned that shows the secret credential rotation started. Checking the web app in the browser, the data coming from the database will be empty.

Give this a few seconds, then query Secrets Manager again to get the value of the new password to ensure the password has been rotated:

aws secretsmanager get-secret-value --secret-id $SECRET_ID --query SecretString --output text | jq

Output:

{
  "dbClusterIdentifier": "rdsstack-postgresrdsserverless",
  "password": "MYSUPERNEWSUPERSECRETPASSWORD123",
  "dbname": "tododb",
  "engine": "postgres",
  "port": 5432,
  "host": "rdsstack-xxxxxxxx.us-west-2.rds.amazonaws.com",
  "username": "postgres"
}

Now that the secret password has been changed, the ECS service running task is still using the now-stale secret. Remember that we are exposing the secret data to the container app via an environment variable called POSTGRES_DATA. In order for the service to pick up the new secret, stop the running task and let the ECS Scheduler bring up a new task which will contain the updated secret.

You can look at the current value of the secret environment variable available to the container:

curl -s $url/env | jq -r '.POSTGRES_DATA' | jq

After stopping the task - give the scheduler a few minutes to launch a new task to get the desired count back to 1.

CLUSTER_ARN=$(aws ecs list-clusters | jq -r .clusterArns[])
TASK_ARN=$(aws ecs list-tasks --cluster $CLUSTER_ARN | jq -r .taskArns[])
aws ecs stop-task --cluster $CLUSTER_ARN --task $TASK_ARN | jq

Once the task scheduler has started a new task, go back to the todo app and refresh, you should see a fully functional app once again. You can also check the state of the environment variables again:

curl -s $url/env | jq -r '.POSTGRES_DATA' | jq

which will now reflect the new secret meaning the rotation was successful.