Actively scanning regularly

As described above, we then have 2 functions which work together. The first one, will list all the repositories in the Registry (note that these are region based), and then send into SQS a list of the repositories that you want to scan.

A second Lambda function, triggered by Lambda via SQS, receives the repositories to scan. The reason for splitting the two functions is to enable parallelism and keep the execution time of these lambda functions very short.

That second function is then going to list the images of the given repository, and describe these in order to get information details about the scan.

If the image was never scanned, it will trigger a scan. If the image was scanned, it will then evaluate how long ago was that last scan, and if above the user defined "expiry duration", will then trigger a new scan for that image.

Reporting findings

As mentioned in the situation analysis, there is not (yet! ... AWS has a habit to make my solutions obsolete) a feature to integrate into notifications systems easily yet to report on the security findings once the scan is complete (or failed).

From SNS, one can do the integration to a number of target, or for integration that might require a little bit more involvement, trigger a similar lambda function to notify on findings.

Participate in the roadmap!

This is an open source project that we would love users to get involved, so please help making the reporter better and open new feature requests on GitHub.

Using a flat file as CFN template

When using your docker-compose file as a CFN template, there are a couple limitations to have in mind:

• You cannot have multiple docker-compose files together (to use override). Therefore, you would need to have a single (potentially longer) docker-compose file.

• After adding the transform section to the template, docker-compose locally will not work because Transform is not a valid docker-compose keyword.

As previously, I have two files:

• docker-compose.yml -- which contains our services definitions. Here, we only have our wordpress service

• aws.yml -- this is our template that sums up all the things Compose-X needs to handle for us in order to deploy the service successfully

So I am going to merge those too files together and add the following at the top (the position does not matter).

Then I add the following to the YAML file. At this point, when one gives that file to Cloudformation as a template, it will need to run the macro to get the rendered parts of the template.

Transform:
  - compose-x

And that is all you had to do. Now we have the "template" and the Macro, we can just create a new stack (or changeset) with AWS CloudFormation.

From the CLI

CAPABILITIES="APABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM"
aws cloudformation create-stack --template-body file://merged.yml --capabilities ${CAPABILITIES} --stack-name wordpress-demo

And that's it. CloudFormation will invoke the CFN macro which will render for us all the templates we need, and return it back to AWS CloudFormation to then create all our resources.

ecs-composex config -f docker-compose.yml -f aws.yml | tee merged.yml

Using files stored in AWS S3

---
# Wordpress demo using ComposeX Macro
Fn::Transform:
  Name: compose-x
  Parameters:
    ComposeFiles:
      - s3://files.compose-x.io/docker-compose.yml
      - s3://files.compose-x.io/aws.yml

Dockerfiles

At the start of the project I had only aimed to build for python3.7 images. But, I realized, why stop there. Given that some extra commands are necessary for installing python3.8 with Amazon Linux, I thought the easiest thing for all is just to have two different files.

But in essence, they are doing the same thing: update the packages installed, install python, set the new python as default.

Batch buildspec definition

AWS CodeBuild supports multiple configurations and is very versatile. Here we want to build 4 docker images, and gather these images in groups of two manifests. So, we are going to have one build per configuration and therefore one image per.

Each build will end by publishing the image build to AWS ECR, which our final stage will use.

Here with a batch-graph configuration, we can define dependencies between builds. So here, our manifest step will only be executed once the others are finished.

batch:
  fast-fail: false
  build-graph:
    - identifier: amd64_py37
      env:
        compute-type: BUILD_GENERAL1_LARGE
        privileged-mode: true
        variables:
          VERSION: 3.7
          ARCH: amd64
      buildspec: build_images.yml

    - identifier: arm64v8_py37
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:2.0
        compute-type: BUILD_GENERAL1_LARGE
        privileged-mode: true
        variables:
          ARCH: arm64v8
          VERSION: 3.7
      buildspec: build_images.yml

    - identifier: amd64_py38
      env:
        compute-type: BUILD_GENERAL1_LARGE
        privileged-mode: true
        variables:
          VERSION: 3.8
          ARCH: amd64
      buildspec: build_images.yml

    - identifier: arm64v8_py38
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:2.0
        compute-type: BUILD_GENERAL1_LARGE
        privileged-mode: true
        variables:
          ARCH: arm64v8
          VERSION: 3.8
      buildspec: build_images.yml

    - identifier: manifest
      env:
        compute-type: BUILD_GENERAL1_LARGE
        privileged-mode: true
      depend-on:
        - amd64_py37
        - arm64v8_py37
        - amd64_py38
        - arm64v8_py38

Once the build has finished, if succeeded, you should see

And that is it, this is really that simple.

Sources

You can find the source files for this project in GitHub

Define local environment

So first of all, we are going to rename our docker-compose.yml file local.yml

In this file we mostly want to keep the mariaDB section in order to test our changes, if any, to the main wordpress docker image.

version: '3.8'
services:
  mariadb:
    image: 'docker.io/bitnami/mariadb:10.3-debian-10'
    volumes:
      - 'mariadb_data:/bitnami/mariadb'
    environment:
      - MARIADB_USER=bn_wordpress
      - MARIADB_DATABASE=bitnami_wordpress
      - ALLOW_EMPTY_PASSWORD=yes

  wordpress:
    depends_on:
      - mariadb
    environment:
      - MARIADB_HOST=mariadb
      - MARIADB_PORT_NUMBER=3306
      - WORDPRESS_DATABASE_USER=bn_wordpress
      - WORDPRESS_DATABASE_NAME=bitnami_wordpress
      - ALLOW_EMPTY_PASSWORD=yes

As a result in our docker-compose.yml file, we now have

version: '3.8'

services:
  wordpress:
    image: public.ecr.aws/bitnami/wordpress:5-debian-10
    ports:
      - '8080:8080'
      - '8443:8443'
    volumes:
      - 'wordpress_data:/bitnami/wordpress'
    environment:
      WORDPRESS_SKIP_INSTALL: "yes"
    deploy:
      resources:
       reservations:
         cpus: 1.0
         memory: 1G

volumes:
  wordpress_data:
    driver: local

To run our wordpress locally we now would simply run

# Verify the configuration of merged files
docker-compose -f docker-compose.yml -f local.yml config
# Deploy continers from the composed set of files
docker-compose -f docker-compose.yml -f local.yml up

Create our AWS Environment definition

This section might seem long, but it only seems so due to a lot of explanations. You might copy-paste and adapt the code-block sections and add these to your aws.yml file.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: dev-vpc
    AppSubnets:
      Tags:
        - vpc::usage: application
        - aws:cloudformation:stack-name: dev-vpc
    PublicSubnets:
      Tags:
        - vpc::usage: public
        - aws:cloudformation:stack-name: dev-vpc
    StorageSubnets:
      Tags:
        - vpc::usage: storage
        - aws:cloudformation:stack-name: dev-vpc

x-elbv2:
  lbA:
    Properties:
      Scheme: internet-facing
      Type: application
    Listeners:
      - Port: 80
        Protocol: HTTP
        Targets:
          - name: wordpress:wordpress
            access: /
    Services:
      - name: wordpress:wordpress
        port: 8080
        protocol: HTTP
        healthcheck: 8080:HTTP:/:7:2:15:5

x-rds:
  wordpress-db:
    Properties:
      Engine: "aurora-mysql"
      EngineVersion: "5.7"
      BackupRetentionPeriod: 1
      DatabaseName: wordpress
      StorageEncrypted: True
      Tags:
        - Key: Name
          Value: "wordpress DB"
    Services:
      - name: wordpress
        access: RW
        SecretsMappings:
          Mappings:
            host: MARIADB_HOST
            port: MARIADB_PORT_NUMBER
            username: WORDPRESS_DATABASE_USER
            password: WORDPRESS_DATABASE_PASSWORD
            dbname: WORDPRESS_DATABASE_NAME

x-s3:
  wp-data-bucket:
    Properties:
      AccessControl: BucketOwnerFullControl
      ObjectLockEnabled: True
      PublicAccessBlockConfiguration:
          BlockPublicAcls: True
          BlockPublicPolicy: True
          IgnorePublicAcls: True
          RestrictPublicBuckets: False
      AccelerateConfiguration:
        AccelerationStatus: Suspended
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
    Services:
      - name: wordpress
        access:
          Bucket: ListOnly
          Objects: RW

docker-compose -f docker-compose.yml -f aws.yml config
ecs-composex config -f docker-compose.yml -f aws.yml

Setup a friendly DNS Name and a SSL certificate

I have a DNS domain which I already have in Route53, so I am going to simply point to it.

x-dns:
  PublicZone:
    Use: ZABCDEFGHIS0123 # Redacted for privacy purposes.
  Records:
    - Properties:
        Name: wordpress.demos.lambda-my-aws.io
        Type: A
      Target: x-elbv2::wordpress-lb

Then we can now create a new ACM Certificate

x-acm:
  wordpress-demo:
    Properties:
      DomainName: wordpress.demos.lambda-my-aws.io
      DomainValidationOptions:
        - HostedZoneId: ZABCDEFGHIS0123 # Redacted for privacy purposes
          DomainName: wordpress.demos.lambda-my-aws.io
      ValidationMethod: DNS

Now we assign this certificate to our existing ALB, simply by editing our Listeners section.

x-elbv2:
  lbA:
    Properties:
      Scheme: internet-facing
      Type: application
    MacroParameters:
      Ingress:
        ExtSources:
          - IPv4: 0.0.0.0/0
            Description: "ANY"
    Listeners:
      - Port: 80
        Protocol: HTTP
        DefaultActions:
          - Redirect: HTTP_TO_HTTPS
      - Port: 443
        Protocol: HTTPS
        Certificates:
          - x-acm: wordpress-demo
        Targets:
          - name: wordpress:wordpress
            access: /
    Services:
      - name: wordpress:wordpress
        port: 8080
        protocol: HTTP
        healthcheck: 8080:HTTP:/:7:2:15:5

We can simply update our existing stack to add our new ACM certificate and DNS names pointing to our ALB.

ecs-composex up -f docker-compose.yml -f aws.yml --format yaml -n wordpress-demo

This update will be rather quick, and thanks to the instruction Redirect: HTTP_TO_HTTPS in our HTTP listener, all requests submitted over HTTP will be redirected to HTTPs. Using an AWS ACM certificate, our clients to the Load-Balancer will be able to get SSL information and match that against our site.

Once this is all done, setup your Wordpress user etc, install the Media Lite plugin, configure it for S3, and you can now use the S3 bucket as storage for your media files.

Here is a little gallery to help you go through the same steps as I did to get Wordpress + S3 working.

What is next ?

At the time of writing this blog post, Troposphere 2.6.4 is pending release to integrate EFS for ECS. This would be the last part of the puzzle to allow some settings to be persistent as they do not live in the database.

Also with the release announcement of AWS Proton, ECS ComposeX will focus on allowing existing Docker-compose users to define environments by using docker-compose syntax and help with the adoption of AWS ECS to run and deploy containerized applications.

Workflow

A good picture is worth 1000 words they say, so here to illustrate the workflow that AWS Codebuild will follow based on our buildspec.yml file:

You can find the full buildspec.yml file here

The most import parts of the buildspec.yml is the build_identifier in the build-graph section for each. When codebuild is done with the two previous artifacts, these will be passed onto the next phase.

In the buildspec_java_apps.yml however, we name the artifacts with the same name as the service, but the name of the identifier is what matters most.

Multi-Inputs build

For those of you used to AWS Codepipeline and AWS Codebuild, you will be familiar with how AWS Codebuild has environment variables CODEBUILD_SRC_{something} which refer to the build artifacts, and more specifically, to the secondary artifacts.

Here, AWS CodeBuild very smartly simply re-used the same principle.

So if we have a build-graph with identifier app01 we end up with CODEBUILD_SRC_app01_AppDefinition (AppDefinition because it is the way it is defined in AWS Codepipeline!).

Now we know that, and given we followed a specific naming convention for our identifiers to match our services names defined in docker-compose files, we can safely gather the outputs as needed.

Here, we only need the JAR files created by maven, and given we have a specific folder for each service source code, we place that JAR file into the folder.

for service in `docker-compose config --services 2>/dev/null`; do
      shortname=`echo $service | sed s'/[^a-zA-Z0-9]//g'`
      dir_env_name="CODEBUILD_SRC_DIR_${shortname}_AppDefinition"
      if ! [ -d ${!dir_env_name} ]; then echo "No output found for $service"; echo ${!dir_env_name}; exit 1 ; fi
      find ${!dir_env_name} -type f -name "${service}*.jar" | ( read path; cp $path ${service}/app.jar ) ;
done

Bundle, publish, deploy

And this is where docker-compose and ComposeX really save us a lot of time and trouble. First off, with docker-compose, we now just build the services images and push them to AWS ECR.

docker-compose build
docker-compose push

Once that is done, we can now use ComposeX to generate our CFN templates and configuration files. We place them into a new artifact which the pipeline will then use.

Back to CodePipeline

Worth pointing out, and I am yet to figure out the differences we might expect between build with batch-graph/matrix/list when it comes to the artifacts, and how they are merged if you so wish to do so.

In this use-case, I am merging the artifacts together.

Indeed, I do not need the JAR. files in the rest of the process, but, for those of you who might want to add some lambda functions in this repository and deploy these to layer or functions, there you go, you already have that JAR file ready in your artifacts!

CodePipeline expects cloudformation template and the config file. Given we bundled things together, CodeBuild has created sub directories for each artifact, named based on the identifier.

We then just have to adapt our path in the CloudFormation action of the Codepipeline stage:

- Name: !Sub 'DeployToDev'
  Actions:
    - Name: CfnDeployToDev
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Provider: CloudFormation
        Version: '1'
      Configuration:
        ActionMode: CREATE_UPDATE
        RoleArn: !ImportValue 'CICD::nonprod::Cfn::Role::Arn'
        StackName: !Sub '${DeploymentName}-dev'
        TemplatePath: !Sub 'AppDefinition::composex/AppDefinition/dev/${DeploymentName}.yaml'
        OutputFileName: outputs.json
        TemplateConfiguration: !Sub 'AppDefinition::composex/AppDefinition/dev/${DeploymentName}.config.json'
        Capabilities: 'CAPABILITY_AUTO_EXPAND,CAPABILITY_IAM'
      InputArtifacts:
        - Name: AppDefinition
      OutputArtifacts:
        - Name: DevStackOutputs
      RunOrder: '1'
      RoleArn: !ImportValue 'CICD::nonprod::Pipeline::Role::Arn'

Some thoughts before you leave

• You could have a repository with your docker compose files etc. and have a repository per microservice instead of a mono repo

  and still achieve the same thing, for example, using git submodules

• If you have shared libraries you want to build first, simply add builds, publish to AWS CodeArtifact / Nexus / else

  then resume the build of your applications

Secrets & ACLs

Least privileges is one of my most important must-have in technical evaluations. AWS IAM is, in my opinion, the most comprehensive system I have seen to date. So putting anything against that often makes me perplex, as things usually do not support RBAC too well.

So, I was very happy to see that Kafka supports ACLs on topics and consumer groups, and cluster level. I was very happy to see that the connect cluster can use 3 different service accounts to manage separately the connect cluster itself, and then use different one for the producers and for the consumers. I was outraged to see that Control Center must have Admin level of access to the kafka cluster. There is no way to limit what people can do with it.

I was very disappointed that equally, the Confluent "Cloud" service registry does not have any notion of ACLs, but I am told by Confluent this is coming.

So, I create one service account for the connect cluster, and decided it was good enough for now to use the same service-account for all three parts (control, producer, consumer).

ccloud service-account create --description connect-cluster
# We retrieve and keep preciously these credentials.
ccloud api-key create --resource cluster-abcd --service-account 123456
# Now allow some access. That is up to you, keeping it rather open for the blog copy-paste.
for access in READ WRITE DESCRIBE; do ccloud kafka acl create --allow --service-account 123456 --operation $access --prefix --topic * ; done
for access in READ WRITE DESCRIBE; do ccloud kafka acl create --allow --service-account 123456 --operation $access --prefix --consumer-group * ; done

Now, we have credentials. So let's use AWS Secrets manager, given ECS integrates very well with it. For that we have two different CFN templates which create the secret for the connect cluster and one for the control center.

Now we have dealt with authorization and authentication, we can move to the easy part. Really, the above was the difficult part.

Build the Docker images

We have the docker images build and published by Confluent, which is a somewhat difficult to follow build process, so not going to rebuild these. But we are going to use them as source to add a few bash scripts in order to deal with the keys of the secret to be individually exposed.

These are then pushed into ECR.

if [ -z ${AWS_ACCOUNT_ID+x} ]; then export AWS_ACCOUNT_ID=`(aws sts get-caller-identity | jq -r .Account)`; fi
docker-compose build
docker-compose push

start.sh

You will have noticed a start.sh file is used to override the start of the services. The reason for it is, until recently AWS Fargate did not support to export each individual secret JSON Key to the container.

I decided to leave it in here instead of using ComposeX to do this (which it can, see ComposeX secrets docs.) to show you how easy it is in a couple lines of bash script to export all these JSON keys at container run-time.

Typically, within your application you would import that secret and parse its JSON to get the values, however, Confluent Images were not built for it, and fortunately, we have environment variables to override settings.

For the control-center, this also allows us to define connect clusters and therefore here, to use the one we are deploying at the same time.

docker-compose files

So, we have a primary docker-compose.yml file which very easily allows us to define what is constant across deployments. Then we have an individual override file per environment. Here, just dev and stg (for staging), but then you could have 20 files if you wanted.

Now all we have to do, is deploy!

# Check the configuration is correct with the override files
ENV_NAME=dev ecs-composex config -f docker-compose.yml -f envs/${ENV_NAME}.yml
mkdir outputs
# Render the CFN templates if you want to double check the content.
ENV_NAME=dev AWS_PROFILE=myprofile ecs-composex render --format yaml -d outputs -n confluent-apps-${ENV_NAME} -f docker-compose.yml -f envs/${ENV_NAME}.yml
# Deploy, using up
ENV_NAME=dev AWS_PROFILE=myprofile ecs-composex up -n confluent-apps-${ENV_NAME} -f docker-compose.yml -f envs/${ENV_NAME}.yml

Here is an example of one of the envs files.

---
version: '3.8'
services:
  controlcenter:
    build:
      context: control-center
      dockerfile: Dockerfile
    deploy:
      resources:
        reservations:
          cpus: "1.0"
          memory: "2G"
    ports:
      - 8080:8080
    environment:
      CONTROL_CENTER_NAME: ${ENV_NAME:-dev}
      CONNECT_CLUSTERS: ${ENV_NAME:-dev}::http://connect.${ENV_NAME:-dev}.lan.internal:8083
      CONTROL_CENTER_KSQL_ENABLE: "false"
    depends_on:
      - connect

  connect:
    ports:
      - "8083:8083"
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "2.0"
          memory: "4G"
    x-scaling:
      TargetScaling:
        range: "1-3"
        memory_target: 75
    x-network:
      Ingress:
        Myself: True

secrets:
  CONNECT_CREDS:
    x-secrets:
        Name: /kafka/${CLUSTER_ID}/confluent.connect.cluster

  CC_CREDS:
    x-secrets:
      Name: /kafka/${CLUSTER_ID}/confluent.controlcenter


x-elbv2:
  controlcenter:
    Properties: {}
    MacroParameters:
      Ingress:
        ExtSources:
          - Ipv4: 0.0.0.0/0
            Name: ANY
            Description: "ANY"
    Listeners:
      - Port: 80
        Protocol: HTTP
        Targets:
          - name: controlcenter:controlcenter
    Services:
      - name: controlcenter:controlcenter
        port: 8080
        protocol: HTTP
        healthcheck: 8080:HTTP:4:2:15:5:/:200

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: demo
    PublicSubnets:
      Tags:
        - vpc::usage: public
    AppSubnets:
      Tags:
        - vpc::usage: "application"
    StorageSubnets:
      Tags:
        - vpc::usage: storage

x-cluster:
  Lookup: default-cluster

x-dns:
  PrivateNamespace:
    Name: ${ENV_NAME:-dev}.lan.internal

x-tags:
  costcentre: lambda-my-aws
  environment: ${ENV_NAME:-dev}

AWS X-Ray overview

AWS X-Ray is what's now one of my very favorite service on AWS. It integrates very well to pretty much any language and has some predefined integration with frameworks such as Flask.

In essence, X-Ray will capture the application metrics which will enable you to identify performances issues, and also provide you with an understanding of how your services communicate together.

It will also allow you to see how your application integrates to AWS Services.

The AWS X-Ray team also made available a Docker image that you can use in your local environments (laptops, Cloud9 etc.) and it will report metrics captured from your local environment, so it really is flexible to integrate anywhere.

How X-Ray is added to your ECS Task

Presently, when ECS ComposeX parses the configuration and services, it will for each service create a task definition which will contain a single container definition. Adding X-Ray was very straight forward, using the pre-defined Docker image provided by AWS, which also comes with recommened compute reservations.

When you enable X-Ray for your service in ECS ComposeX, it simply is going to add that extra container definition.

Secrets are kept secret

Because I care about security, and I am sure you do too, in the code is implemented to ensure that the X-Ray container will not be exposed with Secrets. For example, if you service was linked to a RDS DB, which would expose the secret as an environment variable to the container, the X-Ray container is specifically identified to not have access to that secret too.

IAM policy

The IAM policy that allows the X-Ray container / app to communicate with the X-Ray service is added to the IAM Task Role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords",
                "xray:GetSamplingRules",
                "xray:GetSamplingTargets",
                "xray:GetSamplingStatisticSummaries"
            ],
            "Resource": [
                "arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:group/*",
                "arn:${AWS::Partition}:xray:${AWS::Region}:${AWS::AccountId}:sampling-rule/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Enable X-Ray for your service

Enable or disable locally for a specific service

services:
  serviceA:
    image: link_to_image
    configs:
      x-ray:
        enabled: true

Enable for all services from the top level:

services:
  serviceA:
    image: link_to_image

configs:
  composex:
    x-ray:
      enabled: true

And yes, it is as simple as that.

CICD structure

The below image illustrates how the CI pipeline connects to the CD pipeline and via which artifacts. Obviously, in the CD pipeline, one would also execute the application integration test-suite against the deployed environment.

IAM Structure

The below image tries to illustrate as simply as possible the different relationship between artifacts and IAM roles. It doesn't go to the tiniest of detail but gives a good overview, especially for the cross-account structure, of the different IAM roles and access between these.

App01 - FlaskApp

This application is dumb and purely stateless. It is going to respond with the date when you ping it. As we are going to build new versions we will be able to add more build tests to it. App01 will also communicate with App02 by API message to get the time (fancy, I know!)

App02 - The date teller

The 2nd application will run without any load-balancer, but our App01 will communicate with it to get the date purely using Service Discovery.

The Docker compose repo

We are going to have a separate Docker compose repository which is here to be updated by CodeBuild whenever a build of App01 or App02 is successful and results into a new images in ECR.

ECR Repositories

The ECR repositories will be created with a policy allowing roles from the application/environment accounts (dev/stage/prod etc.) to pull the docker images from there. To create these, I am going to use CloudFormation and assign a resource policy to these. The template for these can be found on Github

We are going to create 2 in the shared/pipeline account.

for count in {01..02}; do aws cloudformation create-stack --stack-name ecr-repo-blog-app-${count} \
    --template-body file://ecr_repos.yml \
    --parameters ParameterKey=RepositoryName,ParameterValue=blog-app-${count};
done

KMS Key

The KMS Key is used to encrypt the artifacts in the Artifacts bucket via CodePipeline. This key allows basic use from the dev and the production account.

aws cloudformation create-stack --stack-name cicd-kms-key --template-body file://shared-kms-cmk.yml

Artifacts and templates bucket

The CI and CD pipelines are going to store artifacts. Artifacts used by CodePipeline to pass outputs from step to step, stage to stage, etc., and templates for Cloudformation.

So we are going to create the buckets first, without creating the IAM policies. Create the roles in your accounts and have their RoleId, we will update that stack with the RoleId for prod and dev roles which will create the bucket policy to allow these roles to access objects in the bucket.

I have not created the SSM parameters for these to show how to input all the parameters. Replace the values accordingly.

aws cloudformation create-stack --stack-name cicd-shared-buckets \
    --template-body file://shared-buckets.yml

Once you have the RoleId for the IAM roles, update the stack to create the Bucket policies.

aws cloudformation update-stack --stack-name cicd-shared-buckets \
    --template-body file://shared-buckets.yml
    --parameters \
        ParameterKey=ProdAccountCfnRoleId,ParameterValue=<ROLE_ID>      \
        ParameterKey=ProdAccountPipelineRoleId,ParameterValue=<ROLE_ID> \
        ParameterKey=DevAccountCfnRoleId,ParameterValue=<ROLE_ID>       \
        ParameterKey=DevAccountPipelineRoleId,ParameterValue=<ROLE_ID>

Cross account role

The cross account role allows the CodePipeline service to assume role into the destination account. Given iam:PassRole cannot be done cross account, this is how we get to run CloudFormation into the external account.

We want this role to be able to:

• Decrypt objects with the KMS Key

• Get objects from the artifacts bucket

• Do everything for CloudFormation

• Pass the CloudFormation role to CFN to create the stacks and resources.

CloudFormation role

As said above, iam:PassRole cannot pass role from one account to another. So once the assume role is done, we still want to pass role to not have this shared role which requires no MFA or external ID to have too many powers. Anyway, I generally prefer to give an IAM role to my CFN stacks anyway as soon as I delegate to a service to invoke CFN Create/Update/Delete.

Create the roles in your accounts

aws cloudformation --capabilities CAPABILITY_IAM                                \
    --stack-name cicd-iam-roles                                                 \
    --template-body file://crossaccount-roles.yml                               \
    --parameters                                                                \
        ParameterKey=CiAccountId,ParameterValue=<012345678912>                  \
        ParameterKey=CiKmsKeyId,ParameterValue=abcd1234-ab12-cd34-ef56-5678wxyz \
        ParameterKey=ArtifactsBucketName,ParameterValue=<BUCKET_NAME>           \
        ParameterKey=CloudformationTemplatesBucketName,ParameterValue=<BUCKET_NAME>

CI - Integration pipeline(s)

Now we have a clearer idea of what we need: we need a constant build project that is in charge of merging / updating the Docker compose file either when its own repository is updated, or, whenever a new image is successfully built.

So we are going to have two more CloudFormation templates for our CodePipelines and CodeBuild projects:

• DockerCompose Build project, which does the same thing across all our applications: merge the docker compose files.

• Applications codebuild to build the app, test it, build the docker image, test it, release it, and onto the Docker compose file merge and update.

for count in {01..02}; do aws cloudformation create-stack \
    --capabilities CAPABILITY_IAM \
    --template-body file://apps_pipeline.yml \
    --parameters file://app${count}-params.json \
    --stack-name app${count}-ci-pipeline;
done

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "codecommit:Git*"
        ],
        "Resource": "arn:aws:codecommit:eu-west-1:373709687836:myapps-compose",
        "Effect": "Allow",
        "Sid": "CodecommitAccessToDockerComposeRepo"
    },
    {
        "Action": [
            "codecommit:GitPull"
        ],
        "Resource": "arn:aws:codecommit:eu-west-1:373709687836:docker-composerx",
        "Effect": "Allow",
        "Sid": "CodecommitAccessToDockerComposerRepo"
    }
]}

CD - Deployment pipeline

aws cloudformation create-stack --capabilities CAPABILITY_IAM \
    --stack-name myapps-cd-pipeline \
    --template-body file://cd_pipeline.yml \
    --parameters \
        ParameterKey=ComposeRepositoryName,ParameterValue=myapps-compose \
        ParameterKey=BranchName,ParameterValue=master \
        ParameterKey=ProdAccountPipelineRoleArn,ParameterValue=<ROLE_ARN>   \
        ParameterKey=ProdAccountCfnRoleArn,ParameterValue=<ROLE_ARN>    \
        ParameterKey=DevAccountCfnRoleArn,ParameterValue=<ROLE_ARN>     \
        ParameterKey=DevAccountPipelineRoleArn,ParameterValue=<ROLE_ARN>

Alternative pipelines

It is completely up to you and your business to decide how you are going to release your applications and therefore decide on what is going to trigger your deployments. Here in this example, I am considering any push on the master branch of my compose repository. You could have with your GIT strategy decided that only a new tag release can trigger a deployment to production and that other pushes, such as new branches etc. would trigger dev environments deployments / updates only.

Room for improvement

As for everything, there is always room for improvement. Please leave your comments and feel free to submit issues or even PRs against ECS ComposeX or this blog's repository for patches and improvements.

Because pipelines and the associated resources are not the most friendly things to generate, I will start working on a similar project than ECS ComposeX which for now is called Pipeline composer, pending a better name possibly.