From mono-repo to multi-services deployment with AWS CodeBuild and AWS Codepipeline

Workflow

A good picture is worth 1000 words they say, so here to illustrate the workflow that AWS Codebuild will follow based on our buildspec.yml file:

You can find the full buildspec.yml file here

The most import parts of the buildspec.yml is the build_identifier in the build-graph section for each. When codebuild is done with the two previous artifacts, these will be passed onto the next phase.

In the buildspec_java_apps.yml however, we name the artifacts with the same name as the service, but the name of the identifier is what matters most.

Multi-Inputs build

For those of you used to AWS Codepipeline and AWS Codebuild, you will be familiar with how AWS Codebuild has environment variables CODEBUILD_SRC_{something} which refer to the build artifacts, and more specifically, to the secondary artifacts.

Here, AWS CodeBuild very smartly simply re-used the same principle.

So if we have a build-graph with identifier app01 we end up with CODEBUILD_SRC_app01_AppDefinition (AppDefinition because it is the way it is defined in AWS Codepipeline!).

Now we know that, and given we followed a specific naming convention for our identifiers to match our services names defined in docker-compose files, we can safely gather the outputs as needed.

Here, we only need the JAR files created by maven, and given we have a specific folder for each service source code, we place that JAR file into the folder.

for service in `docker-compose config --services 2>/dev/null`; do
      shortname=`echo $service | sed s'/[^a-zA-Z0-9]//g'`
      dir_env_name="CODEBUILD_SRC_DIR_${shortname}_AppDefinition"
      if ! [ -d ${!dir_env_name} ]; then echo "No output found for $service"; echo ${!dir_env_name}; exit 1 ; fi
      find ${!dir_env_name} -type f -name "${service}*.jar" | ( read path; cp $path ${service}/app.jar ) ;
done

Bundle, publish, deploy

And this is where docker-compose and ComposeX really save us a lot of time and trouble. First off, with docker-compose, we now just build the services images and push them to AWS ECR.

docker-compose build
docker-compose push

Once that is done, we can now use ComposeX to generate our CFN templates and configuration files. We place them into a new artifact which the pipeline will then use.

Back to CodePipeline

Worth pointing out, and I am yet to figure out the differences we might expect between build with batch-graph/matrix/list when it comes to the artifacts, and how they are merged if you so wish to do so.

In this use-case, I am merging the artifacts together.

Indeed, I do not need the JAR. files in the rest of the process, but, for those of you who might want to add some lambda functions in this repository and deploy these to layer or functions, there you go, you already have that JAR file ready in your artifacts!

CodePipeline expects cloudformation template and the config file. Given we bundled things together, CodeBuild has created sub directories for each artifact, named based on the identifier.

We then just have to adapt our path in the CloudFormation action of the Codepipeline stage:

- Name: !Sub 'DeployToDev'
  Actions:
    - Name: CfnDeployToDev
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Provider: CloudFormation
        Version: '1'
      Configuration:
        ActionMode: CREATE_UPDATE
        RoleArn: !ImportValue 'CICD::nonprod::Cfn::Role::Arn'
        StackName: !Sub '${DeploymentName}-dev'
        TemplatePath: !Sub 'AppDefinition::composex/AppDefinition/dev/${DeploymentName}.yaml'
        OutputFileName: outputs.json
        TemplateConfiguration: !Sub 'AppDefinition::composex/AppDefinition/dev/${DeploymentName}.config.json'
        Capabilities: 'CAPABILITY_AUTO_EXPAND,CAPABILITY_IAM'
      InputArtifacts:
        - Name: AppDefinition
      OutputArtifacts:
        - Name: DevStackOutputs
      RunOrder: '1'
      RoleArn: !ImportValue 'CICD::nonprod::Pipeline::Role::Arn'

Some thoughts before you leave

• You could have a repository with your docker compose files etc. and have a repository per microservice instead of a mono repo

  and still achieve the same thing, for example, using git submodules

• If you have shared libraries you want to build first, simply add builds, publish to AWS CodeArtifact / Nexus / else

  then resume the build of your applications