How to automate deploying to environments with AWS Copilot CLI?

Question

Needless to say, I am new to AWS. Excuse me.

I had a simple static site project (HTML/CSS/JS/PNG). Then put my files from root to /src, and npm init on root and amongst other things installed webpack dependency with which I can compile minimized and optimized files to the /dist directory. npm run build runs webpack. This works perfectly.

Next I installed AWS Copilot CLI and initializes service which generated /copilot/{project-name}/manifest.yml file:

name: {project-name}

type: Static Site

http:
  redirect_to_https: true
  path: '/'
  alias: 'dev-{project-name}.{company-name}.{ltd}'

files:
  - source: dist
    recursive: true

environments:
  stage:
    http:
      alias: 'staging-{project-name}.{company-name}.{ltd}'
  prod:
    http:
      alias: '{project-name}.{company-name}.{ltd}'

Now I can manually build the dist folder with npm run build and then use copilot svc deploy and choose either of the three environments (development, stage, prod), and the generated files automatically show up in their respective S3 buckets, and all three websites work and have the SSL certificate. That is already amazing.

Next step, I would like to automate all three deploys on commit+push/merge to the main branch. I added /copilot/environments/{development/prod/stage}/manifest.yml files, and they are correct. Then with copilot pipeline init two more files were generated, /copilot/pipelines/{project-name}-main/{buildspec/manifest}.yml. I've edited them a bit while playing around, but I cannot seem to get them to work properly. On code commit to BitBucket repo, AWS CodePipeline recognizes the change and starts the pipeline in stages Source -> Build -> DeployTo-development -> DeployTo-stage -> DeployTo-prod.

There are two problems.

1. The pipeline does not respect the requires_approval: true limitation and all stages go through and are successfully completed.
2. Absolutely nothing updates in the S3 buckets.

I don't know if something is wrong in my manifest/buildspec files or is there some hidden configuration in the AWS CodePipeline UI...

/copilot/pipelines/{project-name}-main/manifest.yml

name: {project-name}-main

version: 1

source:
  provider: Bitbucket
  properties:
    branch: main
    repository: https://bitbucket.org/{company-name}/{project-name}
    # connection_name: a-connection

stages:
  - name: development
    # test_commands: [echo 'running tests', make test]
    # deployments:
    #   {project-name}:
    #     template_path: infrastructure/development.env.yml
    #     template_config: infrastructure/development.env.params.json
    #     stack_name: {company-name}-development

  - name: stage
    requires_approval: true
    # deployments:
    #   {project-name}:
    #     template_path: infrastructure/stage.env.yml
    #     template_config: infrastructure/stage.env.params.json
    #     stack_name: {company-name}-stage

  - name: prod
    requires_approval: true
    # deployments:
    #   {project-name}:
    #     template_path: infrastructure/prod.env.yml
    #     template_config: infrastructure/prod.env.params.json
    #     stack_name: {company-name}-prod

/copilot/pipelines/{project-name}-main/buildspec.yml

version: 0.2

env:
  variables:
    NODE_ENV: development
    DEBUG: false
    # Note: This place is NOT meant for secrets (use parameter-store or secrets-manager instead)
    # Note: Any env var defined here overrides existing ones (eg. Docker vars)

phases:
  install:
    runtime-versions:
      nodejs: 20.x
    commands:
      - cd $CODEBUILD_SRC_DIR

      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.33.4
      - mv ./copilot-linux-v1.33.4 ./copilot-linux
      - chmod +x ./copilot-linux

      - npm install

  # pre_build:
  #   commands:
  #     - node --version            # v20.11.1
  #     - npm --version             # 10.2.4
  #     - aws --version             # aws-cli/2.15.43 Python/3.11.8 Linux/4.14.291-218.527.amzn2.x86_64 exec-env/AWS_ECS_EC2 exe/x86_64.amzn.2023 prompt/off
  #     - ./copilot-linux --version # copilot version: v1.33.4
  #     - echo $CODEBUILD_SRC_DIR   # /codebuild/output/src{some-numbers}/src

  build:
    commands:
      - npm run build

      - ls -lah ./dist

  post_build:
    commands:
      - export COLOR="false"
      - export CI="true"
      - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/{project-name}-main/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
      - stages=$(echo $pipeline | jq -r '.stages[].name')
      - >
        for env in $stages; do
          ./copilot-linux env package -n $env --output-dir './infrastructure' --upload-assets --force;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
        done;

      - echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>"
      - echo $CODEBUILD_SRC_DIR
      - ls -lah ./dist
      - ls -lah ./infrastructure

artifacts:
  files:
    - "infrastructure/*"

In the "CodePipeline > Build" output I can see all files are in the /dist folder, despite them not being in the repo, so npm run build works correctly. Any help will be greatly appreciated.

EDIT: AWS CloudWatch logs were mentioned, and I gave myself privileges to see them only to notice it's basically the same thing as the output in the Build stage, so here it is:

[Container] 2024/05/31 06:37:38.048349 Running on CodeBuild On-demand
[Container] 2024/05/31 06:37:38.048361 Waiting for agent ping
[Container] 2024/05/31 06:37:38.249912 Waiting for DOWNLOAD_SOURCE
[Container] 2024/05/31 06:37:39.635098 Phase is DOWNLOAD_SOURCE
[Container] 2024/05/31 06:37:39.636534 CODEBUILD_SRC_DIR=/codebuild/output/src{some-numbers}/src
[Container] 2024/05/31 06:37:39.637003 YAML location is /codebuild/output/src{some-numbers}/src/copilot/pipelines/{project-name}-main/buildspec.yml
[Container] 2024/05/31 06:37:39.639886 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/05/31 06:37:39.639980 Processing environment variables
[Container] 2024/05/31 06:37:39.891055 Resolved 'nodejs' runtime alias '20.x' to '20'.
[Container] 2024/05/31 06:37:39.891074 Selecting 'nodejs' runtime version '20' based on manual selections...
[Container] 2024/05/31 06:37:42.286325 Running command echo "Installing Node.js version 20 ..."
Installing Node.js version 20 ...
[Container] 2024/05/31 06:37:42.328692 Running command n $NODE_20_VERSION
     copying : node/20.11.1
   installed : v20.11.1 (with npm 10.2.4)
[Container] 2024/05/31 06:38:36.688661 Moving to directory /codebuild/output/src{some-numbers}/src
[Container] 2024/05/31 06:38:36.692007 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/05/31 06:38:36.728973 Configuring ssm agent with target id: codebuild:c534f38d-e36a-4eac-8e00-69d6f1e78f9f
[Container] 2024/05/31 06:38:36.767064 Successfully updated ssm agent configuration
[Container] 2024/05/31 06:38:36.767376 Registering with agent
[Container] 2024/05/31 06:38:36.807642 Phases found in YAML: 3
[Container] 2024/05/31 06:38:36.807667  BUILD: 2 commands
[Container] 2024/05/31 06:38:36.807688  POST_BUILD: 9 commands
[Container] 2024/05/31 06:38:36.807692  INSTALL: 5 commands
[Container] 2024/05/31 06:38:36.808004 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/05/31 06:38:36.808017 Phase context status code:  Message: 
[Container] 2024/05/31 06:38:36.880140 Entering phase INSTALL
[Container] 2024/05/31 06:38:36.880615 Running command cd $CODEBUILD_SRC_DIR
[Container] 2024/05/31 06:38:36.886847 Running command wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.33.4
[Container] 2024/05/31 06:38:42.185552 Running command mv ./copilot-linux-v1.33.4 ./copilot-linux
[Container] 2024/05/31 06:38:42.193441 Running command chmod +x ./copilot-linux
[Container] 2024/05/31 06:38:42.200818 Running command npm install
npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
added 714 packages, and audited 715 packages in 16s
137 packages are looking for funding
  run `npm fund` for details
found 0 vulnerabilities
[Container] 2024/05/31 06:38:58.000154 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/05/31 06:38:58.000185 Phase context status code:  Message: 
[Container] 2024/05/31 06:38:58.031454 Entering phase PRE_BUILD
[Container] 2024/05/31 06:38:58.033347 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/05/31 06:38:58.033358 Phase context status code:  Message: 
[Container] 2024/05/31 06:38:58.065394 Entering phase BUILD
[Container] 2024/05/31 06:38:58.065904 Running command npm run build
> {project-name}@1.0.0 build
> webpack
assets by path *.png 195 KiB
  asset 3bcb2032f11742d2680a.png 38.6 KiB [emitted] [immutable] [from: src/assets/favicon-228_x_228.png]
  asset 0a6d7df840b08d4fabad.png 30.4 KiB [emitted] [immutable] [from: src/assets/favicon-196_x_196.png]
  asset 8eb079e7d125996ceb72.png 29.4 KiB [emitted] [immutable] [from: src/assets/favicon-192_x_192.png]
  asset 83464aedc0290b180778.png 26.3 KiB [emitted] [immutable] [from: src/assets/favicon-180_x_180.png]
  asset f8d036bd68f394c5e0b5.png 20.2 KiB [emitted] [immutable] [from: src/assets/favicon-152_x_152.png]
  asset 58fcc430f0671394db92.png 15.2 KiB [emitted] [immutable] [from: src/assets/favicon-128_x_128.png]
  asset 6e7a8f6ab9d9bc06e95e.png 13.6 KiB [emitted] [immutable] [from: src/assets/favicon-120_x_120.png]
  asset 5e3447e529886c883003.png 9.35 KiB [emitted] [immutable] [from: src/assets/favicon-96_x_96.png]
  asset 5fa2633306eddf690966.png 6.38 KiB [emitted] [immutable] [from: src/assets/favicon-76_x_76.png]
  + 2 assets
asset main.js 55.8 KiB [emitted] (name: main) 1 related asset
asset a388d74414c20108664d.css 10.6 KiB [emitted] [immutable] [from: src/index.css]
asset favicon.ico 9.15 KiB [emitted]
asset index.html 4.26 KiB [emitted]
runtime modules 274 bytes 1 module
./src/index.js 54.5 KiB [built] [code generated]
webpack 5.91.0 compiled successfully in 1867 ms
[Container] 2024/05/31 06:39:00.573086 Running command ls -lah ./dist
total 384K
drwxr-xr-x 2 root root 4.0K May 31 06:39 .
drwxr-xr-x 7 root root  267 May 31 06:39 ..
-rw-r--r-- 1 root root  31K May 31 06:39 0a6d7df840b08d4fabad.png
-rw-r--r-- 1 root root  39K May 31 06:39 3bcb2032f11742d2680a.png
-rw-r--r-- 1 root root  16K May 31 06:39 58fcc430f0671394db92.png
-rw-r--r-- 1 root root 9.4K May 31 06:39 5e3447e529886c883003.png
-rw-r--r-- 1 root root 6.4K May 31 06:39 5fa2633306eddf690966.png
-rw-r--r-- 1 root root  14K May 31 06:39 6e7a8f6ab9d9bc06e95e.png
-rw-r--r-- 1 root root  27K May 31 06:39 83464aedc0290b180778.png
-rw-r--r-- 1 root root  30K May 31 06:39 8eb079e7d125996ceb72.png
-rw-r--r-- 1 root root 1.7K May 31 06:39 9de7d859249b7f7808f5.png
-rw-r--r-- 1 root root 4.1K May 31 06:39 a34cf4878a0499c1f897.png
-rw-r--r-- 1 root root  11K May 31 06:39 a388d74414c20108664d.css
-rw-r--r-- 1 root root  21K May 31 06:39 f8d036bd68f394c5e0b5.png
-rw-r--r-- 1 root root 9.2K May 31 06:39 favicon.ico
-rw-r--r-- 1 root root 4.3K May 31 06:39 index.html
-rw-r--r-- 1 root root  56K May 31 06:39 main.js
-rw-r--r-- 1 root root  71K May 31 06:39 main.js.map
[Container] 2024/05/31 06:39:00.604817 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/05/31 06:39:00.604834 Phase context status code:  Message: 
[Container] 2024/05/31 06:39:00.636941 Entering phase POST_BUILD
[Container] 2024/05/31 06:39:00.637475 Running command export COLOR="false"
[Container] 2024/05/31 06:39:00.683840 Running command export CI="true"
[Container] 2024/05/31 06:39:00.702120 Running command pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/{project-name}-main/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
[Container] 2024/05/31 06:39:02.813707 Running command stages=$(echo $pipeline | jq -r '.stages[].name')
[Container] 2024/05/31 06:39:03.003993 Running command for env in $stages; do
  ./copilot-linux env package -n $env --output-dir './infrastructure' --upload-assets --force;
  if [ $? -ne 0 ]; then
    echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
    exit 1;
  fi
done;
[Container] 2024/05/31 06:39:11.886546 Running command echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>"
>>>>>>>>>>>>>>>>>>>>>>>>>>>
[Container] 2024/05/31 06:39:11.893577 Running command echo $CODEBUILD_SRC_DIR
/codebuild/output/src{some-numbers}/src
[Container] 2024/05/31 06:39:11.899600 Running command ls -lah ./dist
total 384K
drwxr-xr-x 2 root root 4.0K May 31 06:39 .
drwxr-xr-x 8 root root  289 May 31 06:39 ..
-rw-r--r-- 1 root root  31K May 31 06:39 0a6d7df840b08d4fabad.png
-rw-r--r-- 1 root root  39K May 31 06:39 3bcb2032f11742d2680a.png
-rw-r--r-- 1 root root  16K May 31 06:39 58fcc430f0671394db92.png
-rw-r--r-- 1 root root 9.4K May 31 06:39 5e3447e529886c883003.png
-rw-r--r-- 1 root root 6.4K May 31 06:39 5fa2633306eddf690966.png
-rw-r--r-- 1 root root  14K May 31 06:39 6e7a8f6ab9d9bc06e95e.png
-rw-r--r-- 1 root root  27K May 31 06:39 83464aedc0290b180778.png
-rw-r--r-- 1 root root  30K May 31 06:39 8eb079e7d125996ceb72.png
-rw-r--r-- 1 root root 1.7K May 31 06:39 9de7d859249b7f7808f5.png
-rw-r--r-- 1 root root 4.1K May 31 06:39 a34cf4878a0499c1f897.png
-rw-r--r-- 1 root root  11K May 31 06:39 a388d74414c20108664d.css
-rw-r--r-- 1 root root  21K May 31 06:39 f8d036bd68f394c5e0b5.png
-rw-r--r-- 1 root root 9.2K May 31 06:39 favicon.ico
-rw-r--r-- 1 root root 4.3K May 31 06:39 index.html
-rw-r--r-- 1 root root  56K May 31 06:39 main.js
-rw-r--r-- 1 root root  71K May 31 06:39 main.js.map
[Container] 2024/05/31 06:39:11.906863 Running command ls -lah ./infrastructure
total 156K
drwxr-xr-x 2 root root 166 May 31 06:39 .
drwxr-xr-x 8 root root 289 May 31 06:39 ..
-rw-r--r-- 1 root root 822 May 31 06:39 development.env.params.json
-rw-r--r-- 1 root root 45K May 31 06:39 development.env.yml
-rw-r--r-- 1 root root 822 May 31 06:39 prod.env.params.json
-rw-r--r-- 1 root root 45K May 31 06:39 prod.env.yml
-rw-r--r-- 1 root root 777 May 31 06:39 stage.env.params.json
-rw-r--r-- 1 root root 45K May 31 06:39 stage.env.yml
[Container] 2024/05/31 06:39:11.914444 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/05/31 06:39:11.914465 Phase context status code:  Message: 
[Container] 2024/05/31 06:39:12.009354 Expanding base directory path: .
[Container] 2024/05/31 06:39:12.013012 Assembling file list
[Container] 2024/05/31 06:39:12.013116 Expanding .
[Container] 2024/05/31 06:39:12.016325 Expanding file paths for base directory .
[Container] 2024/05/31 06:39:12.016339 Assembling file list
[Container] 2024/05/31 06:39:12.016343 Expanding infrastructure/*
[Container] 2024/05/31 06:39:12.019568 Found 6 file(s)
[Container] 2024/05/31 06:39:12.023196 Set report auto-discover timeout to 5 seconds
[Container] 2024/05/31 06:39:12.023258 Expanding base directory path:  .
[Container] 2024/05/31 06:39:12.026457 Assembling file list
[Container] 2024/05/31 06:39:12.026471 Expanding .
[Container] 2024/05/31 06:39:12.029728 Expanding file paths for base directory .
[Container] 2024/05/31 06:39:12.029740 Assembling file list
[Container] 2024/05/31 06:39:12.029744 Expanding **/*
[Container] 2024/05/31 06:39:12.152135 No matching auto-discover report paths found
[Container] 2024/05/31 06:39:12.152180 Report auto-discover file discovery took 0.128984 seconds
[Container] 2024/05/31 06:39:12.152193 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2024/05/31 06:39:12.152200 Phase context status code:  Message: 

Answer 1

Manual Approvals:

• it is not the only thing required to configure approvals.
• it requires more step, please check here in the documentation, first understand it on the console and try to configure https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals-action-add.html

For S3, check once logs on Cloudwatch for detailed information on your code pipeline.