CODEWITHSUNDEEP
amazon-web-servicesaws-codepipelineaws-codecommitaws-codebuild
dnh
dnh

Reputation: 577

AWS CodePipeline build lacks Git history

Context:

I have a CodePipeline set up that uses CodeCommit and CodeBuild as its source and build phases.

My build includes a plugin (com.zoltu.git-versioning) that uses the Git commit history to dynamically create a build version number.

Issue:

This fails on the AWS pipeline because of it cannot find any Git information in the source used to perform the build.

Clearly the action used to checkout the source uses an export which omits the Git metadata and history.

Question:

How do I configure CodeCommit or CodePipeline to do a proper git clone? I've looked in the settings for both these components (as well as CodeBuild) and cannot find any configuration to set the command used by the checkout action.

Has anyone got CodePipeline builds working with a checkout containing full Git metadata?

Upvotes: 14

Views: 7428

Answers (5)

Raphael Setin
Raphael Setin

Reputation: 845

I spent too much time on this poorly documented process, that I decided to create some documentation for myself and future developers. I hope it helps.

CodeBuild + CodePipeline

This will connect CodeBuild and CodePipeline such that changes to your GitHub repository triggers CodePipeline to do a Full clone of your repository, that is then passed to CodeBuild which just transforms the local .git folder metadata to be poiting to the correct branch, and then all of the source code plus the Git metadata is deployed to Elastic Beanstalk.

More information about this process can be found here.

  1. Start creating a CodePipeline pipeline. In the middle of its creation, you wull be prompted to create a CodeBuild project; do it.
  2. Feel free to select a specific location for the Artifact store (custom S3 bucket).
  3. Select GitHub (Version 2) as the source provider, check "Start the pipeline on source code change", and select Full cone as the output artifact format.
  4. Select AWS CodeBuild as the Build provider.
  5. For the Project Name, click onthe "Create project" button and select the below options:
    a. Environment image: Managed image
    b. Operating system: Amazon Linux 2
    c. Runtime(s): Standard
    d. For the Buildspec, select "Insert build commands" and click on "Switch to editor". Then paste the below Buildspec code.
    e. Enable CloudWatch logs.
  6. In the Environment variables, insert:
    BranchName: #{SourceVariables.BranchName} as Plaintext
    CommitId: #{SourceVariables.CommitId} as Plaintext
  7. Select Single build as the Build type.
  8. Select AWS Elastic Beanstalk as the Deploy provider.
  9. Review operation and create the pipeline.
  10. Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-connection-permission and attach the following JSON to it (tutorial):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "codestar-connections:UseConnection",
            "Resource": "arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095"
        }
    ]
}

PS: Change the Resource value arn:aws:codestar-connections:eu-central-1:123456789123:connection/sample-1908-4932-9ecc-2ddacee15095 from the JSON to your connection ARN. To find the connection ARN for your pipeline, open your pipeline and click the (i) icon on your source action.

  1. Create and add a new policy to the newly created CodeBuildServiceRole role. Choose a name, like projectName-s3-access and attach the following JSON to it:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::my-s3-bucket-codepipeline",
                "arn:aws:s3:::my-s3-bucket-codepipeline/*"
            ],
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketAcl",
                "s3:GetBucketLocation"
            ]
        }
    ]
}

PS: Change the Resource values my-s3-bucket-codepipeline to match with your S3 bucket name for your CodePipeline.

  1. Edit the inline policy for your CodePipelineServiceRole role by adding the following object to your Statement array:
{
    "Effect": "Allow",
    "Action": [
        "logs:*"
    ],
    "Resource": "*"
}
  1. Done.

Buildspec code

version: 0.2

#env:
#variables:
   # key: "value"
   # key: "value"
#parameter-store:
   # key: "value"
   # key: "value"
#secrets-manager:
   # key: secret-id:json-key:version-stage:version-id
   # key: secret-id:json-key:version-stage:version-id
#exported-variables:
   # - variable
   # - variable
#git-credential-helper: yes
#batch:
#fast-fail: true
#build-list:
#build-matrix:
#build-graph:
phases:
#install:
  #If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
  #If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
  #runtime-versions:
    # name: version
    # name: version
  #commands:
    # - command
    # - command
#pre_build:
  #commands:
    # - command
    # - command
build:
  commands:
    - echo Branch - $BranchName
    - echo Commit - $CommitId
    - echo Checking out branch - $BranchName
    - git checkout $BranchName
    # - command
    # - command
#post_build:
  #commands:
    # - command
    # - command
#reports:
#report-name-or-arn:
  #files:
    # - location
    # - location
  #base-directory: location
  #discard-paths: yes
  #file-format: JunitXml | CucumberJson
#artifacts:
#files:
  # - location
  # - location
#name: $(date +%Y-%m-%d)
#discard-paths: yes
#base-directory: location
artifacts:
files:
  - '**/*'
#cache:
#paths:
  # - paths

Additional Info

  • Never edit the inline policy that was created by CodePipeline! Only create and add new policies to a role. See this issue.
  • The Environment Variables for CodeBuild must be set from CodePipeline -> Edit: Build -> Environment variables - optional. If you set these variables in CodeBuild -> Edit -> Environment -> Additional configuration -> Environment variables it WON'T WORK!
  • For a bigger list of Environment variables during CodeBuild, see Variables List, Action Variables, and CodeBuild variables.
  • The Git Full clone option on CodePipeline is not available without CodeBuild. This is a known annoying limitation.
  • You can include the buildspec.yml in your root (top level) project directory. See this.
  • The Full clone that CodePipeline does leaves the local repository .git in a detached HEAD state, meaning that in order to get the branch name you will have to either get it with the help of CodeBuild environment variables to retrieve it from CodePipeline, or to execute the following command (see this):
git branch -a --contains HEAD | sed -n 2p | awk '{ printf $1 }'

Upvotes: 1

Ciprian Radu
Ciprian Radu

Reputation: 101

Yes, CodePipeline supports now a Git Full Clone. enter image description here You just need to do some extra steps: https://docs.aws.amazon.com/codepipeline/latest/userguide/troubleshooting.html#codebuild-role-connections

However, CodePipeline does not currently support dynamic branches, Pull Requests. See Dynamically change branches on AWS CodePipeline

Therefore, if you need to extend your pipeline for Pull Requests, I'd recommend the approach posted by Timothy Jones above.

There's one more related thing that's worth mentioning. CodeBuild has the Full Clone option as well. enter image description here

As long as you do not use the Local Source cache option, the Git history is there. enter image description here

When I tried to use the above mentioned cache option, I noticed that .git is not a directory. It's a file containing one line of text, e.g.:

gitdir: /codebuild/local-cache/workspace/9475b907226283405f08daf5401aba99ec6111f966ae2b921e23aa256f52f0aa/.git

I don't know why it's currently implemented like this but, it's confusing (at least for me) and I don't consider it to be the expected behavior.

Upvotes: 6

Neximos
Neximos

Reputation: 71

CodePipeline supports git full clone as of October: https://aws.amazon.com/about-aws/whats-new/2020/09/aws-codepipeline-now-supports-git-clone-for-source-actions/

In your console, go to the source stage and edit. You will have a new option to fully clone your git history. full clone option

In Terraform you will have to add it to the source action's configuration:

      configuration = {
           RepositoryName       = var.repository_name
           BranchName           = "master"
           OutputArtifactFormat = "CODEBUILD_CLONE_REF"
         }

More info:

https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-codecommit-gitclone.html

Upvotes: 7

Timothy Jones
Timothy Jones

Reputation: 22125

Although CodePipeline doesn't natively support this, you can get the information by cloning the repository in CodeBuild.

To do this, you need to set the permissions correctly, then carefully clone the repository.

Permissions

To give the permissions to clone the repository you need to:

  1. Give your CodeBuild role the codecommit:GitPull permission, with the resource ARN of your CodeCommit repository
  2. Put git-credential-helper: yes in the env part of your buildspec file

Cloning the repo

To clone the repo, you'll need to:

  1. know the clone URL and branch (CodeBuild doesn't know this information)

  2. git reset back to the commit that CodeBuild is building (otherwise you'll have a race condition between commits and builds).

    git reset "$CODEBUILD_RESOLVED_SOURCE_VERSION"

If you'd like examples, I've made a detailed writeup of the process, and published an example CodePipeline stack showing it in action.

Upvotes: 2

Aaron
Aaron

Reputation: 1605

This is currently not possible with the CodeCommit action in CodePipeline.

https://forums.aws.amazon.com/thread.jspa?threadID=248267

Upvotes: 6

Related Questions