CODEWITHSUNDEEP
azure-devopsazure-pipelines
AndyD273
AndyD273

Reputation: 7279

Releasing only changed files with Pipelines

I am working on setting up a Azure DevOps project to update our website.
I set up a test website, and got it all working so that it would publish automatically whenever I did a Git Push.

The problem I'm having is that the test website has 2 files, and the real website has many many more, totalling in a little over 500MB.

I'm hoping there is a way to get it to only push out the files that changed, and not every single file.

My build pipeline is using the following script:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:

- task: ArchiveFiles@2
  displayName: 'ArchiveFiles'
  inputs:
    rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
    includeRootFolder: false

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts: drop'

And the release pipeline is doing a IIS Web App Deploy.

Upvotes: 3

Views: 10694

Answers (4)

Sagar Jaiswal
Sagar Jaiswal

Reputation: 23

If you are getting error "unknown revision" then increase the shallow fetch depth of your Azure DevOps YAML pipeline. See the official documentation.

Upvotes: 0

Haresh Shah
Haresh Shah

Reputation: 43

We did face the same issue and had to check a few solutions on Stackoverflow and Google and finally worked out on the following solution, which helped us pick only the last modified files based on GIT and then publish only those files instead of pushing the whole lot.

    # HTML
    # Archive your static HTML project and save it with the build record.
    # Add steps that build, run tests, deploy, and more:
    # https://aka.ms/yaml
    # Updated to pick only modified files and push them for publish by adding "task: PowerShell" (to optimize CICD process)

    trigger:
    - master

    pool:
    vmImage: 'ubuntu-latest'

    steps:
    - task: PowerShell@2
    inputs:
    targetType: 'inline'
        #We fist check the latest modified files based on GIT DIFF command
        #Then we loop through the files using files-split
        #Next we split the file URL to get the file name and parent folder structure of the file separately
        #Based on the file path, we check the path in output directory and create it if its not present
        #Once the folder structure is available in the destination, we then copy the file to that location using copy-item command
        #Force parameter is used copy-item to overwrite any existing files

        script: $files=$(git diff HEAD HEAD~ --name-only);
        $temp=$files-split ' '; 
        $count=$temp.Length;
        echo "Total changed $count files";
        For ($i=0; $i -lt $temp.Length; $i++)
        {
            $name=$temp[$i];
            echo "Modified File -  $name file";
            $filepath = ($name.Split('/'))[0..($name.Split('/').count-2)] -join '/';
            echo "File path - $filepath";          
            $destFile = Join-Path $(Build.ArtifactStagingDirectory) $name;
            $destinationpath = Split-Path $destFile ;
            echo "Destination path - $destinationpath";
            if (!(Test-Path -Path $destinationpath)) {
                New-Item $destinationpath -ItemType Directory
            }
            Copy-Item $name -Destination $destFile -Recurse -Force 
     
        }

        Get-ChildItem -Path $(Build.ArtifactStagingDirectory) -Recurse -Force

    - task: ArchiveFiles@2
    inputs:
        rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
        includeRootFolder: false

    - task: PublishBuildArtifacts@1

Please note that the code above works perfectly fine, only only copy pasting due to tabs/spaces at the start of every line, we need to validate the code before actually running the build.

Happy coding..!

Upvotes: 3

Daniel Rendon
Daniel Rendon

Reputation: 21

I was able to create an ADO powershell task based on the suggestion of another answer in this thread.

  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: 'get-childitem $(build.artifactStagingDirectory) -recurse | Foreach { 
        $lastupdatetime=$_.LastWriteTime; 
        $nowtime=get-date; 
        if(($nowtime - $lastupdatetime).totalhours -le 24) { 
          Write-Host $_.fullname 
        } 
        else{ 
          remove-item $_.fullname -Recurse -Force 
        } 
      }'
      failOnStderr: true
      workingDirectory: '$(build.artifactStagingDirectory)'

The script portion of the task should be in one line, but I expanded it for easier reading.

Upvotes: 2

Leo Liu
Leo Liu

Reputation: 76670

Releasing only changed files with Pipelines

There is no such out of box method to only pick the modified files when using copy task or PublishBuildArtifacts.

As workaround, we could add powershell task to deletes all files (recursive) which have timestamp that is < (Now - x min), with this way Artifact directory contains of ONLY CHANGED files.

Check the details here: TFS 2017 - how build/deliver only changed files?.

I tried to develop this script using my lame PowerShell technology, but I did not success.

Hope this helps.

Upvotes: 2

Related Questions