zipDeploy to Azure App Service does not delete old files. Errors trying to deploy empty zip file

Question

When deploying to an app service plan with the AzureRmWebAppDeployment@4 task and using zipDeploy it should remove files that were in an old .zip file but are not in the new one. But that's not always the case, especially when moving from .NET 4.8 to .NET 6 or higher. Since this doesn't work consistently, we created a process to do it based on a suggestion in How to clean up wwwroot folder on the target Azure Websites Windows Server before each deployment in VSTS.

• We create an empty .zip file and deploy that first. This should remove everything from the deploy folder. In case it doesn't, we run a script as part of the task that deletes everything in the folder.
• Once that is done, we deploy the .zip file with our actual code with a second AzureRmWebAppDeployment@4 task.

It works about 80-90% of the time, but but we randomly get errors in one of the AzureRmWebAppDeployment@4 tasks. I suspect that copying/deleting files is happening asynchronously and we are running into a race condition but I have no way to tell and don't know how we can mae it wait until one task finishes before starting the next.

Can anyone give us any insight into why this fails? Or suggest another way to clean the folder before deploying?

The task that deploys the empty zip file usually looks something like this when it fails:

Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "D:\local\Temp\zipdeploy\extracted" -o "D:\home\site\deployments\tools" --basic --sitePath "D:\local\Temp\zipdeploy\extracted"').
Running deployment command...
Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling Basic Web Site deployment.
KuduSync.NET from: 'D:\local\Temp\zipdeploy\extracted' to: 'D:\home\site\wwwroot'
Deleting file: 'runtimes\browser\lib\net6.0\System.Text.Encodings.Web.dll'
Deleting directory: 'runtimes\browser\lib\net6.0'
Deleting directory: 'runtimes\browser\lib'
Deleting directory: 'runtimes\browser'
Deleting file: 'runtimes\debian-x64\native\libuv.so'
Deleting directory: 'runtimes\debian-x64\native'
Deleting directory: 'runtimes\debian-x64'
Deleting file: 'runtimes\fedora-x64\native\libuv.so'
Deleting directory: 'runtimes\fedora-x64\native'
Deleting directory: 'runtimes\fedora-x64'
Deleting file: 'runtimes\opensuse-x64\native\libuv.so'
Deleting directory: 'runtimes\opensuse-x64\native'
Deleting directory: 'runtimes\opensuse-x64'
Deleting file: 'runtimes\osx\native\libuv.dylib'
Deleting directory: 'runtimes\osx\native'
Deleting directory: 'runtimes\osx'
Deleting file: 'runtimes\osx-x64\native\libcairo.2.dylib'
Deleting file: 'runtimes\osx-x64\native\libfontconfig.1.dylib'
Deleting file: 'runtimes\osx-x64\native\libfreetype.6.dylib'
Deleting file: 'runtimes\osx-x64\native\libgdiplus-lighthouse.dylib'
Deleting file: 'runtimes\osx-x64\native\libgdiplus.dylib'
Deleting file: 'runtimes\osx-x64\native\libgif.7.dylib'
Deleting file: 'runtimes\osx-x64\native\libglib-2.0.0.dylib'
Deleting file: 'runtimes\osx-x64\native\libintl.8.dylib'
Deleting file: 'runtimes\osx-x64\native\libjpeg.9.dylib'
Deleting file: 'runtimes\osx-x64\native\libpcre.1.dylib'
Deleting file: 'runtimes\osx-x64\native\libpixman-1.0.dylib'
Deleting file: 'runtimes\osx-x64\native\libpng16.16.dylib'
Deleting file: 'runtimes\osx-x64\native\libtiff.5.dylib'
Deleting directory: 'runtimes\osx-x64\native'
Deleting directory: 'runtimes\osx-x64'
Deleting file: 'runtimes\rhel-x64\native\libuv.so'
Deleting directory: 'runtimes\rhel-x64\native'
Deleting directory: 'runtimes\rhel-x64'
Deleting file: 'runtimes\unix\lib\net6.0\System.Drawing.Common.dll'
Deleting directory: 'runtimes\unix\lib\net6.0'
Deleting file: 'runtimes\unix\lib\netcoreapp2.1\System.Data.SqlClient.dll'
Deleting directory: 'runtimes\unix\lib\netcoreapp2.1'
Deleting file: 'runtimes\unix\lib\netcoreapp3.1\Microsoft.Data.SqlClient.dll'
Deleting directory: 'runtimes\unix\lib\netcoreapp3.1'
Deleting directory: 'runtimes\unix\lib'
Deleting directory: 'runtimes\unix'
Deleting file: 'runtimes\win\lib\net6.0\Microsoft.Win32.SystemEvents.dll'
Deleting file: 'runtimes\win\lib\net6.0\System.Drawing.Common.dll'
Deleting file: 'runtimes\win\lib\net6.0\System.Runtime.Caching.dll'
Deleting file: 'runtimes\win\lib\net6.0\System.Security.Cryptography.ProtectedData.dll'
Deleting file: 'runtimes\win\lib\net6.0\System.Windows.Extensions.dll'
Deleting directory: 'runtimes\win\lib\net6.0'
Deleting file: 'runtimes\win\lib\netcoreapp2.0\System.Diagnostics.PerformanceCounter.dll'
Omitting next output lines...
Error: The directory is not empty.

Failed exitCode=1, command="kudusync" -v 50  -f "D:\local\Temp\zipdeploy\extracted" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\8d6b23fb74fd4f2fb2827affd893cb46\manifest" -p "D:\home\site\deployments\4481321699452951452\manifest" -i ".git;.hg;.deployment;deploy.cmd"
An error has occurred during web site deployment.
Error: The directory is not empty.\r\nD:\Program Files (x86)\SiteExtensions\Kudu\100.50921.6442\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"
Deployment Failed.
##[error]Failed to deploy web package to App Service.
##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

The task that deploys the real .zip file sometimes gets an error like this:

Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "D:\local\Temp\zipdeploy\extracted" -o "D:\home\site\deployments\tools" --basic --sitePath "D:\local\Temp\zipdeploy\extracted"').
Running deployment command...
Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling Basic Web Site deployment.
KuduSync.NET from: 'D:\local\Temp\zipdeploy\extracted' to: 'D:\home\site\wwwroot'
Copying file: 'wwwroot\appsettings.json'
Copying file: 'wwwroot\appsettings.json.br'
Copying file: 'wwwroot\appsettings.json.gz'
Copying file: 'wwwroot\favicon.ico'
Copying file: 'wwwroot\health.html'
Copying file: 'wwwroot\index.html'
Copying file: 'wwwroot\MacroPoint.Web.Client.styles.css'
Copying file: 'wwwroot\css\Descartes.png'
Copying file: 'wwwroot\css\MainLayout.css'
Copying file: 'wwwroot\css\MainLayoutLite.css'
Copying file: 'wwwroot\css\QuickGrid.css'
Copying file: 'wwwroot\css\select2.min.css'
Copying file: 'wwwroot\css\splash.css'
Copying file: 'wwwroot\css\bootstrap\bootstrap.min.css'
Copying file: 'wwwroot\css\bootstrap\bootstrap.min.css.map'
Copying file: 'wwwroot\css\macropoint\descartes-default-theme-colors.css'
Copying file: 'wwwroot\css\macropoint\extendedElements.css'
Copying file: 'wwwroot\css\macropoint\macropoint-base.css'
Copying file: 'wwwroot\css\macropoint\macropoint-grid.css'
Copying file: 'wwwroot\css\macropoint\macropoint-shared-styles.css'
Copying file: 'wwwroot\css\macropoint\macropoint-theme-default.css'
Copying file: 'wwwroot\css\macropoint\macropoint.css'
Copying file: 'wwwroot\css\open-iconic\FONT-LICENSE'
Copying file: 'wwwroot\css\open-iconic\ICON-LICENSE'
Copying file: 'wwwroot\css\open-iconic\README.md'
Copying file: 'wwwroot\css\open-iconic\font\css\open-iconic-bootstrap.min.css'
Copying file: 'wwwroot\css\open-iconic\font\fonts\open-iconic.eot'
Copying file: 'wwwroot\css\open-iconic\font\fonts\open-iconic.otf'
Copying file: 'wwwroot\css\open-iconic\font\fonts\open-iconic.svg'
Copying file: 'wwwroot\css\open-iconic\font\fonts\open-iconic.ttf'
Copying file: 'wwwroot\css\open-iconic\font\fonts\open-iconic.woff'
Copying file: 'wwwroot\fonts\filter-light.svg'
Copying file: 'wwwroot\logo\Default\DSG_Brand_Icon.svg'
Copying file: 'wwwroot\logo\Default\DSG_MP_Logo_White.svg'
Copying file: 'wwwroot\maps\images\Active Shipments.xls'
Copying file: 'wwwroot\maps\images\stop.png'
Copying file: 'wwwroot\maps\scripts\MacroPointMaps-Entities.js'
Copying file: 'wwwroot\maps\scripts\MacroPointMaps-LoadMap.js'
Copying file: 'wwwroot\maps\scripts\MacroPointMaps-MapManager.js'
Copying file: 'wwwroot\maps\scripts\MacroPointMaps-PingMap.js'
Copying file: 'wwwroot\maps\styles\MacroPointMaps.css'
Copying file: 'wwwroot\scripts\app.js'
Copying file: 'wwwroot\scripts\mpPowerBi.js'
Copying file: 'wwwroot\scripts\mpQuickGrid.js'
Copying file: 'wwwroot\scripts\pendo.js'
Copying file: 'wwwroot\scripts\powerbi.min.js'
Copying file: 'wwwroot\scripts\select2.min.js'
Copying file: 'wwwroot\_content\BlazorApplicationInsights\JsInterop.js'
Copying file: 'wwwroot\_content\MacroPoint.Admin.Client\MacroPoint.Admin.Client.bundle.scp.css'
Omitting next output lines...
Error: Access to the path 'D:\home\site\wwwroot\wwwroot\_content\MacroPoint.Common.Client\flags\4x3\ac.svg' is denied.
Failed exitCode=1, command="kudusync" -v 50  -f "D:\local\Temp\zipdeploy\extracted" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\97227fea81624d40839efbe6b42858ea\manifest" -p "D:\Program Files (x86)\SiteExtensions\Kudu\100.50921.6442\bin\Scripts\firstDeploymentManifest" -i ".git;.hg;.deployment;deploy.cmd"
An error has occurred during web site deployment.
Error: Access to the path 'D:\home\site\wwwroot\wwwroot\_content\MacroPoint.Common.Client\flags\4x3\ac.svg' is denied.\r\nD:\Program Files (x86)\SiteExtensions\Kudu\100.50921.6442\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"
Deployment Failed.
##[error]Failed to deploy web package to App Service.
##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.

This is the template used to deploy the app services.

parameters:
- name: webAppName
  type: string
- name: azureSubscription
  type: string
- name: resourceGroupName
  type: string
- name: packagePath
  type: string
- name: swapSlots
  type: boolean
  default: 'False'
- name: skipDeploy
  type: boolean
  default: 'False'

steps:
- ${{ if eq(parameters.skipDeploy, False) }}:
  - task: PowerShell@2
    displayName: 'Create empty.zip file'
    inputs:
      targetType: 'inline'
      script: |
        $emptyFolder = "$(Pipeline.Workspace)\emptyfolder"
        if (-not (Get-Item -LiteralPath $emptyFolder -ErrorAction SilentlyContinue)) {
          New-Item -Type Directory -Path $emptyFolder
          $emptyZipFile = "$(Pipeline.Workspace)\empty.zip"
          Add-Type -Assembly System.IO.Compression.FileSystem
          [System.IO.Compression.ZipFile]::CreateFromDirectory($emptyFolder, $emptyZipFile)
        }

- ${{ if eq(parameters.skipDeploy, False) }}:
  - task: AzureRmWebAppDeployment@4
    displayName: 'Deploy empty.zip to ${{parameters.webAppName}} slot'
    inputs:
      ConnectionType: 'AzureRM'
      azureSubscription: ${{parameters.azureSubscription}}
      appType: 'webApp'
      WebAppName: '${{parameters.webAppName}}'
      deployToSlotOrASE: true
      ResourceGroupName: '${{parameters.resourceGroupName}}'
      SlotName: 'deploy'
      packageForLinux: '$(Pipeline.Workspace)\empty.zip'
      ScriptType: 'Inline Script'
      InlineScript: |
        dir D:\home\site\wwwroot /s
        rm --dir --force --recursive --verbose D:/home/site/wwwroot/*
        dir D:\home\site\wwwroot /s
        copy nul D:\home\site\wwwroot\app_offline.htm
        dir D:\home\site\wwwroot /s
        sleep 30s
        exit /b 0
      enableCustomDeployment: true
      DeploymentType: 'zipDeploy'

- ${{ if eq(parameters.skipDeploy, False) }}:
  - task: AzureRmWebAppDeployment@4
    displayName: 'Deploy to ${{parameters.webAppName}} slot'
    inputs:
      ConnectionType: 'AzureRM'
      azureSubscription: ${{parameters.azureSubscription}}
      appType: 'webApp'
      WebAppName: '${{parameters.webAppName}}'
      deployToSlotOrASE: true
      ResourceGroupName: '${{parameters.resourceGroupName}}'
      SlotName: 'deploy'
      packageForLinux: '${{parameters.packagePath}}'
      AppSettings: '-BuildNumber $(Build.BuildNumber)'
      enableCustomDeployment: true
      DeploymentType: 'zipDeploy'

- ${{ if and(eq(parameters.swapSlots, True),eq(parameters.skipDeploy, False)) }}:
  - template: swap-app-service.yml
    parameters:
      webAppName: '${{parameters.webAppName}}'
      azureSubscription: '${{parameters.azureSubscription}}'
      resourceGroupName: '${{parameters.resourceGroupName}}'

Answer 1

When using 'zipDeploy' method to run the deployment, the process normally will automatically delete the old files that does not exist in the new package. We generally can see the deleting operation from the task logs. No sure why the old files were not deleted automatically in your deploy process.

---

As a workaround, you can use 'webDeploy' method and enable the option 'RemoveAdditionalFilesFlag'.

- task: AzureRmWebAppDeployment@4
  displayName: 'Azure App Service Deploy: BriRanApp'
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: ${{parameters.azureSubscription}}
    appType: 'webApp'
    WebAppName: '${{parameters.webAppName}}'
    deployToSlotOrASE: true
    ResourceGroupName: '${{parameters.resourceGroupName}}'
    SlotName: deploy
    packageForLinux: '$(Pipeline.Workspace)\empty.zip'
    enableCustomDeployment: true
    RemoveAdditionalFilesFlag: true
    DeploymentType: webDeploy