Azure Devops Pipeline - PublishPipelineArtifact@1 - Symbolic link handling problem

Question

we are using self hosted Linux build agents with the Azure DevOps SaaS solution. The pipeline is running in a Docker container context.

In one pipeline I want to publish the whole workspace folder. Within the workspace folder there is a Python venv folder. When this folder is tried to be published I am getting the below error:

Error:

ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session 98e6e193-0ceb-4b36-ab55-1f94fc84eb3b
##[error]Unable to load symbolic/hard linked file.  Check 'fsutil behavior query SymlinkEvaluation' to ensure proper behavior.
##[warning]RetryHelper encountered task failure, will retry (attempt #: 1 out of 3) after 1000 ms
Artifact name input: variant_config_sets_build
Uploading pipeline artifact from /agent/_work/5/s for build #1266963
Using default max parallelism.
Max dedup parallelism: 192
DomainId: 0
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 9febb015-b994-4533-bde1-2e2cf0cf588b
Hashtype: Dedup64K
DedupManifestArtifactClient will correlate http requests with X-TFS-Session 9febb015-b994-4533-bde1-2e2cf0cf588b
Processing .artifactignore file surfaced 2056 files. Total files under source directory: 7589
2 files processed.
33 files processed.
Failed to calculate hash for the file's reparse point: /agent/_work/5/s/virt_py_environment/bin/python3
Failed to calculate hash for the file's reparse point: /agent/_work/5/s/virt_py_environment/bin/python
Failed to hash a page of files. Exception listed below...
System.IO.FileLoadException: Unable to load symbolic/hard linked file.  Check 'fsutil behavior query SymlinkEvaluation' to ensure proper behavior.
 ---> System.IO.FileNotFoundException: Could not find file '/agent/_work/5/s/virt_py_environment/bin/python3'.
File name: '/agent/_work/5/s/virt_py_environment/bin/python3'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Microsoft.VisualStudio.Services.Content.Common.FileStreamUtils.OpenFileStreamForAsync(String filePath, FileMode mode, FileAccess fileAccess, FileShare fileShare, FileOptions extraOptions)
   at Microsoft.VisualStudio.Services.Content.Common.FileSystem.OpenStreamForFile(String fileFullPath, FileMode fileMode, FileAccess fileAccess, FileShare fileShare)
   at Microsoft.VisualStudio.Services.BlobStore.Common.ChunkerHelper.CreateFromFileAsync(IFileSystem fileSystem, String path, Boolean configureAwait, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.FileBlobDescriptor.CalculateAsync(IFileSystem fileSystem, String rootDirectory, HashType hashType, String relativePath, FileBlobType fileBlobType, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.GetFileBlobDescriptorAsync(String rootDirectory, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, PageItem pageItem, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.GetFileBlobDescriptorAsync(String rootDirectory, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, PageItem pageItem, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.<>c__DisplayClass31_0.<<PaginateAndProcessFilesHelperAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.PaginateAndProcessFilesHelperAsync(String sourceDirectory, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, IEnumerable`1 pathsInDir, Action`1 hashCompleteCallback, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.<>c__DisplayClass26_0.<<PaginateAndProcessFiles>b__0>d.MoveNext()
133 files processed.

What is really strange are the following facts:

Error case

In the error case the Python distribution from which the venv is created is installed via pyenv. And the above listed files are pointing to following folders:

ls -l /agent/_work/5/s/virt_py_environment/bin

lrwxrwxrwx 1 AzDevOps AzDevOps    7 Feb 24 21:37 python -> python3
lrwxrwxrwx 1 AzDevOps AzDevOps   49 Feb 24 21:37 python3 -> /home/AzDevOps/.pyenv/versions/3.8.20/bin/python3

And these folders are accessible since I could run a ls -l on /home/AzDevOps/.pyenv/versions/3.8.20/bin

OK Case

When the Python distribution is a regularly installed Python distribution then I the publish step does not generate any error. I get following information when running ls -l.

ls -l /agent/_work/5/s/virt_py_environment/bin

lrwxrwxrwx 1 AzDevOps AzDevOps    7 Feb 24 21:43 python -> python3
lrwxrwxrwx 1 AzDevOps AzDevOps   16 Feb 24 21:43 python3 -> /usr/bin/python3

As short-term workaround I have added the venv folder to the .artifactignore file. But I would like to understand why the publishing in one case works and not in the other.

Any hint on that?

PS. I have already seen this post: Azure Devops Pipeline - PublishPipelineArtifact@1 - ##[error]Unable to load symbolic/hard linked file

The publish step is invoked like this:

I am using this logic to publish the files:

- task: PublishPipelineArtifact@1
        retryCountOnTaskFailure: 3
        condition: succeededOrFailed()
        inputs:
          targetPath: "$(Build.SourcesDirectory)"
          artifact: "variant_config_sets_build"
          publishLocation: "pipeline"

The runtime of the job is a container. It is specified like this:

stages:
  - stage: Build_Stage
    displayName: Render profiles and templates
    pool:
      name: ${{ parameters.Build_Pool }}

    jobs:
      - job: Build_Variant_Config_Sets
        pool:
          name: ${{ parameters.Build_Pool }}
        container: ${{ variables.imageName }}

        steps:
          - checkout: self
        ...

This is the stacktrace I get when enabling the debug mode:

##[error]Unable to load symbolic/hard linked file.  Check 'fsutil behavior query SymlinkEvaluation' to ensure proper behavior.
##[debug]Processed: ##vso[task.logissue type=error;]Unable to load symbolic/hard linked file.  Check 'fsutil behavior query SymlinkEvaluation' to ensure proper behavior.
##[debug]Processed: ##vso[task.complete result=Failed;]
##[debug]   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.GetFileBlobDescriptorAsync(String rootDirectory, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, PageItem pageItem, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.<>c__DisplayClass31_0.<<PaginateAndProcessFilesHelperAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.PaginateAndProcessFilesHelperAsync(String sourceDirectory, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, IEnumerable`1 pathsInDir, Action`1 hashCompleteCallback, HashType hashType, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.<>c__DisplayClass26_0.<<PaginateAndProcessFiles>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.Content.Common.ExceptionExtensions.ReThrow(Exception ex)
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.<>c__DisplayClass26_0.<<PaginateAndProcessFiles>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.BlobStore.Common.PrecomputedHashesGenerator.PaginateAndProcessFiles(String sourceDirectory, HashType hashType, IEnumerable`1 pages, Boolean shouldPreserveSymbolicLink, Boolean shouldPreservePermissionValue, CancellationToken cancellationToken, Action`1 hashCompleteCallback)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.PublishAsync(String sourceDirectory, IReadOnlyCollection`1 fileInfoList, IReadOnlyDictionary`2 otherFiles, ArtifactPublishOptions artifactPublishOptions, String manifestFileOutputPath, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineArtifactServer.<>c__DisplayClass2_0.<<UploadAsync>b__4>d.MoveNext() in /mnt/vss/_work/1/s/src/Agent.Plugins/Artifact/PipelineArtifactServer.cs:line 80
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Content.Common.ExceptionExtensions.ReThrow(Exception ex)
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Agent.Plugins.PipelineArtifactServer.<>c__DisplayClass2_0.<<UploadAsync>b__3>d.MoveNext() in /mnt/vss/_work/1/s/src/Agent.Plugins/Artifact/PipelineArtifactServer.cs:line 77
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.Services.BlobStore.Common.Telemetry.BlobStoreClientTelemetry.MeasureActionAsync[TResult](BlobStoreTelemetryRecord record, Func`1 actionAsync)
   at Microsoft.VisualStudio.Services.Content.Common.ExceptionExtensions.ReThrow(Exception ex)
   at Microsoft.VisualStudio.Services.BlobStore.Common.Telemetry.BlobStoreClientTelemetry.MeasureActionAsync[TResult](BlobStoreTelemetryRecord record, Func`1 actionAsync)
   at Agent.Plugins.PipelineArtifactServer.UploadAsync(AgentTaskPluginExecutionContext context, Guid projectId, Int32 pipelineId, String name, String source, IDictionary`2 properties, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Plugins/Artifact/PipelineArtifactServer.cs:line 75
   at Agent.Plugins.PipelineArtifact.PublishPipelineArtifactTaskV1.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, CancellationToken token) in /mnt/vss/_work/1/s/src/Agent.Plugins/PipelineArtifact/PipelineArtifactPluginV1.cs:line 146
   at Agent.PluginHost.Program.Main(String[] args) in /mnt/vss/_work/1/s/src/Agent.PluginHost/Program.cs:line 71