CODEWITHSUNDEEP
bashazure-devopscontinuous-integration
PainIsAMaster
PainIsAMaster

Reputation: 2076

Azure DevOps build pipeline self-hosted agent "No space left on device"

I am running a build pipeline on Azure that runs on a private build server (Red Hat Enterprise Linux) running Self Hosted Agent. This build pipeline only has 1 Job and 2 Tasks where the 1st task is it basically SSH's into a Repo server we have (different server that just holds big files) generates an ISO image on that Repo server, then uses curl to put that ISO back on the the build server where the Azure Pipeline agent is running in the stereotypical $(Build.ArtifactStagingDirectory) Azure uses for Artifacts.

This 1st task succeeds, and the ISO is generated and copied over to the build server, but the "Publish Artifact" stage keeps failing. It's trying to publish to the path $(Build.ArtifactStagingDirectory) but produces an error message, with more logs: No space left on device

I already went in a cleared all the directories and files that exceeded > 1GB in this working directory `/home/azure/vsts/_work

I'm not an expert with Linux. When I run df -h and view the filesystem, there are a bunch in the list. Is there a way to know what partition I'm actually using for this Azure pipeline agent that using the /home/azure/vsts/_work directory?

My df -h list looks like: 

Filesystem                                    Size  Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root                    19G   19G   28K 100% /
devtmpfs                                      3.9G     0  3.9G   0% /dev
tmpfs                                         3.9G  8.0K  3.9G   1% /dev/shm
tmpfs                                         3.9G  138M  3.8G   4% /run
tmpfs                                         3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sdc                                      100G  152M  100G   1% /glusterfs
/dev/sda1                                     488M  119M  334M  27% /boot
/dev/mapper/vg_root-lv_var                    997M  106M  891M  11% /var
/dev/mapper/vg_docker-lv_docker                50G  3.1G   44G   7% /var/lib/docker/overlay2
/dev/mapper/vg_root-lv_log                    997M   46M  952M   5% /var/log
/dev/mapper/vg_root-lv_crash                  997M   33M  965M   4% /var/crash
/dev/mapper/vg_root-lv_root_logins             29M  1.8M   27M   6% /var/log/root_logins
/dev/mapper/vg_root-lv_core                   125M  6.6M  119M   6% /var/core
/dev/mapper/vg_root-lv_repo                   997M   83M  915M   9% /var/cache/yum
/dev/mapper/vg_root-lv_home                   997M   33M  965M   4% /export/home
/dev/mapper/vg_root-lv_logins                  93M  5.0M   88M   6% /var/log/logins
/dev/mapper/vg_root-lv_audit                  725M   71M  655M  10% /var/log/audit
tmpfs                                         799M     0  799M   0% /run/user/0
walkie1-ap2.nextgen.com:/hdd-volume0          200G  2.3G  198G  2%  /gluster-hdd

If anyone could provide some insight I'd greatly appreciate it.

End of error log:

[2020-05-06 05:49:09Z ERR  JobRunner] Caught exception from job steps StepsRunner: System.IO.IOException: No space left on device
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode)
   at Microsoft.VisualStudio.Services.Agent.PagingLogger.NewPage()
   at Microsoft.VisualStudio.Services.Agent.PagingLogger.Write(String message)
   at Microsoft.VisualStudio.Services.Agent.Worker.ExecutionContext.Write(String tag, String message)
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunAsync(IExecutionContext jobContext, IList`1 steps)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken)```

Upvotes: 2

Views: 19280

Answers (3)

ランス
ランス

Reputation: 540

The clean option didn't work that well for me. So, what I did was limit the files in the Build and Publish option. Since, we have a different pipeline for the tests. This approach did the trick.

Upvotes: 1

PainIsAMaster
PainIsAMaster

Reputation: 2076

So I was a n00b here, and the solution was simply to clean out space from the main directory we used to store our large ISO files:

/dev/mapper/vg_root-lv_root                    19G   19G   28K 100% /

This is a custom VM we used to run builds on Azure and I wasn't accustomed to the error message. But yes, if anyone says this message and is using a custom build agent, its definitely a space issue.

Upvotes: 1

LoLance
LoLance

Reputation: 28146

I can't reproduce same issue on my side, but I think you can check this article for a trouble-shooting.

As I know this task itself will take extra space when being executed. You can try a bash command to make copy of content under path $(Build.ArtifactStagingDirectory) to double the size of those content, if this action throws same No space left on device error?

And in build pipeline there's one clean option to clean the caches before executing the job, enable it to check if it helps:

enter image description here

If it's yaml pipeline, try something like:

workspace:
    clean: outputs | resources | all # what to clean up before the job runs

and

steps:
- checkout: self | none | repository name # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # if true, run `execute git clean -ffdx && git reset --hard HEAD` before fetching

See Yaml schema.

Upvotes: 3

Related Questions