CODEWITHSUNDEEP
go-cd
dokaspar
dokaspar

Reputation: 8624

How to fetch multiple subdirectories as one artifact with GoCD?

I am using GoCD to build a project with a large number of modules and I have modeled it as a pipeline with two stages (stage 1 is to build the code, stage 2 is to run the tests). The directory structure after a successful build stage looks about like this:

myproject/
|-- myproject-module1
|   |-- build              <-- created by stage 1, required by stage 2
|   `-- src
|-- myproject-module2
|   |-- build              <-- created by stage 1, required by stage 2
|   `-- src
|-- myproject-module3
|   |-- build              <-- created by stage 1, required by stage 2
|   `-- src
`-- ... many more modules ...

In stage 1 I have configured a Build Artifact with source */build and in stage 2 I'm trying to fetch all the build folders again with source * , with the intention that they would end up in the correct location next to the src folder inside each of the project modules.

Unfortunately, I have found no way to achieve this yet. GoCD seems to create a separate ZIP file of all the *\build folders and during the fetch, the file *.zip cannot be found (I assume that it really looks for a file with that exact name, instead of using wildcards). Of course I could hard-code all the module names and individually fetch myproject-module[1:n], but that's exactly what I want to avoid.

Does anyone have some advice on how this could be achieved?

In this discussion from 2014, it is claimed that wildcards cannot be used to fetch artifacts. Is that really still the case?!

Upvotes: 0

Views: 1541

Answers (2)

moritz
moritz

Reputation: 12842

You can select artifacts by wildcard, you just have to give the artifact a destination (attribute dest in the XML config). You can do that multiple times inside the same job, just use a different dest each time:

<artifact src="myproject/myproject-module1/build/*" dest="module1/" />
<artifact src="myproject/myproject-module2/build/*" dest="module2/" />
<artifact src="myproject/myproject-module3/build/*" dest="module3/" />

The corresponding <fetchartifact ...> tags then need to use srcdir="module1" etc.

Upvotes: 0

grundic
grundic

Reputation: 4921

I don't know if it's possible to do using built-in features of GoCD, but it should be definitely be possible using REST API.

Knowing current pipeline name, you can get all available stages of it and calculate previous one. Next, using a possibility to download an artifact directory as zip archive, you can get what you want. So you can add this as a script for the second stage, which will get the zipped artifact and after that you can continue with testing.

To do that, I can recommend my implementation of GoCD API - yagocd. This python library would let you to program aforementioned logic in a natural way. Some tips:

  • you can get current pipeline name from GO_PIPELINE_NAME (there are a lot environment variables at your service)
  • to find pipeline by name you can use PipelineManager: go.pipelines.get($GO_PIPELINE_NAME, $GO_PIPELINE_COUNTER)
  • having pipeline instance, you can iterate stages by pipeline_instance.stages object
  • having a stage, you can get it job and download directory by some path using go.artifacts.directory_wait method

If you would have questions about the implementation I can try to help you.

Upvotes: 1

Related Questions