CODEWITHSUNDEEP
antcontinuous-integrationivybamboo
usr-local-ΕΨΗΕΛΩΝ
usr-local-ΕΨΗΕΛΩΝ

Reputation: 26894

Separate cache folders for Ivy

I want to unleash the power of parallelism by running some Ivy-related Ant tasks on local Bamboo agents. Our Bamboo machine has plenties of CPU horse power and RAM.

If I split my build task into parallel jobs, each that will produce a different artifact based on the result of ivy-retrieve, I have solved my problem in the theory.

In practice, unfortunately, if two Ant tasks happen, for some reason, to run simultaneously on the same machine and the same organization-artifact, they will clash and one gets an XML error.

I don't have the exact error message with me because 1) the problem is randomic to reproduce and 2) I have already done lots of work to put all the jobs into a sequential one. But I have a clear idea of what is happening.

When Ant runs an ivy-retrieve, code below, it will use local user's cache directory, which happens to be /home/bamboo/.ivy2/cache. There I can find lots of resolved-[org]-[artifact]-[version].xml files (each is a different build version of my project). The problem occurs when I want to run the ivy-retrieve task twice, one for example for compile configuration and another for runtime. The two XMLs will clash and Ivy will report a SAX error in reading one of the files, because it looks like it's being written at the moment.

If I run the job on remote agents I expect no problem, but hey I already have 5 local agents and Bamboo won't fire remote agents if the locals are free.

Unfortunately all of my jobs, independent each other, require a different ivy-retrieve. Currently I run them in sequence.

The question is

Is it possible to tell Ivy, running on Bamboo agent, to use a temporary unique cache directory for its work on dependencies.xml files rather than to use global cache? Or at most to synchronize access to files?

The second option would have parallel Ant process read&write the cached dependencies.xml file mutually exclusively. So what they read will be always a consistent file (being the same exact file, I don't care if a process overwrites another)

Upvotes: 1

Views: 593

Answers (1)

Alexey Kiselev
Alexey Kiselev

Reputation: 946

Ivy has 2 caches - repository cache and resolution cache. The second one is overwritten each resolution, and should never be used by multiple processes at the same time.

  1. Set an environment variable pointing to a temporary directory in your bamboo agent.
  2. Create a separate ivysettings.xml file for your project.
  3. User an environment variable in project's ivysettings.xml to setup a cache directory.

Here is an example of ivysettings.xml:

<ivysettings>
  <properties environment="env" />
  <caches resolutionCacheDir="${env.TEMP_RESOLUTION_CACHE}" />

  <settings defaultResolver="local" />

  <statuses default="development">
    <status name="release" integration="false"/>
    <status name="integration" integration="true"/>
    <status name="development" integration="true"/>
  </statuses>

  ...    

</ivysettings>

Or you can give a try to a lock-strategy. I haven't tried it.

Upvotes: 3

Related Questions