CODEWITHSUNDEEP
jenkinslockingjenkins-plugins
Adam Cohen
Adam Cohen

Reputation: 91

Jenkins Dynamic Locking

I am an automation engineer, and I use Jenkins for our automated tests. I have to test each test on multiple platforms, so a build may have these parameters;

OS (Windows 7, Windows 8, XP 64bit, XP 32bit, etc...)

Server (Server of our product, version x, version y, etc...)

Product Version (x, y, etc...)

And more...

The OS chosen determines which VM (Virtual Machine) will be used as the testing grounds.

The thing is, I have many such tests, and those who run the tests do not always check what VM is already in use, or if they set an automatic test during another automatic test's time with a specific VM.

I want the build to wait until the VM is clear to be used.

I tried to play around with the Locks and Latches plugin - changing the plugin to check each lock if it's name appears in the build parameters, and if it does, check it's value. So if the lock's name is "OS Type", and the build has the parameter "OS Type = Windows 7" it would mean the build searches for the lock "Windows 7" to see if it is free or not.

I managed to do the above part - but now when I run the tests, the first test builds it's environment, and the other tests wait for it to finish the entire build, without even checking for locks! Thanks to that, I don't even know if what I did works.

Can anyone help? Did anyone do something like that? I will post the code below, but as I said, I am not sure if it works as intended. Thanks in advance!

public class LockWrapper extends BuildWrapper implements ResourceActivity {
private List<LockWaitConfig> locks;

public LockWrapper(List<LockWaitConfig> locks) {
    for(LockWaitConfig lock : locks)
    {

    }
    this.locks = locks;
}

public List<LockWaitConfig> getLocks() {
    return locks;
}

public void setLocks(List<LockWaitConfig> locks) {
    this.locks = locks;
}

@Override
public Descriptor<BuildWrapper> getDescriptor() {
    return DESCRIPTOR;
}

@Extension
public static final DescriptorImpl DESCRIPTOR = new DescriptorImpl();

/**
 * @see ResourceActivity#getResourceList()
 */
public ResourceList getResourceList() {
    ResourceList resources = new ResourceList();
    for (LockWaitConfig lock : locks) {
        resources.w(new Resource(null, "dynamic-locks/" + lock.getName(), DESCRIPTOR.getWriteLockCount()));
    }
    return resources;
}

@Override
public Environment setUp(AbstractBuild abstractBuild, Launcher launcher, BuildListener buildListener) throws IOException, InterruptedException {
    final List<NamedReentrantLock> backups = new ArrayList<NamedReentrantLock>();
    List<LockWaitConfig> locks = new ArrayList<LockWaitConfig>(this.locks);

    // sort this list of locks so that we _always_ ask for the locks in order
    Collections.sort(locks, new Comparator<LockWaitConfig>() {
        public int compare(LockWaitConfig o1, LockWaitConfig o2) {
            return o1.getName().compareTo(o2.getName());
        }
    });

    // build the list of "real" locks
    for (LockWaitConfig lock : locks) {
        NamedReentrantLock backupLock;
        String varName = lock.getName();
        String temp = varName;
        if(abstractBuild.getBuildVariables().containsKey(varName))
        {
            temp = abstractBuild.getBuildVariables().get(varName).toString();
            buildListener.getLogger().println("Variable " + varName + " found, replacing it with the value '" + temp + "'");
        }
        do {
            backupLock = DESCRIPTOR.backupLocks.get(temp);
            if (backupLock == null) {
                DESCRIPTOR.backupLocks.putIfAbsent(temp, new NamedReentrantLock(temp));
            }
        } while (backupLock == null);
        backups.add(backupLock);
    }

    final StringBuilder locksToGet = new StringBuilder();
    CollectionUtils.forAllDo(backups, new Closure() {
        public void execute(Object input) {
            locksToGet.append(((NamedReentrantLock) input).getName()).append(", ");
        }
    });

    buildListener.getLogger().println("[Dynamic Locks] Locks to get: " + locksToGet.substring(0, locksToGet.length()-2));

    boolean haveAll = false;
    while (!haveAll) {
        haveAll = true;
        List<NamedReentrantLock> locked = new ArrayList<NamedReentrantLock>();

        DESCRIPTOR.lockingLock.lock();
        try {
            for (NamedReentrantLock lock : backups) {
                buildListener.getLogger().print("[Dynamic Locks] Trying to get " + lock.getName() + "... ");
                if (lock.tryLock()) {
                    buildListener.getLogger().println(" Success");
                    locked.add(lock);
                } else {
                    buildListener.getLogger().println(" Failed, releasing all locks");
                    haveAll = false;
                    break;
                }
            }
            if (!haveAll) {
                // release them all
                for (ReentrantLock lock : locked) {
                    lock.unlock();
                }
            }
        } finally {
            DESCRIPTOR.lockingLock.unlock();
        }

        if (!haveAll) {
            buildListener.getLogger().println("[Dynamic Locks] Could not get all the locks, sleeping for 1 minute...");
            TimeUnit.SECONDS.sleep(60);
        }
    }

    buildListener.getLogger().println("[Dynamic Locks] Have all the locks, build can start");

    return new Environment() {
        @Override
        public boolean tearDown(AbstractBuild abstractBuild, BuildListener buildListener) throws IOException, InterruptedException {
            buildListener.getLogger().println("[Dynamic Locks] Releasing all the locks");
            for (ReentrantLock lock : backups) {
                lock.unlock();
            }
            buildListener.getLogger().println("[Dynamic Locks] All the locks released");
            return super.tearDown(abstractBuild, buildListener);
        }
    };
}

public void makeBuildVariables(AbstractBuild build, Map<String,String> variables) {
    final StringBuilder names = new StringBuilder();
    for (LockWaitConfig lock : locks) {
        if (names.length() > 0) {
            names.append(',');
        }
        names.append(lock.getName());
    }
    variables.put("LOCKS", names.toString());
}

public String getDisplayName() {
    return DESCRIPTOR.getDisplayName();
}

public static final class DescriptorImpl extends Descriptor<BuildWrapper> {
    private List<LockConfig> locks;

    /**
     * Required to work around HUDSON-2450.
     */
    private transient ConcurrentMap<String, NamedReentrantLock> backupLocks =
            new ConcurrentHashMap<String, NamedReentrantLock>();

    /**
     * Used to guarantee exclusivity when a build tries to get all its locks.
     */
    private transient ReentrantLock lockingLock = new ReentrantLock();

    DescriptorImpl() {
        super(LockWrapper.class);
        load();
    }

    public String getDisplayName() {
        return "Locks";
    }


    @Override
    public BuildWrapper newInstance(StaplerRequest req, JSONObject formData) throws FormException {
        List<LockWaitConfig> locks = req.bindParametersToList(LockWaitConfig.class, "locks.locks.");
        return new LockWrapper(locks);
    }

    @Override
    public boolean configure(StaplerRequest req, JSONObject formData) throws FormException {
        req.bindParameters(this, "locks.");
        locks = req.bindParametersToList(LockConfig.class, "locks.lock.");
        save();
        return super.configure(req, formData);
    }

    @Override
    public synchronized void save() {
        // let's remove blank locks
        CollectionUtils.filter(getLocks(), new Predicate() {
            public boolean evaluate(Object object) {
                return StringUtils.isNotBlank(((LockConfig) object).getName());
            }
        });

        // now, we can safely sort remaining locks
        Collections.sort(this.locks, new Comparator<LockConfig>() {
            public int compare(LockConfig lock1, LockConfig lock2) {
                return lock1.getName().compareToIgnoreCase(lock2.getName());
            }
        });

        super.save();
    }

    public List<LockConfig> getLocks() {
        if (locks == null) {
            locks = new ArrayList<LockConfig>();
            // provide default if we have none
            locks.add(new LockConfig("(default)"));
        }
        return locks;
    }

    public void setLocks(List<LockConfig> locks) {
        this.locks = locks;
    }

    public LockConfig getLock(String name) {
        for (LockConfig host : locks) {
            if (name.equals(host.getName())) {
                return host;
            }
        }
        return null;
    }

    public String[] getLockNames() {
        getLocks();
        String[] result = new String[locks.size()];
        for (int i = 0; i < result.length; i++) {
            result[i] = locks.get(i).getName();
        }
        return result;
    }

    public void addLock(LockConfig hostConfig) {
        locks.add(hostConfig);
        save();
    }

    /**
     * There wass a bug in the ResourceList.isCollidingWith,
     * this method used to determine the hack workaround if the bug is not fixed, but now only needs to
     * return 1.
     */
    synchronized int getWriteLockCount() {
        return 1;
    }
}

public static final class LockConfig implements Serializable {
    private String name;
    private transient AbstractBuild owner = null;

    public LockConfig() {
    }

    @DataBoundConstructor
    public LockConfig(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        LockConfig that = (LockConfig) o;

        if (name != null ? !name.equals(that.name) : that.name != null) return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result;
        result = (name != null ? name.hashCode() : 0);
        return result;
    }
}

public static final class LockWaitConfig implements Serializable {
    private String name;
    private transient LockConfig lock;

    public LockWaitConfig() {
    }

    @DataBoundConstructor
    public LockWaitConfig(String name) {
        this.name = name;
    }

    public LockConfig getLock() {
        if (lock == null && name != null && !"".equals(name)) {
            setLock(DESCRIPTOR.getLock(name));
        }
        return lock;
    }

    public void setLock(LockConfig lock) {
        this.lock = lock;
    }

    public String getName() {
        if (lock == null) {
            return name;
        }
        return name = lock.getName();
    }

    public void setName(String name) {
        setLock(DESCRIPTOR.getLock(this.name = name));
    }

}

/**
 * Extends {@code ReentrantLock} to add a {@link #name} attribute (mainly
 * for display purposes).
 */
public static final class NamedReentrantLock extends ReentrantLock {
    private String name;

    public NamedReentrantLock(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

private static final Logger LOGGER = Logger.getLogger(LockWrapper.class.getName());

}

What I changed is basically this;

for (LockWaitConfig lock : locks) {
        NamedReentrantLock backupLock;
        String varName = lock.getName();
        String temp = varName;
        if(abstractBuild.getBuildVariables().containsKey(varName))
        {
            temp = abstractBuild.getBuildVariables().get(varName).toString();
            buildListener.getLogger().println("Variable " + varName + " found, replacing it with the value '" + temp + "'");
        }
        do {
            backupLock = DESCRIPTOR.backupLocks.get(temp);
            if (backupLock == null) {
                DESCRIPTOR.backupLocks.putIfAbsent(temp, new NamedReentrantLock(temp));
            }
        } while (backupLock == null);
        backups.add(backupLock);
    }

To clarify with another example (Thank you Peter Schuetze for bringing this up)

I am trying to run different jobs that may have the same resource (testing environment)

For this example I will have two different jobs;

Job A runs some tests on any VM I choose.

Job B runs some other test on any VM I choose.

If I choose Job A to run on VM 'Windows 7', and someone else tries to run Job B on VM 'Windows 7' after Job A started running, I want Job B to be blocked until Job A is finished.

I could have many Job A and Job B variants, each set to work on a different VM, but considering my platform matrix, it would be too much to handle.

If I want to avoid using the Locks plugin, the test list will look like that;

Please consider that in reality I have ... Around 20 jobs right now, each using more or less the same resources (Testing environments, servers, etc...)

Right now I have made it so my job list is like that;

And to make sure no resource is used at the same time by more than one job, I need the locks plugin, and I need it to accept a variable as a parameter.

If you have any further questions or a need for clarification, please feel free to ask :)

Upvotes: 2

Views: 2479

Answers (1)

Peter Schuetze
Peter Schuetze

Reputation: 16305

Just to recap. You modified the logs and latches plugin to accept dynamic labels. You now try to run the different tests with the same job.

Did you configure the job to run concurrently? This setting allows to instances of the same job to run in parallel.

BTW, this sounds like the typical use case for a multi-configuration project.

Upvotes: 1

Related Questions