Jenkins EC2 plugin NullPointerException on Spot Requests

Question

After the weekend I have come in to find my Jenkins a bit worse for wear. It seems the EC2 plugin is failing on something to do with Spot instances.

I use the EC2 Plugin with Spot instances for agents and this has been working happily for a long time.

I have tried removing the plugin and Jenkins loads no issue.

I have tried deleting the plugin and reinstalling it and on restart the same issue occurs. I have tried rolling back to an old version of the plugin (2.0.6) and am seeing the same issues.

Tried adding administrator IAM permissions to the nodes role and that made no difference either.

At bit of a loss where to look next.

The Jenkins is running the latest LTS container which is showing as Jenkins 2.414.1

2023-09-18 11:33:25.098+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2023-09-18 11:33:25.592+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
java.lang.NullPointerException
    at hudson.plugins.ec2.EC2SpotSlave.getSpotRequest(EC2SpotSlave.java:137)
    at hudson.plugins.ec2.EC2SpotSlave.getInstanceId(EC2SpotSlave.java:179)
    at hudson.plugins.ec2.EC2Computer.getInstanceId(EC2Computer.java:69)
    at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:187)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:279)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:53)
    at hudson.model.AbstractCIBase.createNewComputerForNode(AbstractCIBase.java:192)
    at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:153)
    at hudson.model.AbstractCIBase$1.run(AbstractCIBase.java:255)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:238)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1693)
    at jenkins.model.Nodes$6.run(Nodes.java:351)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at jenkins.model.Nodes.load(Nodes.java:346)
    at jenkins.model.Jenkins$12.run(Jenkins.java:3453)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-09-18 11:33:25.597+0000 [id=22]    SEVERE  hudson.util.BootFailure#publish: Failed to initialize Jenkins
java.lang.NullPointerException
    at hudson.plugins.ec2.EC2SpotSlave.getSpotRequest(EC2SpotSlave.java:137)
    at hudson.plugins.ec2.EC2SpotSlave.getInstanceId(EC2SpotSlave.java:179)
    at hudson.plugins.ec2.EC2Computer.getInstanceId(EC2Computer.java:69)
    at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:187)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:279)
    at hudson.plugins.ec2.EC2RetentionStrategy.start(EC2RetentionStrategy.java:53)
    at hudson.model.AbstractCIBase.createNewComputerForNode(AbstractCIBase.java:192)
    at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:153)
    at hudson.model.AbstractCIBase$1.run(AbstractCIBase.java:255)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:238)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1693)
    at jenkins.model.Nodes$6.run(Nodes.java:351)
    at hudson.model.Queue._withLock(Queue.java:1397)
    at hudson.model.Queue.withLock(Queue.java:1271)
    at jenkins.model.Nodes.load(Nodes.java:346)
    at jenkins.model.Jenkins$12.run(Jenkins.java:3453)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:290)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1201)
    at jenkins.model.Jenkins.<init>(Jenkins.java:989)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:247)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:264)
2023-09-18 11:33:25.611+0000 [id=22]    INFO    hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins

Answer 1

Clearing out /var/jenkins_home/nodes did the trick. Never even knew it existed.

(side note, I also had to update my JCasC for EC2 due to a breaking change https://github.com/jenkinsci/ec2-plugin/releases/tag/1628.v6d7b_fc58b_a_1d)