Change legend tick in image.plot

Question

Using this code

library(fields)
rot_mat = matrix(round(runif(25)*3+1),5,5)
image.plot(rot_mat,xaxt= "n", yaxt= "n", frame.plot=F,
             col = c('green', 'blue', 'red', 'yellow'))

I get the plot

Now I want the number 1,2,3,4 on the right side of the legend to strings, for example "a","b","c","d". I want the numbers to be right next to the color so that the label is clearly connected with the certain color.

The closest I have come is:

image.plot(rot_mat,xaxt= "n", yaxt= "n", frame.plot=F,
           col = c('green', 'blue', 'red', 'yellow'),
           lab.breaks=c('a','b','c','d', ""),
           breaks = c(1,2,3,4,5))

This gives me the plot

Which is close but the labels are too far down. You are more than welcome to suggest other methods than image.plot if you think something else is better but I need something that I can use with par(mfrow) so that I can make subplots and it needs to be fast for large (500*500) matrices. The reason I have numbers from 1-4 is that the matrice comes from a comparison of two adjacency matrices and 1-4 is numbers which correspond to true positive, false positive, true negative and false negative "connections".

Answer 1

image.plot(rot_mat,xaxt= "n", yaxt= "n", frame.plot=F,
           col = c('green', 'blue', 'red', 'yellow'), 
           axis.args = list(at = 1:4, labels=c('a','b','c','d')))