Multi Class character segmentation from handwritten words using UNet

Question

So, I have been training an UNet model to segment characters out of words. The model has learned the classes very well with an accuracy of 98% but it is not learning the shape of the content. Instead it gives the shape of the boxes or the masks used to tag the characters. The images I am using are of 144x416 dimension with a handwritten word in every single image and have annotated using VoTT. I trained UNet models with sigmoid activaton, before I was aware of multi class segmentation, and that model was able learn the shape of the content. I am now planning to overlap the output of this model on the output of the multiclass model and cropout the segments to have shape of the content.

I need some suggestions to train the model for multi class segmentation that learn the shape of the content along with the unique classes...