(Keras) My CNN model training progress get stuck

Question

I developed my CNN model based on the repository [https://github.com/matterport/Mask_RCNN]. When I run the program(using cmd: coco.py train --dataset=/DATASET/COCO/2017 --model=None, I commended the loading statement to skip model weights loading), the process went through model building, coco dataset loading and then started to invoke model.train().

    # Create model
    if args.command == "train":
        model = modellib.MeshMask_RCNN(mode="training", config=config,
                                  model_dir=args.logs)
    else:
        model = modellib.MeshMask_RCNN(mode="inference", config=config,
                                  model_dir=args.logs)

    # Select weights file to load
    if args.model.lower() == "coco":
        model_path = COCO_MODEL_PATH
    elif args.model.lower() == "last":
        # Find last trained weights
        model_path = model.find_last()
    elif args.model.lower() == "imagenet":
        # Start from ImageNet trained weights
        model_path = IMAGENET_MODEL_PATH()
    else:
        model_path = args.model

    # Load weights
    print("Loading weights ", model_path)
    # model.load_weights(model_path, by_name=True)

    # Train or evaluate
    if args.command == "train":
        # Training dataset. Use the training set and 35K from the
        # validation set, as as in the Mask RCNN paper.
        dataset_train = CocoDataset()
        dataset_train.load_coco(args.dataset, "train", year=args.year, auto_download=args.download)
        if args.year in '2014':
            dataset_train.load_coco(args.dataset, "valminusminival", year=args.year, auto_download=args.download)
        dataset_train.prepare()

        # Validation dataset
        dataset_val = CocoDataset()
        val_type = "val" if args.year in '2017' else "minival"
        dataset_val.load_coco(args.dataset, val_type, year=args.year, auto_download=args.download)
        dataset_val.prepare()

        # Image Augmentation
        # Right/Left flip 50% of the time
        augmentation = imgaug.augmenters.Fliplr(0.5)

        # *** This training schedule is an example. Update to your needs ***
        # Training - Stage 0
        print("Fine tune all layers")

        #  get stuck when invoking this function #
>         model.train(dataset_train, dataset_val,
>                     learning_rate=config.LEARNING_RATE,
>                     epochs=160,
>                     layers='all',
>                     augmentation=augmentation)

In model.train() , it started to load images from disk, and the memory usage started to increase to about 80GB, and then all the progress got stuck(no training messages and the Cpu/Gpu usage rate are very low). I paused and found the program loops between line 404~406 in multiprocessing/pool.py.

    @staticmethod
    def _handle_workers(pool):
        thread = threading.current_thread()

        # Keep maintaining workers until the cache gets drained, unless the pool
        # is terminated.
404     while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
405         pool._maintain_pool()
406         time.sleep(0.1)
        # send sentinel to stop workers
        pool._taskqueue.put(None)
        util.debug('worker handler exiting')

Does this means there are some resources that had not met the demand, so it got stucked? I'm a newer to keras and tensorflow. Can any one help? Thanks.

amend: When I traced down, I found the exact statement where the program stuck in.

# tensorflow_core/python/client/session.py
class _Callable(object):

  def __init__(self, session, callable_options):
    self._session = session
    self._handle = None
    options_ptr = tf_session.TF_NewBufferFromString(
        compat.as_bytes(callable_options.SerializeToString()))
    try:
>     slef._handle = tf_session.TF_SessionMakeCallable(
>         session._session, options_ptr)

    finally:
      tf_session.TF_DeleteBuffer(options_ptr)

Answer 1

Actually, it's not stuck, it's just consumed too much time. I didn't realize how huge the model I was building was. I thought it got stuck because it cost almost a hour for tf to get ready to proceed after printing "epoch 1/160"(I realized that after leaving it running for a whole night).

The model itself is absolutely not able to train and will throw a OOM error after, so I need to redesign my model. Sorry for my mistake.

Answer 2

Make sure you are using tenorflow gpu:

import tensorflow-gpu

Also, add a device for tensorflow session

with tf.device('/gpu:0'):