Create lmdb for multi label classification

Question

I am trying to create 2 lmdbs. One for my images one for my labels. I want to determine the angle of a picture, to do so I am trying to estimate the horizontal and the vertical angle. I have classes like: 0-10 degree horizontal 10-20 degree horiozntal and so on. And the same for vertical angles. Now I do not know how to create the label db as in how the labels have to be formatted in the lmdb. I have a .txt list file with: /path/pic.png 1 32 entries where 1 means 10-20 degree and 32 means 320-330 degrees. My code looks like this:

for line in fileinput.input(data):
    entries = re.split(' ', line.strip())
    images.append(entries[0])
    labels.append(entries[1:])
.... 
for in_idx, (image, label) in enumerate(inputs):
    im = cv2.imread(image)
    im = im[:,:,::-1]
    im = im.transpose((2,0,1))

    im_dat = caffe.io.array_to_datum(im)
    print im_dat

    images_txn.put('{:0>10d}'.format(in_idx), im_dat.SerializeToString())

    label = np.array(label).astype(int).reshape(1, 1, len(label))

    label_dat = caffe.io.array_to_datum(label)
    labels_txn.put('{:0>10d}'.format(in_idx), label_dat.SerializeToString())

This however seems not to be correct since I got following error when trying to train the network:

Check failed: outer_num_ * inner_num_ == bottom[1]->count() (10 vs. 20) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be NHW, with integer values in {0, 1, ..., C-1}.

My data layer looks like this:

name: "LearnAngle"
layer {
  name: "data"
  type: "Data"
  top: "images"
  include {
    phase: TRAIN
  }
  transform_param {
    mirror: true
    crop_size: 256
    mean_file: "/path/mean_train.binaryproto"
  }
  data_param {
    source: "path/train2_images_lmdb"
    batch_size: 10
    backend: LMDB
  }
}
layer {
  name: "data_label"
  type: "Data"
  top: "labels"
  include {
    phase: TRAIN
  }
  data_param {
    source: "path/train2_labels_lmdb"
    batch_size: 10
    backend: LMDB
  }
}

My last layers looks like

layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 36
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "labels"
  top: "loss"
}

Answer 1

The problem is: your top: "labels" in the 2nd data layer contains 2 kind of labels for the horizontal and vertical angle, while you just used 1 SoftmaxWithLoss layer for the labels.

In fact, to train 2 classification tasks in one network, you can create 2 lmdb databases for the 2 tasks' labels respectivly, and use 2 data layers to parse them to 2 SoftmaxWithLoss layers. Like what's below:

Code for creating lmdb for 2-task classifications:

for line in fileinput.input(data):
    entries = re.split(' ', line.strip())
    images.append(entries[0])
    horizontal_labels.append(entries[1])
    vertical_labels.append(entries[2])

...
for in_idx, (image, label) in enumerate(inputs):
    im = cv2.imread(image)
    im = im[:,:,::-1]
    im = im.transpose((2,0,1))

    im_dat = caffe.io.array_to_datum(im)
    print im_dat

    images_txn.put('{:0>10d}'.format(in_idx), im_dat.SerializeToString())
    horizontal_label = [label[0]]
    vertical_label = [label[1]]

    horizontal_label = np.array(horizontal_label).astype(int).reshape(1, 1, 1)
    vertical_label = np.array(vertical_label).astype(int).reshape(1, 1, 1)

    horizontal_label_dat = caffe.io.array_to_datum(horizontal_label)
    vertical_label_dat = caffe.io.array_to_datum(vertical_label)

    horizontal_labels_txn.put('{:0>10d}'.format(in_idx), horizontal_label_dat.SerializeToString())
    vertical_labels_txn.put('{:0>10d}'.format(in_idx), vertical_label_dat.SerializeToString())

train_val.prototxt:

name: "LearnAngle"
layer {
  name: "data"
  type: "Data"
  top: "images"
  include {
    phase: TRAIN
  }
  ...
}
layer {
  name: "horizontal_label"
  type: "Data"
  top: "horizontal_label"
  include {
    phase: TRAIN
  }
  data_param {
    source: "path/horizontal_labels_lmdb" #created using above python code
    ...
  }
}
layer {
  name: "vertical_label"
  type: "Data"
  top: "vertical_label"
  include {
    phase: TRAIN
  }
  data_param {
    source: "path/vertical_labels_lmdb" #created using above python code
    ...
  }
}
... #follow layers
# branch for horizontal label classification
layer {
  name: "fc_horizon"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc_horizon"
  ...
  inner_product_param {
    num_output: 36
    ...
  }
}

layer {
  name: "loss_horizon"
  type: "SoftmaxWithLoss"
  bottom: "fc_horizon"
  bottom: "horizontal_label"
  top: "loss_horizon"
}
# branch for vertical label classification
layer {
  name: "fc_vertical"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc_vertical"
  ...
  inner_product_param {
    num_output: 36
    ...
  }
}

layer {
  name: "loss_vertical"
  type: "SoftmaxWithLoss"
  bottom: "fc_vertical"
  bottom: "vertical_label"
  top: "loss_vertical"
}