CODEWITHSUNDEEP
pythontensorflowobject-detection-api
kickButtowski
kickButtowski

Reputation: 1

Tensorflow object detection (ssd - mobilnet_v1) not converging

I am trying to train deep-fashion dataset on the tensorflow object detection api. The categories and corresponding number of images in each class are as follows.

{'Blouse': 24562, 'Cutoffs': 1669, 'Cardigan': 13311, 'Button-Down': 330, 'Poncho': 791, 'Chinos': 527, 'Top': 10078, 'Anorak': 160, 'Kimono': 2294, 'Blazer': 7497, 'Robe': 150, 'Parka': 676, 'Jodhpurs': 45, 'Sarong': 32, 'Shorts': 19624, 'Caftan': 54, 'Turtleneck': 146, 'Leggings': 5013, 'Jersey': 727, 'Hoodie': 4010, 'Culottes': 486, 'Sweater': 13123, 'Flannel': 324, 'Jeggings': 594, 'Tank': 15418, 'Onesie': 70, 'Coat': 2120, 'Sweatpants': 3021, 'Henley': 716, 'Jacket': 10425, 'Gauchos': 49, 'Sweatshorts': 1098, 'Romper': 7408, 'Joggers': 4405, 'Trunks': 386, 'Halter': 34, 'Dress': 144316, 'Jeans': 14152, 'Tee': 73333, 'Coverup': 34, 'Capris': 154, 'Kaftan': 252, 'Peacoat': 194, 'Bomber': 618, 'Skirt': 29546, 'Jumpsuit': 12306}

There are 46 classes in total. My total loss graph doesn't seem to go to lower values. The best till now has been around 4. This is my graph. Is this normal for so many classes or should I use another algorithm ?

Another doubt I have is that the model has been training for around 3 days now. I don't have a GPU. Should I stop the training process ?

Edit:

train_config: {
  batch_size: 24
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.004
          decay_steps: 800720
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }

Hardware Information: 

{
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              6
On-line CPU(s) list: 0-5
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           3
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Stepping:            2
CPU MHz:             1997.686
BogoMIPS:            3995.37
Hypervisor vendor:   VMware
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            35840K
NUMA node0 CPU(s):   0-5
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat
}

I am new to tensorflow. Any kind of help will be appreciated. Thanks in advance.

Upvotes: 0

Views: 1092

Answers (1)

netanel-sam
netanel-sam

Reputation: 1912

  1. The value of the total loss is not very informative unless you give the exact loss you're using, and also then it's not indicative unless you have experimented with this loss. Instead, a more informative values are the performance metrics of your model on your test set.
  2. Time is also not very indicative unless you state the hardware you're using. Instead - state the number of iterations and batch size, and the size of your training data or the number of epochs you trained the model.
  3. Have you used a decaying learning rate? If you use a constant learning rate: (1) with large value - then it will less probable to converge; (2) with small value - it may take a long time to converge.

Upvotes: 1

Related Questions