how can I speed up my python code?

Question

It's a program of import multiple images and extract feature using dct and histogram. 1) Import multiple images from folder

2) Make image size 256*256

3) Use image of 64*64 block unit with stride = 32

4) Do dct(8*8 size)

5) make histogram of dct

6) Extract features from dct coefficient histogram

The problem is that it's too slow. I think it's because there's so many "for loop".

This is my full-code in python. How can I change my code to speed up?

I am not familiar with python. Please help me

import numpy as np
from scipy.fftpack import dct
from PIL import Image
import glob
import matplotlib.pyplot as plt

def find_index(x,key):
    for i in range(0,len(x)):
        if x[i] == key :
            return i
        else:
            i = i+1

def image_open(path):
    image_list = []
    #for filename in glob.glob('path/*.jpg'): 
    for filename in glob.glob(path+'/*.jpg'):  
     im=Image.open(filename)
     image_list.append(im)

    return image_list

def dct_2(img):  
    #Get 2D Cosine Transform of Image
    return dct(dct(np.asarray(img).T, norm='ortho').T, norm='ortho')


def return_array(array):
    zero = [0.0, 0.0, 0.0, 0.0, 0.0]

    range = int((max(array)) - min(array))
    x, bins, patch = plt.hist(array, bins=range)
    x = list(zero) + list(x) + list(zero)
    return x


path = 'C:\\Users\\LG\\PycharmProjects\\photo' #folder that contains many images

images = image_open(path)


row = 0
array_matrix = []
label_matrix = []

for i in range(0, len(images)): #access image    
 box3 = (0,0,256,256)
 a = images[i].crop(box3)

 (y,cb,cr) = a.split()  #ycbcr 
 width , height = y.size  
 y.show()

 for q in range(0, height-32 , 32):  #use image 64*64 block unit
  for w in range(0 , width-32 ,32):
     box1 =(q,w,q+64,w+64)
     block = y.crop(box1)
     array1 , array2 , array3 , array4 , array5 , array6 , array7 , array8 ,array9 = [],[],[],[],[],[],[],[],[]



     for j in range(0,64,8):        #dct
       for n in range(0,64,8):
         box2 = (j,n,j+8,n+8)
         temp = block.crop(box2)
         dct_temp = dct_2(temp)


         array1.append(dct_temp[0,1])
         array2.append(dct_temp[1,0])
         array3.append(dct_temp[0,2])
         array4.append(dct_temp[1,1])
         array5.append(dct_temp[2,0])
         array6.append(dct_temp[0,3])
         array7.append(dct_temp[1,2])
         array8.append(dct_temp[2,1])
         array9.append(dct_temp[3,0])


     x1 = return_array(array1)   #extract feature from dct histogram
     index = find_index(x1, max(x1))
     u = [index - 5, index + 5, 1]
     array_matrix.append(x1[u[0]:u[1] + 1:u[2]])

     x2 = return_array(array2)
     index = find_index(x2, max(x2))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x2[u[0]:u[1] + 1:u[2]])

     x3 = return_array(array3)
     index = find_index(x3, max(x3))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x3[u[0]:u[1] + 1:u[2]])

     x4 = return_array(array4)
     index = find_index(x4, max(x4))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x4[u[0]:u[1] + 1:u[2]])

     x5 = return_array(array5)
     index = find_index(x5, max(x5))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x5[u[0]:u[1] + 1:u[2]])

     x6 = return_array(array6)
     index = find_index(x6, max(x6))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x6[u[0]:u[1] + 1:u[2]])

     x7 = return_array(array7)
     index = find_index(x7, max(x7))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x7[u[0]:u[1] + 1:u[2]])

     x8 = return_array(array8)
     index = find_index(x8, max(x8))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x8[u[0]:u[1] + 1:u[2]])

     x9 = return_array(array9)
     index = find_index(x9, max(x9))
     u = [index - 5, index + 5, 1]
     array_matrix[row].extend(x9[u[0]:u[1] + 1:u[2]])

     print(w/32)

     row = row+1

print(array_matrix)

Answer 1

Rather than assuming that a specific section is taking longer than others, I'd recommend profiling your script. A profiler will collect metrics on how long certain parts of your program takes, and also allow you to better see how much any changes affect the code (makes it better, worse, etc).

Once you know where your problem lies, then you can take a more targeted approach at making it faster.

Have a look at the profiling module: https://docs.python.org/2/library/profile.html

Also have a look at some tutorials:

• https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara
• https://zapier.com/engineering/profiling-python-boss/
• https://marcobonzanini.com/2015/01/05/my-python-code-is-slow-tips-for-profiling/