CODEWITHSUNDEEP
pythonnumpytensorflow
Arun
Arun

Reputation: 2478

Reshape Python list to numpy array

I have a use-case where I need to make predictions from a trained LSTM model (TensorFlow = 2.8 and Python = 3.9) where the dataset is large and results in OOM error on my GPU card. The turnaround I have is to predict for each batch and store the resulting tensor in a Python3 list:

# Create TF dataset to iterate over-
train_dataset = tf.data.Dataset.from_tensor_slices(X_train).batch(batch_size)
# batch_size = 256

# Python3 list to contain all model predictions-
preditcions = []

for x in train_dataset:
    y_pred = model_e2d2.predict(x)
    preditcions.append(y_pred)

# Sanity check-
len(preditcions)
# (2048)

# Each batch of prediction has the size: (256, 5, 11).

# Sanity check-
preditcions[0].shape, preditcions[-1].shape
# ((256, 5, 11), (222, 5, 11))

The aim is to convert precitions Python3 list to a numpy array of desired shape: (524254, 5, 11)

?

Upvotes: 0

Views: 159

Answers (1)

MSS
MSS

Reputation: 3633

In general you can do it as follows:

predictions = np.concatenate(predictions, axis=0)

This gives you control on which axis you want to concatenate.

Look at the example given here https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html#numpy-concatenate

>>a = np.array([[1, 2], [3, 4]])
>>b = np.array([[5, 6]])
>>np.concatenate([a, b], axis=0) #slight difference from example.
  # instead of tuple I have passed a list as is OP's requirement
array([[1, 2],
       [3, 4],
       [5, 6]])

All thanks to @hpaulj for correcting my mistake.

Upvotes: 1

Related Questions