CODEWITHSUNDEEP
tensorflowtensorflow-datasetstensorflow-estimatortf.keras
user1779012
user1779012

Reputation: 114

How to build input function for a estimator created from keras model

I'm creating an estimator from keras model as below

estimator = tf.keras.estimator.model_to_estimator(keras_model=keras_model, model_dir=model_dir)

my model is like 

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
main_input (InputLayer)      (None, 8)                 0         
_________________________________________________________________
dense1 (Dense)               (None, 50)                450       
_________________________________________________________________
dense2 (Dense)               (None, 40)                2040      
_________________________________________________________________
dense3 (Dense)               (None, 30)                1230      
_________________________________________________________________
dense4 (Dense)               (None, 20)                620       
_________________________________________________________________
dense5 (Dense)               (None, 10)                210       
_________________________________________________________________
main_output (Dense)          (None, 8)                 88        
=================================================================
Total params: 4,638
Trainable params: 4,638
Non-trainable params: 0

Then I try to create an input_fn for the estimator as 

def train_input_fn():
    dataset = csv_input_fn(training_data_path)
    dataset = dataset.batch(128).repeat(-1)
    train_iterator = dataset.make_one_shot_iterator()
    features, labels = train_iterator.get_next()
    return features, labels

def csv_input_fn(csv_path, batch_size=None, buffer_size=None, repeat=None):
    dataset = tf.data.TextLineDataset(filenames).skip(0)
    dataset = dataset.map(_parse_line)

    if buffer_size is not None:
        dataset = dataset.shuffle(buffer_size=10000)

    if batch_size is not None:
        dataset = dataset.batch(batch_size)

    if buffer_size is not None:
        dataset = dataset.repeat(repeat)

    return dataset

def _parse_line(line):
    fields = tf.decode_csv(line, FIELD_DEFAULTS)
    features = dict(zip(COLUMNS, fields))
    features.pop("DATE")
    label = features.pop("LABEL")
    return features, label

But there is an error

KeyError: "The dictionary passed into features does not have the expected inputs keys defined in the keras model.
Expected keys: {'main_input'}
features keys: {'TURNOVER', 'VOLUME', 'CLOSE', 'P_CHANGE', 'OPEN', 'PRICE_CHANGE', 'LOW', 'HIGH'}
Difference: {'VOLUME', 'CLOSE', 'LOW', 'P_CHANGE', 'main_input', 'OPEN', 'PRICE_CHANGE', 'TURNOVER', 'HIGH'}"

Looks like {'main_input'} is the input name in the keras model {'TURNOVER', 'VOLUME', 'CLOSE', 'P_CHANGE', 'OPEN', 'PRICE_CHANGE', 'LOW', 'HIGH'} are features from my dataset, so they are not matched to each other. Does anyone know how to convert this?

Upvotes: 2

Views: 943

Answers (2)

Sharky
Sharky

Reputation: 4533

Try using tf.data.experimental.make_csv_dataset. It accepts a single csv file or a list of files. It also handles batching and shuffling, so you don't have to apple them explicitly. 

dataset = tf.data.experimental.make_csv_dataset('file.csv', batch, ...)

This will return a batch of type OrderedDict, so you have to apply some parse function.

Another way is to use CsvDataset class

dataset = tf.data.experimental.CsvDataset('file.csv', [dtype]).batch(1)

It requires record_defaults parameter, a list of dtypes of values in file. It is a standard dataset object, so you need to apply shulle, batch, and whatever parse function will suit your data

https://www.tensorflow.org/api_docs/python/tf/data/experimental/CsvDataset https://www.tensorflow.org/versions/r1.13/api_docs/python/tf/data/experimental/make_csv_dataset

Upvotes: 0

Manoj Mohan
Manoj Mohan

Reputation: 6044

Yes, you can convert the feature columns to numpy array and feed it to the model like this.

# Simulate csv data
x = np.random.randn(100,8)
df = pd.DataFrame(data=x, columns=['TURNOVER', 'VOLUME', 'CLOSE', 'P_CHANGE', 'OPEN', 'PRICE_CHANGE', 'LOW', 'HIGH'])

# Convert df to array
train_data = df.to_numpy() # requires pandas 0.24 else use df.values
train_labels = np.zeros((100,8))

train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={model.input_names[0]: train_data},  # input_names[0] would be 'main_input'
y=train_labels,
batch_size=100,
num_epochs=None,
shuffle=True)  

estimator = tf.keras.estimator.model_to_estimator(model)
estimator.train(input_fn=train_input_fn, steps=1)

https://www.tensorflow.org/guide/estimators#creating_estimators_from_keras_models

Upvotes: 1

Related Questions