Maxijazz
Reputation: 185
How can I create irregularly shaped networks in Tensorflow and Keras?
I'm making a neural network with tensorflow 2 and keras, but unlike all the tutorials I have found, my net is sort of irregularly shaped:
It consists of an input layer, 7 hidden layers, and one output layer. However, the input and first 4 hidden layers are split into two, and then later the two parts converge in the 5th hidden layer and from then on the network functions as normal. It is built this way because the network is meant to compare two positions and predict which is better. If you're interested in the concept, it's based on this research paper.
I do not know how to make a network shaped this in keras and tensorflow, but here is my attempt:
import tensorflow as tf
x_train, z_train, y_train = list(int(i) for i in "100000000000010000000000001000000000000100000000000010000000001000000"
"000010000000000100000000000000001000000000001000000000001000000000001"
"000000000001000000000001000000000001000000000001000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000001000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000001000000000001000000000001000000000"
"001000000000000000000000001000000000001000000000001000000100000000000"
"010000000000001000000000000100000000000010000000001000000000010000000"
"00010000001111000000000000000000000000"), \
list(int(i) for i in "100000000000010000000000001000000000000100000000000010000000001000000"
"000010000000000100000000000000001000000000001000000000001000000000001"
"000000000001000000000001000000000001000000000001000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000001000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000001000000000001000000000001000000000"
"001000000000000000000000001000000000001000000000001000000100000000000"
"010000000000001000000000000100000000000010000000001000000000010000000"
"00010000001111000000000000000000000000"), [0, 1]
train_dataset = tf.data.Dataset
network_input = tf.keras.Input(shape=(1594,))
model_a = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output1 = model_a(network_input[:797])
model_b = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output2 = model_b(network_input[797:])
x = tf.keras.layers.Concatenate()([output1, output2])
model_c = tf.keras.Sequential(
[tf.keras.layers.Dense(200, activation="sigmoid"),
tf.keras.layers.Dense(100, activation="sigmoid"),
tf.keras.layers.Dense(2, activation="sigmoid")]
)
output = model_c(x)
model = tf.keras.Model(network_input, output)
# compile the models
model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy'])
# run the models
x_train = tf.convert_to_tensor(x_train)
z_train = tf.convert_to_tensor(z_train)
y_train = tf.convert_to_tensor(y_train)
p = tf.keras.layers.Concatenate()([x_train, z_train]), y_train
model.fit(p)
which gives the following error:
/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 "/Users/max/Desktop/ArmstrongChess/error recreation.py"
2020-11-02 21:29:53.387241: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-02 21:29:53.450960: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff74fd79dd0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-02 21:29:53.450979: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
File "/Users/max/Desktop/ArmstrongChess/error recreation.py", line 65, in <module>
model.fit(p)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1049, in fit
data_handler = data_adapter.DataHandler(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1105, in __init__
self._adapter = adapter_cls(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 282, in __init__
raise ValueError(msg)
ValueError: Data cardinality is ambiguous:
x sizes: 1594, 2
Please provide data which shares the same first dimension.
Process finished with exit code 1
basically, I am trying to create it as three networks, the first two of which feed into the last to replicate the diagram above.
Can anyone explain how to do this? I'm a bit stuck.
Here is the code I tried according to tushv89's answer. I added some comments to let you know about the situation:
import tensorflow as tf
# in my actual project I have a function that creates this training data from a textfile of game PGNs, so what I call is this: "x_train, z_train, y_train = randomize_data(["won", "drawn", "lost"])". The following is just the information for one game so I don't have to paste a massive function here.
x_train, z_train, y_train = list(int(i) for i in "100000000000010000000000001000000000000100000000000010000000001000000"
"000010000000000100000000000000001000000000001000000000001000000000001"
"000000000001000000000001000000000001000000000001000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000001000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000001000000000001000000000001000000000"
"001000000000000000000000001000000000001000000000001000000100000000000"
"010000000000001000000000000100000000000010000000001000000000010000000"
"00010000001111000000000000000000000000"), \
list(int(i) for i in "100000000000010000000000001000000000000100000000000010000000001000000"
"000010000000000100000000000000001000000000001000000000001000000000001"
"000000000001000000000001000000000001000000000001000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000001000000000000000000000000000000000000000"
"000000000000000000000000000000000000000000000000000000000000000000000"
"000000000000000000000000000000000001000000000001000000000001000000000"
"001000000000000000000000001000000000001000000000001000000100000000000"
"010000000000001000000000000100000000000010000000001000000000010000000"
"00010000001111000000000000000000000000"), [0, 1]
# your code, copy pasted from here to the next comment
input = tf.keras.Input(shape=(1594,))
inp1, inp2 = tf.keras.layers.Lambda(lambda x: tf.split(x, 2, axis=1))(input)
model_a = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output1 = model_a(input)
model_b = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output2 = model_b(input)
x = tf.keras.layers.Concatenate()([output1, output2])
model_c = tf.keras.Sequential(
[tf.keras.layers.Dense(200, activation="sigmoid"),
tf.keras.layers.Dense(100, activation="sigmoid"),
tf.keras.layers.Dense(2, activation="sigmoid")]
)
output = model_c(x)
# your code ends here.
# I create the model with the input and output variables from your code
model = tf.keras.Model(inputs=input, outputs=output)
# I compile the model with my preferred settings (although I have no idea which would be the optimised settings, all I know is that I want sigmoid so the predictions are between 0 and 1)
model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy'])
# I convert the training variables to tensors
x_train = tf.convert_to_tensor(x_train)
z_train = tf.convert_to_tensor(z_train)
y_train = tf.convert_to_tensor(y_train)
# I call the network
model.fit(tf.keras.layers.Concatenate()([x_train, z_train]), y_train)
error code:
Traceback (most recent call last):
File "/Users/max/Desktop/ArmstrongChess/error recreation.py", line 61, in <module>
model.fit(tf.keras.layers.Concatenate()([x_train, z_train]), y_train)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1049, in fit
data_handler = data_adapter.DataHandler(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1105, in __init__
self._adapter = adapter_cls(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 282, in __init__
raise ValueError(msg)
ValueError: Data cardinality is ambiguous:
x sizes: 1594
y sizes: 2
Please provide data which shares the same first dimension.
Upvotes: 0
Views: 181
Answers (1)
thushv89
Reputation: 11343
Instead of using standard slicing, use tf.split. It's much cleaner.
input = tf.keras.Input(shape=(1594,))
inp1, inp2 = tf.keras.layers.Lambda(lambda x: tf.split(x, 2, axis=1))(input)
model_a = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output1 = model_a(input)
model_b = tf.keras.Sequential(
[tf.keras.layers.Dense(600, activation="sigmoid"),
tf.keras.layers.Dense(400, activation="sigmoid"),
tf.keras.layers.Dense(200, activation="sigmoid")]
)
output2 = model_b(input)
x = tf.keras.layers.Concatenate()([output1, output2])
model_c = tf.keras.Sequential(
[tf.keras.layers.Dense(200, activation="sigmoid"),
tf.keras.layers.Dense(100, activation="sigmoid"),
tf.keras.layers.Dense(2, activation="sigmoid")]
)
output = model_c(x)
Upvotes: 1
Related Questions
- How to build a neural network by a specific input and output shape layer?
- how to build a custom keras layer witth unkown input shape
- How do I create repeated structures in a Keras network?
- How are input tensors with different shapes fed to neural network?
- Tensorflow: Create a tensor with varying shapes in row dimension
- Can I make a Neural Network output matrices with irregular shapes?
- How to make a neural network with batches with different input shapes
- Simple network for arbitrary shape input
- How to create a custom connected neural network using tensorflow?
- Dynamic tensor shape for tensorflow RNN