How to feed new vectors into recurrent and convolutional keras model for real-time/streaming/live inference?

Question

I have successfully trained a Keras/TensorFlow model consisting of layers SimpleRNN→Conv1D→GRU→Dense. The model is meant to run on an Apple Watch for real time inference, which means I want to feed it with a new feature vector and predict a new output for each time step. My problem is that I don't know how to feed data into it such that the convolutional layer receives the latest k outputs from the RNN layer.

I can see three options:

1. Feed it with one feature vector at a time, i.e. (1,1,6). In this case I assume that the convolutional layer will receive only one time step and hence zero pad for all the previous samples.
2. Feed it with the last k feature vectors for each time step, i.e. (1,9,6), where k = 9 is the CNN kernel length. In this case I assume that the state flow in the recurrent layers will not work.
3. Feed it with the last k feature vectors every k:th time step, again where k = 9 is the CNN kernel length. I assume this would work, but introduces unnecessary latency that I wish to avoid.

What I want is a model that I can feed with a new single feature vector for each time step, and it will automatically feed the last k outputs of the SimpleRNN layer into the following Conv1D layer. Is this possible with my current model? If not, can I work with the layer arguments, or can I introduce some kind of FIFO buffer layer between the SimpleRNN and Conv1D layer?

Here is my current model:

feature_vector_size = 6
model = tf.keras.models.Sequential([
    Input(shape=(None, feature_vector_size)),
    SimpleRNN(16, return_sequences=True, name="rnn"),
    Conv1D(16, 9, padding="causal", activation="relu"),
    GRU(12, return_sequences=True, name="gru"),
    Dropout(0.2),
    Dense(1, activation=tf.nn.sigmoid, name="dense")
])

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
rnn (SimpleRNN)              (None, None, 16)          368       
_________________________________________________________________
conv1d (Conv1D)              (None, None, 16)          2320      
_________________________________________________________________
gru (GRU)                    (None, None, 12)          1080      
_________________________________________________________________
dropout (Dropout)            (None, None, 12)          0         
_________________________________________________________________
dense (Dense)                (None, None, 1)           13        
=================================================================

Edit:

After having researched the problem a bit, I have realized:

• The Conv1D layer will zero pad in all three cases that I described, so option 3 won't work either. Setting padding="valid" solves this particular problem.
• The SimpleRNN and GRU layers must have stateful=True. I found this description of how to make a model stateful after it has been trained stateless: How to implement a forward pass in a Keras RNN in real-time?
• Keras sequence models seem to be made for complete, finite sequences only. The infinite streaming use case with one time step at a time isn't really supported.

However, the original question remains open: How can I build and/or feed new feature vectors into the model such that the convolutional layer receives the latest k outputs from the RNN layer?

Answer 1

For anyone else with the same problem: I couldn't solve the SimpleRNN to Conv1D data flow easily, so I ended up replacing the SimpleRNN layer with another Conv1D layer and setting padding="valid" on both Conv1D layers. The resulting model outputs exactly one time step when fed with a sequence of c * k - 1 time steps, where c is the number of Conv1D layers and k is the convolutional kernel length (c = 2 and k = 9 in my case):

feature_vector_size = 6
model = tf.keras.models.Sequential([
    Input(shape=(None, feature_vector_size)),
    Conv1D(16, 9, padding="valid", name="conv1d1"),
    Conv1D(16, 9, padding="valid", name="conv1d2"),
    GRU(12, return_sequences=True, name="gru"),
    Dropout(0.2),
    Dense(1, activation=tf.nn.sigmoid, name="dense")
])

After training, I make the GRU layer stateful according to How to implement a forward pass in a Keras RNN in real-time?. For real-time inference I keep a FIFO queue of the 17 latest feature vectors and feed all these 17 vectors into the model as an input sequence for each new time step.

I don't know if this is the best possible solution, but at least it works.