How to change the parameter of a tf.keras.Sequential model directly from the variable it is stored in?

Question

Context :

I am working on a text classification problem in tensorflow. To illustrate the nature of my problem consider the following model as an example.

Note : to understand my problem you only need to look at the first 3 layers.

encoder = tf.keras.layers.TextVectorization( standardize = "lower_and_strip_punctuation", 
                                            output_mode = "int",
                                            output_sequence_length = 300) 

# have the encoder learn the vocabulay from the training data set  
encoder.adapt(X_train) 

model = tf.keras.Sequential([tf.keras.Input(shape=(1,), dtype=tf.string),
                             encoder,
                             tf.keras.layers.Embedding(input_dim = encoder.vocabulary_size(),
                                                       output_dim = 128
                                                           ),
                             tf.keras.layers.Conv1D(filters = 32,                             
                                                    kernel_size = 2,
                                                    activation = "relu"),                      
                             tf.keras.layers.Flatten(), 
                             tf.keras.layers.Dense(16, activation = "relu"),
                             tf.keras.layers.Dense(6, activation = "softmax")
                            ]
                           )

As you can see, the way I build this model requires some training data in order for the TextVectorizer layer to learn the vocabulary, and to set the correct dimensions for my embedding layer.

---

At the current stage I am trying to fine tune the parameters of my model. To do this I am using keras-tuner and using hypermodels from that library.

I therefore have to implement the build and fit methods in the class below

class MyHyperModel(kt.HyperModel): # inherits the hyper model class
    
    def build(self, hp): # construct a model
        
        # return a model
        
    def fit(self, hp, model, X, y): # fit the model obtained in "build" method to the data X,y
    
        # returns validation f1_score 

Issue :

This brings us to the issue I have. My model so far has always been built with access to the training data. However I do not have access to that data when implementing the build method.

Currently the build method will return a model as follows

encoder = tf.keras.layers.TextVectorization( standardize = "lower_and_strip_punctuation", 
                                            output_mode = "int",
                                            output_sequence_length = 300) 

# we do NOT adapt encoder to the training data because we do not have access here.
# encoder.adapt(X) 

model = tf.keras.Sequential([tf.keras.Input(shape=(1,), dtype=tf.string),
                             encoder,
                             tf.keras.layers.Embedding(input_dim = encoder.vocabulary_size(),
                                                       output_dim = 128
                                                           ),
                             tf.keras.layers.Conv1D(filters = 32,                             
                                                    kernel_size = 2,
                                                    activation = "relu"),                      
                             tf.keras.layers.Flatten(), 
                             tf.keras.layers.Dense(16, activation = "relu"),
                             tf.keras.layers.Dense(6, activation = "softmax")
                            ]
                           )

In order to fit the model and get the validation score I must therefore modify the model inside of the fit method where the model is stored in the model variable.

Note : technically I could just build a new model from scratch with the parameters I want inside of the fit method and use that one to compute the F1 score but this is not an elegant solution. I would rather modify the existing model.

I tried doing something like

model.get_layer(index = 0).adapt(X)

but this doesn't update the model.

How can I solve this issue?