CODEWITHSUNDEEP
pythonkeraslogistic-regression
Felix
Felix

Reputation: 323

Standard Errors from logistic regression in keras significance test

I have two independent variables (x1,x2), which I use to predict y (binary). With a logistic regression, I could just use the standard errors of each estimated coefficient to test them for significance.

However, I have a deep network that builds on inputA (some text) and inputB (numerical data).

This means, I would have to extract the standard errors from the last layer, to test the coefficient of inputB for significance. Otherwise, it would not be possible to check if inputB actually adds significantly to the model. How do I extract the standard errors from a logistic regression run in an deep learning model (keras)?

#### Network
# define two sets of inputs
inputA = Input(shape=(32,))
inputB = Input(shape=(128,))

# the first branch operates on the first input
x = Dense(8, activation="relu")(inputA)
x = Dense(4, activation="relu")(x)
x = Model(inputs=inputA, outputs=x)

# the second branch opreates on the second input
y = Dense(64, activation="relu")(inputB)
y = Dense(32, activation="relu")(y)
y = Dense(4, activation="relu")(y)
y = Model(inputs=inputB, outputs=y)

# combine the output of the two branches
combined = concatenate([x.output, y.output])

# our model will accept the inputs of the two branches and
# then output a single value
preds = Dense(1, activation='sigmoid',name='output')(combined) 
model = Model(inputs=[x.input, y.input], outputs=[preds])

model.compile(loss='binary_crossentropy',optimizer='adam', metrics=['acc'])

model.fit([x_train,numeric], y_train, epochs=40, batch_size=50)

Edit:

Now, I found this useful link:

https://stats.stackexchange.com/questions/89484/how-to-compute-the-standard-errors-of-a-logistic-regressions-coefficients

So I assume I could use y_pred = model.predict(inputs=[x_train,numeric], verbose=1) # gives probabilities

and then, I have to input combined into the following code... but how do I do that.. or is my approach erroneous?

#### Standard Error testing
# Design matrix -- add column of 1's at the beginning of your X_train matrix
X_design = np.hstack([np.ones((combined.shape[0], 1)), combined])

# Initiate matrix of 0's, fill diagonal with each predicted observation's variance
V = np.diagflat(np.product(y_preds, axis=1))


# Covariance matrix
covLogit = np.linalg.inv(np.dot(np.dot(X_design.T, V), X_design))

Can someone please add some suggestions / verification?

Edit2

The thing that confuses me is that I have two inputs, a numeric-input numeric and a non-numeric input x_train. To test the coefficients, I have to create a matrix in the shape of the combined-input (that's actually filled with the combined input).

Then I can use the models prediction, to test the last layers coefficients for significance (as described by the reference link for coefficient testing).

But how do I input the compined-input.. or am I somewhere mistaken?

Upvotes: 2

Views: 443

Answers (1)

Daniel Möller
Daniel Möller

Reputation: 86630

Not sure this answer is good for you....

What would I do?

  • Test a network with only inputA.
  • Test a network with only inputB.
  • Test the combined network.

Ther winner is the one I'll use.

Getting how much of each input the network is letting through:

If you get the weights of the last layer, you will have two tensors:

  • A (1, 8) weights matrix (or (8, 1), doesn't matter much in this case).
  • A (1,) bias value

Get them:

w, b = model.get_layer("output").get_weights()

Flatten w (it doesn't matter because you have only 1 output unit) and see how much the network is weighting each of the inputs. Following the order in which you concatenated x and y:

w = w.reshape((-1,))
weights_x = w[:4] #the first 4 weigths are multiplying `x.output`   
weights_y = w[4:] #the last 4 weights are multiplying `y.output`

Upvotes: 4

Related Questions