CODEWITHSUNDEEP
neural-networkvowpalwabbit
MaxBenChrist
MaxBenChrist

Reputation: 547

Vowpal Wabbit: Input of neural network?

In the machine learning tool vowpal wabbit (https://github.com/JohnLangford/vowpal_wabbit/), normally a linear estimator y*=wx is trained. However, it is possible to add a forward neural.

My question is: When I use the neural network by the command line option "-nn x", is the linear estimator wx completely replaced by an neural network?

Edit: Thanks Martin and arielf. So apperently the different constellations look like this:

enter image description here

The weights of the models with "--nn" are estimated by backpropagation?

Upvotes: 3

Views: 998

Answers (1)

arielf
arielf

Reputation: 5952

[Edit: corrected answer: original wasn't accurate, thanks Martin]

The 1-layer NN feeds input features into the NN layer (all possible interactions) which are then fed to the output layer.

In order to add pass-through features as-is, without interactions, you should add the --inpass option.

You can look at models created by using --invert_hash to get a readable model on a small example:

$ cat dat.vw
1 | a b
2 | a c

# default linear model, no NN:
$ vw --invert_hash dat.ih dat.vw
...

$ cat dat.ih
...
:0
Constant:116060:0.387717
a:92594:0.387717
b:163331:0.193097
c:185951:0.228943

# Now add --nn 2 (note double-dash in long option)
# to use a 1-layer NN with 2 nodes
$ vw --nn 2 --invert_hash dat-nn.ih dat.vw
...

$ cat dat-nn.ih
...
:0
Constant:202096:-0.270493
Constant[1]:202097:0.214776
a:108232:-0.270493
a[1]:108233:0.214776
b:129036:-0.084952
b[1]:129037:0.047303
c:219516:-0.196927
c[1]:219517:0.172029

Looks like a[N] is the contribution of a to hidden-layer NN node N (starting with base/index zero apparently, the standalone a notation is a bit confusing).

When you add --inpass you get an additional weight per feature (index [2]):

$ vw --nn 2 --inpass --invert_hash dat-nn-ip.ih dat.vw
...
$ cat dat-nn-ip.ih
...
:0
Constant:202096:-0.237726
Constant[1]:202097:0.180595
Constant[2]:202098:0.451169
a:108232:-0.237726
a[1]:108233:0.180595
a[2]:108234:0.451169
b:129036:-0.084570
b[1]:129037:0.047293
b[2]:129038:0.239481
c:219516:-0.167271
c[1]:219517:0.139488
c[2]:219518:0.256326

Upvotes: 4

Related Questions