CODEWITHSUNDEEP
pythonmachine-learningscikit-learnneural-networkbackpropagation
user1368271
user1368271

Reputation:

MLPClassifier parameter setting

I'm developing a project which uses Backpropation algorithm. So I'm learning Backpropagation algorithm in scikit-learn. 

mlp = MLPClassifier(hidden_layer_sizes=(hiddenLayerSize,), solver='lbfgs', learning_rate='constant',learning_rate_init=0.001, max_iter=100000, random_state=1)

There are different solver options as lbfgs, adam and sgd and also activation options. Are there any best practices about which option should be used for backpropagation?

Upvotes: 3

Views: 8242

Answers (2)

Lucian
Lucian

Reputation: 824

Another thing to consider is that the learning rate should not be too large when the activation function is ReLu.

enter image description here

The main issue with the ReLu function is the so called 'Dying Relu' problem. A neuron is considered dead when it is stuck in the negative side and it is most likely to occur when the learning rate is too large.

Upvotes: 0

Ishant Mrinal
Ishant Mrinal

Reputation: 4918

solver is the argument to set the optimization algorithm here. In general setting sgd (stochastic gradient descent) works best, also it achieves faster convergence. While using sgd you apart from setting the learning_rate you also need to set the momentum argument (default value =0.9 works).

activation functions option is for, to introduce non-linearity of the model, if your model has many layers you have to use activation function such as relu (rectified linear unit) to introduce no-linearity, else using multiple layers become useless. relu is the most simplest and most useful activation function.

Upvotes: 3

Related Questions