Partial_fit With Sgdclassifier Gives Fluctuating Accuracy

I have my data in a sparse matrix. I work now first on a subset with ~500k rows before starting the big computation. The data is bigram counts plus entropy and string length, and t

Solution 1:

Usually, partial_fit has seen to be prone to reduction or fluctuation in accuracy. To some extent, this can be slightly mitigated by shuffling and providing only small fractions of the entire dataset. But, for larger data, online training only seems to give reducing accuracies, with SGDClassifier/SVM Classifier.

I tried to experiment with it and discovered that using a low learning rate can help us sometimes. The rough analogy is, on training the same model on large data repeteadly, leads to the model forgetting what it learnt from the previous data. So, using a tiny learning rate slows down the rate of learning as well as forgetting!

Rather than manually providing a rate, we can use adaptive learning rate functionality provided by sklearn. Notice the model initialisation part,

model = SGDClassifier(loss="hinge", penalty="l2", alpha=0.0001, max_iter=3000, tol=None, shuffle=True, verbose=0, learning_rate='adaptive', eta0=0.01, early_stopping=False)

This is described in the [scikit docs] as:

‘adaptive’: eta = eta0, as long as the training keeps decreasing. Each time n_iter_no_change consecutive epochs fail to decrease the training loss by tol or fail to increase validation score by tol if early_stopping is True, the current learning rate is divided by 5.

I got really good results (from initial drop from 98% to 28% in fourth part of the dataset) to 100% model accuracy with the change in learning rate.