Neural Network Accuracy Is Always 0 While Training Classification Problem In Keras

I am making a neural network for the titanic classification problem but my training accuracy is always 0. I checked other solutions but couldn't find a solution that works. The los

Solution 1:

First, you are incorrectly using metrics=['accuracy']. Second, this points to a much deeper bug which I think is unintentional. I have raised an Issue for this on tensorflow repo. Let's hope someone responds.

Keras doesn't identify the metric Accuracy. Keras fails to call the MeanMetricWrapper which is required here properly.

---

Fix for the issue

Fixing that starts showing proper values for the metric.

from tensorflow import keras
from tensorflow.keras import layers

X_train = np.random.random((100,8))
y_train = np.random.randint(0,2,(100,))

model = keras.Sequential(
    [
     layers.Dense(10,activation="relu",input_shape=(8,)),
     layers.Dense(10,activation="relu"),
     layers.Dense(1,activation="sigmoid")
    ]
)

model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
model.fit(X_train,y_train,batch_size=64,epochs=5,verbose=2)

Epoch1/52/2 - 0s - loss: 0.6926 - accuracy:0.5500Epoch2/52/2 - 0s - loss: 0.6915 - accuracy:0.5500Epoch3/52/2 - 0s - loss: 0.6909 - accuracy:0.5600Epoch4/52/2 - 0s - loss: 0.6900 - accuracy:0.5700Epoch5/52/2 - 0s - loss: 0.6894 - accuracy:0.5600<tensorflow.python.keras.callbacks.Historyat0x7f90e5f7d250>

---

Understanding and tracing the issue

@Mathias Müller brought out a valid concern that if Keras doesn't identify the uppercase metric, then it should throw an error instead of running.

I traced the GitHub implementation to see how the current code works. Here are the steps. (I have linked highlighted lines of code from GitHub repo)

1. During compile, when a metric is passed, the parameter is stored in a MetricsContainer object.
2. This container class then calls a function called _get_metric_object from compile_utils.py. The job of this function is to take the input and return a metric object of that metric's class.
3. One of the first things this function does is to check if the input belongs to list ['accuracy', 'acc', 'crossentropy', 'ce'] or not.
   • If YES, then it directly fetches the classes from the metrics.py and calls the MeanMetricWrapper class. The job of this class is to wrap a stateless metric function with the Mean metric. This calculates the mean of the metric you have added.
   • If NO, then it calls the get function from metrics.py. The get function further calls a deserialize function whose job is call a function call deserialize_keras_object function from utils.generic_utils.py. This function's job is to take the string and retrieve the actual object!

Let's see the 2 scenarios now.

Scenario 1: "accuracy"

#With lower case accuracy
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
print(model.metrics)

[<tensorflow.python.keras.metrics.Meanat0x7f90c7ea10d0>,
 <tensorflow.python.keras.metrics.MeanMetricWrapperat0x7f90c7d07e20>]

Since the metric provided belongs to the ['accuracy', 'acc', 'crossentropy', 'ce'], the _get_metric_object function fetches the tf.keras.metrics.Accuracy class and explicitly passes it to the tf.keras.metrics.MeanMetricWrapper. This calculates the mean accuracy as expected.

Scenario 2: "Accuracy"

#With upper case Accuracy
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['Accuracy'])
print(model.metrics)

[<tensorflow.python.keras.metrics.Meanat0x7f90e7285e20>,
 <tensorflow.python.keras.metrics.Accuracyat0x7f90e72fceb0>]

Something interesting happens here. Since the "Accuracy" doesnt belong to the list, _get_metric_object calls the metrics.get() ->metrics.deserialize()-> generic_utils.deserialize_keras_object() function which simply pulls up the tf.keras.metrics.Accuracy and returns that directly, instead of calling tf.keras.metrics.MeanMetricWrapper.

This is why you get incorrect values for accuracy, but it does not throw an error!!