You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let's apply softmax(nomalization) automatically when trying to use CategoricalCrossEntropy loss.
Why
There is no guarantee a pre-trained model has a softmax when trying to apply CategoricalCrossEntropy loss to the model.
To do
Investigate how softmax exists when a circle model is created from Pytorch and Tensorflow.
Pytorch : Circle models don't have softmax. Pytorch does not add softmax to models even when training with CategoricalCrossEntropy.
Tensorflow : Circle models may have softmax if users add softmax. Tensorflow trains well with CategoricalCrossEntropy by applying softmax only once even if there is softmax in the model.
Apply softmax to CategoricalCrossEntropy automatically
To apply normalization(softmax) automatically to categorical cross entropy, we need to consider that sum of labels is not 1.
That consideration will be deal with in another issue later. So, I'm closing this issue since I have completed all other required tasks.
4 cases of circle models pre-trained with CategoricalCrossEntropy can be created:
A model with softmax, trained by executing softmax once for each step (on tensorflow).
A model without softmax, trained by executing softmax once for each step(on tensorflow and pytorch).
A model with softmax, trained by executing softmax twice for each step(on pytorch).
A model without softmax, trained without executing softmax(on tensorflow).
Tensorflow does not allow training without softmax when using CategoricalCrossEntropy loss.
Pytorch only have CrossEntroy instead of CategoricalCrossEntropy. CrossEntropy means that cases other than softmax are allowed. So Pytorch only strictly executes CrossEntory loss logic.
I think It's OK to apply softmax automatically when users try to use CategoricalCrossEntropy loss in onert like tensorlofw.
What
Let's apply softmax(nomalization) automatically when trying to use CategoricalCrossEntropy loss.
Why
There is no guarantee a pre-trained model has a softmax when trying to apply CategoricalCrossEntropy loss to the model.
To do
To apply normalization(softmax) automatically to categorical cross entropy, we need to consider that sum of labels is not 1.
That consideration will be deal with in another issue later. So, I'm closing this issue since I have completed all other required tasks.
Originally posted by @ragmani in #13736 (comment)
The text was updated successfully, but these errors were encountered: