11 Facts About Softmax function

Softmax function, known as softargmax or normalized exponential function, converts a vector of real numbers into a probability distribution of possible outcomes.

The softmax function is often used as the last activation function of a neural network to normalize the output of a network to a probability distribution over predicted output classes, based on Luce's choice axiom.

Softmax function takes as input a vector of real numbers, and normalizes it into a probability distribution consisting of probabilities proportional to the exponentials of the input numbers.

Name "softmax" is misleading; the function is not a smooth maximum, but is rather a smooth approximation to the arg max function: the function whose value is which index has the maximum.

In probability theory, the output of the softargmax Softmax function can be used to represent a categorical distribution – that is, a probability distribution over different possible outcomes.

In statistical mechanics, the softargmax Softmax function is known as the Boltzmann distribution : the index set are the microstates of the system; the inputs are the energies of that state; the denominator is known as the partition Softmax function, often denoted by ; and the factor is called the coldness .

Softmax function is used in various multiclass classification methods, such as multinomial logistic regression [1], multiclass linear discriminant analysis, naive Bayes classifiers, and artificial neural networks.

Standard softmax function is often used in the final layer of a neural network-based classifier.

Standard logistic Softmax function is the special case for a 1-dimensional axis in 2-dimensional space, say the x-axis in the plane.

Softmax function is the gradient of the LogSumExp function, a smooth maximum:.

Softmax function was used in statistical mechanics as the Boltzmann distribution in the foundational paper Boltzmann, formalized and popularized in the influential textbook Gibbs .