Cross Entropy Loss and gradient

Cross Entropy Loss

$$Loss = - \sum_{i=1}^{N}y_{i}\mathrm{log}(p_i)$$

• $y_{i}$는 실제 label이고, $p_i$는 예측된 확률이다. 이때 $p_i$는 softmax함수를 통해 얻어진다.
  $$p_{i}= \frac{\mathrm{exp}(z_i)}{\sum_{j=1}^{N}\mathrm{exp}(z_j)}$$
• $z_i$는 class $i$ 에 대한 logit 값이다.

Gradient of Cross Entropy Loss

참고

$$
\begin{align}
& \frac{\partial Loss}{\partial p_i} = - \frac{y_i}{p_i}
\\
& \frac{\partial p_i}{\partial z_j} =
\begin{cases}
p_i (1 - p_i) & \text{if } i = j \\
-p_i p_j & \text{if } i \neq j
\end{cases}
\\
& \frac{\partial p_i}{\partial z_j} = p_i (\delta_{ij} - p_j)
\end{align}
$$

$\delta_{ij}$는 $i=j$일 때 1이고, $i \neq j$일때 0이다.

chain rule을 적용하여 $\delta_{ij}$는 1이고, 나머지는 0이므로
$$
\begin{align}
\frac{\partial Loss}{\partial z_i} &= \sum_{j=1}^{N} \frac{\partial Loss}{\partial p_j} \frac{\partial p_j}{\partial z_i}
\\
\frac{\partial \text{Loss}}{\partial z_i} &= \sum_{j=1}^{N} \left( -\frac{y_j}{p_j} \right) p_j (\delta_{ij} - p_i)
\\
\frac{\partial Loss}{\partial z_i} &= p_i - y_i
\end{align}
$$

example

$z = [2.0,1.0, 0.1]$ 에 대한 softmax 확률 p
$$
\begin{align}
p_1 &= \frac{\exp(2.0)}{\exp(2.0) + \exp(1.0) + \exp(0.1)} \approx 0.659
\\
p_2 &= \frac{\exp(1.0)}{\exp(2.0) + \exp(1.0) + \exp(0.1)} \approx 0.242
\\
p_3 &= \frac{\exp(0.1)}{\exp(2.0) + \exp(1.0) + \exp(0.1)} \approx 0.099
\end{align}
$$
실제 label이 $[1,0,0]$일 때 gradient 계산
$$
\begin{align}
\frac{\partial \text{Loss}}{\partial z_1} &= p_1 - 1 \approx 0.659 - 1 = -0.341
\\
\frac{\partial \text{Loss}}{\partial z_2} &= p_2 - 0 \approx 0.242 - 0 = 0.242
\\
\frac{\partial \text{Loss}}{\partial z_3} &= p_3 - 0 \approx 0.099 - 0 = 0.099
\end{align}
$$