Differentiation of the Sigmoid activation and cross-entropy loss function

 


A step-by-step differentiation of the Sigmoid activation and cross-entropy loss function is discussed here.
The understanding of derivatives of these two functions is essential in the area of machine learning when performing back-propagation during model training.

Derivative of Sigmoid Function

Sigmoid/ Logistic function is defined as:
$g(x)=\frac{1}{1+e^{-x}} \in (0,1)$

For any value of $x$, the Sigmoid function $g(x)$ falls in the range $(0, 1)$. As the value of  $x$ decreases, g(x) approaches 0, whereas as $x$ grows bigger, $g(x) $tends to 1. Examples,

$g(-5.5)=0.0040$
$g(6.5)=0.9984$
$g(0.4)=0.5986$

It is noted that we can further simplify the derivative and write in term of g(x)


Why do we use this version of the derivative?

In the forward propagation step, you compute the sigmoid function $(g(x))$ and have its value handy. While computing the derivative in the backpropagation step, all you have to do is plug in the value of $g(x$) in the formula derived above.


Here is the plot of sigmoid function and its derivative

Derivative of Cross-Entropy Function

Cross-Entropy loss function is a very important cost function used for classification problems.The concept of cross-entropy traces back into the field of Information Theory where Claude Shannon introduced the concept of entropy in 1948. Before diving into Cross-Entropy cost function, let us introduce entropy .

Entropy

Entropy of a random variable $X$ is the level of uncertainty inherent in the variables possible outcome.

For $p(x)$ — probability distribution and a random variable $X$, entropy is defined as follows



Cross-Entropy Loss Function  is also called logarithmic loss, log loss or logistic loss. Each predicted class probability is compared to the actual class desired output 0 or 1 and a score/loss is calculated that penalizes the probability based on how far it is from the actual expected value. The penalty is logarithmic in nature yielding a large score for large differences close to 1 and small score for small differences tending to 0.

Cross-entropy loss is used when adjusting model weights during training. The aim is to minimize the loss, i.e, the smaller the loss the better the model. A perfect model has a cross-entropy loss of 0.

Cross-entropy is defined as

$E_g= - \sum_{i=1}^n t_i ln (p_i)$
where $t_i$ is the truth value and $p_i$ is the probability of the $i$ᵗʰ class.

For classification with two classes, we have binary cross-entropy loss which is defined as follows

$E_b=-[ t_n ln(\bar{y}) + (1-t) ln (1-\hat{y})]$

The truth label, $t$, on the binary loss is a known value, whereas $\hat{y}$ is a variable. This means that the function will be differentiated with respect to $\hat{y}$ and treat $t$ as a constant. Let’s go ahead and work on the derivative now.

And therefore, the derivative of the binary cross-entropy loss function becomes

Partial Derivative of cost function for linear regression




Partial Derivative of Cost Function for Logistic Regression






Comments

Post a Comment

Popular posts from this blog

NEURAL NETWORKS AND DEEP LEARNING CST 395 CS 5TH SEMESTER HONORS COURSE NOTES - Dr Binu V P, 9847390760

About Me Syllabus Previous Year Question Papers CST 395 Neural Network and Deep Learning Module 1 ( Basics of Machine Learning) Overview of Machine Learning Machine Learning Algorithm Linear Regression Capacity, Overfitting and Underfitting Regularization Hyperparameters and Validation Sets Estimators, Bias , Variance and Consistency Challenges In Machine Learning Linear and Logistic Regression ( Python code) Performance Measures Differentiation of  sigmoid and cross entropy function Gradient Descent, Stochastic, Batch and mini batch gradient descent Regression with Gradient Descent Module-2 (Neural Networks) Introduction to Neural Networks Application of Neural Networks Basic Architecture- Single Layer Neural Network Power of Function Composition- Non Linear Activation XOR Activation Functions Choice of activation and loss functions Multi Layer Neural Network, Back Propagation Back Propagation- Example Implementation of a two layer network(XOR) with sigmoid activation functio...
Read more

Syllabus CST 395 Neural Network and Deep Learning

Syllabus Module - 1 (Basics of Machine Learning ) Machine Learning basics - Learning algorithms - Supervised, Unsupervised, Reinforcement, overfitting, Underfitting, Hyper parameters and Validation sets, Estimators -Bias and Variance. Challenges in machine learning. Simple Linear Regression, Logistic Regression, Performance measures - Confusion matrix, Accuracy, Precision, Recall, Sensitivity, Specificity, Receiver Operating Characteristic curve( ROC), Area Under Curve(AUC). Module -2 (Neural Networks ) Introduction to neural networks -Single layer perceptrons, Multi Layer Perceptrons (MLPs), Representation Power of MLPs, Activation functions - Sigmoid, Tanh, ReLU, Softmax. Risk minimization, Loss function, Training MLPs with backpropagation, Practical issues in neural network training - The Problem of Overfitting, Vanishing and exploding gradient problems, Difficulties in convergence, Local and spurious Optima, Computational Challenges. Applications of neural networks. Module 3 (D...
Read more

Introduction to neural networks

Image
Artificial neural networks are popular machine learning techniques that simulate the mechanism of learning in biological organisms. The human nervous system contains cells, which are referred to as neurons . The neurons are connected to one another with the use of axons and dendrites , and the connecting regions between axons and dendrites are referred to as synapses .The strengths of synaptic connections often change in response to external stimuli. This change is how learning takes place in living organisms. This biological mechanism is simulated in artificial neural networks, which contain computation units that are referred to as neurons . The computational units are connected to one another through weights, which serve the same role as the strengths of synaptic connections in biological organisms. Each input to a  neuron is scaled with a weight, which affects the function computed at that unit. The following figure shows both the biological and artificial neural netw...
Read more