Regression cost functions, like Mean Error or Mean Squared Error, calculate error based on the difference between predicted and actual continuous values, aiming to minimize this difference over all training data . They focus on predicting a single continuous value, and the error is typically computed as numerical deviations from these predictions . In contrast, classification cost functions, such as cross-entropy, focus on calculating error based on the difference between predicted and actual class distributions . These functions guide the adjustment of class-based probability distributions, essential for models dealing with discrete class labels rather than continuous values .

Mean Absolute Error (MAE) addresses the inaccuracies of Mean Error by taking the absolute value of the differences between predicted and actual values, thereby eliminating any cancellation of errors due to positive and negative values . MAE is measured as the average of these absolute differences, making it robust to outliers. Its major advantage over MSE is that MAE does not disproportionately increase error penalties for outliers, which makes it more stable in datasets with noise or extreme values .

Squaring errors in Mean Squared Error (MSE) serves a dual purpose. First, it ensures that the calculated error is always a positive number, avoiding any cancellation effect that might result from negative errors offsetting positive ones . Secondly, it amplifies larger errors, as squaring enhances error magnitude, thus serving as a penalization mechanism for large deviations . This heightened penalty due to squaring directs the optimization process, encouraging the model to focus more on reducing larger errors, which theoretically leads to nuanced improvements in prediction accuracy. However, while this aids meticulous optimization for precise predictions, it also introduces sensitivity to outliers, as extreme cases may drive the optimization process unduly .

Mean Absolute Error (MAE) is characterized by its robustness to outliers, making it a reliable choice for models dealing with noisy data . By computing error as the average of absolute differences between predicted and actual observations, MAE ensures that all deviations are treated equally rather than exaggerating large errors, as seen in MSE . This equitable treatment of error allows MAE to accommodate noise and maintain model stability without disproportionately impacting model performance, which is crucial when datasets contain variability or unexpected deviations resulting from outliers .

Binary Cross Entropy is a specific form of Categorical Cross Entropy used when the classification problem involves only two classes, typically denoted by binary values 0 and 1 . Categorical Cross Entropy, on the other hand, is used when dealing with multiple classes. In Binary Cross Entropy, the loss function is simplified to handle binary output by considering either the negative likelihood for the positive class or the alternative class . Thus, Binary Cross Entropy effectively becomes a special case of Categorical Cross Entropy by focusing on binary class distinctions .

Cross-entropy loss is preferred in multi-class classification problems because it provides a way to measure the distance between the predicted probability distribution and the actual distribution of classes . It adjusts model weights effectively to minimize this distance, thus helping to ensure that the model predictions closely align with actual class assignments. This is crucial in multi-class scenarios where each input belongs to one class, and the wrong probability distribution can significantly affect accuracy . The formula used in cross-entropy accounts for the logarithmic likelihood, which provides a smoother gradient when optimizing model parameters, aiding efficient convergence .

Cost functions are crucial in supervised learning as they quantify the error or difference between predicted outputs and actual outputs during model training . By providing a single real number that reflects this error, cost functions serve as metrics to evaluate and improve the performance of the model . Optimization techniques aim to minimize this cost value, signaling improved accuracy in predictions. Thus, cost functions guide the adjustment of model parameters, leading to optimized predictions over time .

Mean Squared Error (MSE) is not ideal for datasets with significant outliers because the squaring of errors during calculation exaggerates these errors' magnitude . Outliers, which are already large errors, become disproportionately large when squared, which can bias the model's learning process towards fitting these outliers rather than capturing the overall data trend . This emphasis can distort the model's predictions and lead to poor generalization on normal data patterns, compromising performance .

Cross-entropy is derived from information theory and is used as a measure of the dissimilarity between the predicted probability distribution and the true probability distribution over multiple classes . It calculates the negative sum of the predicted probabilities multiplied by the log of the actual probability distribution for each class, thereby determining the difference between expected and predicted values . By minimizing cross-entropy, the model adjusts its parameters to increase prediction accuracy, making it a critical loss function in guiding models towards producing outputs that closely mirror real data class distributions .

Mean Error (ME) calculates the mean of the errors from each training data, which can sometimes lead to a zero mean error due to cancellation of positive and negative errors, making it less robust to identifying actual error trends . In contrast, Mean Squared Error (MSE) computes the average of the squared differences between predicted and actual observations, mainly to avoid negative error possibilities. Squaring the errors highlights even minor deviations, making MSE less robust to outliers because large errors become disproportionately larger when squared .