Order statistics, which are statistics derived from the ordered values in a sample, are crucial in non-parametric inference as they form the foundation of rank-based tests. For instance, in order statistics, the smallest value in a sample is the first order statistic, and the maximum is the nth order statistic. A practical example is the use of order statistics in calculating median, the middle order statistic when sorted, which is an estimator commonly used in non-parametric tests .

Non-parametric tests are suitable for evaluating machine learning models because they do not assume a normal distribution of prediction errors, which is often the case in real-world data. They are robust to outliers and heteroscedasticity, providing reliable results even with non-normal and skewed data. This is particularly advantageous in assessing model performance, where test statistics compare ranks rather than assuming underlying distributions .

The Wilcoxon signed rank test can be applied to compare two machine learning models' cross-validation accuracies by treating the accuracy scores from each fold as paired samples. By ranking the absolute differences in accuracy and considering the signs of these differences, the test evaluates whether the median of these differences is zero, thus indicating if there is a significant accuracy difference between the models across folds. This method uses all available cross-validation information, providing a comprehensive comparison of models .

Non-parametric tests do not assume a specific distribution for the data, making them more flexible in handling non-normally distributed datasets and smaller sample sizes. Parametric tests, however, rely on specific distribution assumptions, like the normal distribution, and are generally more powerful when these assumptions are met. The choice between them depends on the validity of distribution assumptions in your data and the sample size available .

Non-parametric tests benefit real-world data analysis by providing flexibility and robustness when dealing with non-normal data distributions, which are common in many fields. These tests do not rely on specific distribution shapes, making them applicable to a broader range of datasets with various forms of skewness and kurtosis. Their ability to analyze ordinal data and maintain validity despite outliers or heteroscedasticity enhances their practical utility across diverse disciplines .

Designing a non-parametric framework for comparing multiple machine learning models requires careful selection of appropriate tests, such as the Kruskal–Wallis test, which can handle more than two models. It is essential to define objective metrics, such as accuracy or F1-score, and consider the handling of ties and ranks. Robust data handling, consideration of sample independence, and a strategy for dealing with heteroscedasticity and outliers are critical to ensure reliability and validity. Evaluating multiple performance metrics together can also provide a holistic view of model effectiveness .

The Kruskal–Wallis test is a non-parametric method used for determining whether there are statistically significant differences between the medians of three or more independent groups. It is an extension of the Mann–Whitney U test for more than two groups, ranking data across all groups to detect differences in distributions. This makes it especially useful when data do not meet ANOVA assumptions, such as normality and homogeneity of variances .

The Mann–Whitney U test assumes that the two samples are independent, and the measurement scale is ordinal. It assumes that data from both groups are continuous and that the rank order is meaningful and equally spaced. These assumptions make the test robust to violations of normality, but it may become less reliable if these conditions are not respected, particularly in terms of ordinal scale and equality of variance across samples .

The Wilcoxon signed rank test is preferred over the sign test when the sample size is small and the data situation allows for assumptions about symmetry in the differences. It uses more information by considering both the magnitude and the direction of differences, increasing its statistical power compared to the sign test, which only considers the direction .

The Mann–Whitney U test is used for comparing two independent samples and assumes independence within and across groups, focusing on the ranks of the data rather than raw values. In contrast, the Wilcoxon signed rank test compares paired samples or matched pairs and assumes a symmetric distribution of differences. This makes the Mann–Whitney U test suitable for testing whether two independent samples come from the same distribution, while the Wilcoxon test is used for detecting median differences in paired samples .