Locally weighted regression constructs local approximations to the target function by explicitly modeling the function over a local region around the query point. It weights the contribution of nearby points, often using gradient descent to find coefficients that minimize a weighted error function . On the other hand, radial basis function (RBF) networks create global approximations by deploying localized kernel functions, such as Gaussian functions, whose influence diminishes with distance from the query point. RBFs use a two-step approach involving learning kernel parameters and optimizing linear weights, thus blending local specificity with global scope . Both methods aim to leverage locality for improved approximation but differ in their foundational architectures and computational methodologies.

Lazy learning methods, such as k-NN, locally weighted regression, and CBR, differ from eager learning methods in that they delay generalization until new data needs classification. This allows lazy learners to explore a larger hypothesis space since they consider many local approximations. Consequently, lazy methods generally have fast learning times but slow classification stages, as the bulk of computation is deferred to the query phase . In contrast, eager learning methods, like radial basis function networks, commit to a single global approximation during training, leading to slower learning times but faster classification as the model does not change with new queries, operating within a more confined hypothesis space .

k-NN considers all attributes equally while classifying instances, making it sensitive to irrelevant attributes and noise. This often results in diminished performance in such contexts . To address this, one improvement approach is to weigh the attributes differently, using cross-validation to identify optimal weights that reflect attribute importance . Another method involves eliminating the least relevant attributes, again employing cross-validation to determine which attributes can be excluded, thereby reducing sensitivity to noise and improving classification efficiency .

Euclidean distance is the standard metric used in k-NN to measure the similarity between instances, defined as the square root of the sum of squared attribute differences . It is crucial in determining the 'nearness' of training examples to a query point, impacting the selection of neighbors that influence classification outcomes. Using alternative distance metrics, such as Manhattan or Mahalanobis distance, can significantly affect k-NN performance, particularly in handling differently scaled features or correlated attributes. These alternative metrics can provide more robust solutions in certain contexts, especially when Euclidean distance might be sensitive to outliers or irrelevant features .

Kernel functions in Radial Basis Function (RBF) networks help create localized areas of influence that approximate the target function efficiently across input space. These functions, such as Gaussian kernels, diminish in influence with increasing distance, allowing RBF networks to balance between localized specificity and global coverage . The Expectation-Maximization (EM) algorithm plays a key role by determining optimal parameters for these kernels, making the initialization of the network more efficient and ensuring that the network aligns well with the data's intrinsic structures. This results in a more effective and precise global approximation, enhancing RBF network efficiency .

Lazy learning, demonstrated by methods such as Case-Based Reasoning (CBR) and k-NN, significantly affects learning efficiency and decision speed in machine learning systems by delaying computation until a query is made. This approach streamlines the initial learning phase, making it very fast, and requires minimal storage of training instances . However, it often results in slower decision-making since the search and comparison processes occur at query time, leading to increased response times as the system evaluates the relevant data subset then. Consequently, while lazy learning offers flexibility and adaptability, it can suffer from scalability issues and become resource-intensive during classification, particularly with large datasets .

Distance-weighted k-NN improves upon the standard k-NN by assigning greater influence to closer neighbors during classification. This is achieved by modifying the algorithm to weigh the contribution of each neighbor based on its inverse squared distance to the query point, thereby producing a more nuanced and accurate classification . This enhancement allows for smoother decision boundaries and potentially better handling of overlapping classes. However, this weighting comes with additional computational costs as it requires calculating and processing individual distances for each neighbor, which can increase the overall time complexity of the classification phase .

IBL methods excel in problems where the target function, despite its complexity, can be described through a collection of less complex local approximations . This allows IBL to effectively construct different approximations for each distinct query, achieving flexibility and precision in handling complex functions. However, these advantages come at a cost: classifying new instances is computationally intensive as most of the computation is deferred to the classification stage, resulting in high resource consumption during this phase . Furthermore, due to their reliance on all instance attributes, IBL methods are highly susceptible to the curse of dimensionality, which can significantly impair their performance in high-dimensional spaces .

Locally weighted linear regression (LWLR) uses gradient descent to determine coefficients for the best fit line that minimizes an error function focused on local data points. By primarily weighting instances close to the query, LWLR achieves accurate local approximations of the target function . This localized focus allows LWLR to cater to non-linear data patterns within a limited region. However, challenges include choosing appropriate weighting functions and handling the potential increase in computational expense due to frequent adjustments to the model every time a new query is processed. Additionally, determining the size of the local region and appropriate hyperparameter tuning are critical to avoiding overfitting or poor generalizations .

The primary philosophical difference between CBR and k-NN lies in their model representations. CBR employs a rich symbolic representation for instances, which allows it to handle complex, conceptual problems such as mechanical design and legal reasoning . This symbolic nature contrasts with the real-valued point representation used by k-NN. As a result, CBR is inherently more suitable for domains requiring complex relational understanding and conceptual reasoning, while k-NN is limited to data that can be quantitatively measured and compared through distance metrics . This distinction significantly affects their applicability, with CBR being more versatile in symbolic domains and k-NN being preferable for numeric data analysis.