NTU Midterm: Data Analysis & ML Guide
Uploaded by
hamburgerhenry13NTU Midterm: Data Analysis & ML Guide
Uploaded by
hamburgerhenry13Common questions
Powered by AIK-Nearest Neighbors is simple to understand and implement, effective with a small number of classes. It naturally adapts to complex boundaries and doesn't make assumptions about underlying data distributions. Limitations include high computation costs, especially with large datasets, and sensitivity to irrelevant features or noisy data, which can diminish model accuracy and efficiency .
Data cleaning is crucial for removing errors and inconsistencies in data to ensure accurate analysis. Common techniques include: 1) Removing duplicates can cleanse data entry errors, for instance, removing repeated entries in customer data; 2) Handling missing values by methods like imputation (e.g., replacing missing temperatures with the mean temperature); 3) Correcting inconsistencies, such as standardizing date formats. These techniques enhance data quality, leading to more reliable insights and decisions .
Histograms represent data distribution by showing the frequency of data intervals, useful for understanding the shape and spread of the data. Boxplots highlight data distribution through medians and quartiles, revealing outliers and variability. Both are beneficial for quickly grasping data characteristics, spotting trends, and making comparisons, though histograms provide finer granularity, and boxplots summarize key statistical insights efficiently .
Pandas provides data structures like DataFrames for intuitive manipulation and analysis of structured data, enabling efficient operations like filtering, grouping, and merging datasets. Its functions support comprehensive data cleaning and transformation, essential for preparing datasets for analysis and modeling .
Overfitting occurs when a model learns the training data too well, capturing noise and outliers as patterns, thus performing poorly on new data. It is important to prevent it to maintain model generalization. Strategies include using cross-validation, reducing the model complexity, applying regularization techniques like L1 or L2 regularization, and augmenting the training data .
Adding 2SibSp and Parch, representing family aboard, enhances model prediction by introducing socio-familial context as predictors. These factors can influence survival likelihood in historical datasets like Titanic, where social ties may affect access to lifeboats or safety areas, hence improving model robustness and potentially increasing prediction accuracy .
Using separate training and testing datasets ensures that models learn without seeing test instances, allowing unbiased performance evaluation. This separation guards against overfitting by showing how well models generalize to unseen data, providing a more accurate reflection of real-world performance .
Open resources allow students to deepen understanding through wider access to information, fostering critical thinking and problem-solving by cross-referencing multiple sources. However, the risk of diminished originality and potential for academic dishonesty exists if not managed appropriately, highlighting the need for emphasizing ethical practices and skills in synthesizing information .
Tree-based models like random forest and decision tree regressors estimate the pattern in normal data. Although suitable due to their ability to model complex relationships and detect outliers, they might misidentify normal variance as anomalies. Random forests provide robustness by averaging predictions, mitigating false positives, but may not detect anomalies efficiently compared to algorithms specifically designed for anomaly detection such as one-class SVM or Isolation Forest .
Decision trees split data into branches based on feature values leading to decisions or predictions. They are intuitive and easy to interpret but prone to overfitting if deep. Random forests enhance decision trees by creating a 'forest' of trees using bootstrapped datasets and feature randomness, improving accuracy and reducing overfitting. However, they are less interpretable than single decision trees. Decision trees suit simple problems or when interpretability is key, while random forests are apt for complex problems where accuracy is prioritized .