Scribd
Upload
45 views3 pages

Data Analysis Problem Statements

This document contains 16 problem statements related to data analysis using Python and Scala. The problems involve tasks like data preprocessing, outlier detection, data visualization, text processing, machine learning algorithms and Apache Spark applications. The datasets referenced include academic performance data, customer reviews, log files, titanic passenger data and more. The problems require operations such as missing value imputation, data type conversions, word counting, sorting, topic modeling and predictive modeling.

Uploaded by

SHREYANSHU KODILKAR
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
45 views3 pages

Data Analysis Problem Statements

This document contains 16 problem statements related to data analysis using Python and Scala. The problems involve tasks like data preprocessing, outlier detection, data visualization, text processing, machine learning algorithms and Apache Spark applications. The datasets referenced include academic performance data, customer reviews, log files, titanic passenger data and more. The problems require operations such as missing value imputation, data type conversions, word counting, sorting, topic modeling and predictive modeling.

Uploaded by

SHREYANSHU KODILKAR
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Problem Statement No.

01
Consider the “Academic performance” dataset of students (Academic_Performance_Dataset.csv) and perform the
following operations using Python.
a) Scan all variables for missing values and inconsistencies. If there are missing values and/or inconsistencies, use any
of the suitable techniques to deal with them.
b) Scan all numeric variables for outliers. If there are outliers, use any of the suitable techniques to deal with them.
c) Apply data transformations on categorical variables to convert it into numerical variables.
Reason and document your approach properly.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 02
Perform the following operations on Age-Income dataset ([Link])
Provide summary statistics (mean, median, minimum, maximum, standard deviation) for numeric variables with and
without using any library functions.
Provide summary statistics of income grouped by the age groups. Create a list that contains a numeric value for each
response to the categorical variable.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 03
Write a Python program to display some basic statistical details like percentile, mean, standard deviation etc. of the
species of ‘Iris-setosa’, ‘Iris-versicolor’ and ‘Iris-virginica’ of [Link] dataset.
Calculate the measures of variability. Calculate and provide the visualization of the Correlation among the variables.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 04
Consider the Bangalore House Price Data. Perform following operations.
a) Find and replace null values in the data using appropriate technique.
b) Transform the ‘Size’ column to numerical values. For Example: 2 BHK to be converted as 2
c) Transform the ‘total_sqft’ column to contain numerical values on same scale. If the range is given average value of
the range to be taken.
d) Calculate and add one more column as ‘Price_Per_Sqft’
e) Remove the outliers from Price_Per_Sqft and BHK Size column if any.
f) Apply the Linear Regression model to the data and display the training and testing performance measures as Mean
Squared Error and Accuracy
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 05
Write a Scala Program to process a log file of a system and perform following analytics on the given dataset.
(I) Display the list of top 10 frequent hosts.
(II) Display the list of top 5 URLs or paths
(III) Display the number of unique Hosts
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 06
Write a Scala Program to process a log file of a system and perform following analytics on the given dataset.
(I) Display the count of 404 Response Codes
(II) Display the list of Top Twenty-five 404 Response Code Hosts
(III) Display the number of Unique Daily Hosts
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 07
Perform the following operations using Python on any open source dataset (e.g., [Link])
1. Import all the required Python Libraries.
2. Load the Dataset into pandas data frame.
4. Data Preprocessing: check for missing values in the data to get some initial statistics. Provide variable descriptions.
Types of variables etc. Check the dimensions of the data frame.
5. Data Formatting and Data Normalization: Summarize the types of variables by checking the data types (i.e.,
character, numeric, integer, factor, and logical) of the variables in the data set. If variables are not in the correct data
type, apply proper type conversions.
6. Turn categorical variables into quantitative variables in Python. In addition to the codes and outputs, explain every
operation that you do in the above steps and explain everything that you do to import/read/scrape the data set.
Use [Link], [Link]
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 08
1. Implement logistic regression using Python/R to perform classification on Social_Network_Ads.csv dataset.
2. Compute Confusion matrix to find TP, FP, TN, FN, Accuracy, Error rate, Precision, Recall on the given dataset
Use :Social_Network_Ads.csv
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 09
1. Implement Simple Naïve Bayes classification algorithm using Python/R on [Link] dataset.
2. Compute Confusion matrix to find TP, FP, TN, FN, Accuracy, Error rate, Precision, Recall on the given dataset
Use [Link] [Link]
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 10
1. Use the dataset 'titanic'. Use the Seaborn library to see if we can find any patterns in the data.
2. Write a code to check how the price of the ticket (column name: 'fare') for each passenger is distributed by plotting a
histogram.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 11
1. Use the dataset 'titanic'. Plot a box plot for distribution of age with respect to each gender along with the information
about whether they survived or not. (Column names : 'sex' and 'age')
2. Write observations on the inference from the above statistics. Use voilin plot and others plots from seaborn package
Use :[Link]
Problem Statement No. 12
Scan the Iris dataset and give the inference as:
1. List down the features and their types (e.g., numeric, nominal) available in the dataset.
2. Create a histogram for each feature in the dataset to illustrate the feature distributions.
3. Create a boxplot for each feature in the dataset. 4. Compare distributions and identify outliers.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 13
Write a code in SCALA for a simple WordCount application that counts the number of occurrences of each word in a
given input set using the APACHE SPARK MapReduce framework on local-standalone set-up.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 14
Write a code in SCALA for a simple Sort application that sort the occurrences of each word in a given input set using
the APACHE SPARK MapReduce framework on local-standalone set-up.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 15
Consider the Amazon Alexa Reviews Dataset. This dataset consists of a nearly 3000 Amazon customer reviews (input
text), star ratings, date of review, variant and feedback of various amazon Alexa products like Alexa Echo, Echo dots,
Alexa Firesticks etc. Perform following operations on this dataset.
(I) Plot a graph of Positive and Negative Feedback (1 = Positive Feedback, 0 = Negative Feedback)
(II) Convert the review text into lowercase.
(III) Remove all punctuations from review text.
(IV) Remove emoticons and emojis from the text
(V) Tokenize the review text into words.
(VI) Remove the Stopwords from the tokenized text.
-----------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement No. 16
Consider the Amazon Alexa Reviews Dataset. This dataset consists of a nearly 3000 Amazon customer reviews (input
text), star ratings, date of review, variant and feedback of various amazon Alexa products like Alexa Echo, Echo dots,
Alexa Firesticks etc. Perform following operations on this dataset.
(I) Remove all punctuations from review text.
(II) Tokenize the review text into words.
(III) Remove the Stopwords from the tokenized text.
(IV) Perform stemming & lemmatization on the review text.
(V) Perform the word vectorization on review text using Bag of Words technique.
(VI) Create representation of Review Text by calculating Term Frequency and Inverse Document Frequency (TF-IDF)

Common questions

Powered by AI

TF-IDF represents text data by calculating a score that signifies the importance of a word in a document relative to the corpus. It helps in reducing the noise from frequently occurring words by down-weighting them, thus emphasizing more relevant words that could indicate sentiment, enhancing the accuracy of sentiment classification models .

One-hot encoding allows models to understand the categorical distinctiveness by creating binary columns, avoiding ordinal implications but increasing dimensionality, which could lead to a sparse matrix in high cardinality features. Label encoding, on the other hand, is simpler and does not increase dimensionality but implicitly introduces ordinality in categorical features, which can mislead some models .

Outliers can be identified using IQR, Z-score, or visualization techniques like box plots. They distort statistical measures such as mean and standard deviation and impact the predictions of models like Linear Regression. Techniques for treatment include removing outliers, capping them, or transforming the data. Addressing outliers ensures robust statistical analyses and accurate model predictions .

Insights on the correlation between age and income might reveal trends or patterns such as income growth with age or stabilization at certain age groups. Without libraries, this can be done by manually calculating correlation coefficients using covariance and standard deviations formulas. Such analysis uncovers the relationship strength and direction between age and income .

A confusion matrix provides a summary of prediction results by displaying True Positives, False Positives, True Negatives, and False Negatives. For a Naive Bayes classifier on the Iris dataset, it can assess accuracy, error rate, precision, and recall, highlighting the model's capability to correctly classify the iris species and pinpoint areas needing improvement .

Transforming 'total_sqft' to a consistent numeric format is crucial to avoid biases and ensure model interpretability and efficiency. Range values like '1200-1300' should be resolved by using the average, thus standardizing inputs into a model-ready numerical format and maintaining data integrity .

Stemming and lemmatization reduce words to their base forms, facilitating uniformity in text data which enhances feature extraction and model training efficiency by reducing dimensionality and noise. This increases the relevance of features and improves analysis, as variations of a word are treated as equivalent .

Normalizing data types ensures uniformity and compatibility for subsequent analysis, especially statistical and machine learning models expecting specific data types. Skipping this step can lead to errors in computations, incorrect analysis, and model training issues as operations on mismatched data types may yield incorrect results or fail to execute .

Missing values can lead to biases and distortions in model predictions, affecting the accuracy of a Linear Regression model. Techniques to handle them include using mean, median, or mode for imputation, employing algorithms like k-NN for imputation, or removing rows/columns with missing data if they are below a certain threshold .

Outliers in 'Price_Per_Sqft' can skew the regression line, leading to inaccurate model predictions. Strategies to mitigate impact include outlier removal, data transformation using logarithmic or other scaling techniques, and robust regression methods that lessen influence on extreme values .

You might also like