0% found this document useful (0 votes)
11 views1 page

Data Forecasting & Text Analysis Lab

The document outlines two exercises for a data analytics lab. Exercise 1 involves performing time series analysis and visualization on unemployment rate data to explore trends, seasonality, and correlations between industries. Exercise 2 involves downloading Amazon baby product review data and cleaning it by removing special characters and punctuation before analyzing review lengths, word counts, polarity by product and rating, and visualizing the distribution of unigrams, bigrams and trigrams.

Uploaded by

Mukta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
11 views1 page

Data Forecasting & Text Analysis Lab

The document outlines two exercises for a data analytics lab. Exercise 1 involves performing time series analysis and visualization on unemployment rate data to explore trends, seasonality, and correlations between industries. Exercise 2 involves downloading Amazon baby product review data and cleaning it by removing special characters and punctuation before analyzing review lengths, word counts, polarity by product and rating, and visualizing the distribution of unigrams, bigrams and trigrams.

Uploaded by

Mukta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

DSE 2141– Data Analytics Lab

Lab 7 – Date: 20th Septmber 2023


EXERCISE 1: Time Series Analysis

Use the “[Link]” data set and perform time series analysis and visualization through the
following questions.
1. Convert datestamp column to a datetime object and Set the datestamp columns as the index of
your DataFrame. Check if there are missing values in each column.
2. Generate a boxplot to find the distribution of unemployment rate for every industry .
3. Using line chart Visualize the unemployment rate of workers by industry .
4. Plot the monthly and yearly trends .
5. Apply time series decomposition to your dataset to visualize the trend and seasonality .
6. Visualize the seasonality of Agriculture, Health and Finance sector.
7. Visualize the seasonality of multiple time series and the correlation between each time series
in the dataset.

EXERCISE 2 : Text Analysis

Download the amazon_baby.zip file and answer the following:

1. Check the number of the reviews received for each product.

2. Check the products that have more than 15 reviews.

3. Find any missing review are present or not, If present remove those data.

4. Clean the data and remove the special characters and replace the contractions with its
expansion by converting the uppercase character to lower case. Also, remove the
punctuations.

5. Add the Polarity, length of the review, the word count and average word length of
each review.

6. Visualize the distribution of the word count, review length, and polarity.

7. Visualize polarity considering the rating.

8. Visualize the count of the reviews of each rating available in the dataset.

9. List the Top 20 products based on the polarity.

10. Visualize to check whether the review length changes with rating.

11. Visualize the distribution of Top 25 Unigram, Bigram and Trigram.

Common questions

Powered by AI

Distributions of unigrams, bigrams, and trigrams highlight the most common words and phrases used in reviews, revealing frequent subjects of either praise or complaint. For instance, common phrases might signal key features or issues, helping in sentiment analysis to understand customer perception and prioritize product development or marketing strategies .

Time series decomposition separates a series into trend, seasonal, and residual components. This provides insights into underlying patterns such as long-term trends indicating whether unemployment is generally increasing or decreasing. Seasonal components reveal recurring patterns at specific intervals, which can help in understanding industry-specific cycles of unemployment, such as peak seasons in agriculture or finance .

Analyzing unemployment rates alongside customer reviews can yield comprehensive insights for policymakers and businesses by correlating economic conditions with consumer sentiment. For instance, a rise in unemployment might coincide with negative reviews or reduced spending, informing targeted economic interventions or business strategies. Understanding these dynamics aids in crafting policies and strategies that align with economic trends and consumer attitudes .

Data cleaning in text analysis involves: 1) Identifying and removing missing reviews to maintain data integrity; 2) Converting text to lowercase to maintain uniformity; 3) Removing special characters and punctuations to simplify the text; 4) Replacing contractions with full words for clarity; 5) Reviewing and removing irrelevant data that could skew analysis like very short entries .

Identifying missing values in a time series dataset typically involves checking each column to see if any entries are null or NaN. This can be done using functions like `isnull()` or `isna()` in Python's pandas library. Once identified, missing values can be addressed by either imputing them using statistical methods such as forward fill, backward fill, or interpolation, or by removing the rows/columns if they're not essential .

To compare correlations between multiple time series, one can use statistical approaches like Pearson or Spearman correlation coefficients to measure linear relationships. Cross-correlation functions can reveal lags between the series. Visualization techniques like scatterplots and heatmaps can also be applied to visually assess the correlation, highlighting potential dependencies or causal relationships .

Enhancing text reviews with attributes like polarity, word count, review length, and average word length provides depth to the analysis by quantifying sentiment (polarity), measuring verbosity (word count and length), and understanding the intricacy of the language (average word length). These attributes help in clustering, trend analysis, and offer insights into customer sentiment linked to quantitative data points .

Visualizing seasonality can be done using line plots for each sector to compare seasonal trends. By plotting the seasonal components separately for agriculture, health, and finance, we observe specific cyclical patterns. For instance, agriculture might show seasonality influenced by planting and harvest periods, whereas health might reflect monthly or yearly cycles related to healthcare demands. Correlating these patterns identifies interdependencies and unique trends across the sectors .

Visualizing review polarity against product ratings provides a nuanced understanding of customer satisfaction beyond numerical ratings. It exposes discrepancies between stated sentiment and rating scores, highlighting possibly misleading reviews and helping in identifying consistent sentiment trends that could impact product perception and sales strategy .

A boxplot is significant in visualizing unemployment rates as it provides a summary of the data's central tendency, variability, and skewness. It highlights the median, quartiles, and potential outliers, allowing for a clear comparison of the distribution of unemployment rates across different industries. This can help in identifying which industries have higher variability in unemployment rates and potential anomalies .

You might also like