UNIT II

I. Data Science Concepts (Key components and lifecycle)

Data science is an interdisciplinary field that uses scientific techniques, procedures, algorithms, and
structures to extract know-how and insights from established and unstructured information.

Key Components of Data Science

1. Data and Data Collections
The first step in every data science endeavor is to get the necessary datasets needed to address
the business problem at hand or answer a specific question. Structured data and unstructured
data are two major categories of data.
2. Data Engineering
Data engineering designs, develops, and manages the infrastructure for storing, and
processing data efficiently. Real-world data obtained from businesses could be more
consistent and complete. Data cleaning and preparation is an important step performed to
transform raw data accessed from diverse sources into high-quality datasets ready for analysis.
Some common data issues that need to be resolved are: Missing values which could indicate a
data capture or an extraction issue, Incorrect data types like text when a numerical value was
expected, Duplicates which can skew analysis, Data inconsistencies due to mergers, system
migrations, etc and outliers that fall outside expected statistical distributions.
3. Statistics
Statistics is a foundational pillar of data science, providing the theoretical framework for data
analysis and interpretation. As a crucial component, it encompasses methods for
summarizing and interpreting data, inferential techniques for drawing conclusions, and
hypothesis testing for validating insights. Statistical methods aid in uncovering patterns,
trends, and relationships within datasets, facilitating informed decision-making. Descriptive
statistics illuminate the central tendencies and distributions of data, while inferential
statistics enable generalizations and predictions.
4. Machine Learning
Machine learning serves as an indispensable component within the broader field of data
science, representing a paradigm shift in analytical methodologies. It involves the utilization
of sophisticated algorithms to enable systems to learn and adapt autonomously based on data
patterns, without explicit programming. This transformative capability allows for the
extraction of meaningful insights, predictive modeling, and informed decision-making.
[Link] languages (Python, R, SQL)
Programming languages such as Python, R, and SQL serve as integral components in the
toolkit of a data scientist.
6. Big Data
Big data refers to extremely large and diverse collections of data that are: voluminous (the size
of the data is massive, often in terabytes or even petabytes), varied (structured (e.g.,
databases), semi-structured (e.g., JSON files), and unstructured (e.g., text documents, images,
videos)) and fast-growing. The volume, variety, and velocity (speed of data generation) of big
data are constantly increasing, posing challenges in storage, processing, and analysis.
II. Data Science Lifecycle
Data Science Lifecycle revolves around the use of machine learning and different analytical strategies
to produce insights and predictions from information in order to acquire a commercial enterprise
objective.
1. Business Understanding: The complete cycle revolves around the enterprise goal. It is
extraordinarily essential to apprehend the commercial enterprise goal sincerely due to the fact
that will be the ultimate aim of the analysis.
2. Data Understanding: After enterprise understanding, the subsequent step is data
understanding. This includes a series of all the reachable data. This step includes describing
the data, their structure, their relevance, their records type. Explore the information using
graphical plots.
3. Preparation of Data: Next comes the data preparation stage. This consists of steps like
choosing the applicable data, integrating the data by means of merging the data sets, cleaning
it, treating the lacking values through either eliminating them or imputing them, treating
inaccurate data through eliminating them, additionally testing for outliers, constructing new
data, deriving new elements from present ones, formatting the data into the preferred
structure, eliminating undesirable columns and features.
4. Exploratory Data Analysis: This step includes getting some concept about the answer and
elements affecting it, earlier than constructing the real model. Relations between distinct
aspects are captured via graphical representations like scatter plots and warmth maps.
5. Data Modeling: Data modeling is the coronary heart of data analysis. A model takes the
organized data as input and gives the preferred output. This step consists of selecting the
suitable kind of model, whether the problem is a classification problem, or a regression
problem or a clustering problem. Then, the hyperparameters of every model are tuned to
obtain the preferred performance.
6. Model Evaluation: Here the model is evaluated for checking if it is geared up to be deployed.
The model is examined on an unseen data, evaluated on a cautiously thought out set of
assessment metrics. Any data science solution, a machine learning model, simply like a
human, must evolve, must be capable to enhance itself with new data, adapt to a new
evaluation metric. The model assessment helps us select and construct an ideal model.
7. Model Deployment: The model after a rigorous assessment is at the end deployed in the
preferred structure and channel. This is the last step in the data science life cycle.
III. Structured vs Unstructured data
Structured data has a fixed schema and fits neatly into rows and columns, such as names and phone
numbers. Unstructured data has no fixed schema and can have a more complex format, such as
audio files, images and web pages.
Here are key areas where structured and unstructured data differ:
Format: Structured data has a strict, predefined data model. Unstructured data does not have
a predefined format.
Storage: Structured data storage systems have rigid schemas, such as those in relational
databases or data warehouses. Unstructured data is often stored in its native format
in nonrelational databases or data lakes.
Use cases: Organizations can use both structured and unstructured data across artificial
intelligence (AI) and analytics use cases. Structured data is often used in machine
learning (ML) and drives ML algorithms. Unstructured data is often used in natural language
processing (NLP) and is a rich and diverse data source for generative AI (gen AI) models.
Complexity: Structured data is easier to manipulate and analyze for general business users
with traditional tools. Unstructured data can be more complex and requires specialized skills
and tools to parse and analyze.

IV. Time-series in electrical systems

A time series is a sequence of data points collected or recorded at regular time intervals. Unlike
random data, time series data carries an inherent temporal order, making it valuable for
understanding trends, patterns, and dependencies over time. In electrical engineering, where signals,
power loads, and voltage fluctuations are monitored over time, time series techniques offer a
powerful framework for analysis, forecasting, and anomaly detection.
Examples in Electrical Engineering:
 Voltage and current measurements over time in a circuit.
 Load demand data from power grids.
 Temperature readings from electrical components.
 Signal amplitude variations in communication systems.

Common Characteristics of Time Series

1. Trend
A trend refers to a long-term upward or downward movement in the data. This could be
linear or nonlinear. Example: In a growing city, the long-term trend in electricity
consumption might show a steady increase due to population growth and industrial
development. Similarly, the gradual degradation of a battery over time might show a
downward trend in voltage output.
2. Seasonality
Seasonality indicates repeating patterns or cycles that occur at regular intervals — daily,
weekly, monthly, or yearly. Example: Household electricity usage typically increases in the
evening and drops overnight — this is a daily seasonal pattern. On a larger scale, power
demand often spikes in summer and winter due to air conditioning and heating.

3. Cyclic Behavior
 While seasonality is regular and tied to fixed time intervals, cyclic behavior refers to
fluctuations that do not follow a fixed schedule but instead occur over irregular durations.
Example: Power consumption in an industrial facility may follow an economic cycle —
expanding during boom periods and contracting during downturns.
4. Noise
Noise is the random variation in a time series that cannot be explained by trend, seasonality,
or cycles. It often results from measurement errors, sensor inaccuracies, or unpredictable
external influences. Example: Sudden, brief fluctuations in voltage readings from a sensor due
to electromagnetic interference or poor signal transmission are considered noise
5. Stationarity
A stationary time series is one whose statistical properties (mean, variance, autocorrelation,
etc.) do not change over time. Many modeling techniques assume stationarity because it
simplifies analysis. Example: A stationary signal might be the output of a voltage regulator
under constant load.

Common Time Series Models

1. Classical Statistical Models
These models assume a linear structure in the data and often rely on assumptions such as
stationarity and normality.
AR (Auto Regressive): Models current values as a linear combination of past values.
MA (Moving Average): Models the error term as a linear combination of past errors.
ARMA: Combines AR and MA models for stationary series
ARIMA (Auto Regressive Integrated Moving Average): Handles non stationary data
with differencing
SARIMA (Seasonal ARIMA): Extends ARIMA with seasonal components.
2. Machine Learning Models
These models make fewer assumptions about the data and can capture complex nonlinear
patterns through feature engineering.
Linear Regression: Uses logged features to fit a straight line to the data
SVR (Support Vector Regression): Margin based regression method, good with small
non-linear datasets.
Random Forest / XGBoost: Tree based models that work well with engineered time
features.
LSTM (Long Short Term Memory): A type of RNN that captures long-term
dependencies in sequence data
3. State-Space and Probabilistic Models
These models consider the system’s hidden internal states and are used when observations are
indirect or noisy.
Kalman Filter: A recursive filter that estimates unobserved states from noisy
measurements.
Hidden Markov Model (HMM): Models systems with hidden states and observed
outputs.
Dynamic Linear Models: Generalization of Kalman Filters with time-varying
parameters.
V. Data Science Applications in Electrical Engineering: Load forecasting, Smart grids,
Fault prediction

Time Series in Practice: Electrical Engineering Use Cases

1. Power Load Forecasting

Electricity consumption varies by time of day, season, weather conditions, and user behavior.
Utility companies rely heavily on time series forecasting models
like ARIMA, SARIMA, Prophet, and LSTM neural networks to predict short-term and long-
term electricity demand, optimize power plant operations and energy dispatch, minimize
costs by avoiding under- or over-production.
Example: Using hourly consumption data from smart meters, an LSTM model can forecast
next-day peak load, allowing grid operators to balance supply and demand dynamically.

2. Fault Detection in Power Systems

Faults such as short circuits, insulation failure, or transformer issues can be catastrophic if not
detected early. Time series analysis of real-time current and voltage signals helps:
identify spikes, drifts, and anomalies in waveforms, trigger automated alarms or safety shut-
offs and support predictive analytics for condition-based maintenance.
Example: Monitoring voltage dips over time using an XGBoost-based anomaly detector to
detect partial discharge activity in underground cables.

3. Signal Processing & System Identification

Electrical signals often carry noise or interference that needs filtering. Time-domain and
frequency-domain time series techniques are used to filter noise (e.g., with Kalman filters or
wavelet transforms), detect harmonic distortion or modulation patterns and model and
simulate system dynamics (e.g., via autoregressive models).
Example: Time-domain analysis of EEG signals in biomedical electronics or frequency
decomposition of motor vibrations for fault diagnosis.

4. Preventive and Predictive Maintenance

Maintenance planning is shifting from reactive to predictive approaches using sensor data
collected over time. Time series models help monitor trends in temperature, vibration, and
load current, estimate Remaining Useful Life (RUL) of electrical components, reduce
downtime and improve asset lifespan.
Example: Using time-stamped thermal imaging data and SVR models to predict transformer
overheating trends and plan cooling system upgrades.

5. Renewable Energy Forecasting

Time series forecasting is also vital in solar and wind power systems, where generation is
inherently variable and can be used for wind speed forecasting using SARIMA, solar
irradiance prediction for optimizing photovoltaic outputs and grid balancing using multi-step
forecasts.
Example: Combining satellite weather data and past power output to train a hybrid LSTM-
ARIMA model for solar farm generation forecasting.
Selecting the appropriate time series model
Selection in electrical engineering is essential for ensuring accurate analysis and effective system
management.

Accuracy in Prediction:
Different models capture various patterns such as trends, seasonality, and noise differently.
Choosing the right model ensures precise forecasting of electrical loads, system behavior, and
fault occurrences.

System Reliability:
Accurate modeling helps detect anomalies and predict failures early, reducing downtime and
enhancing the stability of power grids and electrical equipment.

Cost Efficiency:
Reliable forecasts enable optimized resource allocation and preventive maintenance,
minimizing costly repairs and energy wastage.

Interpretability and Decision Making:

A well-chosen model provides clear insights into system dynamics, allowing engineers to
make informed decisions and implement effective control strategies.

Real-time Responsiveness:
The right model supports fast and efficient processing of streaming data, critical for real-time
monitoring and control in modern electrical systems.

Future Trends and Advancements in Time Series Forecasting

Future trends and advancements in time series analysis within the electrical engineering field are
rapidly evolving, driven by technological innovations and increasing data availability. Key areas of
development include:

Advanced Machine Learning Models:

The adoption of deep learning architectures such as recurrent neural networks (RNNs), long
short-term memory (LSTM), and transformer models is enhancing the accuracy of time series
forecasting in power systems and fault detection.

Edge Computing and IoT Integration:

With the proliferation of smart sensors and IoT devices, time series data can be processed
locally at the edge, reducing latency and enabling real-time monitoring and control of
electrical grids.

Hybrid Modeling Approaches:

Combining physics-based models with data-driven techniques is improving the
interpretability and robustness of predictions, particularly in complex electrical machinery
and renewable energy systems.

Cloud and Big Data Analytics:

Scalable cloud platforms facilitate the handling of massive time series datasets, enabling more
comprehensive analysis, anomaly detection, and predictive maintenance across large-scale
electrical networks.
VI. Roles in data science: Analyst, Engineer, Scientist
The three major data roles that organizations hire for their data teams are: data engineers, data
analysts, and data scientists.

Data Engineer
A data engineer is responsible for figuring out how to gather data, organize it, and maintain it,
so they are a vital role to have on a data team. Data engineers frequently have to contend with
messy or incomplete data and make decisions on how that data will be processed and
maintained. The engineer needs to know how data applications are structured, test data
pipelines, and monitor how data is being used. Over the past few years, cloud-based data
storage solutions like Amazon Redshift and Google BigQuery can flexibly manage the
manipulation of large amounts of data. It is not inconceivable that a data analyst or scientist
can address an organization’s data needs without a data engineer by setting up and leveraging
these tools.

Data Architect

The general distinction between a data engineer and a data architect is that a data architect
has responsibility for planning the architecture or framework in which the data will be
processed and stored. The architect dictates the ways in which data should be collected,
stored, and made available to users at a high level, while anticipating and adapting to the
changing needs of those users. The data engineer, then, will be responsible for
implementation and maintenance of the data pipeline following the architect’s plan.

Data Analyst

Data engineers and data architects are responsible for operations before the data is collected,
while data analysts and data scientists are responsible for operations after the data is collected.
A top data analyst will have the curiosity and skills to investigate the data from multiple
angles, performing cleaning and transformation operations and to look for trends in the data.
They may find new paths for the company to explore, possibly identifying areas where more
data could be collected for deeper analysis. Data analysts not only mine through data, but also
to report their findings to others. An analyst should be able to create visualizations or use
tools to create dashboards that convey to others what they have found.

Data Scientist

Data analysts tend to develop performance metrics, report what is there, and convey those
observations to others, while data scientists are geared towards making sure those
observations actually carry statistical significance. A data scientist should be able to sift
through data in the same way as an analyst, but also be able to apply statistical techniques in
order to differentiate between signal and noise. Another factor that differentiates data
scientists from analysts is in their ability to apply machine learning to data. Machine learning
can be used in combination with other statistical techniques to move beyond descriptive
analytics and into the realm of predictive analytics, making predictions about future events or
outcomes.
VII. Overview of tools: Python, R, Jupyter, Excel
Data analysis tools are software programs, applications, and other aids that professionals use to
analyze data sets in ways that characterize the big picture of the information and provide usable
information for meaningful insights, predictions, and decision-making purposes.

Analysts commonly use tools during the following stages of the data analysis process:

Data mining:
Data mining helps users find the key characteristics of their data so they can apply this
knowledge to real-world problems, and data mining software helps automate this process by
looking for patterns and trends within the data.

Data visualization:
Data visualization is a powerful way to transform raw data into meaningful and
comprehensive visual representations. It provides us with a way to understand complex data
patterns, trends, and insights that people might miss in text-based data.

Business intelligence:
Data analysis is a powerful tool for understanding the story data tells and using it to make
informed decisions. Businesses can use these insights to enhance their performance, improve
customer satisfaction, gain a competitive advantage, and benefit the overall health of their
company.

Some powerful data analysis tools and software

1. Python
Primary use: Data mining and visualization
Python is routinely ranked as the most popular programming language in the world today.
Unlike other programming languages, Python is relatively easy to learn and can be used for a
wide range of tasks, including software and web development, and data analysis. In the world
of data, Python is used to streamline, model, visualize, and analyze data using its built-in data
analytics tools. One of the key features of Python that appeal to data analytics professionals is
its many libraries, such as Pandas and Numpy, which offer a variety of powerful tools for
many analytics needs.

2. R
Primary use: Data mining and visualization
R is an open-source programming language used for statistical computing and graphics. Like
Python, R is considered a relatively easy-to-learn programming language. Typically, it’s used
for statistical analysis, data visualization, and data manipulation. The statistical focus of R
means that it’s well-suited to statistical calculations, while the visualization tools included
within R make it a great language for creating compelling graphics like scatter plots and
graphs. Alongside Python, R is one of the most important programming languages used in
data analysis.
3. MySQL
Primary use: Data mining
MySQL is an open-source relational database management system (RDBMS) used for storing
application data, particularly web-based ones. MySQL is popular among websites and is used
by such well-known sites as Facebook, X (formerly Twitter), and YouTube.

4. Microsoft Excel and Power BI

Primary use: Business intelligence
Microsoft Excel, fundamentally a spreadsheet software, also has noteworthy data analytics
capabilities. Because of the wide enterprise-level adoption of Microsoft products, many
businesses find they already have access to it. Excel can be used to construct at least 20
distinct chart types using spreadsheet data. These range from standard options such as bar
charts and scatter plots to more complex options like radar charts and treemaps. Excel also
has many streamlined options for businesses to find insights into their data and use modern
business analytics formulas. If business needs more robust data visualization tools within the
Microsoft ecosystem, Power BI is a great option. Designed specifically for data analytics and
visualization, Power BI can import data from an array of sources and produce visualizations
in various formats.

5. Google Analytics
Primary use: Business intelligence
Google Analytics is a tool that helps businesses understand how people interact with their
websites and apps. To use it, we add a special Javascript code to our web pages. This code
collects information when someone visits the website, like which pages they see, what device
they’re using, and how they found the site. It then sends this data to Google Analytics, where
it is organized into reports. These reports help us see patterns, like which products are most
popular or which ads are bringing people to our site.

6. Jupyter Notebook
Primary use: Data mining and visualization
Jupyter Notebook is a web-based interactive environment used to share computational
documents or “notebooks.” Data analysts use Jupyter Notebooks to write and run code, clean
data, data visualization, machine learning, statistical analysis, and many other forms of data
analysis. Furthermore, Juypter Notebook allows users to combine data visualizations, code,
comments, and numerous different programming languages in one place, allowing for an
improved space to document a data analysis process and share it with others.