Module 5
Strategies to Organize Data for
Analytics
Linked Analytical Datasets (LAD) can significantly enhance the speed and efficiency of
machine learning model development by leveraging the strengths of both analytical
datasets and relational databases.
LADs integrate the structured and relational nature of databases with the comprehensive,
feature-rich characteristics of analytical datasets. They streamline the process of preparing
data for machine learning, making it easier and faster to move from raw data to actionable
insights.
Relational Databases:
● Store data in tables with defined relationships between them.
● Enable efficient querying and data retrieval.
● Ensure data integrity and consistency.
Analytical Datasets:
● Combine multiple features into each record for comprehensive analysis.
● Focus on including all relevant data needed for machine learning models.
● Ensure the data is clean, well-formatted, and ready for analysis.
Accelerated Data Preparation:
● By linking datasets, data scientists can quickly access and integrate the necessary data without extensive manual processing.
● Simplifies the process of feature engineering by providing a well-organized structure.
Improved Data Quality:
● Relational databases ensure data integrity, reducing errors and inconsistencies.
● Clean and well-structured data improves model accuracy and reliability.
Enhanced Productivity:
● Reduces the time spent on data wrangling, allowing data scientists to focus more on model development and analysis.
● Streamlines repetitive tasks through automation and standardized processes.
Maintainability:
● Ensures that datasets are easier to update and maintain over time.
● Changes in the underlying data are automatically reflected in the linked analytical datasets, reducing the need for manual updates.
Define Relationships:
● Identify and define the relationships between different data tables in your database.
● Ensure that the relational structure supports efficient data retrieval and integration.
Create Analytical Views:
● Develop views or queries that combine data from multiple tables into comprehensive datasets.
● These views should include all the relevant features needed for your machine learning models.
Automate Data Integration:
● Use tools and scripts to automate the process of creating and updating analytical datasets.
● Ensure that new data is automatically incorporated into the LAD without manual intervention.
Monitor and Maintain:
● Regularly monitor the quality and integrity of your linked datasets.
● Implement processes for maintaining and updating the relational structure and analytical views as needed.
By effectively implementing Linked Analytical Datasets, organizations can dramatically
improve the speed and efficiency of their machine learning model development processes.
This approach balances the need for maintainable, high-quality data with the productivity
demands of data scientists, ultimately leading to faster and more accurate analytical
insights.
Creating an analytical dataset for machine learning involves preparing data in a way that makes it easy for an algorithm to learn from
it. Here’s a simplified explanation:
1. Gathering Data: Collect raw data from various sources. This could be anything from numbers in a spreadsheet, text from
documents, or sensor readings from IoT devices.
2. Cleaning Data: Fix or remove any incorrect, incomplete, or irrelevant data. This ensures that the dataset is accurate and
useful. For instance, if you're working with IoT data, you might need to handle missing values.
3. Combining Data: Merge data from different sources into one comprehensive dataset. This might involve joining tables or
datasets together so that all relevant information is in one place.
4. Feature Selection: Identify and select the most important pieces of information (features) that will help the machine learning
model make predictions. Features are individual measurable properties or characteristics of the data.
5. Transforming Data: Convert the data into a suitable format for analysis. This might involve normalizing numerical values,
encoding categorical data, or creating new features that are derived from the original data.
6. Labeling Data (if necessary): For supervised learning, add labels to the data. Labels are the outcomes or target values that
you want the machine learning model to predict.
In essence, an analytical dataset is a well-organized collection of data that contains all the necessary information, cleaned and
formatted, to train a machine learning model effectively. Each row in this dataset is a single instance, and each column is a feature
that helps the model learn patterns and make accurate predictions.
We'll use a retail store as an example, where we want to predict customer
churn (whether a customer will stop shopping at the store).
Steps to Build the Analytical Dataset
1. IdentifyKey Entities and Relationships
Entities are the core objects that store data. In this example, the key entities are:
1. Customers:
Customer ID: A unique identifier for each customer.
Name: The name of the customer.
Age: The age of the customer.
Gender: The gender of the customer.
2. Transactions:
1. Transaction ID: A unique identifier for each transaction.
2. Customer ID: The ID of the customer who made the transaction (links to the Customers entity).
3. Transaction Date: The date when the transaction occurred.
4. Amount Spent: The total amount of money spent in the transaction.
5. Items Purchased: The list of items purchased in the transaction.
3. Products:
1. Product ID: A unique identifier for each product.
2. Product Name: The name of the product.
3. Category: The category to which the product belongs.
4. Price: The price of the product.
CustomerID: Unique identifier for each customer.
Name, Age, Gender, City: Demographic information about the customer.
TotalSpent: Sum of all amounts spent by the customer.
NumTransactions: Total number of transactions made by the customer.
LastTransactionDate: Date of the last transaction made by the customer.
MostFrequentCategory: The product category that the customer buys most frequently.
AvgSpentPerTransaction: Average amount spent per transaction by the customer.
Concept of Linked Analytical Datasets (LAD)
Linking Datasets:
● Identify Natural Bridges: Find fields that can act as links between different datasets. These are often unique identifiers like
CustomerID, ProductID, or TransactionID.
● Create or Use Existing Identifier Keys: Use the same identifier key across multiple datasets to make linking them easy.
Minimize Joins:
● Include Descriptive Fields: Instead of just including IDs, also include descriptions (like product names or customer details) to
avoid unnecessary joins.
● Reduce Data Transformation Work: Preprocess data and include useful features directly in the dataset.
Trade-offs:
● Data Size: Datasets might be larger because of the inclusion of descriptive fields.
● ETL Complexity: Initial extraction, transformation, and loading (ETL) process might be more complex to ensure all useful
features are included.
● Data Duplication: Some data may be duplicated across datasets to simplify analysis.
Scenario 1: An analyst wants to predict customer churn. They can directly use the Customers Analytical Dataset without needing to join
with the Transactions dataset to get necessary features.
Scenario 2: An analyst wants to analyze product performance. They can use the Products Analytical Dataset, which already has
aggregated sales and transaction numbers.
How They Link
● CustomerID: Connects Customer and Transaction data.
● ProductsBought: Lists product names directly in Transactions to avoid joining with Product data.
Advantages
1. Easy Access: Analysts get ready-to-use data with all necessary details.
2. Save Time: No need for complex data joins; everything needed is in one place.
3. Flexible: Data can be quickly linked for various analyses without extra processing.
By organizing data this way, analysts can focus more on analyzing and building models rather than spending time on data preparation.
Example Scenario: Retail Store
1. Customer Data:
○ CustomerID: Unique identifier.
○ Details: Name, Age, Gender, City.
○ Purchase Info: TotalSpent, NumTransactions, LastTransactionDate, MostFrequentCategory.
2. Product Data:
○ ProductID: Unique identifier.
○ Details: ProductName, Category, Price.
○ Sales Info: TotalSold, NumTransactions.
3. Transaction Data:
○ TransactionID: Unique identifier.
○ Details: CustomerID, Date, AmountSpent, ProductsBought.
Managing data lakes
Imagine you have a giant pool where you dump all kinds of water from different
sources—rainwater, river water, and even water from your house. This pool is like
a data lake where you store all kinds of data—information from IoT devices,
company records, customer service logs, etc. The idea is to have everything in
one place so you can easily access and use the data.
In the case of Hadoop, the data lake would be stored in HDFS and probably
accessed through Hive or Spark.
However, if you just keep pouring water into this pool without organizing it or using
it properly, it becomes stagnant and gross, like a swamp. In the same way, if you
keep adding raw data to your data lake without any organization or use, it turns
into a "data swamp." This makes it hard for analysts to find useful information
because the data is messy and [Link] can find it difficult to access
due to security restrictions.
Data Refineries
To prevent this, you need to refine the water, just like how a refinery processes crude oil into useful products like gasoline and plastic.
● Data Refinery: Think of it as a system that takes in raw data (like crude oil) and processes it into useful information (like
gasoline or plastic).
● Process: You take raw data, clean it, organize it, and transform it into high-value, finished products that are easy to use for
analysis and decision-making.
Developing a progression process
Scenario: Baking a New Cake Recipe
Imagine you love baking and you want to create a new cake recipe using various ingredients.
Step-by-Step Process:
1. Experimentation Stage:
○ Collect Ingredients: You gather different ingredients like flour, sugar, eggs, and some unique ones like blueberries and matcha powder.
○ Create Ad Hoc Recipes: You start experimenting by mixing different combinations of these ingredients. You might try a blueberry-matcha cake,
or a matcha cake with a sugar glaze.
2. Regular Reviews:
○ Set Review Meetings: Every week, you and your friends taste the cakes and discuss which recipes are good and which need improvement.
○ Evaluate Progress: During these meetings, you decide which cake recipes are worth refining and which ones aren't tasty.
3. Stages of Development:
○ Early Development: Raw experiments. For example, you try different ingredient combinations without worrying too much about the exact
measurements.
○ Refinement: Start perfecting the recipes. You adjust the amount of sugar, flour, or blueberries to make the cake taste better.
○ Stable Recipes: Finalize the recipes that everyone loves. These recipes are now reliable and can be used repeatedly without further changes.
4. Requirements for Each Stage:
○ Early Development: Just try out different ideas without worrying about perfection.
○ Refinement: Start fine-tuning the recipes, adjusting measurements, and removing ingredients that don't work well.
○ Stable Recipes: Ensure the recipes are perfect, easy to follow, and consistently delicious.
Sure, let's simplify this concept with an easy-to-understand example:
Scenario: Experimenting with IoT Data in a Retail Store
Imagine you are a data scientist at a retail store. You have sensors (IoT devices) that collect data on customer movements, temperature, and
product shelf levels.
Step-by-Step Process:
Experimentation Stage:
Collect Data: You start by collecting raw data from the sensors.
Create Ad Hoc Datasets: You experiment with different ways to organize and analyze this data. For example, you might look at how
temperature affects customer movements.
Regular Reviews:
Set Review Meetings: Every month, you and your team meet to review the datasets you have created.
Evaluate Progress: During these meetings, you decide which datasets are useful and should be improved and which ones are not helpful.
.
Stages of Development:
Early Development: Raw data and initial experiments. For example, you notice a pattern where more
customers visit the store when the temperature is comfortable.
Refinement: Clean and organize the data more. Combine data from different sensors to get a clearer
picture. For example, you combine temperature data with customer movement data to create a dataset
that shows the optimal temperature for customer visits.
Stable Data Products: Finalize the datasets that are most useful and reliable. These datasets are
now ready for regular use in analysis and decision-making. For example, you create a stable dataset that
store managers can use to adjust the temperature to increase customer visits.
Requirements for Each Stage:
Early Development: Just collect and start experimenting with data.
Refinement: Clean the data, remove errors, and combine it with other relevant data.
Stable Data Products: Ensure the data is reliable, easy to use, and regularly updated
Segmentation of Data Lake
Sandbox
● Purpose: This area is like a playground where data scientists and analysts can freely experiment without worrying about
affecting production data.
● Access: Each data scientist may have their own sandbox for individual experimentation, and there's also a shared sandbox for
team collaboration.
● Activities:
○ Initial Experimentation: Trying out new ideas, exploring different datasets, and developing initial models.
○ Model Development: Building and refining machine learning models.
● Example: Imagine each data scientist has a sandbox where they can create and test different algorithms for predicting
customer behavior based on historical sales data. They can try different approaches without affecting the operational data used
by the company.
Mature
● Purpose: This area is where more mature and refined datasets and models are stored. It's still flexible for development but
more organized and controlled.
● Access: Data scientists have full access, but there's no individual sandbox; the team shares this area.
● Control:
○ Source Code Control: All code and scripts used to generate datasets are managed under version control (like Git).
○ Organization: Data sets are structured and documented, making them easier to understand and use collaboratively.
● Example: In the mature area, data scientists work on improving models that have shown promise in the sandbox. They
collaborate on refining algorithms and optimizing performance based on feedback and additional data sources.
Production
Purpose: This area is for finalized, production-ready datasets and models that are used for real-world applications.
Access: Data scientists have read-only access; they can view and analyze data but cannot modify it.
Control:
● Testing: Datasets and models are thoroughly tested to ensure accuracy and reliability.
● Change Control: A formal process is in place for making updates or changes to datasets and models, ensuring stability and
consistency.
Example: Once a model developed in the sandbox and matured in the mature area is ready for deployment, it moves to the
production area. Here, it's used by operational teams to make business decisions, such as predicting customer demand for products
based on real-time sales data.
● Sandbox: Playground for initial experimentation and model development.
● Mature: Area for refined datasets and models, managed with source code control.
● Production: Stable area for tested and finalized datasets and models, with strict change control processes.
By segmenting the data lake in this way, organizations can balance innovation and stability, ensuring that experimental ideas can
evolve into reliable solutions without compromising operational integrity.
The suggested progression process to move datasets between areas is here
Establish a regular recurring review of your datasets
Review the datasets in all three areas for any that should be deleted
Review Sandbox datasets that are ready to move to the Mature area
Review Mature datasets for ones that are ready to move to Production