Scribd
Upload
6 views2 pages

EDA and Modelling Helper Functions

This module provides helper functions for exploratory data analysis (EDA) and modeling exercises, including functions to read and clean demand and promotion data. It also includes functionality to merge demand with promotions and extend promotions over multiple days. Additionally, there is a function to aggregate data to a weekly format.

Uploaded by

salimshaik045
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
6 views2 pages

EDA and Modelling Helper Functions

This module provides helper functions for exploratory data analysis (EDA) and modeling exercises, including functions to read and clean demand and promotion data. It also includes functionality to merge demand with promotions and extend promotions over multiple days. Additionally, there is a function to aggregate data to a weekly format.

Uploaded by

salimshaik045
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

"""

This module contains helper functions for the EDA and modelling
exercises. Feel free to use them to get you started more quickly.

"""
import pandas as pd
import numpy as np
from datetime import datetime

def parse_time(s):
return [Link](s, "%Y-%m-%d").date()

def read_demand(path):
df = pd.read_csv(path)
df = [Link](date=lambda df: [Link](parse_time))
df = df.set_index("date")
[Link] = [Link]([Link])
return df

def read_promotions(path):
df = pd.read_csv(path, index_col=0)
df = [Link](promotion_date=lambda df:
df.promotion_date.apply(parse_time))
df = df.set_index("promotion_date")
[Link] = [Link]([Link])
return df

def clean(ts: [Link]) -> [Link]:


# Replaces missing values
return [Link]().fillna([Link]())

def clean_demand_per_group(demand: [Link]) -> [Link]:


"""TODO add docstring"""
sus = [Link]()
skus = [Link]()
for su in sus:
for sku in skus:
[Link][([Link] == sku) & ([Link] == su),
"demand"] = clean([Link][([Link] == sku) & ([Link] ==
su), "demand"])
return demand

def merge(demand: [Link], promotions: [Link]) ->


[Link]:
promotions = promotions.rename_axis("date").assign(promotion=True)
demand = [Link](
promotions,
on=["supermarket", "sku", "date"],
how="outer",
)
demand = [Link](promotion=lambda df:
[Link](False))
return demand

def extend_promotions_days(promotions, n_days):


""" Extends the promotions to have multiple rows for a specific
number of days.
The input promotions is assumed be specified with a single row with a
starting date.
The output extends the input promotions with multiple days, one row
for each day of the promotion.
"""
n_promotions = len(promotions)
initial_promotions = [Link]()
promotion_id = [Link](n_promotions)
extended_promotions =
[Link]().assign(promotion_id=promotion_id)
for days_to_add in range(1, n_days):
additional_promotion_days =
initial_promotions.copy().assign(promotion_id=promotion_id)
additional_promotion_days.index += [Link](days_to_add, "d")
extended_promotions =
extended_promotions.append(additional_promotion_days)
return extended_promotions

def aggregate_to_weekly(df):
grouped = [Link](["sku", "supermarket"])
# Performs a simplistic aggregation of promotion. If a promotion
occured during the week this variable will be true.
weekly = [Link](lambda df: [Link]("W").agg({"demand":
"sum", "promotion": "max"}))
weekly = weekly.reset_index().set_index("date")
return weekly

Common questions

Powered by AI

Setting the date as an index and converting it into a DateTimeIndex is crucial as it optimizes the DataFrame for time series operations, facilitating efficient resampling, rolling aggregations, and time-based indexing and slicing. This enhances performance and ease in temporal analyses.

The 'read_demand' function reads a CSV file into a DataFrame, transforms the 'date' column to a datetime format using the 'parse_time' function, sets this column as the index, and ensures the index is a DateTimeIndex. This prepares the DataFrame for time series operations.

The 'aggregate_to_weekly' function groups the DataFrame by 'sku' and 'supermarket', then resamples the data to weekly frequency. It sums the 'demand' and takes the maximum of 'promotion' within each week. This transformation allows for analysis on a broader temporal scale, smoothing daily fluctuations while highlighting weeks with any promotions.

The 'clean_demand_per_group' function methodically applies the cleaning process to individual groups formed by unique combinations of supermarkets and SKUs. This ensures that each group retains its intrinsic demand characteristics while missing data is cleaned group-specifically to preserve the unique variance and demand patterns intrinsic to each group.

The 'merge' function integrates demand and promotion DataFrames by performing an outer join on 'supermarket', 'sku', and 'date', ensuring that all entries are included regardless of a match. Missing promotion values are then set to False, indicating no promotion occurred, thereby maintaining data integrity while incorporating promotional context.

The 'extend_promotions_days' function duplicates each promotion entry for a specified number of days by adding new rows for each day a promotion lasts, incremented by days using Timedelta. 'Promotion_id' serves as a unique identifier for each promotion event, ensuring traceability of these extended entries across the time span of the promotion.

The 'clean' function aims to address missing values in a time series. It first fills any gaps by propagating the last valid observation forward to the next valid observation (backfill), and any remaining missing values are replaced with the series' mean. This approach balances preserving trends and filling gaps effectively.

Using backfill and mean filling can smooth out short-term data volatility and fill in gaps which might bias prediction models by diluting pronounced patterns in the data. This might impact the model's sensitivity to detect swift changes or anomalies essential in demand forecasting or anomaly detection tasks.

Resetting the index in 'aggregate_to_weekly' ensures that 'date' becomes an explicit column instead of an index after aggregating by week. This restores the DataFrame structure for further operations or merges where 'date' as a linearly accessible column is crucial.

Extending promotion days simulates the ongoing impact of a promotion beyond its initial day, potentially capturing the extended influence on demand patterns across time. This can lead to models that better understand lagged effects of promotions, allowing for more accurate prediction of sustained increases in demand beyond promotional periods.

You might also like