Contents

FreshData — API & Key Features Reference 1

What Is FreshData? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Core API (70 public exports) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Enterprise API (89 lazy exports) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Integrations (lazy exports) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Semantic Models ([Link]) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Experimental — AI Copilot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Key Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
CleanConfig — Key Options (61 total) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
CLI Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Quick Reference — Common Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
API Surface Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

FreshData — API & Key Features Reference

Version: 1.1.1
PyPI: freshdata-cleaner · Import: import freshdata as fd
Repo: FreshCode-Org/freshdata

What Is FreshData?

FreshData is an explainable, automated data-cleaning library for pandas (and optional Polars/DuckDB/Spark).
Unlike simple fillna wrappers, it runs a per-column decision engine that profiles each column and chooses
the right action — with a full audit trail of every change (rationale, risk level, confidence score).

import pandas as pd
import freshdata as fd

df = pd.read_csv("messy_export.csv")
cleaned = [Link](df)
cleaned, report = [Link](df, return_report=True)
print([Link]())

1
Installation

pip install freshdata-cleaner

Extra Adds
ml KNN / MissForest imputation, isolation forest
polars Polars DataFrame in/out
duckdb Out-of-core DuckDB execution
spark PySpark execution
viz Interactive HTML reports
privacy PII detection & anonymization
enterprise Trust score, lineage, compliance, quality-ops
semantic Local embedding model for semantic cleaning
all Everything above

Core API (70 public exports)

Cleaning

API Description
[Link](df, **options) Main one-call cleaner. Returns new DataFrame (input
never mutated by default).
fd.clean_csv(path, ...) Read CSV → clean → optionally write output.
fd.clean_domain_file(path, domain=..., Parse domain file → validate → clean.
format=...)
fd.clean_timeseries(df, ...) Time-series-aware cleaning (ordering, gaps,
anomalies).
[Link](**options) Reusable configured cleaner for batch jobs.
[Link] Frozen, validated configuration object (61 options).

Common options:

[Link](
df,
strategy="balanced", # "conservative" | "balanced" | "aggressive"
impute="median", # override engine imputation
outliers="clip", # override outlier handling
drop_duplicates=True,
fix_dtypes=True,
column_names=True, # snake_case + dedupe
verbose=True,
return_report=True, # returns (df, CleanReport)
progress_callback=callback # optional pipeline progress events
)

Profiling & Inspection

2
API Description
[Link](df) Read-only column/dataset profiling (same inference as
clean).
fd.infer_roles(df) Detect id / target / text / numeric / datetime roles.
fd.explain_clean(before, after) Reverse-engineer what cleaning changed.
[Link](df, domain=...) Domain validation without full clean.

Context Policies (NL Rules)

API Description
fd.compile_context(rules, df=...) Compile plain-English rules → ContextPolicy.
[Link] Structured policy object.
[Link] Per-column constraint from policy.

Example:

policy = fd.compile_context(
"CustomerID is unique. Never modify revenue. Email must be masked.",
df=df,
)
cleaned, report = [Link](df, policy=policy, return_report=True)

Plan & Repair Workflow

API Description
fd.suggest_plan(df) Dry-run column repair plans → CleanPlan.
fd.apply_plan(df, plan) Execute an approved repair plan.
fd.compare_plans(plan_a, plan_b) Diff two plans.
fd.compare_clean(run_a, run_b) Compare two clean runs.

Reports & Results

API Description
[Link] Full audit log: actions, warnings, row/col counts,
duration.
[Link] Wrapper with .report(), .summary(),
.visualize().
[Link] Single cleaning action (column, count, rationale, risk,
confidence).
[Link] / [Link] Profiling output types.
[Link] Explain-clean output.

Simple API (function-first)

Lightweight helpers for common tasks without the full engine:

3
fd.fill_missing(df, method="median") # auto, mean, median, mode, constant, ffill, bfill
fd.detect_outliers(df) # IQR / z-score
fd.remove_outliers(df) # drop outlier rows
fd.resolve_duplicates(df, keep="first") # first / last / drop
fd.group_aggregate(df, by="col", agg="mean")
fd.pipeline_clean(df) # chained fill → outliers → dedupe

Diagnostics

API Description
fd.cdc_profile(df, event_time=..., key=...) CDC/event-time defect profiling (stale, late,
out-of-order, etc.).
fd.lint_text_encoding(df) Unicode/text quality lint (mojibake, mixed script,
etc.).
fd.evaluate_quality_debt(df) 9-dimension quality debt ledger with SQLite
persistence.
fd.insight_report(report) JSON/HTML insight report.
fd.trust_gate_report(report) Trust-gate presentation.
fd.stakeholder_summary(report) Business-language summary.

Compliance

API Description
fd.generate_compliance_report(report, Map clean report to regulatory frameworks.
frameworks=[...])

Frameworks: 21cfr_11, gdpr_30, alcoa_plus, sox_404, hipaa_safe_harbor

Memory & Learning

API Description
fd.learn_cleaning_memory(messy, clean) Learn semantic repair memory from example pair.
fd.load_cleaning_memory(path) Load memory artifact.
[Link] Replay compatible repairs during clean.
[Link](messy, clean) Learn .fdprofile (lazy export).
fd.save_profile(profile, path) Persist learned profile.
fd.load_profile(path) Load profile artifact.

Plugins

API Description
fd.register_expert(cls) Register semantic expert plugin.
fd.register_backend(cls) Register semantic backend plugin.
fd.register_validator(cls) Register validator plugin.
fd.registered_plugins() List all registered plugins.

4
Parsing & Domains

API Description
fd.parse_domain(path, format=...) Parse HL7/FHIR/GPX/SDMX/EDIFACT →
DataFrames.
[Link](df, domain="finance") Apply industry domain pack (validate + repair).

Domain packs: finance, retail, transport, healthcare, education, agriculture, media, energy
Format parsers: fhir, hl7v2, gpx, sdmx, edifact

Streaming & Time-Series

API Description
[Link] Micro-batch cleaner with bounded state.
[Link] Window size, warmup, drift thresholds, trust gate.
[Link] Cross-batch state container.
[Link] Timestamp ordering, interpolation, seasonal impute,
anomalies.

Execution Backends

[Link](df, engine="duckdb", output_format="pandas")

[Link](df, engine="polars", output_format="polars-lazy")

Engine Output formats

pandas (default) pandas, polars, arrow
polars pandas, polars, polars-lazy, arrow
duckdb pandas, duckdb (lazy relation)
spark pandas, spark
freshcore pandas
auto picks best for input type

Enterprise API (89 lazy exports)

Accessed as fd.<name> or from [Link] import ... — loaded only when used.

Enterprise Cleaning & Trust

result = fd.clean_enterprise(df, enterprise=[Link](...))

score = fd.compute_trust_score(df) # 0–100 Data Trust Score
report = fd.build_quality_report(df)

Lineage

tracker = [Link]()
schema = fd.schema_of(df) # schema fingerprint

5
Clustering & Masking

fd.cluster_column(df, "vendor")
fd.merge_clusters(df, cols=("vendor",))
fd.mask_dataframe(df, rules=[[Link](...)])

PII & Privacy

scan = fd.detect_pii(df)
anonymized = [Link](df, config=...)
fd.check_k_anonymity(df, quasi_identifiers=[...])
vault = fd.make_vault("sqlite", path="[Link]")
token = fd.tokenize_value("secret", vault=vault)
fd.apply_privacy_policy(df, policy=[Link](...))

Data Contracts & Drift

baseline = fd.build_baseline(df)
fd.save_baseline(baseline, "[Link]")
drift = fd.compare_to_baseline(df, baseline)
violations = fd.monitor_contract(df, contract=[Link](...))

Entity Resolution

result = fd.resolve_entities(df_a, df_b, config=[Link](...))

fd.link_entities(left, right, ...)
queue = fd.build_review_queue(result)

Semantic Validation

fd.run_semantic_validation(df, configs=[[Link](...)])
fd.validate_columns(df, validators=[[Link](...)])

Join Assistant & Templates

fd.suggest_join_keys(left, right)
template = fd.get_template("healthcare")

Integrations (lazy exports)

API Platform
fd.export_dbt_tests(report, path=...) dbt
fd.export_gx_suite(report, suite_name=..., Great Expectations
path=...)
fd.export_quality_ops(report, ...) Multi-format quality-ops export
fd.build_exception_table(report) Exception/quarantine table
FreshDataResource Dagster
freshdata_asset_check Dagster asset checks

6
API Platform
FreshDataCleanOperator Airflow

Semantic Models ([Link])

[Link]() # installed model status
[Link]() # download local semantic model
[Link].list_available() # available models

Requires [semantic] extra.

Experimental — AI Copilot
from [Link].ai_copilot import analyze_dataset

report = analyze_dataset(
df,
goal="Prepare for analytics and ML",
privacy="mask_pii_before_reasoning",
context_policy={"email": "must_mask", "age": "must_be_between_0_and_120"},
)
print([Link])
print(report.cleaning_plan)
print(report.recommended_code)

Deterministic, offline, privacy-first. No API key required.

Key Features
1. Two-Layer Cleaning Architecture

Layer What it does Default

Representation repair Whitespace, sentinels, dtypes, Always on
empty rows/cols, duplicates
Decision engine Per-column missing/outlier strategy="balanced"
imputation based on role +
thresholds

2. Per-Column Decision Engine

Profiles every column (missing ratio, skewness, cardinality, inferred role) and applies explicit rules:

• Missing: preserve, drop column, mean, median, mode, KNN, MissForest, time-fill
• Outliers: auto, cap (winsorize), remove, flag — via IQR, z-score, or isolation forest
• Protected: IDs never imputed; targets never modified; outliers ignored on IDs

7
3. Explainable Audit Trail

Every action in CleanReport includes:

• Column name and affected row count

• Human-readable rationale
• Risk level (low / medium / high)
• Confidence score (0–1)
• Warnings and manual-review recommendations

4. Pipeline Stages (16 steps)

context → column_names → strings → empty_columns → empty_rows → dtypes

→ constant_columns → duplicates → semantic → engine_missing → engine_outliers
→ missing → outliers → memory → protected_verify → index → complete

5. Semantic Cleaning (optional)

10 built-in experts repair values like spelled numbers, boolean synonyms, currency strings, units, category syn-
onyms, dates, emails, phones — with modes:

• assist — report proposals only

• review — apply zero-risk repairs
• auto — apply high-confidence repairs

6. Context Policies

Write rules in plain English; FreshData compiles them into column constraints:

“CustomerID is unique. Never modify revenue. Normalize city spelling.”

7. Domain Packs (8 industries)

Built-in validate + repair for finance, retail, transport, healthcare, education, agriculture, media, energy.

8. Out-of-Core Execution

Clean datasets larger than memory via DuckDB, Polars, or Spark backends without changing the API.

9. Streaming

Micro-batch cleaning for CSV/Parquet files and Kafka topics with bounded memory, drift detection, and rolling
trust scores.

10. Enterprise Governance

Trust scoring, PII masking, entity resolution, data contracts, drift monitoring, lineage tracking, compliance reports
— all optional.

8
11. Quality Debt Ledger

Tracks 9 quality dimensions across runs in SQLite; escalates from warn → fail on repeated issues.

12. Plugin Ecosystem

Extend with custom semantic experts, backends, validators, domain packs, and format parsers via entry points.

CleanConfig — Key Options (61 total)

Structure & Text

column_names, drop_empty_rows, drop_empty_columns, drop_constant_columns, strip_whitespace,

normalize_sentinels, extra_sentinels, string_case, reset_index

Dtype Parsing

fix_dtypes, numeric_threshold, datetime_threshold, preserve_leading_zeros, dayfirst, decimal,

thousands

Strategy & Thresholds

strategy, missing_threshold_low, missing_threshold_medium, missing_threshold_high, duplicate_threshold

Missing Values

impute, impute_strategy, advanced_imputation, missing_indicators, MissForest options (5 fields)

Outliers

outlier_action, outliers, outlier_method, outlier_factor

Role Protection

preserve_columns, target_column, id_columns, preserve_original

Duplicates

drop_duplicates, duplicate_subset, duplicate_keep, allow_timeseries_duplicates

Semantic Layer

semantic_mode, semantic_auto_threshold, semantic_review_threshold, semantic_backends, semantic_context,

semantic_privacy_policy, semantic_budget

9
Context & Observability

context, policy, strict, verbose, progress_callback, optimize_memory

CLI Commands

freshdata clean FILE # Clean file + quality/lineage reports

freshdata plan FILE # Suggest repair plan JSON
freshdata apply-plan FILE PLAN # Execute approved plan
freshdata profile FILE # Profile (audit/diff/merge sub-modes)
freshdata learn --messy A --clean B # Learn .fdprofile
freshdata trust FILE # Data Trust Score
freshdata quality-ops FILE # Export dbt/GX/exceptions/lineage
freshdata policy compile RULES # Compile NL rules to policy JSON
freshdata models status # Semantic model status
freshdata models pull # Download semantic model
freshdata stream FILE # Micro-batch CSV/Parquet clean
freshdata stream-kafka TOPIC # Kafka topic clean
freshdata benchmark-stream # Streaming memory benchmark

Quick Reference — Common Patterns

Clean with report

cleaned, report = [Link](df, return_report=True)

print([Link]())
for action in report:
print([Link], [Link], [Link])

Reusable cleaner for a directory

cleaner = [Link](strategy="balanced", drop_constant_columns=True)

for path in paths:
df = pd.read_csv(path)
cleaned = [Link](df)
print(cleaner.report_.summary())

Domain-aware clean

cleaned, report = [Link](df, domain="healthcare", return_report=True)

Progress callback

events = []
cleaned = [Link](df, progress_callback=[Link], verbose=False)
for e in events:
print(e["step"], e["rows"], e["columns"])

10
Polars in/out

import polars as pl
cleaned = [Link](pl_df) # returns [Link] when input is Polars

Compliance report

_, report = [Link](df, return_report=True)

bundle = fd.generate_compliance_report(report, frameworks=["gdpr_30", "hipaa_safe_harbor"])

API Surface Summary

Layer Count
Core public exports (__all__) 70
Enterprise lazy exports 89
Integration + learning lazy exports 9
Pipeline stages 16
CleanConfig options 61
Simple API functions 6
Domain packs 8
Format parsers 5
Semantic experts 10
Execution engines 6
Compliance frameworks 5
CLI commands 12

Further Reading

• Official docs
• Quickstart
• API reference (auto-generated)
• Feature overview
• Cleaning engine deep-dive
• Context policies
• Enterprise / compliance

11