Smart Organizer for Messy Archive
String Cleaning & Standardization
1. Trim whitespace (left/right/both/middle/all)
sc.trim_whitespace()
2. Change Cases
sc.change_case()
3. Strip accents (e.g., “café” → “cafe”)
sc.change_accents()
4. Remove non-ASCII characters
sc.remove_non_ascii()
5. Remove punctuation
sc.remove_punctuations()
6. Remove numbers from strings
sc.remove_numbers()
7. Remove emojis/symbols
Column Name Cleaning
11.Remove special characters
12.Make all column names lowercase
13.Replace spaces with underscores
sc.space_to_underscore()
14.Remove units from names (e.g., “Revenue (INR)” → “revenue”)
sc.space_to_underscore()
15.Rename columns using a dictionary/map
16.Auto-detect and rename duplicate columns
17.Reorder columns alphabetically
18.Move ID/name columns to the front
19.Add prefixes/suffixes to column names
Numerical Data Cleaning
21.Remove commas from numbers
sc.space_to_underscore()
22.Convert character to numeric
23.Replace out-of-bound values
24.Cap/floor outliers (winsorizing)
25.Impute missing numeric with mean
26.Impute with median
27.Standardize (z-score)
28.Normalize (0–1)
29.Detect and remove outliers (IQR method)
Missing Data Handling
31.Replace “NA”, “null”, “N/A”, “”, “.” with NA
sc.standardize_na()
32.Detect columns with all missing
sc.rm_ghost_columns()
33.Drop rows with high NA %
34.Drop columns with high NA %
35.Flag rows with any missing value
36.Visual summary of missing values
37.Impute using regression
38.Interpolate missing values (time series)
39.Fill down/up (last observation carried forward)
40.Replace NA with custom value per column
Date & Time Cleaning
41.Convert text to Date
42.Convert text to POSIXct (datetime)
43.Parse inconsistent date formats
44.Split datetime into date and time
45.Extract year/month/day from Date
46.Calculate age from DOB
47.Impute missing dates with median
48.Detect future dates and correct
49.Standardize date format (yyyy-mm-dd)
50.Fill missing months/years
Categorical Cleaning
51.Lowercase all levels
52.Remove unused factor levels
53.Merge similar levels (e.g., “M” & “Male”)
54.Replace levels using mapping
55.Sort levels
56.Relevel to set base category
57.Convert categorical to numeric
58.Encode using one-hot encoding
59.Encode with label encoding
60.Detect rare categories and group as “Other”
Duplicate & Consistency Checks
61.Remove exact duplicate rows
62.Remove based on subset of columns
63.Keep first/last occurrence
64.Flag duplicates instead of removing
65.Detect inconsistent spellings (e.g., “Kolkata” vs “Kolkatta”)
66.Merge rows with minor differences
67.Detect inconsistent capitalization
68.Check key uniqueness (ID duplication)
69.Suggest key columns for uniqueness
70.Remove empty rows
Data Summary & Reports
71.Summary of data types
72.Count missing values
73.Count unique values per column
74.Most frequent value per column
75.% of zeros per column
76.Column-wise correlation matrix
77.Skewness/kurtosis per column
78.Visual heatmap of missingness
79. Full EDA report (like DataExplorer)
80.Print top 5/10 rows per column (wide preview)
Validation Checks
81.Validate emails
sc.validate_email()
82.Validate phone numbers
sc.validate_mobile()
83.Check for negative values where not allowed
84.Validate postcodes
85.Validate IP addresses
86.Validate URLs
87.Check date range validity
88.Check numeric value ranges
89.Detect typos using fuzzy matching
90.Validate data using custom rules
Text & NLP Preprocessing
91.Remove stop words
92.Tokenize text
93.Lemmatize/stem words
94.Count words per row
95.Detect language of text
96.Remove HTML tags
97.Clean hashtags/mentions
98.Remove URLs from text
99.Calculate TF-IDF
100. Extract keywords or named entities
Unit & Scale Related
101. standardize_units() – Convert mixed units to one standard
102. detect_mixed_units() – Flag columns with inconsistent units
103. convert_temperature_units() – Celsius ↔ Fahrenheit ↔ Kelvin
104. convert_currency() – Normalize currencies using exchange rate
sc.standardize_currency()
105. scale_by_column() – Scale values column-wise with custom
multipliers
Validation & Rules
106. validate_custom_rules() – Apply user-defined data rules
107. validate_identifier_format() – E.g., PAN, Aadhaar, SSN
108. check_duplicate_keys() – Validate primary key uniqueness
109. validate_column_dependencies() – e.g., "if A=yes, then B must not
be null"
110. check_value_coherence() – Detect illogical combinations (e.g., age
= 200)
Advanced Numerical Cleaning
111. log_transform() – Apply log(x+1) or similar
112. bin_numerical_column() – Discretize into bins (equal
width/frequency)
113. detect_constant_columns() – Columns with same value throughout
114. detect_near_zero_variance() – Flag uninformative columns
115. detect_high_correlation() – Flag columns highly correlated
String Enhancements
116. remove_html_entities() – Convert & etc.
117. unescape_unicode() – Turn \u2019 into '
118. smart_case_conversion() – Use ML to guess correct case
119. spell_check_column() – Flag likely misspellings
120. remove_brackets_content() – Remove text inside (), [], {}
Cross-column Cleaning
121. coalesce_columns() – Combine multiple columns into one with
priority
122. compare_columns() – Flag differences in duplicate columns
123. create_flag_columns() – Add flag for logic-based checks (e.g.,
is_senior)
124. harmonize_multiple_columns() – Normalize synonymous fields
125. combine_date_parts() – Combine year/month/day into full date
Rare and Powerful Tools
126. split_column_by_delimiter() – Turn comma-separated text into lists
sc.split_by_delimiter()
127. merge_rows_same_id() – Combine duplicate IDs by aggregating
128. extract_regex_matches() – Extract using custom patterns
Smart Detection
129. detect_column_type() – Auto classify column (date, text, numeric
etc.)
130. detect_language_column() – Find language used in free text
131. detect_outlier_patterns() – Use ML to detect anomaly rows
132. detect_potential_keys() – Suggest key columns for joins
133. detect_inverted_columns() – e.g., swapped first_name/last_name
Reporting & Audits
134. generate_data_audit() – Full pre-cleaning report
135. generate_post_clean_summary() – Before-after comparison
136. change_tracking_log() – Log all cleaning steps
137. flag_suspect_rows() – Heuristic scoring for messy rows
138. summarize_by_group() – Quick stats grouped by a column
Automation & Integration
139. auto_clean_all() – One-click smart cleaner (calls common functions)
140. profile_column() – Deep dive into 1 column
141. autodetect_column_roles() – Suggest ID, time, target, feature
columns
142. apply_cleaning_pipeline() – Apply saved pipeline of steps
143. auto_generate_clean_code() – Return R code for steps applied
International & Locale
144. convert_locale_format() – Handle comma/period decimal issues
145. translate_column_values() – Use dictionary or API to translate
146. standardize_address_fields() – Normalize address columns
147. parse_phone_number() – Extract country code, number
148. convert_local_timezones() – Normalize across time zones