totally fair — regex feels alien at first.

let’s make it super concrete with tiny, repeatable patterns +

“english first” explanations. we’ll use only four tools:
 col("...").rlike("pattern") → returns True/False
 regexp_extract(col("..."), "pattern", group_index) → pull a piece out
 regexp_replace(col("..."), "pattern", "replacement") → clean/transform
 split(col("..."), "pattern") → split by a regex delimiter
tip for PySpark strings:
 in Python, write regex as raw strings: r"\d+" so \d stays \d
 in SQL expr strings, escape backslashes: "\\d+"

1) 80/20 mini-dictionary (with plain English)

Pattern Read it as Matches example
^/$ start / end of string ^hi$ matches exactly "hi"
. any single char a.c matches "abc","axc"
\d / \D digit / non-digit \d\d → 2 digits
\w / \W “word” (letters/digits/underscore) / not \w+ → word chunk
\s / \S whitespace / not \s+ → one or more spaces
[abc] one of a, b, or c [ae] in "tea" hits e
[^abc] anything except a/b/c [^0-9] → not a digit
`a b` a or b
?/*/+ 0 or 1 / 0+ / 1+ \d+ → 1+ digits
{m} / {m,n} exactly m / between m and n \d{10} → 10 digits
(...) capture group (\d{3})-(\d{2})

2) Tiny dataset to practice

from [Link] import col, regexp_extract, regexp_replace, split, expr, when

data = [
(1, "Alice Johnson", "alice.j@[Link]", "+91-98765-43210",
"[Link]
(2, "Bob Singh", "bob_singh@[Link]", "9876543210", "[Link]
(3, "cara.k", "cara.k at mail dot com", "(040) 2345-6789", "[Link]
(4, " Dan O'Neil ", "[Link]+test@[Link]", "91 98765 43210",
"[Link]
(5, "Eve", None, None, None),
]
df = [Link](data, ["id","name","email","phone","url"])

3) 10 “recipes” (each shows: pattern → plain English → result)

3.1 Is this a basic email shape?
 Pattern: ^[^@\s]+@[^@\s]+\.[A-Za-z]{2,}$
 English: start→ “some non-space, non-@” → @ → “some non-space, non-@” → dot → 2+
letters → end
df1 = [Link](
"email_ok",
col("email").rlike(r"^[^@\s]+@[^@\s]+\.[A-Za-z]{2,}$")
)
[Link]("id","email","email_ok").show(truncate=False)
3.2 Extract the email domain
 Pattern: @([^@\s]+)$
 English: at-sign, then capture all non-space, non-@ until end
df2 = [Link]("email_domain", regexp_extract(col("email"), r"@([^@\s]+)$", 1))
[Link]("id","email","email_domain").show(truncate=False)
3.3 Clean “name with extra spaces”
 Pattern 1: \s+ → collapse multiple spaces to one
 Pattern 2: ^\s+|\s+$ → trim leading OR trailing spaces
df3 = (df
.withColumn("name_1", regexp_replace(col("name"), r"\s+", " "))
.withColumn("name_clean", regexp_replace(col("name_1"), r"^\s+|\s+$", ""))
)
[Link]("id","name","name_clean").show(truncate=False)
3.4 Keep only digits from phone
 Pattern: \D
 English: any non-digit → replace with nothing
df4 = [Link]("digits", regexp_replace(col("phone"), r"\D", ""))
[Link]("id","phone","digits").show(truncate=False)
3.5 Format a 10-digit phone as XXX-XXX-XXXX
 Pattern: ^(\d{3})(\d{3})(\d{4})$ with replacement $1-$2-$3
df5 = (df4
.withColumn("phone_fmt",
when(col("digits").rlike(r"^\d{10}$"),
regexp_replace(col("digits"), r"^(\d{3})(\d{3})(\d{4})$", r"$1-$2-$3"))
.otherwise(col("digits"))
)
)
[Link]("id","digits","phone_fmt").show(truncate=False)
3.6 Does URL start with http/https/ftp?
 Pattern: ^(?i)(https?|ftp)://
 English: start → (case-insensitive) http or https or ftp → ://
df6 = [Link]("has_protocol", col("url").rlike(r"^(?i)(https?|ftp)://"))
[Link]("id","url","has_protocol").show(truncate=False)
3.7 Extract path from URL (stuff after the domain)
 Pattern: ^(?i)(?:https?|ftp)://[^/]+(/.*)$
 English: protocol:// then non-slash host, then capture /... until end
df7 = [Link]("url_path", regexp_extract(col("url"), r"^(?i)(?:https?|ftp)://[^/]+(/.*)$", 1))
[Link]("id","url","url_path").show(truncate=False)
3.8 Remove query string (? and after)
 Pattern: \?.*$
 English: a ? and then anything till end
df8 = [Link]("url_base", regexp_replace(col("url"), r"\?.*$", ""))
[Link]("id","url","url_base").show(truncate=False)
3.9 Fix obfuscated email: "name at mail dot com" → name@[Link]
 Patterns: \s+at\s+ → @, then \s+dot\s+ → .
df9 = [Link](
"email_fixed",
 regexp_replace(
regexp_replace(col("email"), r"\s+at\s+", "@"),
r"\s+dot\s+", "."
)
)
[Link]("id","email","email_fixed").show(truncate=False)
3.10 Split URL path into segments
 Pattern: remove protocol+host, then split by /+
df10 = (df
.withColumn("path_only", regexp_replace(col("url"), r"^(?i)(?:https?|ftp)://[^/]+/?", ""))
.withColumn("segments", split(col("path_only"), r"/+"))
)
[Link]("id","url","segments").show(truncate=False)

4) How to “read” a regex (training wheels)

1. Look for anchors ^ (start) and $ (end) — they set boundaries.
2. Identify chunks in parentheses — those are groups you can extract.
3. Replace cryptic classes with words in your head:
o \d{10} → “ten digits”
o \s+ → “one or more spaces”
o [A-Za-z]{2,} → “2+ letters”
4. If it says ?i, think “ignore case.”
5. For replace patterns with $1, $2, imagine you’re re-arranging the captured chunks.

5) When strings get picky (escaping)

 In Python, prefer raw strings: r"\d{3}" (so \d stays intact).
 In SQL via expr/selectExpr, you often need double backslashes: "\\d{3}".
Example:
 [Link]("regexp_extract(email, '@([^@\\s]+)$', 1) as domain").show()

6) Quick practice set (with answers)

Q1. Make a flag name_clean_ok that is True if name contains only letters, spaces, or apostrophes
after collapsing spaces.
Answer (one way):
names = (df
.withColumn("n1", regexp_replace(col("name"), r"\s+", " "))
.withColumn("n2", regexp_replace(col("n1"), r"^\s+|\s+$", ""))
.withColumn("name_clean_ok", col("n2").rlike(r"^[A-Za-z' ]+$"))
)
[Link]("id","name","n2","name_clean_ok").show(truncate=False)
Q2. Extract the subdomain from URLs like [Link] → sub (empty if none).
Answer:
subs = [Link]("subdomain", regexp_extract(col("url"), r"^(?i)(?:https?|ftp)://([^.]+)\.[^/]+\.",
1))
[Link]("id","url","subdomain").show(truncate=False)
Q3. Mask email username to first letter + ***** + domain (e.g., a*****@[Link]).
Answer:
masked = (df
 .withColumn("user", regexp_extract(col("email"), r"^([^@]+)@", 1))
.withColumn("dom", regexp_extract(col("email"), r"@([^@\s]+)$", 1))
.withColumn("email_masked", when((col("user")!="") & (col("dom")!=""),
expr("concat(substring(user,1,1),'*****','@',dom)")))
)
[Link]("id","email","email_masked").show(truncate=False)

7) Mental model to carry forward

 rlike → “does it look like this?” (True/False)
 regexp_extract → “give me the part inside (…) group #k”
 regexp_replace → “find X, change to Y (everywhere)”
 split → “cut the string by regex delimiter into an array”
Want me to turn this into a printable cheat-sheet PDF or a Databricks notebook with markdown
explanations between cells?