Skip to content

Implement denormalize()/defuzzify() and extractOriginalURL hook (#396) - #409

Merged
benoit74 merged 2 commits into
mainfrom
benoit74/denormalize-extract-original-url
Nov 17, 2026
Merged

benoit74 merged 2 commits into
mainfrom
benoit74/denormalize-extract-original-url

Conversation

@benoit74

Copy link
Copy Markdown
Collaborator

Summary

Implements item 3 of #396 (denormalization):

  • ArticleUrlRewriter.denormalize()/defuzzify() (and their JS ports in wombatSetup.js): best-effort reverse of normalize()/the fuzzy rules, reconstructing a plausible (not guaranteed exact) original URL from a ZIM path.
  • rules.yaml gains an unreplace_pattern/unreplace per rule (hand-authored, mirroring pattern/replace); generate_rules.py codegens DEFUZZY_RULES/defuzzyRules into Python and JS, with matching generated tests.
  • Patches vendored wombat.js so extractOriginalURL defers to a new wb_info.extract_original_url_function hook, the same way rewriteUrl already defers to wb_info.rewrite_function, letting it call denormalize/defuzzify for cases its own pywb-oriented heuristic can never handle (our rewriting scheme doesn't embed an absolute URL the way pywb's does): CSS url() reads, document.referrer, dynamically created elements, and location.replace()/.assign()/updateLocation.
  • Removes the old, narrower #382/wombat#244 Location-getter patch, superseded by the above.
  • A second commit fixes a real bug found while testing this against a real MDN page: urlRewriteFunction/isAlreadyPrefixed/static_prefix assumed prefix never carried REPLAY_MARKER, but head_insert.html deliberately computes it with the marker (needed to keep wombat.js's wb_rel_prefix non-degenerate, per Rewriting: prepend a constant marker to every ZIM path #408). Every fresh runtime rewrite of a never-before-seen URL (a dynamic import() specifier, a location assignment, ...) was getting the marker added twice, producing a ZIM path with no matching entry. Also fixes extractOriginalURLFunction/denormalize to account for wombat.js's extractOriginalURL handing the hook a value already past the full prefix (marker included) and still percent-encoded as served.

Adds ArticleUrlRewriter.denormalize()/defuzzify() (and their JS ports) as a
best-effort reverse of normalize()/the fuzzy rules, and patches vendored
wombat.js so extractOriginalURL() defers to a new
wb_info.extract_original_url_function hook, the same way rewriteUrl() already
defers to wb_info.rewrite_function. This lets CSS url() reads,
document.referrer, dynamically created elements, and
location.replace()/.assign()/updateLocation recover a plausible original URL,
none of which the previous, narrower Location-getter patch (superseded here)
ever covered.

Closes #396.
head_insert.html's prefix deliberately already ends with REPLAY_MARKER (so
wombat.js's own wb_rel_prefix stays non-degenerate), but urlRewriteFunction
and isAlreadyPrefixed still assumed it didn't, adding the marker a second
time whenever a never-before-seen URL got rewritten at runtime (a dynamic
import() specifier, a location assignment, ...), producing a ZIM path with
no matching entry. static_prefix had the opposite problem, needing the
marker-free root since static assets are never normalize()'d.

Also fixes extractOriginalURLFunction/denormalize, exercised while
reproducing the above against a real page: wombat.js's extractOriginalURL
hook hands it the value already stripped of the full prefix (REPLAY_MARKER
included, for the same reason), not merely the serving root, and that value
is still percent-encoded as served (only '/', '=' and ',' literal) rather
than the plain ZIM path denormalize() expects.
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (55223c3) to head (829b3fb).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #409   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           42        42           
  Lines         2961      2989   +28     
  Branches       453       457    +4     
=========================================
+ Hits          2961      2989   +28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benoit74 benoit74 self-assigned this Nov 17, 2026
@benoit74
benoit74 merged commit 5cb9bd3 into main Nov 17, 2026
9 of 10 checks passed
@benoit74
benoit74 deleted the benoit74/denormalize-extract-original-url branch November 17, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant