Repository navigation
Implement denormalize()/defuzzify() and extractOriginalURL hook (#396) - #409
Merged
Merged
Conversation
Adds ArticleUrlRewriter.denormalize()/defuzzify() (and their JS ports) as a best-effort reverse of normalize()/the fuzzy rules, and patches vendored wombat.js so extractOriginalURL() defers to a new wb_info.extract_original_url_function hook, the same way rewriteUrl() already defers to wb_info.rewrite_function. This lets CSS url() reads, document.referrer, dynamically created elements, and location.replace()/.assign()/updateLocation recover a plausible original URL, none of which the previous, narrower Location-getter patch (superseded here) ever covered. Closes #396.
head_insert.html's prefix deliberately already ends with REPLAY_MARKER (so wombat.js's own wb_rel_prefix stays non-degenerate), but urlRewriteFunction and isAlreadyPrefixed still assumed it didn't, adding the marker a second time whenever a never-before-seen URL got rewritten at runtime (a dynamic import() specifier, a location assignment, ...), producing a ZIM path with no matching entry. static_prefix had the opposite problem, needing the marker-free root since static assets are never normalize()'d. Also fixes extractOriginalURLFunction/denormalize, exercised while reproducing the above against a real page: wombat.js's extractOriginalURL hook hands it the value already stripped of the full prefix (REPLAY_MARKER included, for the same reason), not merely the serving root, and that value is still percent-encoded as served (only '/', '=' and ',' literal) rather than the plain ZIM path denormalize() expects.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #409 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 42 42
Lines 2961 2989 +28
Branches 453 457 +4
=========================================
+ Hits 2961 2989 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements item 3 of #396 (denormalization):
ArticleUrlRewriter.denormalize()/defuzzify()(and their JS ports inwombatSetup.js): best-effort reverse ofnormalize()/the fuzzy rules, reconstructing a plausible (not guaranteed exact) original URL from a ZIM path.rules.yamlgains anunreplace_pattern/unreplaceper rule (hand-authored, mirroringpattern/replace);generate_rules.pycodegensDEFUZZY_RULES/defuzzyRulesinto Python and JS, with matching generated tests.wombat.jssoextractOriginalURLdefers to a newwb_info.extract_original_url_functionhook, the same wayrewriteUrlalready defers towb_info.rewrite_function, letting it calldenormalize/defuzzifyfor cases its own pywb-oriented heuristic can never handle (our rewriting scheme doesn't embed an absolute URL the way pywb's does): CSSurl()reads,document.referrer, dynamically created elements, andlocation.replace()/.assign()/updateLocation.#382/wombat#244 Location-getter patch, superseded by the above.urlRewriteFunction/isAlreadyPrefixed/static_prefixassumedprefixnever carriedREPLAY_MARKER, buthead_insert.htmldeliberately computes it with the marker (needed to keep wombat.js'swb_rel_prefixnon-degenerate, per Rewriting: prepend a constant marker to every ZIM path #408). Every fresh runtime rewrite of a never-before-seen URL (a dynamicimport()specifier, alocationassignment, ...) was getting the marker added twice, producing a ZIM path with no matching entry. Also fixesextractOriginalURLFunction/denormalizeto account for wombat.js'sextractOriginalURLhanding the hook a value already past the full prefix (marker included) and still percent-encoded as served.