* fix: resolve DNS before SSRF check and block redirects in parse-url
isPrivateUrl() did string-only hostname matching and never resolved DNS,
so a public-looking name that maps to an internal IP (e.g.
127-0-0-1.sslip.io -> 127.0.0.1) passed the check while fetch/extract
later resolved it and reached internal services (GHSA-wqcv-5qvx-vx75).
- isPrivateUrl is now async: it keeps the fast string/literal-IP path,
then resolves the hostname via DNS and rejects if any address is private.
- parse-url now fetches the page itself with redirect: "error" and parses
via extractFromHtml(), since article-extractor follows redirects
internally and drops a redirect option, which allowed a public URL to
302 to an internal host.
- Update validate-model call site to await; add regression tests.
* fix: preserve charset detection and block CGNAT range in parse-url SSRF fix
Follow-up to the multi-reviewer review of the SSRF fix:
- Restore charset handling lost when switching from extract() to
response.text(): non-UTF-8 pages (Shift_JIS/GBK/EUC/Big5, common on CJK
sites this project targets) decoded as mojibake. Now read the body as
bytes, detect charset from Content-Type / <meta charset>, and decode
with TextDecoder before extractFromHtml.
- Wrap extractFromHtml in try/catch: it throws (not returns null) on
empty/non-HTML bodies, which previously surfaced as a 500 instead of the
intended 400.
- Add 100.64.0.0/10 (RFC 6598 CGNAT) to isPrivateIp; it is routable inside
some cloud internal networks and was a residual SSRF target.
- Add tests for CGNAT, its boundaries, 0.0.0.0, and DNS-resolved IPv6.
* fix: block private IPv6 URLs
* fix: cover full fe80::/10 link-local range and :: unspecified
- Replace startsWith("fe80:") with a check covering the full fe80::/10
range (fe80 through febf) per RFC 4291.
- Add :: (unspecified) to the localhost block.
- Drop the dead 0:0:0:0:0:0:0:1 branch (URL parser normalizes it to ::1).
- Add tests for fe9f::1, febf::1, and ::.
---------
Co-authored-by: dayuan.jiang <jdy.toh@gmail.com>
