fix: preserve charset detection and block CGNAT range in parse-url SSRF fix

Follow-up to the multi-reviewer review of the SSRF fix:

- Restore charset handling lost when switching from extract() to
  response.text(): non-UTF-8 pages (Shift_JIS/GBK/EUC/Big5, common on CJK
  sites this project targets) decoded as mojibake. Now read the body as
  bytes, detect charset from Content-Type / <meta charset>, and decode
  with TextDecoder before extractFromHtml.
- Wrap extractFromHtml in try/catch: it throws (not returns null) on
  empty/non-HTML bodies, which previously surfaced as a 500 instead of the
  intended 400.
- Add 100.64.0.0/10 (RFC 6598 CGNAT) to isPrivateIp; it is routable inside
  some cloud internal networks and was a residual SSRF target.
- Add tests for CGNAT, its boundaries, 0.0.0.0, and DNS-resolved IPv6.
This commit is contained in:
dayuan.jiang
2026-06-28 00:32:24 +09:00
parent 73862f6108
commit 886e748aa7
3 changed files with 54 additions and 2 deletions

View File

@@ -41,6 +41,7 @@ function isPrivateIp(ip: string): boolean {
if (a === 169 && b === 254) return true // 169.254.0.0/16 (link-local)
if (a === 127) return true // 127.0.0.0/8 (loopback)
if (a === 0) return true // 0.0.0.0/8
if (a === 100 && b >= 64 && b <= 127) return true // 100.64.0.0/10 (CGNAT, used by some cloud internal networks)
}
return false