* fix: resolve DNS before SSRF check and block redirects in parse-url
isPrivateUrl() did string-only hostname matching and never resolved DNS,
so a public-looking name that maps to an internal IP (e.g.
127-0-0-1.sslip.io -> 127.0.0.1) passed the check while fetch/extract
later resolved it and reached internal services (GHSA-wqcv-5qvx-vx75).
- isPrivateUrl is now async: it keeps the fast string/literal-IP path,
then resolves the hostname via DNS and rejects if any address is private.
- parse-url now fetches the page itself with redirect: "error" and parses
via extractFromHtml(), since article-extractor follows redirects
internally and drops a redirect option, which allowed a public URL to
302 to an internal host.
- Update validate-model call site to await; add regression tests.
* fix: preserve charset detection and block CGNAT range in parse-url SSRF fix
Follow-up to the multi-reviewer review of the SSRF fix:
- Restore charset handling lost when switching from extract() to
response.text(): non-UTF-8 pages (Shift_JIS/GBK/EUC/Big5, common on CJK
sites this project targets) decoded as mojibake. Now read the body as
bytes, detect charset from Content-Type / <meta charset>, and decode
with TextDecoder before extractFromHtml.
- Wrap extractFromHtml in try/catch: it throws (not returns null) on
empty/non-HTML bodies, which previously surfaced as a 500 instead of the
intended 400.
- Add 100.64.0.0/10 (RFC 6598 CGNAT) to isPrivateIp; it is routable inside
some cloud internal networks and was a residual SSRF target.
- Add tests for CGNAT, its boundaries, 0.0.0.0, and DNS-resolved IPv6.
/api/parse-url accepted any URL the user submitted, fetched it via
@extractus/article-extractor, and returned the body as Markdown. With
ALLOW_PRIVATE_URLS unset (the default after #600) the SSRF guard
short-circuited entirely, so an unauthenticated POST could probe
container ports, read AWS IMDS / GCP metadata, and reach same-VPC
internal services.
- parse-url now always rejects private URLs regardless of
ALLOW_PRIVATE_URLS. The flag's only legitimate use case is local
LLM provider baseUrl overrides (validate-model, chat); article
extraction has no business fetching internal hosts. Local LLM
setups (Ollama, LM Studio, etc.) are unaffected.
- Strip a trailing dot from the hostname before equality checks so
the FQDN form "localhost." (which still resolves to 127.0.0.1) is
caught by the existing string match.
Known follow-ups (not addressed here):
- DNS rebinding: hostnames are matched as strings; a public domain
resolving to 127.0.0.1 (e.g. localtest.me) is not caught.
- HTTP redirects: @extractus/article-extractor uses cross-fetch with
default redirect: "follow" and exposes no hook, so a public URL
302-ing to an internal host still leaks.
* fix: allow private URLs by default for reverse proxy setups
Fixes#588 - Users with reverse proxy setups (e.g., Antigravity tools)
were getting "Invalid base URL" errors due to SSRF protection blocking
private/internal URLs.
Changes:
- Add ALLOW_PRIVATE_URLS env var (defaults to true)
- Set to "false" to enable strict SSRF protection if needed
* refactor: extract isPrivateUrl to shared utility
* feat: add URL content extraction for AI diagram generation
* Changes made as recommended by Claude:
1. Added a request timeout to prevent server resources from being tied up (route.ts)
2. Implemented runtime validation for the API response shape (url-utils.ts)
3. Removed hardcoded English error messages and replaced them with localized strings (url-input-dialog.tsx)
4. Fixed the incorrect i18n namespace (changed from pdf.* to url.*) (url-input-dialog.tsx and en/ja/zh.json)
* chore: restore package.json and package-lock.json
* fix: use i18n strings for URL dialog error messages
---------
Co-authored-by: dayuan.jiang <jdy.toh@gmail.com>
