mirror of
https://github.com/fawney19/Aether.git
synced 2026-09-12 14:10:19 +08:00
feat(proxy): 重构 Proxy 节点管理与隧道系统
- 重构 proxy_nodes 管理端,支持节点注册、心跳、隧道生命周期管理 - 增强 tunnel 嵌入式 hub 和隧道协议 - 重构 aether-proxy 配置、隧道客户端、心跳和调度机制 - 调整 admin OAuth/配额/导入等处理器的参数传递 - 扩展数据迁移模块 - 补充 proxy nodes、OAuth、配额、系统导入等测试 - 更新前端 proxy nodes 视图和 API
This commit is contained in:
@@ -1,10 +0,0 @@
|
||||
FROM debian:bookworm-slim
|
||||
|
||||
ARG TARGETARCH
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY build/linux-${TARGETARCH}/aether-proxy /usr/local/bin/aether-proxy
|
||||
|
||||
ENTRYPOINT ["aether-proxy"]
|
||||
+15
-17
@@ -10,14 +10,6 @@ Tunnel 模式下代理节点**无需对外监听端口**,仅需出站连接到
|
||||
- 常规 Linux 发行版:`systemd`
|
||||
- Alpine Linux:`OpenRC`
|
||||
|
||||
### Docker Compose 部署
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# 编辑 .env 填入 AETHER_PROXY_AETHER_URL 和 AETHER_PROXY_MANAGEMENT_TOKEN
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 下载预编译二进制
|
||||
|
||||
<!-- DOWNLOAD_TABLE_START -->
|
||||
@@ -82,23 +74,30 @@ sudo aether-proxy uninstall
|
||||
| `--node-name` | `AETHER_PROXY_NODE_NAME` | **必填** | 节点名称标识 |
|
||||
| `--public-ip` | `AETHER_PROXY_PUBLIC_IP` | 自动检测 | 公网 IP |
|
||||
| `--node-region` | `AETHER_PROXY_NODE_REGION` | 自动检测 | 地区标识 |
|
||||
| `--heartbeat-interval` | `AETHER_PROXY_HEARTBEAT_INTERVAL` | `30` | 心跳间隔(秒) |
|
||||
| `--heartbeat-interval` | `AETHER_PROXY_HEARTBEAT_INTERVAL` | `5` | 心跳间隔(秒) |
|
||||
| `--allowed-ports` | `AETHER_PROXY_ALLOWED_PORTS` | `80,443,8080,8443` | 允许代理的目标端口 |
|
||||
|
||||
#### Tunnel 连接
|
||||
|
||||
| 参数 | 环境变量 | 默认值 | 说明 |
|
||||
|------|----------|--------|------|
|
||||
| `--tunnel-connections` | `AETHER_PROXY_TUNNEL_CONNECTIONS` | `3` | 到 Aether 的连接池大小 |
|
||||
| `--tunnel-connections` | `AETHER_PROXY_TUNNEL_CONNECTIONS` | 自动(硬件估算) | 最小连接池大小;显式设置后默认固定为该值 |
|
||||
| `--tunnel-connections-max` | `AETHER_PROXY_TUNNEL_CONNECTIONS_MAX` | 自动(硬件估算) | 连接池自动扩容上限;大于 `tunnel_connections` 时启用 autoscale |
|
||||
| `--tunnel-max-streams` | `AETHER_PROXY_TUNNEL_MAX_STREAMS` | 自动(硬件估算) | 单连接最大并发 stream 数 |
|
||||
| `--tunnel-connect-timeout-secs` | `AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_SECS` | `15` | TCP + TLS 握手超时(秒) |
|
||||
| `--tunnel-ping-interval-ms` | `AETHER_PROXY_TUNNEL_PING_INTERVAL_MS` | `250` | fast-fail 探测周期(毫秒) |
|
||||
| `--tunnel-connect-timeout-ms` | `AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_MS` | `800` | fast-reconnect 建连超时(毫秒) |
|
||||
| `--tunnel-stale-timeout-ms` | `AETHER_PROXY_TUNNEL_STALE_TIMEOUT_MS` | `900` | 无入站数据断连阈值(毫秒) |
|
||||
| `--tunnel-scale-check-interval-ms` | `AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS` | `1000` | autoscale 采样周期(毫秒) |
|
||||
| `--tunnel-scale-up-threshold-percent` | `AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT` | `70` | 单 tunnel 占用率超过该值时扩容 |
|
||||
| `--tunnel-scale-down-threshold-percent` | `AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT` | `35` | 单 tunnel 占用率持续低于该值时允许缩容 |
|
||||
| `--tunnel-scale-down-grace-secs` | `AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS` | `15` | 低负载持续时间达到该值后才回收次级 tunnel |
|
||||
| `--tunnel-tcp-keepalive-secs` | `AETHER_PROXY_TUNNEL_TCP_KEEPALIVE_SECS` | `30` | TCP keepalive 初始延迟(秒) |
|
||||
| `--tunnel-tcp-nodelay` | `AETHER_PROXY_TUNNEL_TCP_NODELAY` | `true` | 禁用 Nagle 算法 |
|
||||
| `--tunnel-ping-interval-secs` | `AETHER_PROXY_TUNNEL_PING_INTERVAL_SECS` | `15` | WebSocket Ping 频率(秒) |
|
||||
| `--tunnel-stale-timeout-secs` | `AETHER_PROXY_TUNNEL_STALE_TIMEOUT_SECS` | `45` | 无数据断连阈值(秒) |
|
||||
| `--tunnel-reconnect-base-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS` | `500` | 指数退避基础延迟(毫秒) |
|
||||
| `--tunnel-reconnect-base-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS` | `50` | 指数退避基础延迟(毫秒) |
|
||||
| `--tunnel-reconnect-max-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS` | `30000` | 指数退避上限(毫秒) |
|
||||
|
||||
省略 `tunnel_connections` 时,proxy 会按设备能力自动计算一个基线值和扩容上限;如果显式设置了 `tunnel_connections` 但没有设置 `tunnel_connections_max`,则保持固定连接池,不自动扩缩。
|
||||
|
||||
#### 上游 HTTP 请求
|
||||
|
||||
| 参数 | 环境变量 | 默认值 | 说明 |
|
||||
@@ -141,12 +140,12 @@ sudo aether-proxy uninstall
|
||||
- 默认 `AETHER_PROXY_LOG_DESTINATION=stdout`,日志交给容器日志驱动或宿主机服务管理器
|
||||
- 需要落盘时改成 `file` 或 `both`,并设置 `AETHER_PROXY_LOG_DIR`;setup TUI 里用 `Save Logs to File` 开关即可
|
||||
- 文件日志固定写普通文本,并支持 `hourly/daily` 轮转;默认按天轮换、保留 7 天,最多保留 30 个文件
|
||||
- `docker compose` 默认保持 `stdout`,避免和容器自带日志重复;以 `systemd` 或 `OpenRC` 安装时默认会额外打开文件日志到 `/var/log/aether-proxy`
|
||||
- 以 `systemd` 或 `OpenRC` 安装时默认会额外打开文件日志到 `/var/log/aether-proxy`
|
||||
- OpenRC 安装时,`aether-proxy logs` 实际读取 `/var/log/aether-proxy/current.log` 和 `/var/log/aether-proxy/error.log`;这些文件通常需要用 `sudo aether-proxy logs` 查看
|
||||
|
||||
### 多服务器配置
|
||||
|
||||
在 `aether-proxy.toml` 中使用 `[[servers]]` 配置多个 Aether 服务器:
|
||||
在 `aether-proxy.toml` 中使用 `[[servers]]` 配置 Aether 服务器。即使只有一个服务器,也必须写成一个 `[[servers]]` 条目;旧的顶层单服务器写法已不再支持。
|
||||
|
||||
```toml
|
||||
[[servers]]
|
||||
@@ -164,7 +163,6 @@ node_name = "jp-proxy-02"
|
||||
|
||||
推送 `proxy-v*` 格式的 tag,GitHub Actions 会自动:
|
||||
- 编译所有平台二进制并发布到 Releases
|
||||
- 构建 Docker 镜像并推送到 GHCR 和 Docker Hub
|
||||
- 更新 README 中的下载链接表格
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
services:
|
||||
aether-proxy:
|
||||
image: ghcr.io/fawney19/aether-proxy:latest
|
||||
container_name: aether-proxy
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
AETHER_PROXY_LOG_JSON: ${AETHER_PROXY_LOG_JSON:-true}
|
||||
AETHER_PROXY_LOG_DESTINATION: ${AETHER_PROXY_LOG_DESTINATION:-stdout}
|
||||
AETHER_PROXY_LOG_DIR: ${AETHER_PROXY_LOG_DIR:-/var/log/aether-proxy}
|
||||
AETHER_PROXY_LOG_ROTATION: ${AETHER_PROXY_LOG_ROTATION:-daily}
|
||||
AETHER_PROXY_LOG_RETENTION_DAYS: ${AETHER_PROXY_LOG_RETENTION_DAYS:-7}
|
||||
AETHER_PROXY_LOG_MAX_FILES: ${AETHER_PROXY_LOG_MAX_FILES:-30}
|
||||
volumes:
|
||||
- ./logs:/var/log/aether-proxy
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "50m"
|
||||
max-file: "3"
|
||||
+352
-29
@@ -1,8 +1,9 @@
|
||||
//! Application lifecycle: initialization, task orchestration, and shutdown.
|
||||
|
||||
use std::sync::atomic::AtomicU64;
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Arc, RwLock};
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use aether_http::{jittered_delay_for_retry, HttpRetryConfig};
|
||||
use aether_runtime::{
|
||||
@@ -14,7 +15,7 @@ use tokio::sync::{watch, Mutex};
|
||||
use tokio::task::JoinHandle;
|
||||
use tracing::{error, info, warn};
|
||||
|
||||
use crate::config::{Config, ServerEntry};
|
||||
use crate::config::{Config, ServerEntry, TunnelPoolSizing};
|
||||
use crate::net;
|
||||
use crate::registration::client::AetherClient;
|
||||
use crate::runtime::{self, DynamicConfig};
|
||||
@@ -24,6 +25,52 @@ use crate::{hardware, target_filter, tunnel};
|
||||
|
||||
type TaskHandles = Arc<Mutex<Vec<JoinHandle<()>>>>;
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct TunnelPoolPolicy {
|
||||
min_connections: usize,
|
||||
max_connections: usize,
|
||||
max_streams_per_tunnel: usize,
|
||||
scale_check_interval: Duration,
|
||||
scale_up_threshold_percent: u32,
|
||||
scale_down_threshold_percent: u32,
|
||||
scale_down_grace: Duration,
|
||||
}
|
||||
|
||||
impl TunnelPoolPolicy {
|
||||
fn from_config(config: &Config, sizing: TunnelPoolSizing) -> Self {
|
||||
Self {
|
||||
min_connections: sizing.initial_connections.max(1) as usize,
|
||||
max_connections: sizing
|
||||
.max_connections
|
||||
.max(sizing.initial_connections)
|
||||
.max(1) as usize,
|
||||
max_streams_per_tunnel: config.tunnel_max_streams.unwrap_or(128).max(1) as usize,
|
||||
scale_check_interval: Duration::from_millis(config.tunnel_scale_check_interval_ms),
|
||||
scale_up_threshold_percent: config.tunnel_scale_up_threshold_percent,
|
||||
scale_down_threshold_percent: config.tunnel_scale_down_threshold_percent,
|
||||
scale_down_grace: Duration::from_secs(config.tunnel_scale_down_grace_secs),
|
||||
}
|
||||
}
|
||||
|
||||
fn scale_up_high_water_mark(&self) -> u64 {
|
||||
occupancy_threshold(self.max_streams_per_tunnel, self.scale_up_threshold_percent)
|
||||
}
|
||||
|
||||
fn scale_down_low_water_mark(&self) -> u64 {
|
||||
occupancy_threshold(
|
||||
self.max_streams_per_tunnel,
|
||||
self.scale_down_threshold_percent,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
struct ManagedTunnel {
|
||||
slot_id: usize,
|
||||
drain_tx: watch::Sender<bool>,
|
||||
handle: JoinHandle<()>,
|
||||
draining: bool,
|
||||
}
|
||||
|
||||
/// Run the full application lifecycle after config has been parsed.
|
||||
pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Result<()> {
|
||||
config.validate()?;
|
||||
@@ -63,6 +110,19 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
|
||||
"auto-detected tunnel_max_streams from hardware"
|
||||
);
|
||||
}
|
||||
let tunnel_pool_sizing = config.resolve_tunnel_pool_sizing(&hw_info)?;
|
||||
let tunnel_pool_policy = TunnelPoolPolicy::from_config(&config, tunnel_pool_sizing);
|
||||
info!(
|
||||
tunnel_connections_initial = tunnel_pool_policy.min_connections,
|
||||
tunnel_connections_max = tunnel_pool_policy.max_connections,
|
||||
tunnel_max_streams = tunnel_pool_policy.max_streams_per_tunnel,
|
||||
scale_check_interval_ms = tunnel_pool_policy.scale_check_interval.as_millis(),
|
||||
scale_up_threshold_percent = tunnel_pool_policy.scale_up_threshold_percent,
|
||||
scale_down_threshold_percent = tunnel_pool_policy.scale_down_threshold_percent,
|
||||
scale_down_grace_secs = tunnel_pool_policy.scale_down_grace.as_secs(),
|
||||
auto_sizing = config.tunnel_connections.is_none(),
|
||||
"resolved tunnel pool policy"
|
||||
);
|
||||
|
||||
info!(
|
||||
max_concurrency = hw_info.estimated_max_concurrency,
|
||||
@@ -177,15 +237,14 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
|
||||
"running in tunnel mode"
|
||||
);
|
||||
|
||||
// Spawn tunnel connections per server (pool_size connections each)
|
||||
let pool_size = state.config.tunnel_connections.max(1) as usize;
|
||||
// Spawn tunnel pool manager per server.
|
||||
let tunnel_handles: TaskHandles = Arc::new(Mutex::new(Vec::new()));
|
||||
let retry_handles: TaskHandles = Arc::new(Mutex::new(Vec::new()));
|
||||
for server in server_contexts.lock().await.iter() {
|
||||
spawn_tunnel_pool(
|
||||
spawn_tunnel_pool_manager(
|
||||
Arc::clone(&state),
|
||||
Arc::clone(server),
|
||||
pool_size,
|
||||
tunnel_pool_policy,
|
||||
shutdown_rx.clone(),
|
||||
Arc::clone(&tunnel_handles),
|
||||
)
|
||||
@@ -200,7 +259,7 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
|
||||
failed_entries,
|
||||
public_ip.clone(),
|
||||
hw_info.clone(),
|
||||
pool_size,
|
||||
tunnel_pool_policy,
|
||||
shutdown_rx.clone(),
|
||||
Arc::clone(&tunnel_handles),
|
||||
Arc::clone(&retry_handles),
|
||||
@@ -240,7 +299,7 @@ async fn spawn_registration_recovery_tasks(
|
||||
failed: Vec<(String, ServerEntry)>,
|
||||
public_ip: String,
|
||||
hw_info: crate::hardware::HardwareInfo,
|
||||
pool_size: usize,
|
||||
tunnel_pool_policy: TunnelPoolPolicy,
|
||||
shutdown: watch::Receiver<bool>,
|
||||
tunnel_handles: TaskHandles,
|
||||
retry_handles: TaskHandles,
|
||||
@@ -261,7 +320,7 @@ async fn spawn_registration_recovery_tasks(
|
||||
entry,
|
||||
retry_public_ip,
|
||||
retry_hw_info,
|
||||
pool_size,
|
||||
tunnel_pool_policy,
|
||||
retry_shutdown,
|
||||
retry_tunnels,
|
||||
)
|
||||
@@ -281,7 +340,7 @@ async fn retry_failed_registration(
|
||||
entry: ServerEntry,
|
||||
public_ip: String,
|
||||
hw_info: crate::hardware::HardwareInfo,
|
||||
pool_size: usize,
|
||||
tunnel_pool_policy: TunnelPoolPolicy,
|
||||
mut shutdown: watch::Receiver<bool>,
|
||||
tunnel_handles: TaskHandles,
|
||||
) {
|
||||
@@ -329,10 +388,10 @@ async fn retry_failed_registration(
|
||||
node_id,
|
||||
);
|
||||
server_contexts.lock().await.push(Arc::clone(&server));
|
||||
spawn_tunnel_pool(
|
||||
spawn_tunnel_pool_manager(
|
||||
Arc::clone(&state),
|
||||
server,
|
||||
pool_size,
|
||||
tunnel_pool_policy,
|
||||
shutdown,
|
||||
tunnel_handles,
|
||||
)
|
||||
@@ -386,23 +445,239 @@ fn build_server_context(
|
||||
})
|
||||
}
|
||||
|
||||
async fn spawn_tunnel_pool(
|
||||
async fn spawn_tunnel_pool_manager(
|
||||
state: Arc<AppState>,
|
||||
server: Arc<ServerContext>,
|
||||
pool_size: usize,
|
||||
policy: TunnelPoolPolicy,
|
||||
shutdown: watch::Receiver<bool>,
|
||||
tunnel_handles: TaskHandles,
|
||||
) {
|
||||
let mut handles = Vec::with_capacity(pool_size);
|
||||
for conn_idx in 0..pool_size {
|
||||
let s = Arc::clone(&state);
|
||||
let srv = Arc::clone(&server);
|
||||
let rx = shutdown.clone();
|
||||
handles.push(tokio::spawn(async move {
|
||||
tunnel::run(&s, &srv, conn_idx, rx).await;
|
||||
}));
|
||||
let handle = tokio::spawn(async move {
|
||||
run_tunnel_pool_manager(state, server, policy, shutdown).await;
|
||||
});
|
||||
tunnel_handles.lock().await.push(handle);
|
||||
}
|
||||
|
||||
async fn run_tunnel_pool_manager(
|
||||
state: Arc<AppState>,
|
||||
server: Arc<ServerContext>,
|
||||
policy: TunnelPoolPolicy,
|
||||
mut shutdown: watch::Receiver<bool>,
|
||||
) {
|
||||
let mut tunnels = BTreeMap::<usize, ManagedTunnel>::new();
|
||||
ensure_tunnel_capacity(
|
||||
&mut tunnels,
|
||||
policy.min_connections,
|
||||
&policy,
|
||||
&state,
|
||||
&server,
|
||||
&shutdown,
|
||||
);
|
||||
let mut ticker = tokio::time::interval(policy.scale_check_interval);
|
||||
ticker.tick().await;
|
||||
let mut low_load_since: Option<Instant> = None;
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
_ = shutdown.changed() => {
|
||||
info!(server = %server.server_label, "tunnel pool manager shutting down");
|
||||
break;
|
||||
}
|
||||
_ = ticker.tick() => {
|
||||
reap_finished_tunnels(&mut tunnels).await;
|
||||
|
||||
let available = tunnels.values().filter(|tunnel| !tunnel.draining).count();
|
||||
if available < policy.min_connections {
|
||||
ensure_tunnel_capacity(
|
||||
&mut tunnels,
|
||||
policy.min_connections,
|
||||
&policy,
|
||||
&state,
|
||||
&server,
|
||||
&shutdown,
|
||||
);
|
||||
low_load_since = None;
|
||||
continue;
|
||||
}
|
||||
|
||||
let active_connections = server.active_connections.load(Ordering::Acquire);
|
||||
let desired_connections = desired_tunnel_connections(active_connections, &policy);
|
||||
if desired_connections > available {
|
||||
ensure_tunnel_capacity(
|
||||
&mut tunnels,
|
||||
desired_connections,
|
||||
&policy,
|
||||
&state,
|
||||
&server,
|
||||
&shutdown,
|
||||
);
|
||||
info!(
|
||||
server = %server.server_label,
|
||||
active_connections,
|
||||
available_connections = available,
|
||||
target_connections = desired_connections,
|
||||
"scaled tunnel pool up"
|
||||
);
|
||||
low_load_since = None;
|
||||
continue;
|
||||
}
|
||||
|
||||
if should_scale_down(active_connections, available, &policy) {
|
||||
match low_load_since {
|
||||
Some(since) if since.elapsed() >= policy.scale_down_grace => {
|
||||
if request_tunnel_drain(&mut tunnels, policy.min_connections) {
|
||||
info!(
|
||||
server = %server.server_label,
|
||||
active_connections,
|
||||
available_connections = available,
|
||||
"requested tunnel drain for scale-down"
|
||||
);
|
||||
}
|
||||
low_load_since = None;
|
||||
}
|
||||
None => {
|
||||
low_load_since = Some(Instant::now());
|
||||
}
|
||||
Some(_) => {}
|
||||
}
|
||||
} else {
|
||||
low_load_since = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
tunnel_handles.lock().await.extend(handles);
|
||||
|
||||
for tunnel in tunnels.values_mut() {
|
||||
let _ = tunnel.drain_tx.send(true);
|
||||
tunnel.draining = true;
|
||||
}
|
||||
while !tunnels.is_empty() {
|
||||
reap_finished_tunnels(&mut tunnels).await;
|
||||
if !tunnels.is_empty() {
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn ensure_tunnel_capacity(
|
||||
tunnels: &mut BTreeMap<usize, ManagedTunnel>,
|
||||
target_connections: usize,
|
||||
policy: &TunnelPoolPolicy,
|
||||
state: &Arc<AppState>,
|
||||
server: &Arc<ServerContext>,
|
||||
shutdown: &watch::Receiver<bool>,
|
||||
) {
|
||||
let target_connections = target_connections.min(policy.max_connections);
|
||||
while tunnels.values().filter(|tunnel| !tunnel.draining).count() < target_connections {
|
||||
let Some(slot_id) = next_available_tunnel_slot(tunnels, policy.max_connections) else {
|
||||
break;
|
||||
};
|
||||
tunnels.insert(
|
||||
slot_id,
|
||||
spawn_managed_tunnel(
|
||||
Arc::clone(state),
|
||||
Arc::clone(server),
|
||||
slot_id,
|
||||
shutdown.clone(),
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn spawn_managed_tunnel(
|
||||
state: Arc<AppState>,
|
||||
server: Arc<ServerContext>,
|
||||
slot_id: usize,
|
||||
shutdown: watch::Receiver<bool>,
|
||||
) -> ManagedTunnel {
|
||||
let (drain_tx, drain_rx) = watch::channel(false);
|
||||
let handle = tokio::spawn(async move {
|
||||
tunnel::run(&state, &server, slot_id, shutdown, drain_rx).await;
|
||||
});
|
||||
ManagedTunnel {
|
||||
slot_id,
|
||||
drain_tx,
|
||||
handle,
|
||||
draining: false,
|
||||
}
|
||||
}
|
||||
|
||||
async fn reap_finished_tunnels(tunnels: &mut BTreeMap<usize, ManagedTunnel>) {
|
||||
let finished_slots = tunnels
|
||||
.iter()
|
||||
.filter_map(|(slot_id, tunnel)| tunnel.handle.is_finished().then_some(*slot_id))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
for slot_id in finished_slots {
|
||||
if let Some(tunnel) = tunnels.remove(&slot_id) {
|
||||
let _ = tunnel.handle.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn next_available_tunnel_slot(
|
||||
tunnels: &BTreeMap<usize, ManagedTunnel>,
|
||||
max_connections: usize,
|
||||
) -> Option<usize> {
|
||||
(0..max_connections).find(|slot_id| !tunnels.contains_key(slot_id))
|
||||
}
|
||||
|
||||
fn request_tunnel_drain(
|
||||
tunnels: &mut BTreeMap<usize, ManagedTunnel>,
|
||||
min_connections: usize,
|
||||
) -> bool {
|
||||
let available = tunnels.values().filter(|tunnel| !tunnel.draining).count();
|
||||
if available <= min_connections {
|
||||
return false;
|
||||
}
|
||||
|
||||
let Some((_, tunnel)) = tunnels
|
||||
.iter_mut()
|
||||
.rev()
|
||||
.find(|(_, tunnel)| tunnel.slot_id != 0 && !tunnel.draining)
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
|
||||
if tunnel.drain_tx.send(true).is_ok() {
|
||||
tunnel.draining = true;
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn desired_tunnel_connections(active_connections: u64, policy: &TunnelPoolPolicy) -> usize {
|
||||
let required = div_ceil_u64(active_connections.max(1), policy.scale_up_high_water_mark());
|
||||
required.clamp(policy.min_connections as u64, policy.max_connections as u64) as usize
|
||||
}
|
||||
|
||||
fn should_scale_down(
|
||||
active_connections: u64,
|
||||
available_connections: usize,
|
||||
policy: &TunnelPoolPolicy,
|
||||
) -> bool {
|
||||
if available_connections <= policy.min_connections {
|
||||
return false;
|
||||
}
|
||||
active_connections
|
||||
<= (available_connections as u64)
|
||||
.saturating_sub(1)
|
||||
.saturating_mul(policy.scale_down_low_water_mark())
|
||||
}
|
||||
|
||||
fn occupancy_threshold(max_streams_per_tunnel: usize, percent: u32) -> u64 {
|
||||
div_ceil_u64(
|
||||
(max_streams_per_tunnel as u64).saturating_mul(percent as u64),
|
||||
100,
|
||||
)
|
||||
.max(1)
|
||||
}
|
||||
|
||||
fn div_ceil_u64(value: u64, divisor: u64) -> u64 {
|
||||
if divisor == 0 {
|
||||
return value;
|
||||
}
|
||||
value.saturating_add(divisor.saturating_sub(1)) / divisor
|
||||
}
|
||||
|
||||
async fn await_all_handles(handles: &TaskHandles) {
|
||||
@@ -469,6 +744,8 @@ mod tests {
|
||||
},
|
||||
)];
|
||||
let (shutdown_tx, shutdown_rx) = watch::channel(false);
|
||||
let tunnel_pool_policy =
|
||||
TunnelPoolPolicy::from_config(&state.config, sample_tunnel_pool_sizing());
|
||||
|
||||
spawn_registration_recovery_tasks(
|
||||
Arc::clone(&state),
|
||||
@@ -476,7 +753,7 @@ mod tests {
|
||||
failed,
|
||||
"127.0.0.1".to_string(),
|
||||
sample_hardware_info(),
|
||||
1,
|
||||
tunnel_pool_policy,
|
||||
shutdown_rx.clone(),
|
||||
Arc::clone(&tunnel_handles),
|
||||
Arc::clone(&retry_handles),
|
||||
@@ -508,6 +785,40 @@ mod tests {
|
||||
gateway_handle.abort();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn desired_tunnel_connections_expands_when_load_crosses_high_water() {
|
||||
let policy = TunnelPoolPolicy {
|
||||
min_connections: 1,
|
||||
max_connections: 6,
|
||||
max_streams_per_tunnel: 1024,
|
||||
scale_check_interval: Duration::from_secs(1),
|
||||
scale_up_threshold_percent: 70,
|
||||
scale_down_threshold_percent: 35,
|
||||
scale_down_grace: Duration::from_secs(15),
|
||||
};
|
||||
|
||||
assert_eq!(desired_tunnel_connections(1, &policy), 1);
|
||||
assert_eq!(desired_tunnel_connections(2_000, &policy), 3);
|
||||
assert_eq!(desired_tunnel_connections(5_000, &policy), 6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn should_scale_down_requires_load_to_fit_remaining_tunnels() {
|
||||
let policy = TunnelPoolPolicy {
|
||||
min_connections: 1,
|
||||
max_connections: 6,
|
||||
max_streams_per_tunnel: 1024,
|
||||
scale_check_interval: Duration::from_secs(1),
|
||||
scale_up_threshold_percent: 70,
|
||||
scale_down_threshold_percent: 35,
|
||||
scale_down_grace: Duration::from_secs(15),
|
||||
};
|
||||
|
||||
assert!(!should_scale_down(800, 3, &policy));
|
||||
assert!(should_scale_down(600, 3, &policy));
|
||||
assert!(!should_scale_down(200, 1, &policy));
|
||||
}
|
||||
|
||||
async fn wait_for_registered_server(
|
||||
server_contexts: &Arc<Mutex<Vec<Arc<ServerContext>>>>,
|
||||
) -> Arc<ServerContext> {
|
||||
@@ -595,6 +906,13 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_tunnel_pool_sizing() -> TunnelPoolSizing {
|
||||
TunnelPoolSizing {
|
||||
initial_connections: 1,
|
||||
max_connections: 1,
|
||||
}
|
||||
}
|
||||
|
||||
fn sample_state(config: Config) -> Arc<ProxyAppState> {
|
||||
let config = Arc::new(config);
|
||||
let dns_cache = Arc::new(DnsCache::new(Duration::from_secs(60), 128));
|
||||
@@ -656,13 +974,18 @@ mod tests {
|
||||
log_max_files: 30,
|
||||
tunnel_reconnect_base_ms: 50,
|
||||
tunnel_reconnect_max_ms: 250,
|
||||
tunnel_ping_interval_secs: 1,
|
||||
tunnel_ping_interval_ms: 1_000,
|
||||
tunnel_max_streams: Some(8),
|
||||
tunnel_connect_timeout_secs: 2,
|
||||
tunnel_connect_timeout_ms: 2_000,
|
||||
tunnel_tcp_keepalive_secs: 30,
|
||||
tunnel_tcp_nodelay: true,
|
||||
tunnel_stale_timeout_secs: 5,
|
||||
tunnel_connections: 1,
|
||||
tunnel_stale_timeout_ms: 5_000,
|
||||
tunnel_connections: Some(1),
|
||||
tunnel_connections_max: Some(1),
|
||||
tunnel_scale_check_interval_ms: 1_000,
|
||||
tunnel_scale_up_threshold_percent: 70,
|
||||
tunnel_scale_down_threshold_percent: 35,
|
||||
tunnel_scale_down_grace_secs: 15,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+433
-151
@@ -1,10 +1,13 @@
|
||||
use std::fmt;
|
||||
use std::path::Path;
|
||||
use std::time::Duration;
|
||||
|
||||
use aether_runtime::{FileLoggingConfig, LogDestination, LogRotation, ServiceRuntimeConfig};
|
||||
use clap::Parser;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::hardware::HardwareInfo;
|
||||
|
||||
/// Fields that existed in 0.1.x but were removed in 0.2.0.
|
||||
const LEGACY_ONLY_KEYS: &[&str] = &[
|
||||
"hmac_key",
|
||||
@@ -16,6 +19,12 @@ const LEGACY_ONLY_KEYS: &[&str] = &[
|
||||
"tls_cert",
|
||||
"tls_key",
|
||||
];
|
||||
const REMOVED_TUNNEL_SECONDS_KEYS: &[&str] = &[
|
||||
"tunnel_ping_interval_secs",
|
||||
"tunnel_connect_timeout_secs",
|
||||
"tunnel_stale_timeout_secs",
|
||||
];
|
||||
const REMOVED_SINGLE_SERVER_KEYS: &[&str] = &["aether_url", "management_token"];
|
||||
|
||||
/// Fields renamed from 0.1.x `delegate_*` to 0.2.0 `upstream_*`.
|
||||
const DELEGATE_TO_UPSTREAM: &[(&str, &str)] = &[
|
||||
@@ -38,13 +47,31 @@ const DELEGATE_TO_UPSTREAM: &[(&str, &str)] = &[
|
||||
/// Default bytes buffered before a tunnel request becomes non-replayable for
|
||||
/// 307/308 redirects. Kept aligned with the current admin-side request size
|
||||
/// default, but exposed as an independent proxy transport budget.
|
||||
pub const DEFAULT_HEARTBEAT_INTERVAL_SECS: u64 = 30;
|
||||
pub const DEFAULT_HEARTBEAT_INTERVAL_SECS: u64 = 5;
|
||||
#[allow(dead_code)]
|
||||
pub const DEFAULT_REDIRECT_REPLAY_BUDGET_BYTES: usize = 5_242_880;
|
||||
pub const DEFAULT_REDIRECT_REPLAY_BUDGET_HUMAN: &str = "5M";
|
||||
pub const DEFAULT_LOG_RETENTION_DAYS: u64 = 7;
|
||||
pub const DEFAULT_LOG_MAX_FILES: usize = 30;
|
||||
pub const DEFAULT_TUNNEL_PING_INTERVAL_MS: u64 = 250;
|
||||
pub const DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS: u64 = 800;
|
||||
pub const DEFAULT_TUNNEL_STALE_TIMEOUT_MS: u64 = 900;
|
||||
pub const DEFAULT_TUNNEL_SCALE_CHECK_INTERVAL_MS: u64 = 1_000;
|
||||
pub const DEFAULT_TUNNEL_SCALE_UP_THRESHOLD_PERCENT: u32 = 70;
|
||||
pub const DEFAULT_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT: u32 = 35;
|
||||
pub const DEFAULT_TUNNEL_SCALE_DOWN_GRACE_SECS: u64 = 15;
|
||||
const AUTO_TUNNEL_CONNECTIONS_BASE_CAP: u64 = 4;
|
||||
const AUTO_TUNNEL_CONNECTIONS_MAX_CAP: u64 = 8;
|
||||
|
||||
const TUNNEL_PING_INTERVAL_MS_ENV: &str = "AETHER_PROXY_TUNNEL_PING_INTERVAL_MS";
|
||||
const TUNNEL_CONNECT_TIMEOUT_MS_ENV: &str = "AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_MS";
|
||||
const TUNNEL_STALE_TIMEOUT_MS_ENV: &str = "AETHER_PROXY_TUNNEL_STALE_TIMEOUT_MS";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct TunnelPoolSizing {
|
||||
pub initial_connections: u32,
|
||||
pub max_connections: u32,
|
||||
}
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum ByteSizeValue {
|
||||
Text(String),
|
||||
@@ -473,7 +500,7 @@ pub struct Config {
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS",
|
||||
default_value_t = 500
|
||||
default_value_t = 50
|
||||
)]
|
||||
pub tunnel_reconnect_base_ms: u64,
|
||||
|
||||
@@ -485,21 +512,25 @@ pub struct Config {
|
||||
)]
|
||||
pub tunnel_reconnect_max_ms: u64,
|
||||
|
||||
/// WebSocket tunnel ping interval in seconds
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_PING_INTERVAL", default_value_t = 15)]
|
||||
pub tunnel_ping_interval_secs: u64,
|
||||
/// WebSocket tunnel ping interval in milliseconds
|
||||
#[arg(
|
||||
long,
|
||||
env = TUNNEL_PING_INTERVAL_MS_ENV,
|
||||
default_value_t = DEFAULT_TUNNEL_PING_INTERVAL_MS
|
||||
)]
|
||||
pub tunnel_ping_interval_ms: u64,
|
||||
|
||||
/// Maximum concurrent streams over tunnel (auto-detected from hardware if omitted)
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_MAX_STREAMS")]
|
||||
pub tunnel_max_streams: Option<u32>,
|
||||
|
||||
/// WebSocket tunnel TCP connect timeout in seconds
|
||||
/// WebSocket tunnel TCP connect timeout in milliseconds
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
|
||||
default_value_t = 15
|
||||
env = TUNNEL_CONNECT_TIMEOUT_MS_ENV,
|
||||
default_value_t = DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS
|
||||
)]
|
||||
pub tunnel_connect_timeout_secs: u64,
|
||||
pub tunnel_connect_timeout_ms: u64,
|
||||
|
||||
/// WebSocket tunnel TCP keepalive in seconds (0 disables)
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_KEEPALIVE", default_value_t = 30)]
|
||||
@@ -509,13 +540,55 @@ pub struct Config {
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_NODELAY", default_value_t = true)]
|
||||
pub tunnel_tcp_nodelay: bool,
|
||||
|
||||
/// Tunnel connection staleness timeout in seconds (triggers reconnect if no data received)
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_STALE_TIMEOUT", default_value_t = 45)]
|
||||
pub tunnel_stale_timeout_secs: u64,
|
||||
/// Tunnel connection staleness timeout in milliseconds
|
||||
#[arg(
|
||||
long,
|
||||
env = TUNNEL_STALE_TIMEOUT_MS_ENV,
|
||||
default_value_t = DEFAULT_TUNNEL_STALE_TIMEOUT_MS
|
||||
)]
|
||||
pub tunnel_stale_timeout_ms: u64,
|
||||
|
||||
/// Number of parallel WebSocket tunnel connections per server (connection pool)
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS", default_value_t = 3)]
|
||||
pub tunnel_connections: u32,
|
||||
/// Minimum number of parallel WebSocket tunnel connections per server.
|
||||
/// If omitted, a device-aware value is auto-detected at startup.
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS")]
|
||||
pub tunnel_connections: Option<u32>,
|
||||
|
||||
/// Maximum number of WebSocket tunnel connections per server.
|
||||
/// When larger than `tunnel_connections`, the proxy may autoscale up to this limit.
|
||||
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS_MAX")]
|
||||
pub tunnel_connections_max: Option<u32>,
|
||||
|
||||
/// Autoscale evaluation interval for the tunnel pool.
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS",
|
||||
default_value_t = DEFAULT_TUNNEL_SCALE_CHECK_INTERVAL_MS
|
||||
)]
|
||||
pub tunnel_scale_check_interval_ms: u64,
|
||||
|
||||
/// Per-tunnel occupancy percentage that triggers scale-up.
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT",
|
||||
default_value_t = DEFAULT_TUNNEL_SCALE_UP_THRESHOLD_PERCENT
|
||||
)]
|
||||
pub tunnel_scale_up_threshold_percent: u32,
|
||||
|
||||
/// Per-tunnel occupancy percentage that allows scale-down after the grace window.
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT",
|
||||
default_value_t = DEFAULT_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT
|
||||
)]
|
||||
pub tunnel_scale_down_threshold_percent: u32,
|
||||
|
||||
/// Low-load grace window before a secondary tunnel is drained.
|
||||
#[arg(
|
||||
long,
|
||||
env = "AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS",
|
||||
default_value_t = DEFAULT_TUNNEL_SCALE_DOWN_GRACE_SECS
|
||||
)]
|
||||
pub tunnel_scale_down_grace_secs: u64,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
@@ -539,22 +612,52 @@ impl Config {
|
||||
anyhow::bail!("allowed_ports: port 0 is not valid");
|
||||
}
|
||||
}
|
||||
if self.tunnel_connect_timeout_secs == 0 {
|
||||
anyhow::bail!("tunnel_connect_timeout_secs must be > 0");
|
||||
let tunnel_connect_timeout = self.tunnel_connect_timeout()?;
|
||||
if tunnel_connect_timeout.is_zero() {
|
||||
anyhow::bail!("effective tunnel connect timeout must be > 0");
|
||||
}
|
||||
if self.tunnel_ping_interval_secs == 0 {
|
||||
anyhow::bail!("tunnel_ping_interval_secs must be > 0");
|
||||
let tunnel_ping_interval = self.tunnel_ping_interval()?;
|
||||
if tunnel_ping_interval.is_zero() {
|
||||
anyhow::bail!("effective tunnel ping interval must be > 0");
|
||||
}
|
||||
if self.tunnel_stale_timeout_secs <= self.tunnel_ping_interval_secs {
|
||||
let tunnel_stale_timeout = self.tunnel_stale_timeout()?;
|
||||
if tunnel_stale_timeout <= tunnel_ping_interval {
|
||||
anyhow::bail!(
|
||||
"tunnel_stale_timeout_secs ({}) must be > tunnel_ping_interval_secs ({})",
|
||||
self.tunnel_stale_timeout_secs,
|
||||
self.tunnel_ping_interval_secs
|
||||
"effective tunnel stale timeout ({:?}) must be > effective tunnel ping interval ({:?})",
|
||||
tunnel_stale_timeout,
|
||||
tunnel_ping_interval
|
||||
);
|
||||
}
|
||||
if self.tunnel_connections == 0 {
|
||||
if matches!(self.tunnel_connections, Some(0)) {
|
||||
anyhow::bail!("tunnel_connections must be > 0");
|
||||
}
|
||||
if matches!(self.tunnel_connections_max, Some(0)) {
|
||||
anyhow::bail!("tunnel_connections_max must be > 0");
|
||||
}
|
||||
if let (Some(min_connections), Some(max_connections)) =
|
||||
(self.tunnel_connections, self.tunnel_connections_max)
|
||||
{
|
||||
if max_connections < min_connections {
|
||||
anyhow::bail!("tunnel_connections_max must be >= tunnel_connections");
|
||||
}
|
||||
}
|
||||
if self.tunnel_scale_check_interval_ms == 0 {
|
||||
anyhow::bail!("tunnel_scale_check_interval_ms must be > 0");
|
||||
}
|
||||
if self.tunnel_scale_down_grace_secs == 0 {
|
||||
anyhow::bail!("tunnel_scale_down_grace_secs must be > 0");
|
||||
}
|
||||
if !(1..=100).contains(&self.tunnel_scale_up_threshold_percent) {
|
||||
anyhow::bail!("tunnel_scale_up_threshold_percent must be within 1..=100");
|
||||
}
|
||||
if !(1..100).contains(&self.tunnel_scale_down_threshold_percent) {
|
||||
anyhow::bail!("tunnel_scale_down_threshold_percent must be within 1..100");
|
||||
}
|
||||
if self.tunnel_scale_down_threshold_percent >= self.tunnel_scale_up_threshold_percent {
|
||||
anyhow::bail!(
|
||||
"tunnel_scale_down_threshold_percent must be < tunnel_scale_up_threshold_percent"
|
||||
);
|
||||
}
|
||||
if self.aether_retry_max_attempts == 0 {
|
||||
anyhow::bail!("aether_retry_max_attempts must be >= 1");
|
||||
}
|
||||
@@ -600,6 +703,54 @@ impl Config {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn tunnel_ping_interval(&self) -> anyhow::Result<Duration> {
|
||||
Ok(Duration::from_millis(self.tunnel_ping_interval_ms))
|
||||
}
|
||||
|
||||
pub fn tunnel_connect_timeout(&self) -> anyhow::Result<Duration> {
|
||||
Ok(Duration::from_millis(self.tunnel_connect_timeout_ms))
|
||||
}
|
||||
|
||||
pub fn tunnel_stale_timeout(&self) -> anyhow::Result<Duration> {
|
||||
Ok(Duration::from_millis(self.tunnel_stale_timeout_ms))
|
||||
}
|
||||
|
||||
pub fn resolve_tunnel_pool_sizing(
|
||||
&self,
|
||||
hw_info: &HardwareInfo,
|
||||
) -> anyhow::Result<TunnelPoolSizing> {
|
||||
let per_tunnel_capacity = u64::from(self.tunnel_max_streams.unwrap_or(128).max(1));
|
||||
let estimated = hw_info.estimated_max_concurrency.max(per_tunnel_capacity);
|
||||
let cpu_cap = u64::from(hw_info.cpu_cores).clamp(1, AUTO_TUNNEL_CONNECTIONS_MAX_CAP);
|
||||
|
||||
let auto_initial = div_ceil_u64(estimated, per_tunnel_capacity.saturating_mul(8))
|
||||
.clamp(1, AUTO_TUNNEL_CONNECTIONS_BASE_CAP)
|
||||
.min(cpu_cap);
|
||||
let auto_max = div_ceil_u64(estimated, per_tunnel_capacity.saturating_mul(4))
|
||||
.clamp(auto_initial, AUTO_TUNNEL_CONNECTIONS_MAX_CAP)
|
||||
.min(cpu_cap.max(auto_initial));
|
||||
|
||||
let initial_connections = u64::from(self.tunnel_connections.unwrap_or(auto_initial as u32));
|
||||
let max_connections = match self.tunnel_connections_max {
|
||||
Some(explicit) => u64::from(explicit),
|
||||
None if self.tunnel_connections.is_some() => initial_connections,
|
||||
None => auto_max,
|
||||
};
|
||||
|
||||
if max_connections < initial_connections {
|
||||
anyhow::bail!(
|
||||
"effective tunnel_connections_max ({max_connections}) must be >= tunnel_connections ({initial_connections})"
|
||||
);
|
||||
}
|
||||
|
||||
Ok(TunnelPoolSizing {
|
||||
initial_connections: u32::try_from(initial_connections)
|
||||
.expect("effective tunnel initial connections should fit in u32"),
|
||||
max_connections: u32::try_from(max_connections)
|
||||
.expect("effective tunnel max connections should fit in u32"),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn service_runtime_config(&self) -> anyhow::Result<ServiceRuntimeConfig> {
|
||||
let mut config = ServiceRuntimeConfig::new("aether-proxy", "aether_proxy=info")
|
||||
.with_log_format(aether_runtime::LogFormat::Pretty)
|
||||
@@ -629,6 +780,7 @@ impl Config {
|
||||
|
||||
/// Per-server connection config (used in multi-server TOML `[[servers]]`).
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ServerEntry {
|
||||
pub aether_url: String,
|
||||
pub management_token: String,
|
||||
@@ -643,11 +795,8 @@ pub struct ServerEntry {
|
||||
/// Serializable config for TOML file persistence.
|
||||
/// All fields are optional -- only populated values are written.
|
||||
#[derive(Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ConfigFile {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub aether_url: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub management_token: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub public_ip: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
@@ -717,23 +866,31 @@ pub struct ConfigFile {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_reconnect_max_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_ping_interval_secs: Option<u64>,
|
||||
pub tunnel_ping_interval_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_max_streams: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_connect_timeout_secs: Option<u64>,
|
||||
pub tunnel_connect_timeout_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_tcp_keepalive_secs: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_tcp_nodelay: Option<bool>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_stale_timeout_secs: Option<u64>,
|
||||
pub tunnel_stale_timeout_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_connections: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_connections_max: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_scale_check_interval_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_scale_up_threshold_percent: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_scale_down_threshold_percent: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub tunnel_scale_down_grace_secs: Option<u64>,
|
||||
|
||||
/// Multi-server config: each entry connects to a separate Aether instance.
|
||||
/// When present, top-level aether_url/management_token are ignored for
|
||||
/// tunnel connections (but still injected as env for clap compatibility).
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub servers: Vec<ServerEntry>,
|
||||
}
|
||||
@@ -742,6 +899,7 @@ impl ConfigFile {
|
||||
/// Load from a TOML file.
|
||||
pub fn load(path: &Path) -> anyhow::Result<Self> {
|
||||
let content = std::fs::read_to_string(path)?;
|
||||
reject_removed_config_keys(&content)?;
|
||||
Ok(toml::from_str(&content)?)
|
||||
}
|
||||
|
||||
@@ -752,102 +910,6 @@ impl ConfigFile {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Detect and migrate a 0.1.x config file to 0.2.0 format in-place.
|
||||
///
|
||||
/// Returns `true` if migration was performed, `false` if already current.
|
||||
/// The original file is backed up as `<name>.v1.bak` before rewriting.
|
||||
pub fn migrate_legacy(path: &Path) -> anyhow::Result<bool> {
|
||||
let content = match std::fs::read_to_string(path) {
|
||||
Ok(c) => c,
|
||||
Err(_) => return Ok(false),
|
||||
};
|
||||
let mut table: toml::map::Map<String, toml::Value> = toml::from_str(&content)?;
|
||||
|
||||
// Detect legacy format: presence of any 0.1.x-only key.
|
||||
let is_legacy = LEGACY_ONLY_KEYS.iter().any(|k| table.contains_key(*k))
|
||||
|| DELEGATE_TO_UPSTREAM
|
||||
.iter()
|
||||
.any(|(old, _)| table.contains_key(*old));
|
||||
|
||||
if !is_legacy {
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
// 1. Rename delegate_* -> upstream_* (carry over user-customized values)
|
||||
for &(old, new) in DELEGATE_TO_UPSTREAM {
|
||||
if let Some(val) = table.remove(old) {
|
||||
table.entry(new.to_string()).or_insert(val);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Build [[servers]] from top-level aether_url + management_token + node_name
|
||||
if !table.contains_key("servers") {
|
||||
let aether_url = table.get("aether_url").and_then(|v| v.as_str());
|
||||
let management_token = table.get("management_token").and_then(|v| v.as_str());
|
||||
if let (Some(url), Some(token)) = (aether_url, management_token) {
|
||||
let mut entry = toml::map::Map::new();
|
||||
entry.insert("aether_url".into(), toml::Value::String(url.to_string()));
|
||||
entry.insert(
|
||||
"management_token".into(),
|
||||
toml::Value::String(token.to_string()),
|
||||
);
|
||||
if let Some(name) = table.get("node_name").and_then(|v| v.as_str()) {
|
||||
entry.insert("node_name".into(), toml::Value::String(name.to_string()));
|
||||
}
|
||||
table.insert(
|
||||
"servers".into(),
|
||||
toml::Value::Array(vec![toml::Value::Table(entry)]),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Remove top-level fields that are now in [[servers]] or obsolete
|
||||
table.remove("aether_url");
|
||||
table.remove("management_token");
|
||||
table.remove("node_name");
|
||||
for &key in LEGACY_ONLY_KEYS {
|
||||
table.remove(key);
|
||||
}
|
||||
|
||||
// 4. Backup original file (abort migration if backup fails)
|
||||
let backup_path = path.with_extension("v1.bak");
|
||||
std::fs::copy(path, &backup_path).map_err(|e| {
|
||||
anyhow::anyhow!(
|
||||
"failed to backup config before migration: {} -> {}: {}",
|
||||
path.display(),
|
||||
backup_path.display(),
|
||||
e
|
||||
)
|
||||
})?;
|
||||
|
||||
// 5. Write migrated config
|
||||
let new_content = toml::to_string_pretty(&table)?;
|
||||
std::fs::write(path, &new_content)?;
|
||||
|
||||
eprintln!(" Config migrated from 0.1.x to 0.2.0 format.");
|
||||
eprintln!(" Backup saved: {}", backup_path.display());
|
||||
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Resolve the effective server list.
|
||||
///
|
||||
/// If `[[servers]]` is present, use it. Otherwise fall back to the
|
||||
/// top-level `aether_url` + `management_token` as a single server.
|
||||
pub fn effective_servers(&self) -> Vec<ServerEntry> {
|
||||
if !self.servers.is_empty() {
|
||||
return self.servers.clone();
|
||||
}
|
||||
match (&self.aether_url, &self.management_token) {
|
||||
(Some(url), Some(token)) => vec![ServerEntry {
|
||||
aether_url: url.clone(),
|
||||
management_token: token.clone(),
|
||||
node_name: None,
|
||||
}],
|
||||
_ => vec![],
|
||||
}
|
||||
}
|
||||
|
||||
/// Inject values as environment variables so clap picks them up.
|
||||
///
|
||||
/// Only sets variables that are **not** already present in the
|
||||
@@ -874,18 +936,9 @@ impl ConfigFile {
|
||||
};
|
||||
}
|
||||
|
||||
// When top-level fields are absent, fall back to the first [[servers]]
|
||||
// entry so that clap's required `aether_url` / `management_token` are
|
||||
// satisfied even with the new config format.
|
||||
let first_server = self.servers.first();
|
||||
let aether_url = self
|
||||
.aether_url
|
||||
.as_deref()
|
||||
.or(first_server.map(|s| s.aether_url.as_str()));
|
||||
let management_token = self
|
||||
.management_token
|
||||
.as_deref()
|
||||
.or(first_server.map(|s| s.management_token.as_str()));
|
||||
let aether_url = first_server.map(|s| s.aether_url.as_str());
|
||||
let management_token = first_server.map(|s| s.management_token.as_str());
|
||||
let node_name = self
|
||||
.node_name
|
||||
.as_deref()
|
||||
@@ -988,25 +1041,39 @@ impl ConfigFile {
|
||||
"AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS",
|
||||
self.tunnel_reconnect_max_ms
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_PING_INTERVAL",
|
||||
self.tunnel_ping_interval_secs
|
||||
);
|
||||
set!(TUNNEL_PING_INTERVAL_MS_ENV, self.tunnel_ping_interval_ms);
|
||||
set!("AETHER_PROXY_TUNNEL_MAX_STREAMS", self.tunnel_max_streams);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
|
||||
self.tunnel_connect_timeout_secs
|
||||
TUNNEL_CONNECT_TIMEOUT_MS_ENV,
|
||||
self.tunnel_connect_timeout_ms
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_TCP_KEEPALIVE",
|
||||
self.tunnel_tcp_keepalive_secs
|
||||
);
|
||||
set!("AETHER_PROXY_TUNNEL_TCP_NODELAY", self.tunnel_tcp_nodelay);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_STALE_TIMEOUT",
|
||||
self.tunnel_stale_timeout_secs
|
||||
);
|
||||
set!(TUNNEL_STALE_TIMEOUT_MS_ENV, self.tunnel_stale_timeout_ms);
|
||||
set!("AETHER_PROXY_TUNNEL_CONNECTIONS", self.tunnel_connections);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_CONNECTIONS_MAX",
|
||||
self.tunnel_connections_max
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS",
|
||||
self.tunnel_scale_check_interval_ms
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT",
|
||||
self.tunnel_scale_up_threshold_percent
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT",
|
||||
self.tunnel_scale_down_threshold_percent
|
||||
);
|
||||
set!(
|
||||
"AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS",
|
||||
self.tunnel_scale_down_grace_secs
|
||||
);
|
||||
|
||||
// allowed_ports needs special handling (comma-separated)
|
||||
if let Some(ref ports) = self.allowed_ports {
|
||||
@@ -1022,11 +1089,70 @@ impl ConfigFile {
|
||||
}
|
||||
}
|
||||
|
||||
fn reject_removed_config_keys(content: &str) -> anyhow::Result<()> {
|
||||
let value: toml::Value = toml::from_str(content)?;
|
||||
let Some(table) = value.as_table() else {
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
let removed_seconds = REMOVED_TUNNEL_SECONDS_KEYS
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|key| table.contains_key(*key))
|
||||
.collect::<Vec<_>>();
|
||||
if !removed_seconds.is_empty() {
|
||||
anyhow::bail!(
|
||||
"removed tunnel config keys detected: {}. Use *_ms variants instead",
|
||||
removed_seconds.join(", ")
|
||||
);
|
||||
}
|
||||
|
||||
let removed_single_server = REMOVED_SINGLE_SERVER_KEYS
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|key| table.contains_key(*key))
|
||||
.collect::<Vec<_>>();
|
||||
if !removed_single_server.is_empty() {
|
||||
anyhow::bail!(
|
||||
"single-server top-level config keys are no longer supported: {}. Use [[servers]] entries instead",
|
||||
removed_single_server.join(", ")
|
||||
);
|
||||
}
|
||||
|
||||
let removed_legacy = LEGACY_ONLY_KEYS
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|key| table.contains_key(*key))
|
||||
.chain(
|
||||
DELEGATE_TO_UPSTREAM
|
||||
.iter()
|
||||
.map(|(old, _)| *old)
|
||||
.filter(|key| table.contains_key(*key)),
|
||||
)
|
||||
.collect::<Vec<_>>();
|
||||
if !removed_legacy.is_empty() {
|
||||
anyhow::bail!(
|
||||
"legacy config keys are no longer supported: {}",
|
||||
removed_legacy.join(", ")
|
||||
);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn div_ceil_u64(value: u64, divisor: u64) -> u64 {
|
||||
if divisor == 0 {
|
||||
return value;
|
||||
}
|
||||
value.saturating_add(divisor.saturating_sub(1)) / divisor
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use clap::CommandFactory;
|
||||
use clap::{CommandFactory, Parser};
|
||||
|
||||
use super::*;
|
||||
use crate::hardware::HardwareInfo;
|
||||
|
||||
#[test]
|
||||
fn parse_byte_size_supports_human_units() {
|
||||
@@ -1056,6 +1182,36 @@ mod tests {
|
||||
assert_eq!(stringy.redirect_replay_budget_bytes.as_deref(), Some("6M"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_file_rejects_removed_tunnel_seconds_keys() {
|
||||
let error = reject_removed_config_keys("tunnel_ping_interval_secs = 5")
|
||||
.expect_err("removed tunnel seconds keys should be rejected");
|
||||
assert!(
|
||||
error.to_string().contains("tunnel_ping_interval_secs"),
|
||||
"error should mention removed key"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_file_rejects_top_level_single_server_keys() {
|
||||
let error = reject_removed_config_keys("aether_url = \"https://example.com\"")
|
||||
.expect_err("top-level single-server key should be rejected");
|
||||
assert!(
|
||||
error.to_string().contains("aether_url"),
|
||||
"error should mention removed single-server key"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_file_rejects_legacy_keys() {
|
||||
let error = reject_removed_config_keys("delegate_connect_timeout_secs = 10")
|
||||
.expect_err("legacy delegate key should be rejected");
|
||||
assert!(
|
||||
error.to_string().contains("delegate_connect_timeout_secs"),
|
||||
"error should mention removed legacy key"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_requires_node_name() {
|
||||
let command = Config::command();
|
||||
@@ -1067,4 +1223,130 @@ mod tests {
|
||||
assert!(node_name.is_required_set());
|
||||
assert!(node_name.get_default_values().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tunnel_fast_recovery_defaults_use_millisecond_values() {
|
||||
let config = Config::parse_from([
|
||||
"aether-proxy",
|
||||
"--aether-url",
|
||||
"https://example.com",
|
||||
"--management-token",
|
||||
"ae_test",
|
||||
"--node-name",
|
||||
"proxy-test",
|
||||
]);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_ping_interval()
|
||||
.expect("ping interval should resolve"),
|
||||
Duration::from_millis(DEFAULT_TUNNEL_PING_INTERVAL_MS)
|
||||
);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_connect_timeout()
|
||||
.expect("connect timeout should resolve"),
|
||||
Duration::from_millis(DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS)
|
||||
);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_stale_timeout()
|
||||
.expect("stale timeout should resolve"),
|
||||
Duration::from_millis(DEFAULT_TUNNEL_STALE_TIMEOUT_MS)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tunnel_millisecond_flags_take_effect_when_explicitly_set() {
|
||||
let config = Config::parse_from([
|
||||
"aether-proxy",
|
||||
"--aether-url",
|
||||
"https://example.com",
|
||||
"--management-token",
|
||||
"ae_test",
|
||||
"--node-name",
|
||||
"proxy-test",
|
||||
"--tunnel-ping-interval-ms",
|
||||
"100",
|
||||
"--tunnel-connect-timeout-ms",
|
||||
"200",
|
||||
"--tunnel-stale-timeout-ms",
|
||||
"300",
|
||||
]);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_ping_interval()
|
||||
.expect("ping interval should resolve"),
|
||||
Duration::from_millis(100)
|
||||
);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_connect_timeout()
|
||||
.expect("connect timeout should resolve"),
|
||||
Duration::from_millis(200)
|
||||
);
|
||||
assert_eq!(
|
||||
config
|
||||
.tunnel_stale_timeout()
|
||||
.expect("stale timeout should resolve"),
|
||||
Duration::from_millis(300)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auto_tunnel_pool_sizing_uses_hardware_capacity() {
|
||||
let config = Config::parse_from([
|
||||
"aether-proxy",
|
||||
"--aether-url",
|
||||
"https://example.com",
|
||||
"--management-token",
|
||||
"ae_test",
|
||||
"--node-name",
|
||||
"proxy-test",
|
||||
"--tunnel-max-streams",
|
||||
"1024",
|
||||
]);
|
||||
let hw = HardwareInfo {
|
||||
cpu_cores: 12,
|
||||
total_memory_mb: 20_480,
|
||||
os_info: "test".to_string(),
|
||||
fd_limit: 1_048_576,
|
||||
estimated_max_concurrency: 24_000,
|
||||
};
|
||||
|
||||
let sizing = config
|
||||
.resolve_tunnel_pool_sizing(&hw)
|
||||
.expect("sizing should resolve");
|
||||
assert_eq!(sizing.initial_connections, 3);
|
||||
assert_eq!(sizing.max_connections, 6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_tunnel_connections_keep_fixed_pool_without_max_override() {
|
||||
let config = Config::parse_from([
|
||||
"aether-proxy",
|
||||
"--aether-url",
|
||||
"https://example.com",
|
||||
"--management-token",
|
||||
"ae_test",
|
||||
"--node-name",
|
||||
"proxy-test",
|
||||
"--tunnel-max-streams",
|
||||
"512",
|
||||
"--tunnel-connections",
|
||||
"2",
|
||||
]);
|
||||
let hw = HardwareInfo {
|
||||
cpu_cores: 12,
|
||||
total_memory_mb: 20_480,
|
||||
os_info: "test".to_string(),
|
||||
fd_limit: 1_048_576,
|
||||
estimated_max_concurrency: 24_000,
|
||||
};
|
||||
|
||||
let sizing = config
|
||||
.resolve_tunnel_pool_sizing(&hw)
|
||||
.expect("sizing should resolve");
|
||||
assert_eq!(sizing.initial_connections, 2);
|
||||
assert_eq!(sizing.max_connections, 2);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,12 +61,15 @@ async fn main() -> anyhow::Result<()> {
|
||||
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
|
||||
let config_path = std::path::Path::new(&config_file_path);
|
||||
if config_path.exists() {
|
||||
// Migrate legacy 0.1.x config to 0.2.0 format if needed
|
||||
if let Err(e) = config::ConfigFile::migrate_legacy(config_path) {
|
||||
eprintln!(" WARNING: config migration failed: {}", e);
|
||||
}
|
||||
if let Ok(file_cfg) = config::ConfigFile::load(config_path) {
|
||||
file_cfg.inject_env();
|
||||
match config::ConfigFile::load(config_path) {
|
||||
Ok(file_cfg) => file_cfg.inject_env(),
|
||||
Err(error) => {
|
||||
eprintln!(
|
||||
" WARNING: failed to load config {}: {}",
|
||||
config_path.display(),
|
||||
error
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -148,21 +151,19 @@ async fn run_proxy(config: Config) -> anyhow::Result<()> {
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
// Resolve server list: prefer [[servers]] from TOML, fall back to CLI/env single server.
|
||||
// Resolve server list: if a config file exists, it must use [[servers]].
|
||||
// Otherwise fall back to CLI/env single-server mode.
|
||||
let config_path =
|
||||
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
|
||||
let servers = if std::path::Path::new(&config_path).exists() {
|
||||
config::ConfigFile::load(std::path::Path::new(&config_path))
|
||||
.ok()
|
||||
.map(|f| f.effective_servers())
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| {
|
||||
vec![config::ServerEntry {
|
||||
aether_url: config.aether_url.clone(),
|
||||
management_token: config.management_token.clone(),
|
||||
node_name: None,
|
||||
}]
|
||||
})
|
||||
let file_cfg = config::ConfigFile::load(std::path::Path::new(&config_path))?;
|
||||
if file_cfg.servers.is_empty() {
|
||||
anyhow::bail!(
|
||||
"config file {} must contain at least one [[servers]] entry",
|
||||
config_path
|
||||
);
|
||||
}
|
||||
file_cfg.servers.clone()
|
||||
} else {
|
||||
vec![config::ServerEntry {
|
||||
aether_url: config.aether_url.clone(),
|
||||
|
||||
@@ -172,7 +172,7 @@ impl App {
|
||||
value: DEFAULT_HEARTBEAT_INTERVAL_SECS.to_string(),
|
||||
kind: FieldKind::Text,
|
||||
required: false,
|
||||
help: "Heartbeat interval in seconds; default is 30",
|
||||
help: "Heartbeat interval in seconds; default is 5",
|
||||
},
|
||||
Field {
|
||||
label: "Redirect Replay Budget",
|
||||
@@ -264,22 +264,11 @@ impl App {
|
||||
}
|
||||
|
||||
// Server tabs
|
||||
let servers = cfg.effective_servers();
|
||||
let servers = cfg.servers.clone();
|
||||
if servers.is_empty() {
|
||||
let mut tab = ServerTab::new();
|
||||
// Single-server fallback: use top-level node_name
|
||||
if let Some(ref name) = cfg.node_name {
|
||||
tab.fields[2].value = name.clone();
|
||||
}
|
||||
self.server_tabs = vec![tab];
|
||||
self.server_tabs = vec![ServerTab::new()];
|
||||
} else {
|
||||
self.server_tabs = servers.iter().map(ServerTab::from_entry).collect();
|
||||
// For single-server mode, node_name might be in top-level only
|
||||
if self.server_tabs.len() == 1 && self.server_tabs[0].fields[2].value.is_empty() {
|
||||
if let Some(ref name) = cfg.node_name {
|
||||
self.server_tabs[0].fields[2].value = name.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
self.active_tab = 0;
|
||||
self.selected = 0;
|
||||
@@ -387,7 +376,7 @@ impl App {
|
||||
..ConfigFile::default()
|
||||
};
|
||||
|
||||
// Always write [[servers]] format; old top-level fields are read-only compat
|
||||
// Always write [[servers]] format.
|
||||
cfg.servers = self
|
||||
.server_tabs
|
||||
.iter()
|
||||
|
||||
@@ -31,6 +31,7 @@ pub async fn connect_and_run(
|
||||
server: &Arc<ServerContext>,
|
||||
conn_idx: usize,
|
||||
shutdown: &mut watch::Receiver<bool>,
|
||||
drain: watch::Receiver<bool>,
|
||||
) -> Result<TunnelOutcome, anyhow::Error> {
|
||||
let ws_url = build_tunnel_url(server);
|
||||
info!(url = %ws_url, conn = conn_idx, "connecting tunnel");
|
||||
@@ -53,8 +54,7 @@ pub async fn connect_and_run(
|
||||
http::HeaderValue::from_str(&dynamic_node_name)?,
|
||||
);
|
||||
// Advertise per-connection max concurrent streams so the backend can
|
||||
// respect the proxy's capacity limit (backward-compatible: old backends
|
||||
// ignore this header).
|
||||
// respect the proxy's capacity limit.
|
||||
let max_streams = state.config.tunnel_max_streams.unwrap_or(128);
|
||||
headers.insert("X-Tunnel-Max-Streams", http::HeaderValue::from(max_streams));
|
||||
|
||||
@@ -67,13 +67,16 @@ pub async fn connect_and_run(
|
||||
let port = uri.port_u16().unwrap_or(if is_tls { 443 } else { 80 });
|
||||
|
||||
// TCP connect with timeout
|
||||
let connect_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
|
||||
let connect_timeout = state
|
||||
.config
|
||||
.tunnel_connect_timeout()
|
||||
.expect("validated config should resolve tunnel connect timeout");
|
||||
let tcp_stream = tokio::time::timeout(connect_timeout, TcpStream::connect((host, port)))
|
||||
.await
|
||||
.map_err(|_| {
|
||||
anyhow::anyhow!(
|
||||
"tunnel TCP connect timeout ({}s)",
|
||||
connect_timeout.as_secs()
|
||||
"tunnel TCP connect timeout ({}ms)",
|
||||
connect_timeout.as_millis()
|
||||
)
|
||||
})??;
|
||||
|
||||
@@ -96,7 +99,7 @@ pub async fn connect_and_run(
|
||||
max_message_size: Some(64 << 20),
|
||||
..Default::default()
|
||||
};
|
||||
let handshake_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
|
||||
let handshake_timeout = connect_timeout;
|
||||
let (ws_stream, _response) = tokio::time::timeout(
|
||||
handshake_timeout,
|
||||
tokio_tungstenite::client_async_tls_with_config(
|
||||
@@ -109,16 +112,25 @@ pub async fn connect_and_run(
|
||||
.await
|
||||
.map_err(|_| {
|
||||
anyhow::anyhow!(
|
||||
"tunnel WebSocket handshake timeout ({}s)",
|
||||
handshake_timeout.as_secs()
|
||||
"tunnel WebSocket handshake timeout ({}ms)",
|
||||
handshake_timeout.as_millis()
|
||||
)
|
||||
})??;
|
||||
let stale_timeout = state
|
||||
.config
|
||||
.tunnel_stale_timeout()
|
||||
.expect("validated config should resolve tunnel stale timeout");
|
||||
let ping_interval = state
|
||||
.config
|
||||
.tunnel_ping_interval()
|
||||
.expect("validated config should resolve tunnel ping interval");
|
||||
info!(
|
||||
conn = conn_idx,
|
||||
tcp_keepalive_secs = state.config.tunnel_tcp_keepalive_secs,
|
||||
tcp_nodelay = state.config.tunnel_tcp_nodelay,
|
||||
connect_timeout_secs = state.config.tunnel_connect_timeout_secs,
|
||||
stale_timeout_secs = state.config.tunnel_stale_timeout_secs,
|
||||
connect_timeout_ms = connect_timeout.as_millis(),
|
||||
stale_timeout_ms = stale_timeout.as_millis(),
|
||||
ping_interval_ms = ping_interval.as_millis(),
|
||||
"tunnel connected"
|
||||
);
|
||||
|
||||
@@ -129,8 +141,8 @@ pub async fn connect_and_run(
|
||||
let (ws_sink, ws_read) = futures_util::StreamExt::split(ws_stream);
|
||||
|
||||
// Spawn writer task (with WebSocket ping keepalive)
|
||||
let ping_interval = Duration::from_secs(state.config.tunnel_ping_interval_secs);
|
||||
let (frame_tx, mut writer_handle) = writer::spawn_writer(ws_sink, ping_interval);
|
||||
let drain_signal = spawn_drain_signal(conn_idx, frame_tx.clone(), drain.clone());
|
||||
|
||||
// Spawn heartbeat task (only for primary connection to avoid
|
||||
// resetting shared atomic metrics via swap(0))
|
||||
@@ -153,7 +165,14 @@ pub async fn connect_and_run(
|
||||
let state_clone = Arc::clone(state);
|
||||
let server_clone = Arc::clone(server);
|
||||
let outcome = tokio::select! {
|
||||
result = dispatcher::run(state_clone, server_clone, ws_read, frame_tx.clone(), hb_handle) => {
|
||||
result = dispatcher::run(
|
||||
state_clone,
|
||||
server_clone,
|
||||
ws_read,
|
||||
frame_tx.clone(),
|
||||
hb_handle,
|
||||
drain.clone(),
|
||||
) => {
|
||||
match result {
|
||||
Ok(()) => TunnelOutcome::Disconnected,
|
||||
Err(e) => return Err(e),
|
||||
@@ -181,6 +200,10 @@ pub async fn connect_and_run(
|
||||
// Drop our sender; the writer will exit once all stream handler clones
|
||||
// are also dropped (i.e. after they finish their in-flight work).
|
||||
drop(frame_tx);
|
||||
if !drain_signal.is_finished() {
|
||||
drain_signal.abort();
|
||||
let _ = drain_signal.await;
|
||||
}
|
||||
|
||||
// Wait for the writer task to finish with a generous timeout — the
|
||||
// dispatcher already waits up to 30s for stream handlers, so 35s here
|
||||
@@ -194,6 +217,35 @@ pub async fn connect_and_run(
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
fn spawn_drain_signal(
|
||||
conn_idx: usize,
|
||||
frame_tx: writer::FrameSender,
|
||||
mut drain: watch::Receiver<bool>,
|
||||
) -> tokio::task::JoinHandle<()> {
|
||||
tokio::spawn(async move {
|
||||
if !*drain.borrow() {
|
||||
loop {
|
||||
if drain.changed().await.is_err() {
|
||||
return;
|
||||
}
|
||||
if *drain.borrow() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
debug!(conn = conn_idx, "sending GOAWAY for tunnel drain");
|
||||
let _ = tokio::time::timeout(
|
||||
Duration::from_millis(250),
|
||||
frame_tx.send(super::protocol::Frame::control(
|
||||
super::protocol::MsgType::GoAway,
|
||||
bytes::Bytes::new(),
|
||||
)),
|
||||
)
|
||||
.await;
|
||||
})
|
||||
}
|
||||
|
||||
/// Configure TCP keepalive and NODELAY on an established socket.
|
||||
fn configure_tcp_socket(stream: &TcpStream, state: &Arc<AppState>) {
|
||||
let sock_ref = socket2::SockRef::from(stream);
|
||||
|
||||
@@ -7,6 +7,7 @@ use std::time::Duration;
|
||||
use bytes::Bytes;
|
||||
use futures_util::StreamExt;
|
||||
use tokio::sync::mpsc;
|
||||
use tokio::sync::watch;
|
||||
use tokio::task::JoinHandle;
|
||||
use tokio_tungstenite::tungstenite::Message;
|
||||
use tracing::{debug, error, info, warn};
|
||||
@@ -25,6 +26,7 @@ pub async fn run<S>(
|
||||
mut ws_stream: S,
|
||||
frame_tx: FrameSender,
|
||||
heartbeat: HeartbeatHandle,
|
||||
mut drain: watch::Receiver<bool>,
|
||||
) -> Result<(), anyhow::Error>
|
||||
where
|
||||
S: StreamExt<Item = Result<Message, tokio_tungstenite::tungstenite::Error>>
|
||||
@@ -38,12 +40,21 @@ where
|
||||
let mut handler_handles: Vec<JoinHandle<()>> = Vec::new();
|
||||
let max_streams = state.config.tunnel_max_streams.unwrap_or(128) as usize;
|
||||
let mut frames_since_cleanup: u32 = 0;
|
||||
let stale_timeout = Duration::from_secs(state.config.tunnel_stale_timeout_secs);
|
||||
let stale_timeout = state
|
||||
.config
|
||||
.tunnel_stale_timeout()
|
||||
.expect("validated config should resolve tunnel stale timeout");
|
||||
|
||||
// Track last time we received any data to detect stale connections
|
||||
let mut last_data_at = tokio::time::Instant::now();
|
||||
let mut draining = *drain.borrow();
|
||||
|
||||
let read_err = loop {
|
||||
if draining && streams.is_empty() {
|
||||
info!("tunnel drained after in-flight streams completed");
|
||||
break None;
|
||||
}
|
||||
|
||||
let msg_result = tokio::select! {
|
||||
msg = ws_stream.next() => {
|
||||
match msg {
|
||||
@@ -51,9 +62,19 @@ where
|
||||
None => break None,
|
||||
}
|
||||
}
|
||||
changed = drain.changed() => {
|
||||
if changed.is_err() {
|
||||
continue;
|
||||
}
|
||||
if *drain.borrow() {
|
||||
info!("tunnel drain requested, waiting for in-flight streams");
|
||||
draining = true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
_ = tokio::time::sleep_until(last_data_at + stale_timeout) => {
|
||||
warn!(
|
||||
stale_secs = stale_timeout.as_secs(),
|
||||
stale_ms = stale_timeout.as_millis(),
|
||||
"tunnel connection stale, no data received"
|
||||
);
|
||||
break None;
|
||||
@@ -92,6 +113,24 @@ where
|
||||
|
||||
match frame.msg_type {
|
||||
MsgType::RequestHeaders => {
|
||||
if draining {
|
||||
if frame_tx
|
||||
.try_send(Frame::new(
|
||||
frame.stream_id,
|
||||
MsgType::StreamError,
|
||||
0,
|
||||
Bytes::from("tunnel draining"),
|
||||
))
|
||||
.is_err()
|
||||
{
|
||||
warn!(
|
||||
stream_id = frame.stream_id,
|
||||
"writer channel full, StreamError dropped during drain"
|
||||
);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Decompress if the frame is gzip-compressed, then parse metadata
|
||||
let payload = match decompress_if_gzip(&frame) {
|
||||
Ok(p) => p,
|
||||
@@ -176,6 +215,10 @@ where
|
||||
let _ = tx.send(frame).await;
|
||||
if is_end {
|
||||
streams.remove(&sid);
|
||||
if draining && streams.is_empty() {
|
||||
info!("tunnel drained after request body completion");
|
||||
break None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -184,6 +227,10 @@ where
|
||||
// Client-side cancellation or end
|
||||
if let Some(tx) = streams.remove(&frame.stream_id) {
|
||||
let _ = tx.send(frame).await;
|
||||
if draining && streams.is_empty() {
|
||||
info!("tunnel drained after stream termination");
|
||||
break None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ static NON_ROOT_UPGRADE_WARNED: AtomicBool = AtomicBool::new(false);
|
||||
|
||||
enum AckDecision {
|
||||
Accept {
|
||||
heartbeat_id: Option<u64>,
|
||||
heartbeat_id: u64,
|
||||
upgrade_to: Option<String>,
|
||||
},
|
||||
Ignore,
|
||||
@@ -143,16 +143,8 @@ pub fn spawn(
|
||||
upgrade_to,
|
||||
} => {
|
||||
if let Some((pending_id, _)) = pending {
|
||||
match ack_id {
|
||||
Some(id) if id == pending_id => {
|
||||
pending = None;
|
||||
}
|
||||
None => {
|
||||
// Backward-compatible with servers that don't echo
|
||||
// heartbeat_id in ACK payload yet.
|
||||
pending = None;
|
||||
}
|
||||
_ => {}
|
||||
if ack_id == pending_id {
|
||||
pending = None;
|
||||
}
|
||||
}
|
||||
maybe_trigger_upgrade(upgrade_to);
|
||||
@@ -266,6 +258,7 @@ async fn build_heartbeat_payload(
|
||||
"node_id": node_id,
|
||||
"heartbeat_session_id": heartbeat_session_id,
|
||||
"heartbeat_id": heartbeat_id,
|
||||
"heartbeat_interval": server.dynamic.load().heartbeat_interval,
|
||||
"active_connections": server.active_connections.load(Ordering::Acquire),
|
||||
"total_requests": snapshot.requests,
|
||||
"avg_latency_ms": avg_latency_ms,
|
||||
@@ -283,10 +276,8 @@ async fn build_heartbeat_payload(
|
||||
|
||||
fn handle_ack(server: &ServerContext, payload: &[u8]) -> AckDecision {
|
||||
if payload.is_empty() {
|
||||
return AckDecision::Accept {
|
||||
heartbeat_id: None,
|
||||
upgrade_to: None,
|
||||
};
|
||||
warn!("received empty heartbeat ACK");
|
||||
return AckDecision::Ignore;
|
||||
}
|
||||
|
||||
#[derive(serde::Deserialize)]
|
||||
@@ -295,8 +286,7 @@ fn handle_ack(server: &ServerContext, payload: &[u8]) -> AckDecision {
|
||||
remote_config: Option<RemoteConfig>,
|
||||
#[serde(default)]
|
||||
config_version: u64,
|
||||
#[serde(default)]
|
||||
heartbeat_id: Option<u64>,
|
||||
heartbeat_id: u64,
|
||||
#[serde(default)]
|
||||
upgrade_to: Option<String>,
|
||||
}
|
||||
@@ -371,3 +361,74 @@ fn maybe_trigger_upgrade(version: Option<String>) {
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::atomic::AtomicU64;
|
||||
use std::sync::{Arc, RwLock};
|
||||
|
||||
use arc_swap::ArcSwap;
|
||||
use clap::Parser;
|
||||
|
||||
use super::{handle_ack, AckDecision};
|
||||
use crate::registration::client::AetherClient;
|
||||
use crate::runtime::DynamicConfig;
|
||||
use crate::state::{ProxyMetrics, ServerContext};
|
||||
|
||||
fn sample_server() -> Arc<ServerContext> {
|
||||
let config = Arc::new(crate::config::Config::parse_from([
|
||||
"aether-proxy",
|
||||
"--aether-url",
|
||||
"https://example.com",
|
||||
"--management-token",
|
||||
"ae_test",
|
||||
"--node-name",
|
||||
"proxy-test",
|
||||
]));
|
||||
Arc::new(ServerContext {
|
||||
server_label: "heartbeat-test".to_string(),
|
||||
aether_url: config.aether_url.clone(),
|
||||
management_token: config.management_token.clone(),
|
||||
node_name: config.node_name.clone(),
|
||||
node_id: Arc::new(RwLock::new("node-123".to_string())),
|
||||
aether_client: Arc::new(AetherClient::new(
|
||||
&config,
|
||||
&config.aether_url,
|
||||
&config.management_token,
|
||||
)),
|
||||
dynamic: Arc::new(ArcSwap::from_pointee(DynamicConfig::from_config(&config))),
|
||||
active_connections: Arc::new(AtomicU64::new(0)),
|
||||
metrics: Arc::new(ProxyMetrics::new()),
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_ack_requires_heartbeat_id() {
|
||||
let server = sample_server();
|
||||
let decision = handle_ack(
|
||||
&server,
|
||||
br#"{"config_version":1,"remote_config":{"heartbeat_interval":9}}"#,
|
||||
);
|
||||
|
||||
assert!(matches!(decision, AckDecision::Ignore));
|
||||
assert_eq!(server.dynamic.load().heartbeat_interval, 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_ack_applies_remote_config_with_heartbeat_id() {
|
||||
let server = sample_server();
|
||||
let decision = handle_ack(
|
||||
&server,
|
||||
br#"{"heartbeat_id":7,"config_version":1,"remote_config":{"heartbeat_interval":9}}"#,
|
||||
);
|
||||
|
||||
assert!(matches!(
|
||||
decision,
|
||||
AckDecision::Accept {
|
||||
heartbeat_id: 7,
|
||||
upgrade_to: None
|
||||
}
|
||||
));
|
||||
assert_eq!(server.dynamic.load().heartbeat_interval, 9);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,10 +36,16 @@ pub async fn run(
|
||||
server: &Arc<ServerContext>,
|
||||
conn_idx: usize,
|
||||
mut shutdown: watch::Receiver<bool>,
|
||||
mut drain: watch::Receiver<bool>,
|
||||
) {
|
||||
info!(server = %server.server_label, conn = conn_idx, "starting tunnel");
|
||||
let reconnect_salt = compute_connection_salt(server, conn_idx);
|
||||
|
||||
if *drain.borrow() {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested before startup");
|
||||
return;
|
||||
}
|
||||
|
||||
let startup_delay = compute_startup_stagger(conn_idx, reconnect_salt);
|
||||
if !startup_delay.is_zero() {
|
||||
info!(
|
||||
@@ -54,14 +60,24 @@ pub async fn run(
|
||||
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during startup stagger");
|
||||
return;
|
||||
}
|
||||
_ = drain.changed() => {
|
||||
if *drain.borrow() {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested during startup stagger");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut consecutive_failures: u32 = 0;
|
||||
|
||||
loop {
|
||||
if *drain.borrow() {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel drained, exiting slot");
|
||||
return;
|
||||
}
|
||||
let started_at = Instant::now();
|
||||
match client::connect_and_run(state, server, conn_idx, &mut shutdown).await {
|
||||
match client::connect_and_run(state, server, conn_idx, &mut shutdown, drain.clone()).await {
|
||||
Ok(client::TunnelOutcome::Shutdown) => {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel shut down gracefully");
|
||||
return;
|
||||
@@ -78,6 +94,10 @@ pub async fn run(
|
||||
info!(server = %server.server_label, conn = conn_idx, "shutdown requested, not reconnecting");
|
||||
return;
|
||||
}
|
||||
if *drain.borrow() {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel drained after disconnect");
|
||||
return;
|
||||
}
|
||||
|
||||
// Reset backoff after a stable session to keep recovery snappy when
|
||||
// failures are only occasional.
|
||||
@@ -108,6 +128,12 @@ pub async fn run(
|
||||
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during reconnect wait");
|
||||
return;
|
||||
}
|
||||
_ = drain.changed() => {
|
||||
if *drain.borrow() {
|
||||
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested during reconnect wait");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -266,8 +292,9 @@ mod tests {
|
||||
let proxy_task = tokio::spawn({
|
||||
let state = Arc::clone(&state);
|
||||
let server = Arc::clone(&server);
|
||||
let (_drain_tx, drain_rx) = watch::channel(false);
|
||||
async move {
|
||||
run(&state, &server, 0, shutdown_rx).await;
|
||||
run(&state, &server, 0, shutdown_rx, drain_rx).await;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -486,13 +513,18 @@ mod tests {
|
||||
log_max_files: 30,
|
||||
tunnel_reconnect_base_ms: 50,
|
||||
tunnel_reconnect_max_ms: 250,
|
||||
tunnel_ping_interval_secs: 1,
|
||||
tunnel_ping_interval_ms: 1_000,
|
||||
tunnel_max_streams: Some(8),
|
||||
tunnel_connect_timeout_secs: 2,
|
||||
tunnel_connect_timeout_ms: 2_000,
|
||||
tunnel_tcp_keepalive_secs: 30,
|
||||
tunnel_tcp_nodelay: true,
|
||||
tunnel_stale_timeout_secs: 5,
|
||||
tunnel_connections: 1,
|
||||
tunnel_stale_timeout_ms: 5_000,
|
||||
tunnel_connections: Some(1),
|
||||
tunnel_connections_max: Some(1),
|
||||
tunnel_scale_check_interval_ms: 1_000,
|
||||
tunnel_scale_up_threshold_percent: 70,
|
||||
tunnel_scale_down_threshold_percent: 35,
|
||||
tunnel_scale_down_grace_secs: 15,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1594,13 +1594,18 @@ mod tests {
|
||||
log_max_files: 30,
|
||||
tunnel_reconnect_base_ms: 500,
|
||||
tunnel_reconnect_max_ms: 30_000,
|
||||
tunnel_ping_interval_secs: 15,
|
||||
tunnel_ping_interval_ms: 15_000,
|
||||
tunnel_max_streams: Some(8),
|
||||
tunnel_connect_timeout_secs: 15,
|
||||
tunnel_connect_timeout_ms: 15_000,
|
||||
tunnel_tcp_keepalive_secs: 30,
|
||||
tunnel_tcp_nodelay: true,
|
||||
tunnel_stale_timeout_secs: 45,
|
||||
tunnel_connections: 1,
|
||||
tunnel_stale_timeout_ms: 45_000,
|
||||
tunnel_connections: Some(1),
|
||||
tunnel_connections_max: Some(1),
|
||||
tunnel_scale_check_interval_ms: 1_000,
|
||||
tunnel_scale_up_threshold_percent: 70,
|
||||
tunnel_scale_down_threshold_percent: 35,
|
||||
tunnel_scale_down_grace_secs: 15,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user