feat(proxy): 重构 Proxy 节点管理与隧道系统

- 重构 proxy_nodes 管理端,支持节点注册、心跳、隧道生命周期管理
- 增强 tunnel 嵌入式 hub 和隧道协议
- 重构 aether-proxy 配置、隧道客户端、心跳和调度机制
- 调整 admin OAuth/配额/导入等处理器的参数传递
- 扩展数据迁移模块
- 补充 proxy nodes、OAuth、配额、系统导入等测试
- 更新前端 proxy nodes 视图和 API
This commit is contained in:
fawney19
2026-04-14 22:51:02 +08:00
parent fb31928e44
commit a4e7ac1df6
55 changed files with 3477 additions and 892 deletions
-10
View File
@@ -1,10 +0,0 @@
FROM debian:bookworm-slim
ARG TARGETARCH
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY build/linux-${TARGETARCH}/aether-proxy /usr/local/bin/aether-proxy
ENTRYPOINT ["aether-proxy"]
+15 -17
View File
@@ -10,14 +10,6 @@ Tunnel 模式下代理节点**无需对外监听端口**,仅需出站连接到
- 常规 Linux 发行版:`systemd`
- Alpine Linux`OpenRC`
### Docker Compose 部署
```bash
cp .env.example .env
# 编辑 .env 填入 AETHER_PROXY_AETHER_URL 和 AETHER_PROXY_MANAGEMENT_TOKEN
docker compose up -d
```
### 下载预编译二进制
<!-- DOWNLOAD_TABLE_START -->
@@ -82,23 +74,30 @@ sudo aether-proxy uninstall
| `--node-name` | `AETHER_PROXY_NODE_NAME` | **必填** | 节点名称标识 |
| `--public-ip` | `AETHER_PROXY_PUBLIC_IP` | 自动检测 | 公网 IP |
| `--node-region` | `AETHER_PROXY_NODE_REGION` | 自动检测 | 地区标识 |
| `--heartbeat-interval` | `AETHER_PROXY_HEARTBEAT_INTERVAL` | `30` | 心跳间隔(秒) |
| `--heartbeat-interval` | `AETHER_PROXY_HEARTBEAT_INTERVAL` | `5` | 心跳间隔(秒) |
| `--allowed-ports` | `AETHER_PROXY_ALLOWED_PORTS` | `80,443,8080,8443` | 允许代理的目标端口 |
#### Tunnel 连接
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--tunnel-connections` | `AETHER_PROXY_TUNNEL_CONNECTIONS` | `3` | 到 Aether 的连接池大小 |
| `--tunnel-connections` | `AETHER_PROXY_TUNNEL_CONNECTIONS` | 自动(硬件估算) | 最小连接池大小;显式设置后默认固定为该值 |
| `--tunnel-connections-max` | `AETHER_PROXY_TUNNEL_CONNECTIONS_MAX` | 自动(硬件估算) | 连接池自动扩容上限;大于 `tunnel_connections` 时启用 autoscale |
| `--tunnel-max-streams` | `AETHER_PROXY_TUNNEL_MAX_STREAMS` | 自动(硬件估算) | 单连接最大并发 stream 数 |
| `--tunnel-connect-timeout-secs` | `AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_SECS` | `15` | TCP + TLS 握手超时(秒) |
| `--tunnel-ping-interval-ms` | `AETHER_PROXY_TUNNEL_PING_INTERVAL_MS` | `250` | fast-fail 探测周期(毫秒) |
| `--tunnel-connect-timeout-ms` | `AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_MS` | `800` | fast-reconnect 建连超时(毫秒) |
| `--tunnel-stale-timeout-ms` | `AETHER_PROXY_TUNNEL_STALE_TIMEOUT_MS` | `900` | 无入站数据断连阈值(毫秒) |
| `--tunnel-scale-check-interval-ms` | `AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS` | `1000` | autoscale 采样周期(毫秒) |
| `--tunnel-scale-up-threshold-percent` | `AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT` | `70` | 单 tunnel 占用率超过该值时扩容 |
| `--tunnel-scale-down-threshold-percent` | `AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT` | `35` | 单 tunnel 占用率持续低于该值时允许缩容 |
| `--tunnel-scale-down-grace-secs` | `AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS` | `15` | 低负载持续时间达到该值后才回收次级 tunnel |
| `--tunnel-tcp-keepalive-secs` | `AETHER_PROXY_TUNNEL_TCP_KEEPALIVE_SECS` | `30` | TCP keepalive 初始延迟(秒) |
| `--tunnel-tcp-nodelay` | `AETHER_PROXY_TUNNEL_TCP_NODELAY` | `true` | 禁用 Nagle 算法 |
| `--tunnel-ping-interval-secs` | `AETHER_PROXY_TUNNEL_PING_INTERVAL_SECS` | `15` | WebSocket Ping 频率(秒) |
| `--tunnel-stale-timeout-secs` | `AETHER_PROXY_TUNNEL_STALE_TIMEOUT_SECS` | `45` | 无数据断连阈值(秒) |
| `--tunnel-reconnect-base-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS` | `500` | 指数退避基础延迟(毫秒) |
| `--tunnel-reconnect-base-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS` | `50` | 指数退避基础延迟(毫秒) |
| `--tunnel-reconnect-max-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS` | `30000` | 指数退避上限(毫秒) |
省略 `tunnel_connections` 时,proxy 会按设备能力自动计算一个基线值和扩容上限;如果显式设置了 `tunnel_connections` 但没有设置 `tunnel_connections_max`,则保持固定连接池,不自动扩缩。
#### 上游 HTTP 请求
| 参数 | 环境变量 | 默认值 | 说明 |
@@ -141,12 +140,12 @@ sudo aether-proxy uninstall
- 默认 `AETHER_PROXY_LOG_DESTINATION=stdout`,日志交给容器日志驱动或宿主机服务管理器
- 需要落盘时改成 `file``both`,并设置 `AETHER_PROXY_LOG_DIR`setup TUI 里用 `Save Logs to File` 开关即可
- 文件日志固定写普通文本,并支持 `hourly/daily` 轮转;默认按天轮换、保留 7 天,最多保留 30 个文件
- `docker compose` 默认保持 `stdout`,避免和容器自带日志重复;`systemd``OpenRC` 安装时默认会额外打开文件日志到 `/var/log/aether-proxy`
-`systemd``OpenRC` 安装时默认会额外打开文件日志到 `/var/log/aether-proxy`
- OpenRC 安装时,`aether-proxy logs` 实际读取 `/var/log/aether-proxy/current.log``/var/log/aether-proxy/error.log`;这些文件通常需要用 `sudo aether-proxy logs` 查看
### 多服务器配置
`aether-proxy.toml` 中使用 `[[servers]]` 配置多个 Aether 服务器
`aether-proxy.toml` 中使用 `[[servers]]` 配置 Aether 服务器。即使只有一个服务器,也必须写成一个 `[[servers]]` 条目;旧的顶层单服务器写法已不再支持。
```toml
[[servers]]
@@ -164,7 +163,6 @@ node_name = "jp-proxy-02"
推送 `proxy-v*` 格式的 tagGitHub Actions 会自动:
- 编译所有平台二进制并发布到 Releases
- 构建 Docker 镜像并推送到 GHCR 和 Docker Hub
- 更新 README 中的下载链接表格
```bash
-21
View File
@@ -1,21 +0,0 @@
services:
aether-proxy:
image: ghcr.io/fawney19/aether-proxy:latest
container_name: aether-proxy
restart: unless-stopped
env_file:
- .env
environment:
AETHER_PROXY_LOG_JSON: ${AETHER_PROXY_LOG_JSON:-true}
AETHER_PROXY_LOG_DESTINATION: ${AETHER_PROXY_LOG_DESTINATION:-stdout}
AETHER_PROXY_LOG_DIR: ${AETHER_PROXY_LOG_DIR:-/var/log/aether-proxy}
AETHER_PROXY_LOG_ROTATION: ${AETHER_PROXY_LOG_ROTATION:-daily}
AETHER_PROXY_LOG_RETENTION_DAYS: ${AETHER_PROXY_LOG_RETENTION_DAYS:-7}
AETHER_PROXY_LOG_MAX_FILES: ${AETHER_PROXY_LOG_MAX_FILES:-30}
volumes:
- ./logs:/var/log/aether-proxy
logging:
driver: json-file
options:
max-size: "50m"
max-file: "3"
+352 -29
View File
@@ -1,8 +1,9 @@
//! Application lifecycle: initialization, task orchestration, and shutdown.
use std::sync::atomic::AtomicU64;
use std::collections::BTreeMap;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, RwLock};
use std::time::Duration;
use std::time::{Duration, Instant};
use aether_http::{jittered_delay_for_retry, HttpRetryConfig};
use aether_runtime::{
@@ -14,7 +15,7 @@ use tokio::sync::{watch, Mutex};
use tokio::task::JoinHandle;
use tracing::{error, info, warn};
use crate::config::{Config, ServerEntry};
use crate::config::{Config, ServerEntry, TunnelPoolSizing};
use crate::net;
use crate::registration::client::AetherClient;
use crate::runtime::{self, DynamicConfig};
@@ -24,6 +25,52 @@ use crate::{hardware, target_filter, tunnel};
type TaskHandles = Arc<Mutex<Vec<JoinHandle<()>>>>;
#[derive(Debug, Clone, Copy)]
struct TunnelPoolPolicy {
min_connections: usize,
max_connections: usize,
max_streams_per_tunnel: usize,
scale_check_interval: Duration,
scale_up_threshold_percent: u32,
scale_down_threshold_percent: u32,
scale_down_grace: Duration,
}
impl TunnelPoolPolicy {
fn from_config(config: &Config, sizing: TunnelPoolSizing) -> Self {
Self {
min_connections: sizing.initial_connections.max(1) as usize,
max_connections: sizing
.max_connections
.max(sizing.initial_connections)
.max(1) as usize,
max_streams_per_tunnel: config.tunnel_max_streams.unwrap_or(128).max(1) as usize,
scale_check_interval: Duration::from_millis(config.tunnel_scale_check_interval_ms),
scale_up_threshold_percent: config.tunnel_scale_up_threshold_percent,
scale_down_threshold_percent: config.tunnel_scale_down_threshold_percent,
scale_down_grace: Duration::from_secs(config.tunnel_scale_down_grace_secs),
}
}
fn scale_up_high_water_mark(&self) -> u64 {
occupancy_threshold(self.max_streams_per_tunnel, self.scale_up_threshold_percent)
}
fn scale_down_low_water_mark(&self) -> u64 {
occupancy_threshold(
self.max_streams_per_tunnel,
self.scale_down_threshold_percent,
)
}
}
struct ManagedTunnel {
slot_id: usize,
drain_tx: watch::Sender<bool>,
handle: JoinHandle<()>,
draining: bool,
}
/// Run the full application lifecycle after config has been parsed.
pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Result<()> {
config.validate()?;
@@ -63,6 +110,19 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
"auto-detected tunnel_max_streams from hardware"
);
}
let tunnel_pool_sizing = config.resolve_tunnel_pool_sizing(&hw_info)?;
let tunnel_pool_policy = TunnelPoolPolicy::from_config(&config, tunnel_pool_sizing);
info!(
tunnel_connections_initial = tunnel_pool_policy.min_connections,
tunnel_connections_max = tunnel_pool_policy.max_connections,
tunnel_max_streams = tunnel_pool_policy.max_streams_per_tunnel,
scale_check_interval_ms = tunnel_pool_policy.scale_check_interval.as_millis(),
scale_up_threshold_percent = tunnel_pool_policy.scale_up_threshold_percent,
scale_down_threshold_percent = tunnel_pool_policy.scale_down_threshold_percent,
scale_down_grace_secs = tunnel_pool_policy.scale_down_grace.as_secs(),
auto_sizing = config.tunnel_connections.is_none(),
"resolved tunnel pool policy"
);
info!(
max_concurrency = hw_info.estimated_max_concurrency,
@@ -177,15 +237,14 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
"running in tunnel mode"
);
// Spawn tunnel connections per server (pool_size connections each)
let pool_size = state.config.tunnel_connections.max(1) as usize;
// Spawn tunnel pool manager per server.
let tunnel_handles: TaskHandles = Arc::new(Mutex::new(Vec::new()));
let retry_handles: TaskHandles = Arc::new(Mutex::new(Vec::new()));
for server in server_contexts.lock().await.iter() {
spawn_tunnel_pool(
spawn_tunnel_pool_manager(
Arc::clone(&state),
Arc::clone(server),
pool_size,
tunnel_pool_policy,
shutdown_rx.clone(),
Arc::clone(&tunnel_handles),
)
@@ -200,7 +259,7 @@ pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Resul
failed_entries,
public_ip.clone(),
hw_info.clone(),
pool_size,
tunnel_pool_policy,
shutdown_rx.clone(),
Arc::clone(&tunnel_handles),
Arc::clone(&retry_handles),
@@ -240,7 +299,7 @@ async fn spawn_registration_recovery_tasks(
failed: Vec<(String, ServerEntry)>,
public_ip: String,
hw_info: crate::hardware::HardwareInfo,
pool_size: usize,
tunnel_pool_policy: TunnelPoolPolicy,
shutdown: watch::Receiver<bool>,
tunnel_handles: TaskHandles,
retry_handles: TaskHandles,
@@ -261,7 +320,7 @@ async fn spawn_registration_recovery_tasks(
entry,
retry_public_ip,
retry_hw_info,
pool_size,
tunnel_pool_policy,
retry_shutdown,
retry_tunnels,
)
@@ -281,7 +340,7 @@ async fn retry_failed_registration(
entry: ServerEntry,
public_ip: String,
hw_info: crate::hardware::HardwareInfo,
pool_size: usize,
tunnel_pool_policy: TunnelPoolPolicy,
mut shutdown: watch::Receiver<bool>,
tunnel_handles: TaskHandles,
) {
@@ -329,10 +388,10 @@ async fn retry_failed_registration(
node_id,
);
server_contexts.lock().await.push(Arc::clone(&server));
spawn_tunnel_pool(
spawn_tunnel_pool_manager(
Arc::clone(&state),
server,
pool_size,
tunnel_pool_policy,
shutdown,
tunnel_handles,
)
@@ -386,23 +445,239 @@ fn build_server_context(
})
}
async fn spawn_tunnel_pool(
async fn spawn_tunnel_pool_manager(
state: Arc<AppState>,
server: Arc<ServerContext>,
pool_size: usize,
policy: TunnelPoolPolicy,
shutdown: watch::Receiver<bool>,
tunnel_handles: TaskHandles,
) {
let mut handles = Vec::with_capacity(pool_size);
for conn_idx in 0..pool_size {
let s = Arc::clone(&state);
let srv = Arc::clone(&server);
let rx = shutdown.clone();
handles.push(tokio::spawn(async move {
tunnel::run(&s, &srv, conn_idx, rx).await;
}));
let handle = tokio::spawn(async move {
run_tunnel_pool_manager(state, server, policy, shutdown).await;
});
tunnel_handles.lock().await.push(handle);
}
async fn run_tunnel_pool_manager(
state: Arc<AppState>,
server: Arc<ServerContext>,
policy: TunnelPoolPolicy,
mut shutdown: watch::Receiver<bool>,
) {
let mut tunnels = BTreeMap::<usize, ManagedTunnel>::new();
ensure_tunnel_capacity(
&mut tunnels,
policy.min_connections,
&policy,
&state,
&server,
&shutdown,
);
let mut ticker = tokio::time::interval(policy.scale_check_interval);
ticker.tick().await;
let mut low_load_since: Option<Instant> = None;
loop {
tokio::select! {
_ = shutdown.changed() => {
info!(server = %server.server_label, "tunnel pool manager shutting down");
break;
}
_ = ticker.tick() => {
reap_finished_tunnels(&mut tunnels).await;
let available = tunnels.values().filter(|tunnel| !tunnel.draining).count();
if available < policy.min_connections {
ensure_tunnel_capacity(
&mut tunnels,
policy.min_connections,
&policy,
&state,
&server,
&shutdown,
);
low_load_since = None;
continue;
}
let active_connections = server.active_connections.load(Ordering::Acquire);
let desired_connections = desired_tunnel_connections(active_connections, &policy);
if desired_connections > available {
ensure_tunnel_capacity(
&mut tunnels,
desired_connections,
&policy,
&state,
&server,
&shutdown,
);
info!(
server = %server.server_label,
active_connections,
available_connections = available,
target_connections = desired_connections,
"scaled tunnel pool up"
);
low_load_since = None;
continue;
}
if should_scale_down(active_connections, available, &policy) {
match low_load_since {
Some(since) if since.elapsed() >= policy.scale_down_grace => {
if request_tunnel_drain(&mut tunnels, policy.min_connections) {
info!(
server = %server.server_label,
active_connections,
available_connections = available,
"requested tunnel drain for scale-down"
);
}
low_load_since = None;
}
None => {
low_load_since = Some(Instant::now());
}
Some(_) => {}
}
} else {
low_load_since = None;
}
}
}
}
tunnel_handles.lock().await.extend(handles);
for tunnel in tunnels.values_mut() {
let _ = tunnel.drain_tx.send(true);
tunnel.draining = true;
}
while !tunnels.is_empty() {
reap_finished_tunnels(&mut tunnels).await;
if !tunnels.is_empty() {
tokio::time::sleep(Duration::from_millis(50)).await;
}
}
}
fn ensure_tunnel_capacity(
tunnels: &mut BTreeMap<usize, ManagedTunnel>,
target_connections: usize,
policy: &TunnelPoolPolicy,
state: &Arc<AppState>,
server: &Arc<ServerContext>,
shutdown: &watch::Receiver<bool>,
) {
let target_connections = target_connections.min(policy.max_connections);
while tunnels.values().filter(|tunnel| !tunnel.draining).count() < target_connections {
let Some(slot_id) = next_available_tunnel_slot(tunnels, policy.max_connections) else {
break;
};
tunnels.insert(
slot_id,
spawn_managed_tunnel(
Arc::clone(state),
Arc::clone(server),
slot_id,
shutdown.clone(),
),
);
}
}
fn spawn_managed_tunnel(
state: Arc<AppState>,
server: Arc<ServerContext>,
slot_id: usize,
shutdown: watch::Receiver<bool>,
) -> ManagedTunnel {
let (drain_tx, drain_rx) = watch::channel(false);
let handle = tokio::spawn(async move {
tunnel::run(&state, &server, slot_id, shutdown, drain_rx).await;
});
ManagedTunnel {
slot_id,
drain_tx,
handle,
draining: false,
}
}
async fn reap_finished_tunnels(tunnels: &mut BTreeMap<usize, ManagedTunnel>) {
let finished_slots = tunnels
.iter()
.filter_map(|(slot_id, tunnel)| tunnel.handle.is_finished().then_some(*slot_id))
.collect::<Vec<_>>();
for slot_id in finished_slots {
if let Some(tunnel) = tunnels.remove(&slot_id) {
let _ = tunnel.handle.await;
}
}
}
fn next_available_tunnel_slot(
tunnels: &BTreeMap<usize, ManagedTunnel>,
max_connections: usize,
) -> Option<usize> {
(0..max_connections).find(|slot_id| !tunnels.contains_key(slot_id))
}
fn request_tunnel_drain(
tunnels: &mut BTreeMap<usize, ManagedTunnel>,
min_connections: usize,
) -> bool {
let available = tunnels.values().filter(|tunnel| !tunnel.draining).count();
if available <= min_connections {
return false;
}
let Some((_, tunnel)) = tunnels
.iter_mut()
.rev()
.find(|(_, tunnel)| tunnel.slot_id != 0 && !tunnel.draining)
else {
return false;
};
if tunnel.drain_tx.send(true).is_ok() {
tunnel.draining = true;
return true;
}
false
}
fn desired_tunnel_connections(active_connections: u64, policy: &TunnelPoolPolicy) -> usize {
let required = div_ceil_u64(active_connections.max(1), policy.scale_up_high_water_mark());
required.clamp(policy.min_connections as u64, policy.max_connections as u64) as usize
}
fn should_scale_down(
active_connections: u64,
available_connections: usize,
policy: &TunnelPoolPolicy,
) -> bool {
if available_connections <= policy.min_connections {
return false;
}
active_connections
<= (available_connections as u64)
.saturating_sub(1)
.saturating_mul(policy.scale_down_low_water_mark())
}
fn occupancy_threshold(max_streams_per_tunnel: usize, percent: u32) -> u64 {
div_ceil_u64(
(max_streams_per_tunnel as u64).saturating_mul(percent as u64),
100,
)
.max(1)
}
fn div_ceil_u64(value: u64, divisor: u64) -> u64 {
if divisor == 0 {
return value;
}
value.saturating_add(divisor.saturating_sub(1)) / divisor
}
async fn await_all_handles(handles: &TaskHandles) {
@@ -469,6 +744,8 @@ mod tests {
},
)];
let (shutdown_tx, shutdown_rx) = watch::channel(false);
let tunnel_pool_policy =
TunnelPoolPolicy::from_config(&state.config, sample_tunnel_pool_sizing());
spawn_registration_recovery_tasks(
Arc::clone(&state),
@@ -476,7 +753,7 @@ mod tests {
failed,
"127.0.0.1".to_string(),
sample_hardware_info(),
1,
tunnel_pool_policy,
shutdown_rx.clone(),
Arc::clone(&tunnel_handles),
Arc::clone(&retry_handles),
@@ -508,6 +785,40 @@ mod tests {
gateway_handle.abort();
}
#[test]
fn desired_tunnel_connections_expands_when_load_crosses_high_water() {
let policy = TunnelPoolPolicy {
min_connections: 1,
max_connections: 6,
max_streams_per_tunnel: 1024,
scale_check_interval: Duration::from_secs(1),
scale_up_threshold_percent: 70,
scale_down_threshold_percent: 35,
scale_down_grace: Duration::from_secs(15),
};
assert_eq!(desired_tunnel_connections(1, &policy), 1);
assert_eq!(desired_tunnel_connections(2_000, &policy), 3);
assert_eq!(desired_tunnel_connections(5_000, &policy), 6);
}
#[test]
fn should_scale_down_requires_load_to_fit_remaining_tunnels() {
let policy = TunnelPoolPolicy {
min_connections: 1,
max_connections: 6,
max_streams_per_tunnel: 1024,
scale_check_interval: Duration::from_secs(1),
scale_up_threshold_percent: 70,
scale_down_threshold_percent: 35,
scale_down_grace: Duration::from_secs(15),
};
assert!(!should_scale_down(800, 3, &policy));
assert!(should_scale_down(600, 3, &policy));
assert!(!should_scale_down(200, 1, &policy));
}
async fn wait_for_registered_server(
server_contexts: &Arc<Mutex<Vec<Arc<ServerContext>>>>,
) -> Arc<ServerContext> {
@@ -595,6 +906,13 @@ mod tests {
}
}
fn sample_tunnel_pool_sizing() -> TunnelPoolSizing {
TunnelPoolSizing {
initial_connections: 1,
max_connections: 1,
}
}
fn sample_state(config: Config) -> Arc<ProxyAppState> {
let config = Arc::new(config);
let dns_cache = Arc::new(DnsCache::new(Duration::from_secs(60), 128));
@@ -656,13 +974,18 @@ mod tests {
log_max_files: 30,
tunnel_reconnect_base_ms: 50,
tunnel_reconnect_max_ms: 250,
tunnel_ping_interval_secs: 1,
tunnel_ping_interval_ms: 1_000,
tunnel_max_streams: Some(8),
tunnel_connect_timeout_secs: 2,
tunnel_connect_timeout_ms: 2_000,
tunnel_tcp_keepalive_secs: 30,
tunnel_tcp_nodelay: true,
tunnel_stale_timeout_secs: 5,
tunnel_connections: 1,
tunnel_stale_timeout_ms: 5_000,
tunnel_connections: Some(1),
tunnel_connections_max: Some(1),
tunnel_scale_check_interval_ms: 1_000,
tunnel_scale_up_threshold_percent: 70,
tunnel_scale_down_threshold_percent: 35,
tunnel_scale_down_grace_secs: 15,
}
}
+433 -151
View File
@@ -1,10 +1,13 @@
use std::fmt;
use std::path::Path;
use std::time::Duration;
use aether_runtime::{FileLoggingConfig, LogDestination, LogRotation, ServiceRuntimeConfig};
use clap::Parser;
use serde::{Deserialize, Serialize};
use crate::hardware::HardwareInfo;
/// Fields that existed in 0.1.x but were removed in 0.2.0.
const LEGACY_ONLY_KEYS: &[&str] = &[
"hmac_key",
@@ -16,6 +19,12 @@ const LEGACY_ONLY_KEYS: &[&str] = &[
"tls_cert",
"tls_key",
];
const REMOVED_TUNNEL_SECONDS_KEYS: &[&str] = &[
"tunnel_ping_interval_secs",
"tunnel_connect_timeout_secs",
"tunnel_stale_timeout_secs",
];
const REMOVED_SINGLE_SERVER_KEYS: &[&str] = &["aether_url", "management_token"];
/// Fields renamed from 0.1.x `delegate_*` to 0.2.0 `upstream_*`.
const DELEGATE_TO_UPSTREAM: &[(&str, &str)] = &[
@@ -38,13 +47,31 @@ const DELEGATE_TO_UPSTREAM: &[(&str, &str)] = &[
/// Default bytes buffered before a tunnel request becomes non-replayable for
/// 307/308 redirects. Kept aligned with the current admin-side request size
/// default, but exposed as an independent proxy transport budget.
pub const DEFAULT_HEARTBEAT_INTERVAL_SECS: u64 = 30;
pub const DEFAULT_HEARTBEAT_INTERVAL_SECS: u64 = 5;
#[allow(dead_code)]
pub const DEFAULT_REDIRECT_REPLAY_BUDGET_BYTES: usize = 5_242_880;
pub const DEFAULT_REDIRECT_REPLAY_BUDGET_HUMAN: &str = "5M";
pub const DEFAULT_LOG_RETENTION_DAYS: u64 = 7;
pub const DEFAULT_LOG_MAX_FILES: usize = 30;
pub const DEFAULT_TUNNEL_PING_INTERVAL_MS: u64 = 250;
pub const DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS: u64 = 800;
pub const DEFAULT_TUNNEL_STALE_TIMEOUT_MS: u64 = 900;
pub const DEFAULT_TUNNEL_SCALE_CHECK_INTERVAL_MS: u64 = 1_000;
pub const DEFAULT_TUNNEL_SCALE_UP_THRESHOLD_PERCENT: u32 = 70;
pub const DEFAULT_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT: u32 = 35;
pub const DEFAULT_TUNNEL_SCALE_DOWN_GRACE_SECS: u64 = 15;
const AUTO_TUNNEL_CONNECTIONS_BASE_CAP: u64 = 4;
const AUTO_TUNNEL_CONNECTIONS_MAX_CAP: u64 = 8;
const TUNNEL_PING_INTERVAL_MS_ENV: &str = "AETHER_PROXY_TUNNEL_PING_INTERVAL_MS";
const TUNNEL_CONNECT_TIMEOUT_MS_ENV: &str = "AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_MS";
const TUNNEL_STALE_TIMEOUT_MS_ENV: &str = "AETHER_PROXY_TUNNEL_STALE_TIMEOUT_MS";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TunnelPoolSizing {
pub initial_connections: u32,
pub max_connections: u32,
}
#[derive(Debug, Clone, PartialEq, Eq)]
enum ByteSizeValue {
Text(String),
@@ -473,7 +500,7 @@ pub struct Config {
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS",
default_value_t = 500
default_value_t = 50
)]
pub tunnel_reconnect_base_ms: u64,
@@ -485,21 +512,25 @@ pub struct Config {
)]
pub tunnel_reconnect_max_ms: u64,
/// WebSocket tunnel ping interval in seconds
#[arg(long, env = "AETHER_PROXY_TUNNEL_PING_INTERVAL", default_value_t = 15)]
pub tunnel_ping_interval_secs: u64,
/// WebSocket tunnel ping interval in milliseconds
#[arg(
long,
env = TUNNEL_PING_INTERVAL_MS_ENV,
default_value_t = DEFAULT_TUNNEL_PING_INTERVAL_MS
)]
pub tunnel_ping_interval_ms: u64,
/// Maximum concurrent streams over tunnel (auto-detected from hardware if omitted)
#[arg(long, env = "AETHER_PROXY_TUNNEL_MAX_STREAMS")]
pub tunnel_max_streams: Option<u32>,
/// WebSocket tunnel TCP connect timeout in seconds
/// WebSocket tunnel TCP connect timeout in milliseconds
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
default_value_t = 15
env = TUNNEL_CONNECT_TIMEOUT_MS_ENV,
default_value_t = DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS
)]
pub tunnel_connect_timeout_secs: u64,
pub tunnel_connect_timeout_ms: u64,
/// WebSocket tunnel TCP keepalive in seconds (0 disables)
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_KEEPALIVE", default_value_t = 30)]
@@ -509,13 +540,55 @@ pub struct Config {
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_NODELAY", default_value_t = true)]
pub tunnel_tcp_nodelay: bool,
/// Tunnel connection staleness timeout in seconds (triggers reconnect if no data received)
#[arg(long, env = "AETHER_PROXY_TUNNEL_STALE_TIMEOUT", default_value_t = 45)]
pub tunnel_stale_timeout_secs: u64,
/// Tunnel connection staleness timeout in milliseconds
#[arg(
long,
env = TUNNEL_STALE_TIMEOUT_MS_ENV,
default_value_t = DEFAULT_TUNNEL_STALE_TIMEOUT_MS
)]
pub tunnel_stale_timeout_ms: u64,
/// Number of parallel WebSocket tunnel connections per server (connection pool)
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS", default_value_t = 3)]
pub tunnel_connections: u32,
/// Minimum number of parallel WebSocket tunnel connections per server.
/// If omitted, a device-aware value is auto-detected at startup.
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS")]
pub tunnel_connections: Option<u32>,
/// Maximum number of WebSocket tunnel connections per server.
/// When larger than `tunnel_connections`, the proxy may autoscale up to this limit.
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS_MAX")]
pub tunnel_connections_max: Option<u32>,
/// Autoscale evaluation interval for the tunnel pool.
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS",
default_value_t = DEFAULT_TUNNEL_SCALE_CHECK_INTERVAL_MS
)]
pub tunnel_scale_check_interval_ms: u64,
/// Per-tunnel occupancy percentage that triggers scale-up.
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT",
default_value_t = DEFAULT_TUNNEL_SCALE_UP_THRESHOLD_PERCENT
)]
pub tunnel_scale_up_threshold_percent: u32,
/// Per-tunnel occupancy percentage that allows scale-down after the grace window.
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT",
default_value_t = DEFAULT_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT
)]
pub tunnel_scale_down_threshold_percent: u32,
/// Low-load grace window before a secondary tunnel is drained.
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS",
default_value_t = DEFAULT_TUNNEL_SCALE_DOWN_GRACE_SECS
)]
pub tunnel_scale_down_grace_secs: u64,
}
impl Config {
@@ -539,22 +612,52 @@ impl Config {
anyhow::bail!("allowed_ports: port 0 is not valid");
}
}
if self.tunnel_connect_timeout_secs == 0 {
anyhow::bail!("tunnel_connect_timeout_secs must be > 0");
let tunnel_connect_timeout = self.tunnel_connect_timeout()?;
if tunnel_connect_timeout.is_zero() {
anyhow::bail!("effective tunnel connect timeout must be > 0");
}
if self.tunnel_ping_interval_secs == 0 {
anyhow::bail!("tunnel_ping_interval_secs must be > 0");
let tunnel_ping_interval = self.tunnel_ping_interval()?;
if tunnel_ping_interval.is_zero() {
anyhow::bail!("effective tunnel ping interval must be > 0");
}
if self.tunnel_stale_timeout_secs <= self.tunnel_ping_interval_secs {
let tunnel_stale_timeout = self.tunnel_stale_timeout()?;
if tunnel_stale_timeout <= tunnel_ping_interval {
anyhow::bail!(
"tunnel_stale_timeout_secs ({}) must be > tunnel_ping_interval_secs ({})",
self.tunnel_stale_timeout_secs,
self.tunnel_ping_interval_secs
"effective tunnel stale timeout ({:?}) must be > effective tunnel ping interval ({:?})",
tunnel_stale_timeout,
tunnel_ping_interval
);
}
if self.tunnel_connections == 0 {
if matches!(self.tunnel_connections, Some(0)) {
anyhow::bail!("tunnel_connections must be > 0");
}
if matches!(self.tunnel_connections_max, Some(0)) {
anyhow::bail!("tunnel_connections_max must be > 0");
}
if let (Some(min_connections), Some(max_connections)) =
(self.tunnel_connections, self.tunnel_connections_max)
{
if max_connections < min_connections {
anyhow::bail!("tunnel_connections_max must be >= tunnel_connections");
}
}
if self.tunnel_scale_check_interval_ms == 0 {
anyhow::bail!("tunnel_scale_check_interval_ms must be > 0");
}
if self.tunnel_scale_down_grace_secs == 0 {
anyhow::bail!("tunnel_scale_down_grace_secs must be > 0");
}
if !(1..=100).contains(&self.tunnel_scale_up_threshold_percent) {
anyhow::bail!("tunnel_scale_up_threshold_percent must be within 1..=100");
}
if !(1..100).contains(&self.tunnel_scale_down_threshold_percent) {
anyhow::bail!("tunnel_scale_down_threshold_percent must be within 1..100");
}
if self.tunnel_scale_down_threshold_percent >= self.tunnel_scale_up_threshold_percent {
anyhow::bail!(
"tunnel_scale_down_threshold_percent must be < tunnel_scale_up_threshold_percent"
);
}
if self.aether_retry_max_attempts == 0 {
anyhow::bail!("aether_retry_max_attempts must be >= 1");
}
@@ -600,6 +703,54 @@ impl Config {
Ok(())
}
pub fn tunnel_ping_interval(&self) -> anyhow::Result<Duration> {
Ok(Duration::from_millis(self.tunnel_ping_interval_ms))
}
pub fn tunnel_connect_timeout(&self) -> anyhow::Result<Duration> {
Ok(Duration::from_millis(self.tunnel_connect_timeout_ms))
}
pub fn tunnel_stale_timeout(&self) -> anyhow::Result<Duration> {
Ok(Duration::from_millis(self.tunnel_stale_timeout_ms))
}
pub fn resolve_tunnel_pool_sizing(
&self,
hw_info: &HardwareInfo,
) -> anyhow::Result<TunnelPoolSizing> {
let per_tunnel_capacity = u64::from(self.tunnel_max_streams.unwrap_or(128).max(1));
let estimated = hw_info.estimated_max_concurrency.max(per_tunnel_capacity);
let cpu_cap = u64::from(hw_info.cpu_cores).clamp(1, AUTO_TUNNEL_CONNECTIONS_MAX_CAP);
let auto_initial = div_ceil_u64(estimated, per_tunnel_capacity.saturating_mul(8))
.clamp(1, AUTO_TUNNEL_CONNECTIONS_BASE_CAP)
.min(cpu_cap);
let auto_max = div_ceil_u64(estimated, per_tunnel_capacity.saturating_mul(4))
.clamp(auto_initial, AUTO_TUNNEL_CONNECTIONS_MAX_CAP)
.min(cpu_cap.max(auto_initial));
let initial_connections = u64::from(self.tunnel_connections.unwrap_or(auto_initial as u32));
let max_connections = match self.tunnel_connections_max {
Some(explicit) => u64::from(explicit),
None if self.tunnel_connections.is_some() => initial_connections,
None => auto_max,
};
if max_connections < initial_connections {
anyhow::bail!(
"effective tunnel_connections_max ({max_connections}) must be >= tunnel_connections ({initial_connections})"
);
}
Ok(TunnelPoolSizing {
initial_connections: u32::try_from(initial_connections)
.expect("effective tunnel initial connections should fit in u32"),
max_connections: u32::try_from(max_connections)
.expect("effective tunnel max connections should fit in u32"),
})
}
pub fn service_runtime_config(&self) -> anyhow::Result<ServiceRuntimeConfig> {
let mut config = ServiceRuntimeConfig::new("aether-proxy", "aether_proxy=info")
.with_log_format(aether_runtime::LogFormat::Pretty)
@@ -629,6 +780,7 @@ impl Config {
/// Per-server connection config (used in multi-server TOML `[[servers]]`).
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ServerEntry {
pub aether_url: String,
pub management_token: String,
@@ -643,11 +795,8 @@ pub struct ServerEntry {
/// Serializable config for TOML file persistence.
/// All fields are optional -- only populated values are written.
#[derive(Debug, Default, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ConfigFile {
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub management_token: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub public_ip: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -717,23 +866,31 @@ pub struct ConfigFile {
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_reconnect_max_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_ping_interval_secs: Option<u64>,
pub tunnel_ping_interval_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_max_streams: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_connect_timeout_secs: Option<u64>,
pub tunnel_connect_timeout_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_tcp_keepalive_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_tcp_nodelay: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_stale_timeout_secs: Option<u64>,
pub tunnel_stale_timeout_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_connections: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_connections_max: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_scale_check_interval_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_scale_up_threshold_percent: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_scale_down_threshold_percent: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_scale_down_grace_secs: Option<u64>,
/// Multi-server config: each entry connects to a separate Aether instance.
/// When present, top-level aether_url/management_token are ignored for
/// tunnel connections (but still injected as env for clap compatibility).
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub servers: Vec<ServerEntry>,
}
@@ -742,6 +899,7 @@ impl ConfigFile {
/// Load from a TOML file.
pub fn load(path: &Path) -> anyhow::Result<Self> {
let content = std::fs::read_to_string(path)?;
reject_removed_config_keys(&content)?;
Ok(toml::from_str(&content)?)
}
@@ -752,102 +910,6 @@ impl ConfigFile {
Ok(())
}
/// Detect and migrate a 0.1.x config file to 0.2.0 format in-place.
///
/// Returns `true` if migration was performed, `false` if already current.
/// The original file is backed up as `<name>.v1.bak` before rewriting.
pub fn migrate_legacy(path: &Path) -> anyhow::Result<bool> {
let content = match std::fs::read_to_string(path) {
Ok(c) => c,
Err(_) => return Ok(false),
};
let mut table: toml::map::Map<String, toml::Value> = toml::from_str(&content)?;
// Detect legacy format: presence of any 0.1.x-only key.
let is_legacy = LEGACY_ONLY_KEYS.iter().any(|k| table.contains_key(*k))
|| DELEGATE_TO_UPSTREAM
.iter()
.any(|(old, _)| table.contains_key(*old));
if !is_legacy {
return Ok(false);
}
// 1. Rename delegate_* -> upstream_* (carry over user-customized values)
for &(old, new) in DELEGATE_TO_UPSTREAM {
if let Some(val) = table.remove(old) {
table.entry(new.to_string()).or_insert(val);
}
}
// 2. Build [[servers]] from top-level aether_url + management_token + node_name
if !table.contains_key("servers") {
let aether_url = table.get("aether_url").and_then(|v| v.as_str());
let management_token = table.get("management_token").and_then(|v| v.as_str());
if let (Some(url), Some(token)) = (aether_url, management_token) {
let mut entry = toml::map::Map::new();
entry.insert("aether_url".into(), toml::Value::String(url.to_string()));
entry.insert(
"management_token".into(),
toml::Value::String(token.to_string()),
);
if let Some(name) = table.get("node_name").and_then(|v| v.as_str()) {
entry.insert("node_name".into(), toml::Value::String(name.to_string()));
}
table.insert(
"servers".into(),
toml::Value::Array(vec![toml::Value::Table(entry)]),
);
}
}
// 3. Remove top-level fields that are now in [[servers]] or obsolete
table.remove("aether_url");
table.remove("management_token");
table.remove("node_name");
for &key in LEGACY_ONLY_KEYS {
table.remove(key);
}
// 4. Backup original file (abort migration if backup fails)
let backup_path = path.with_extension("v1.bak");
std::fs::copy(path, &backup_path).map_err(|e| {
anyhow::anyhow!(
"failed to backup config before migration: {} -> {}: {}",
path.display(),
backup_path.display(),
e
)
})?;
// 5. Write migrated config
let new_content = toml::to_string_pretty(&table)?;
std::fs::write(path, &new_content)?;
eprintln!(" Config migrated from 0.1.x to 0.2.0 format.");
eprintln!(" Backup saved: {}", backup_path.display());
Ok(true)
}
/// Resolve the effective server list.
///
/// If `[[servers]]` is present, use it. Otherwise fall back to the
/// top-level `aether_url` + `management_token` as a single server.
pub fn effective_servers(&self) -> Vec<ServerEntry> {
if !self.servers.is_empty() {
return self.servers.clone();
}
match (&self.aether_url, &self.management_token) {
(Some(url), Some(token)) => vec![ServerEntry {
aether_url: url.clone(),
management_token: token.clone(),
node_name: None,
}],
_ => vec![],
}
}
/// Inject values as environment variables so clap picks them up.
///
/// Only sets variables that are **not** already present in the
@@ -874,18 +936,9 @@ impl ConfigFile {
};
}
// When top-level fields are absent, fall back to the first [[servers]]
// entry so that clap's required `aether_url` / `management_token` are
// satisfied even with the new config format.
let first_server = self.servers.first();
let aether_url = self
.aether_url
.as_deref()
.or(first_server.map(|s| s.aether_url.as_str()));
let management_token = self
.management_token
.as_deref()
.or(first_server.map(|s| s.management_token.as_str()));
let aether_url = first_server.map(|s| s.aether_url.as_str());
let management_token = first_server.map(|s| s.management_token.as_str());
let node_name = self
.node_name
.as_deref()
@@ -988,25 +1041,39 @@ impl ConfigFile {
"AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS",
self.tunnel_reconnect_max_ms
);
set!(
"AETHER_PROXY_TUNNEL_PING_INTERVAL",
self.tunnel_ping_interval_secs
);
set!(TUNNEL_PING_INTERVAL_MS_ENV, self.tunnel_ping_interval_ms);
set!("AETHER_PROXY_TUNNEL_MAX_STREAMS", self.tunnel_max_streams);
set!(
"AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
self.tunnel_connect_timeout_secs
TUNNEL_CONNECT_TIMEOUT_MS_ENV,
self.tunnel_connect_timeout_ms
);
set!(
"AETHER_PROXY_TUNNEL_TCP_KEEPALIVE",
self.tunnel_tcp_keepalive_secs
);
set!("AETHER_PROXY_TUNNEL_TCP_NODELAY", self.tunnel_tcp_nodelay);
set!(
"AETHER_PROXY_TUNNEL_STALE_TIMEOUT",
self.tunnel_stale_timeout_secs
);
set!(TUNNEL_STALE_TIMEOUT_MS_ENV, self.tunnel_stale_timeout_ms);
set!("AETHER_PROXY_TUNNEL_CONNECTIONS", self.tunnel_connections);
set!(
"AETHER_PROXY_TUNNEL_CONNECTIONS_MAX",
self.tunnel_connections_max
);
set!(
"AETHER_PROXY_TUNNEL_SCALE_CHECK_INTERVAL_MS",
self.tunnel_scale_check_interval_ms
);
set!(
"AETHER_PROXY_TUNNEL_SCALE_UP_THRESHOLD_PERCENT",
self.tunnel_scale_up_threshold_percent
);
set!(
"AETHER_PROXY_TUNNEL_SCALE_DOWN_THRESHOLD_PERCENT",
self.tunnel_scale_down_threshold_percent
);
set!(
"AETHER_PROXY_TUNNEL_SCALE_DOWN_GRACE_SECS",
self.tunnel_scale_down_grace_secs
);
// allowed_ports needs special handling (comma-separated)
if let Some(ref ports) = self.allowed_ports {
@@ -1022,11 +1089,70 @@ impl ConfigFile {
}
}
fn reject_removed_config_keys(content: &str) -> anyhow::Result<()> {
let value: toml::Value = toml::from_str(content)?;
let Some(table) = value.as_table() else {
return Ok(());
};
let removed_seconds = REMOVED_TUNNEL_SECONDS_KEYS
.iter()
.copied()
.filter(|key| table.contains_key(*key))
.collect::<Vec<_>>();
if !removed_seconds.is_empty() {
anyhow::bail!(
"removed tunnel config keys detected: {}. Use *_ms variants instead",
removed_seconds.join(", ")
);
}
let removed_single_server = REMOVED_SINGLE_SERVER_KEYS
.iter()
.copied()
.filter(|key| table.contains_key(*key))
.collect::<Vec<_>>();
if !removed_single_server.is_empty() {
anyhow::bail!(
"single-server top-level config keys are no longer supported: {}. Use [[servers]] entries instead",
removed_single_server.join(", ")
);
}
let removed_legacy = LEGACY_ONLY_KEYS
.iter()
.copied()
.filter(|key| table.contains_key(*key))
.chain(
DELEGATE_TO_UPSTREAM
.iter()
.map(|(old, _)| *old)
.filter(|key| table.contains_key(*key)),
)
.collect::<Vec<_>>();
if !removed_legacy.is_empty() {
anyhow::bail!(
"legacy config keys are no longer supported: {}",
removed_legacy.join(", ")
);
}
Ok(())
}
fn div_ceil_u64(value: u64, divisor: u64) -> u64 {
if divisor == 0 {
return value;
}
value.saturating_add(divisor.saturating_sub(1)) / divisor
}
#[cfg(test)]
mod tests {
use clap::CommandFactory;
use clap::{CommandFactory, Parser};
use super::*;
use crate::hardware::HardwareInfo;
#[test]
fn parse_byte_size_supports_human_units() {
@@ -1056,6 +1182,36 @@ mod tests {
assert_eq!(stringy.redirect_replay_budget_bytes.as_deref(), Some("6M"));
}
#[test]
fn config_file_rejects_removed_tunnel_seconds_keys() {
let error = reject_removed_config_keys("tunnel_ping_interval_secs = 5")
.expect_err("removed tunnel seconds keys should be rejected");
assert!(
error.to_string().contains("tunnel_ping_interval_secs"),
"error should mention removed key"
);
}
#[test]
fn config_file_rejects_top_level_single_server_keys() {
let error = reject_removed_config_keys("aether_url = \"https://example.com\"")
.expect_err("top-level single-server key should be rejected");
assert!(
error.to_string().contains("aether_url"),
"error should mention removed single-server key"
);
}
#[test]
fn config_file_rejects_legacy_keys() {
let error = reject_removed_config_keys("delegate_connect_timeout_secs = 10")
.expect_err("legacy delegate key should be rejected");
assert!(
error.to_string().contains("delegate_connect_timeout_secs"),
"error should mention removed legacy key"
);
}
#[test]
fn config_requires_node_name() {
let command = Config::command();
@@ -1067,4 +1223,130 @@ mod tests {
assert!(node_name.is_required_set());
assert!(node_name.get_default_values().is_empty());
}
#[test]
fn tunnel_fast_recovery_defaults_use_millisecond_values() {
let config = Config::parse_from([
"aether-proxy",
"--aether-url",
"https://example.com",
"--management-token",
"ae_test",
"--node-name",
"proxy-test",
]);
assert_eq!(
config
.tunnel_ping_interval()
.expect("ping interval should resolve"),
Duration::from_millis(DEFAULT_TUNNEL_PING_INTERVAL_MS)
);
assert_eq!(
config
.tunnel_connect_timeout()
.expect("connect timeout should resolve"),
Duration::from_millis(DEFAULT_TUNNEL_CONNECT_TIMEOUT_MS)
);
assert_eq!(
config
.tunnel_stale_timeout()
.expect("stale timeout should resolve"),
Duration::from_millis(DEFAULT_TUNNEL_STALE_TIMEOUT_MS)
);
}
#[test]
fn tunnel_millisecond_flags_take_effect_when_explicitly_set() {
let config = Config::parse_from([
"aether-proxy",
"--aether-url",
"https://example.com",
"--management-token",
"ae_test",
"--node-name",
"proxy-test",
"--tunnel-ping-interval-ms",
"100",
"--tunnel-connect-timeout-ms",
"200",
"--tunnel-stale-timeout-ms",
"300",
]);
assert_eq!(
config
.tunnel_ping_interval()
.expect("ping interval should resolve"),
Duration::from_millis(100)
);
assert_eq!(
config
.tunnel_connect_timeout()
.expect("connect timeout should resolve"),
Duration::from_millis(200)
);
assert_eq!(
config
.tunnel_stale_timeout()
.expect("stale timeout should resolve"),
Duration::from_millis(300)
);
}
#[test]
fn auto_tunnel_pool_sizing_uses_hardware_capacity() {
let config = Config::parse_from([
"aether-proxy",
"--aether-url",
"https://example.com",
"--management-token",
"ae_test",
"--node-name",
"proxy-test",
"--tunnel-max-streams",
"1024",
]);
let hw = HardwareInfo {
cpu_cores: 12,
total_memory_mb: 20_480,
os_info: "test".to_string(),
fd_limit: 1_048_576,
estimated_max_concurrency: 24_000,
};
let sizing = config
.resolve_tunnel_pool_sizing(&hw)
.expect("sizing should resolve");
assert_eq!(sizing.initial_connections, 3);
assert_eq!(sizing.max_connections, 6);
}
#[test]
fn explicit_tunnel_connections_keep_fixed_pool_without_max_override() {
let config = Config::parse_from([
"aether-proxy",
"--aether-url",
"https://example.com",
"--management-token",
"ae_test",
"--node-name",
"proxy-test",
"--tunnel-max-streams",
"512",
"--tunnel-connections",
"2",
]);
let hw = HardwareInfo {
cpu_cores: 12,
total_memory_mb: 20_480,
os_info: "test".to_string(),
fd_limit: 1_048_576,
estimated_max_concurrency: 24_000,
};
let sizing = config
.resolve_tunnel_pool_sizing(&hw)
.expect("sizing should resolve");
assert_eq!(sizing.initial_connections, 2);
assert_eq!(sizing.max_connections, 2);
}
}
+19 -18
View File
@@ -61,12 +61,15 @@ async fn main() -> anyhow::Result<()> {
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
let config_path = std::path::Path::new(&config_file_path);
if config_path.exists() {
// Migrate legacy 0.1.x config to 0.2.0 format if needed
if let Err(e) = config::ConfigFile::migrate_legacy(config_path) {
eprintln!(" WARNING: config migration failed: {}", e);
}
if let Ok(file_cfg) = config::ConfigFile::load(config_path) {
file_cfg.inject_env();
match config::ConfigFile::load(config_path) {
Ok(file_cfg) => file_cfg.inject_env(),
Err(error) => {
eprintln!(
" WARNING: failed to load config {}: {}",
config_path.display(),
error
);
}
}
}
@@ -148,21 +151,19 @@ async fn run_proxy(config: Config) -> anyhow::Result<()> {
std::process::exit(1);
}
// Resolve server list: prefer [[servers]] from TOML, fall back to CLI/env single server.
// Resolve server list: if a config file exists, it must use [[servers]].
// Otherwise fall back to CLI/env single-server mode.
let config_path =
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
let servers = if std::path::Path::new(&config_path).exists() {
config::ConfigFile::load(std::path::Path::new(&config_path))
.ok()
.map(|f| f.effective_servers())
.filter(|s| !s.is_empty())
.unwrap_or_else(|| {
vec![config::ServerEntry {
aether_url: config.aether_url.clone(),
management_token: config.management_token.clone(),
node_name: None,
}]
})
let file_cfg = config::ConfigFile::load(std::path::Path::new(&config_path))?;
if file_cfg.servers.is_empty() {
anyhow::bail!(
"config file {} must contain at least one [[servers]] entry",
config_path
);
}
file_cfg.servers.clone()
} else {
vec![config::ServerEntry {
aether_url: config.aether_url.clone(),
+4 -15
View File
@@ -172,7 +172,7 @@ impl App {
value: DEFAULT_HEARTBEAT_INTERVAL_SECS.to_string(),
kind: FieldKind::Text,
required: false,
help: "Heartbeat interval in seconds; default is 30",
help: "Heartbeat interval in seconds; default is 5",
},
Field {
label: "Redirect Replay Budget",
@@ -264,22 +264,11 @@ impl App {
}
// Server tabs
let servers = cfg.effective_servers();
let servers = cfg.servers.clone();
if servers.is_empty() {
let mut tab = ServerTab::new();
// Single-server fallback: use top-level node_name
if let Some(ref name) = cfg.node_name {
tab.fields[2].value = name.clone();
}
self.server_tabs = vec![tab];
self.server_tabs = vec![ServerTab::new()];
} else {
self.server_tabs = servers.iter().map(ServerTab::from_entry).collect();
// For single-server mode, node_name might be in top-level only
if self.server_tabs.len() == 1 && self.server_tabs[0].fields[2].value.is_empty() {
if let Some(ref name) = cfg.node_name {
self.server_tabs[0].fields[2].value = name.clone();
}
}
}
self.active_tab = 0;
self.selected = 0;
@@ -387,7 +376,7 @@ impl App {
..ConfigFile::default()
};
// Always write [[servers]] format; old top-level fields are read-only compat
// Always write [[servers]] format.
cfg.servers = self
.server_tabs
.iter()
+64 -12
View File
@@ -31,6 +31,7 @@ pub async fn connect_and_run(
server: &Arc<ServerContext>,
conn_idx: usize,
shutdown: &mut watch::Receiver<bool>,
drain: watch::Receiver<bool>,
) -> Result<TunnelOutcome, anyhow::Error> {
let ws_url = build_tunnel_url(server);
info!(url = %ws_url, conn = conn_idx, "connecting tunnel");
@@ -53,8 +54,7 @@ pub async fn connect_and_run(
http::HeaderValue::from_str(&dynamic_node_name)?,
);
// Advertise per-connection max concurrent streams so the backend can
// respect the proxy's capacity limit (backward-compatible: old backends
// ignore this header).
// respect the proxy's capacity limit.
let max_streams = state.config.tunnel_max_streams.unwrap_or(128);
headers.insert("X-Tunnel-Max-Streams", http::HeaderValue::from(max_streams));
@@ -67,13 +67,16 @@ pub async fn connect_and_run(
let port = uri.port_u16().unwrap_or(if is_tls { 443 } else { 80 });
// TCP connect with timeout
let connect_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
let connect_timeout = state
.config
.tunnel_connect_timeout()
.expect("validated config should resolve tunnel connect timeout");
let tcp_stream = tokio::time::timeout(connect_timeout, TcpStream::connect((host, port)))
.await
.map_err(|_| {
anyhow::anyhow!(
"tunnel TCP connect timeout ({}s)",
connect_timeout.as_secs()
"tunnel TCP connect timeout ({}ms)",
connect_timeout.as_millis()
)
})??;
@@ -96,7 +99,7 @@ pub async fn connect_and_run(
max_message_size: Some(64 << 20),
..Default::default()
};
let handshake_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
let handshake_timeout = connect_timeout;
let (ws_stream, _response) = tokio::time::timeout(
handshake_timeout,
tokio_tungstenite::client_async_tls_with_config(
@@ -109,16 +112,25 @@ pub async fn connect_and_run(
.await
.map_err(|_| {
anyhow::anyhow!(
"tunnel WebSocket handshake timeout ({}s)",
handshake_timeout.as_secs()
"tunnel WebSocket handshake timeout ({}ms)",
handshake_timeout.as_millis()
)
})??;
let stale_timeout = state
.config
.tunnel_stale_timeout()
.expect("validated config should resolve tunnel stale timeout");
let ping_interval = state
.config
.tunnel_ping_interval()
.expect("validated config should resolve tunnel ping interval");
info!(
conn = conn_idx,
tcp_keepalive_secs = state.config.tunnel_tcp_keepalive_secs,
tcp_nodelay = state.config.tunnel_tcp_nodelay,
connect_timeout_secs = state.config.tunnel_connect_timeout_secs,
stale_timeout_secs = state.config.tunnel_stale_timeout_secs,
connect_timeout_ms = connect_timeout.as_millis(),
stale_timeout_ms = stale_timeout.as_millis(),
ping_interval_ms = ping_interval.as_millis(),
"tunnel connected"
);
@@ -129,8 +141,8 @@ pub async fn connect_and_run(
let (ws_sink, ws_read) = futures_util::StreamExt::split(ws_stream);
// Spawn writer task (with WebSocket ping keepalive)
let ping_interval = Duration::from_secs(state.config.tunnel_ping_interval_secs);
let (frame_tx, mut writer_handle) = writer::spawn_writer(ws_sink, ping_interval);
let drain_signal = spawn_drain_signal(conn_idx, frame_tx.clone(), drain.clone());
// Spawn heartbeat task (only for primary connection to avoid
// resetting shared atomic metrics via swap(0))
@@ -153,7 +165,14 @@ pub async fn connect_and_run(
let state_clone = Arc::clone(state);
let server_clone = Arc::clone(server);
let outcome = tokio::select! {
result = dispatcher::run(state_clone, server_clone, ws_read, frame_tx.clone(), hb_handle) => {
result = dispatcher::run(
state_clone,
server_clone,
ws_read,
frame_tx.clone(),
hb_handle,
drain.clone(),
) => {
match result {
Ok(()) => TunnelOutcome::Disconnected,
Err(e) => return Err(e),
@@ -181,6 +200,10 @@ pub async fn connect_and_run(
// Drop our sender; the writer will exit once all stream handler clones
// are also dropped (i.e. after they finish their in-flight work).
drop(frame_tx);
if !drain_signal.is_finished() {
drain_signal.abort();
let _ = drain_signal.await;
}
// Wait for the writer task to finish with a generous timeout — the
// dispatcher already waits up to 30s for stream handlers, so 35s here
@@ -194,6 +217,35 @@ pub async fn connect_and_run(
Ok(outcome)
}
fn spawn_drain_signal(
conn_idx: usize,
frame_tx: writer::FrameSender,
mut drain: watch::Receiver<bool>,
) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
if !*drain.borrow() {
loop {
if drain.changed().await.is_err() {
return;
}
if *drain.borrow() {
break;
}
}
}
debug!(conn = conn_idx, "sending GOAWAY for tunnel drain");
let _ = tokio::time::timeout(
Duration::from_millis(250),
frame_tx.send(super::protocol::Frame::control(
super::protocol::MsgType::GoAway,
bytes::Bytes::new(),
)),
)
.await;
})
}
/// Configure TCP keepalive and NODELAY on an established socket.
fn configure_tcp_socket(stream: &TcpStream, state: &Arc<AppState>) {
let sock_ref = socket2::SockRef::from(stream);
+49 -2
View File
@@ -7,6 +7,7 @@ use std::time::Duration;
use bytes::Bytes;
use futures_util::StreamExt;
use tokio::sync::mpsc;
use tokio::sync::watch;
use tokio::task::JoinHandle;
use tokio_tungstenite::tungstenite::Message;
use tracing::{debug, error, info, warn};
@@ -25,6 +26,7 @@ pub async fn run<S>(
mut ws_stream: S,
frame_tx: FrameSender,
heartbeat: HeartbeatHandle,
mut drain: watch::Receiver<bool>,
) -> Result<(), anyhow::Error>
where
S: StreamExt<Item = Result<Message, tokio_tungstenite::tungstenite::Error>>
@@ -38,12 +40,21 @@ where
let mut handler_handles: Vec<JoinHandle<()>> = Vec::new();
let max_streams = state.config.tunnel_max_streams.unwrap_or(128) as usize;
let mut frames_since_cleanup: u32 = 0;
let stale_timeout = Duration::from_secs(state.config.tunnel_stale_timeout_secs);
let stale_timeout = state
.config
.tunnel_stale_timeout()
.expect("validated config should resolve tunnel stale timeout");
// Track last time we received any data to detect stale connections
let mut last_data_at = tokio::time::Instant::now();
let mut draining = *drain.borrow();
let read_err = loop {
if draining && streams.is_empty() {
info!("tunnel drained after in-flight streams completed");
break None;
}
let msg_result = tokio::select! {
msg = ws_stream.next() => {
match msg {
@@ -51,9 +62,19 @@ where
None => break None,
}
}
changed = drain.changed() => {
if changed.is_err() {
continue;
}
if *drain.borrow() {
info!("tunnel drain requested, waiting for in-flight streams");
draining = true;
}
continue;
}
_ = tokio::time::sleep_until(last_data_at + stale_timeout) => {
warn!(
stale_secs = stale_timeout.as_secs(),
stale_ms = stale_timeout.as_millis(),
"tunnel connection stale, no data received"
);
break None;
@@ -92,6 +113,24 @@ where
match frame.msg_type {
MsgType::RequestHeaders => {
if draining {
if frame_tx
.try_send(Frame::new(
frame.stream_id,
MsgType::StreamError,
0,
Bytes::from("tunnel draining"),
))
.is_err()
{
warn!(
stream_id = frame.stream_id,
"writer channel full, StreamError dropped during drain"
);
}
continue;
}
// Decompress if the frame is gzip-compressed, then parse metadata
let payload = match decompress_if_gzip(&frame) {
Ok(p) => p,
@@ -176,6 +215,10 @@ where
let _ = tx.send(frame).await;
if is_end {
streams.remove(&sid);
if draining && streams.is_empty() {
info!("tunnel drained after request body completion");
break None;
}
}
}
}
@@ -184,6 +227,10 @@ where
// Client-side cancellation or end
if let Some(tx) = streams.remove(&frame.stream_id) {
let _ = tx.send(frame).await;
if draining && streams.is_empty() {
info!("tunnel drained after stream termination");
break None;
}
}
}
+78 -17
View File
@@ -24,7 +24,7 @@ static NON_ROOT_UPGRADE_WARNED: AtomicBool = AtomicBool::new(false);
enum AckDecision {
Accept {
heartbeat_id: Option<u64>,
heartbeat_id: u64,
upgrade_to: Option<String>,
},
Ignore,
@@ -143,16 +143,8 @@ pub fn spawn(
upgrade_to,
} => {
if let Some((pending_id, _)) = pending {
match ack_id {
Some(id) if id == pending_id => {
pending = None;
}
None => {
// Backward-compatible with servers that don't echo
// heartbeat_id in ACK payload yet.
pending = None;
}
_ => {}
if ack_id == pending_id {
pending = None;
}
}
maybe_trigger_upgrade(upgrade_to);
@@ -266,6 +258,7 @@ async fn build_heartbeat_payload(
"node_id": node_id,
"heartbeat_session_id": heartbeat_session_id,
"heartbeat_id": heartbeat_id,
"heartbeat_interval": server.dynamic.load().heartbeat_interval,
"active_connections": server.active_connections.load(Ordering::Acquire),
"total_requests": snapshot.requests,
"avg_latency_ms": avg_latency_ms,
@@ -283,10 +276,8 @@ async fn build_heartbeat_payload(
fn handle_ack(server: &ServerContext, payload: &[u8]) -> AckDecision {
if payload.is_empty() {
return AckDecision::Accept {
heartbeat_id: None,
upgrade_to: None,
};
warn!("received empty heartbeat ACK");
return AckDecision::Ignore;
}
#[derive(serde::Deserialize)]
@@ -295,8 +286,7 @@ fn handle_ack(server: &ServerContext, payload: &[u8]) -> AckDecision {
remote_config: Option<RemoteConfig>,
#[serde(default)]
config_version: u64,
#[serde(default)]
heartbeat_id: Option<u64>,
heartbeat_id: u64,
#[serde(default)]
upgrade_to: Option<String>,
}
@@ -371,3 +361,74 @@ fn maybe_trigger_upgrade(version: Option<String>) {
}
});
}
#[cfg(test)]
mod tests {
use std::sync::atomic::AtomicU64;
use std::sync::{Arc, RwLock};
use arc_swap::ArcSwap;
use clap::Parser;
use super::{handle_ack, AckDecision};
use crate::registration::client::AetherClient;
use crate::runtime::DynamicConfig;
use crate::state::{ProxyMetrics, ServerContext};
fn sample_server() -> Arc<ServerContext> {
let config = Arc::new(crate::config::Config::parse_from([
"aether-proxy",
"--aether-url",
"https://example.com",
"--management-token",
"ae_test",
"--node-name",
"proxy-test",
]));
Arc::new(ServerContext {
server_label: "heartbeat-test".to_string(),
aether_url: config.aether_url.clone(),
management_token: config.management_token.clone(),
node_name: config.node_name.clone(),
node_id: Arc::new(RwLock::new("node-123".to_string())),
aether_client: Arc::new(AetherClient::new(
&config,
&config.aether_url,
&config.management_token,
)),
dynamic: Arc::new(ArcSwap::from_pointee(DynamicConfig::from_config(&config))),
active_connections: Arc::new(AtomicU64::new(0)),
metrics: Arc::new(ProxyMetrics::new()),
})
}
#[test]
fn heartbeat_ack_requires_heartbeat_id() {
let server = sample_server();
let decision = handle_ack(
&server,
br#"{"config_version":1,"remote_config":{"heartbeat_interval":9}}"#,
);
assert!(matches!(decision, AckDecision::Ignore));
assert_eq!(server.dynamic.load().heartbeat_interval, 5);
}
#[test]
fn heartbeat_ack_applies_remote_config_with_heartbeat_id() {
let server = sample_server();
let decision = handle_ack(
&server,
br#"{"heartbeat_id":7,"config_version":1,"remote_config":{"heartbeat_interval":9}}"#,
);
assert!(matches!(
decision,
AckDecision::Accept {
heartbeat_id: 7,
upgrade_to: None
}
));
assert_eq!(server.dynamic.load().heartbeat_interval, 9);
}
}
+38 -6
View File
@@ -36,10 +36,16 @@ pub async fn run(
server: &Arc<ServerContext>,
conn_idx: usize,
mut shutdown: watch::Receiver<bool>,
mut drain: watch::Receiver<bool>,
) {
info!(server = %server.server_label, conn = conn_idx, "starting tunnel");
let reconnect_salt = compute_connection_salt(server, conn_idx);
if *drain.borrow() {
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested before startup");
return;
}
let startup_delay = compute_startup_stagger(conn_idx, reconnect_salt);
if !startup_delay.is_zero() {
info!(
@@ -54,14 +60,24 @@ pub async fn run(
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during startup stagger");
return;
}
_ = drain.changed() => {
if *drain.borrow() {
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested during startup stagger");
return;
}
}
}
}
let mut consecutive_failures: u32 = 0;
loop {
if *drain.borrow() {
info!(server = %server.server_label, conn = conn_idx, "tunnel drained, exiting slot");
return;
}
let started_at = Instant::now();
match client::connect_and_run(state, server, conn_idx, &mut shutdown).await {
match client::connect_and_run(state, server, conn_idx, &mut shutdown, drain.clone()).await {
Ok(client::TunnelOutcome::Shutdown) => {
info!(server = %server.server_label, conn = conn_idx, "tunnel shut down gracefully");
return;
@@ -78,6 +94,10 @@ pub async fn run(
info!(server = %server.server_label, conn = conn_idx, "shutdown requested, not reconnecting");
return;
}
if *drain.borrow() {
info!(server = %server.server_label, conn = conn_idx, "tunnel drained after disconnect");
return;
}
// Reset backoff after a stable session to keep recovery snappy when
// failures are only occasional.
@@ -108,6 +128,12 @@ pub async fn run(
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during reconnect wait");
return;
}
_ = drain.changed() => {
if *drain.borrow() {
info!(server = %server.server_label, conn = conn_idx, "tunnel drain requested during reconnect wait");
return;
}
}
}
}
}
@@ -266,8 +292,9 @@ mod tests {
let proxy_task = tokio::spawn({
let state = Arc::clone(&state);
let server = Arc::clone(&server);
let (_drain_tx, drain_rx) = watch::channel(false);
async move {
run(&state, &server, 0, shutdown_rx).await;
run(&state, &server, 0, shutdown_rx, drain_rx).await;
}
});
@@ -486,13 +513,18 @@ mod tests {
log_max_files: 30,
tunnel_reconnect_base_ms: 50,
tunnel_reconnect_max_ms: 250,
tunnel_ping_interval_secs: 1,
tunnel_ping_interval_ms: 1_000,
tunnel_max_streams: Some(8),
tunnel_connect_timeout_secs: 2,
tunnel_connect_timeout_ms: 2_000,
tunnel_tcp_keepalive_secs: 30,
tunnel_tcp_nodelay: true,
tunnel_stale_timeout_secs: 5,
tunnel_connections: 1,
tunnel_stale_timeout_ms: 5_000,
tunnel_connections: Some(1),
tunnel_connections_max: Some(1),
tunnel_scale_check_interval_ms: 1_000,
tunnel_scale_up_threshold_percent: 70,
tunnel_scale_down_threshold_percent: 35,
tunnel_scale_down_grace_secs: 15,
}
}
@@ -1594,13 +1594,18 @@ mod tests {
log_max_files: 30,
tunnel_reconnect_base_ms: 500,
tunnel_reconnect_max_ms: 30_000,
tunnel_ping_interval_secs: 15,
tunnel_ping_interval_ms: 15_000,
tunnel_max_streams: Some(8),
tunnel_connect_timeout_secs: 15,
tunnel_connect_timeout_ms: 15_000,
tunnel_tcp_keepalive_secs: 30,
tunnel_tcp_nodelay: true,
tunnel_stale_timeout_secs: 45,
tunnel_connections: 1,
tunnel_stale_timeout_ms: 45_000,
tunnel_connections: Some(1),
tunnel_connections_max: Some(1),
tunnel_scale_check_interval_ms: 1_000,
tunnel_scale_up_threshold_percent: 70,
tunnel_scale_down_threshold_percent: 35,
tunnel_scale_down_grace_secs: 15,
}
}