## Root Cause: Envoy Filter Chain Ordering The "missing trailer" error is caused by **Envoy's HTTP filter chain ordering** in the `grpc_web_listener` configuration. ### The Problem In `lava-k8s/helm/envoygateway/templates/envoy-configmap.yaml:70-152`, the filter order is: ``` 1. header_to_metadata 2. lua 3. cors 4. JwtAuthentication ← rejects here with 401 5. grpc_web ← never processes the response 6. router ``` When the JWT filter (#4) rejects a request (expired token), it generates a **local reply** (HTTP 401). In Envoy, local replies only flow back through filters that already processed the request (filters 1-3). **The `grpc_web` filter (#5) never sees this response**, so the 401 is sent as a plain HTTP response — not encoded in gRPC-web format. The gRPC-web protocol requires a **trailer frame** (flag byte `0x80`) in the response body. A plain HTTP 401 has no such frame, so `@connectrpc/connect-web` throws `"missing trailer"` with code `Unknown` (because `grpc-status` is also absent from the headers). ### Evidence from Datadog The log sequence confirms this: | Timestamp | Message | |---|---| | 15:34:05.300 | **Token refresh failed, rejecting queued requests** | | 15:34:05.301 | missing trailer | | 15:34:05.302 | missing trailer | | 15:34:06.449 | **Token refresh failed, rejecting queued requests** | | 15:34:06.451 | missing trailer | | 15:34:06.452 | missing trailer | | ... | (repeats ~8 times in 9 seconds) | Pattern: JWT expires → API calls get rejected by Envoy's JWT filter → plain 401 bypasses gRPC-web encoding → "missing trailer" → app tries token refresh → refresh fails → queued requests rejected → retry loop. The specific log entry shows: - **URL**: `grpcweb.mainnet.lava.xyz/credit/GetCreditSummary` - **Code**: `Unknown` (no `grpc-status` header = response was not gRPC at all) - **isAuthError**: `false` (the client can't detect it's an auth error because the gRPC error code is lost) - **Component**: `ApiClient-Reactive` ### Why It Only Happens on Web **Native** (iOS/Android): Uses native gRPC over HTTP/2. The app connects through a different transport that handles HTTP/2 trailers natively. Even if Envoy returns a 401, the native gRPC client can interpret the HTTP status code directly. **Web**: Uses gRPC-web over HTTP/1.1. Trailers must be encoded as a special frame in the response body by Envoy's `grpc_web` filter. When this filter is bypassed (JWT rejection), the browser has no way to parse the response. ### The Fix **Move the `grpc_web` filter BEFORE `JwtAuthentication`** in the filter chain at `envoy-configmap.yaml:70-152`: ```yaml http_filters: - name: envoy.filters.http.header_to_metadata # ... - name: envoy.filters.http.lua # ... - name: envoy.filters.http.cors # ... - name: envoy.filters.http.grpc_web # ← moved BEFORE JWT typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb - name: envoy.filters.http.JwtAuthentication # ... - name: envoy.filters.http.router # ... ``` With this order, when the JWT filter rejects a request, the response flows back through `grpc_web → cors → lua → header_to_metadata`. The `grpc_web` filter will now properly encode the 401 rejection as a gRPC-web response with the trailer frame, and the browser client will receive a proper `UNAUTHENTICATED` gRPC error instead of "missing trailer". ### Secondary Benefit This fix will also allow the web client to properly detect auth errors (`isAuthError` will become `true`), enabling the token refresh flow to work correctly rather than entering a retry loop. ### Other Scenarios This Fixes Any Envoy local reply generated by the JWT filter will be affected. This includes: - Expired JWT tokens - Invalid JWT signatures - Missing JWT on protected endpoints - JWKS fetch failures (if `jwks-server` is down) ### `ExceptionUtils.kt` Note The `ExceptionUtils.kt` correctly adds trailers to gRPC error responses. The issue is NOT with the backend services — they properly send trailers. The problem is that **Envoy's infrastructure-level rejection** (JWT filter) generates a response that bypasses the gRPC-web encoding entirely, before the request ever reaches the backend.