## Root Cause: Envoy Filter Chain Ordering
The "missing trailer" error is caused by **Envoy's HTTP filter chain ordering** in the `grpc_web_listener` configuration.
### The Problem
In `lava-k8s/helm/envoygateway/templates/envoy-configmap.yaml:70-152`, the filter order is:
```
1. header_to_metadata
2. lua
3. cors
4. JwtAuthentication ← rejects here with 401
5. grpc_web ← never processes the response
6. router
```
When the JWT filter (#4) rejects a request (expired token), it generates a **local reply** (HTTP 401). In Envoy, local replies only flow back through filters that already processed the request (filters 1-3). **The `grpc_web` filter (#5) never sees this response**, so the 401 is sent as a plain HTTP response — not encoded in gRPC-web format.
The gRPC-web protocol requires a **trailer frame** (flag byte `0x80`) in the response body. A plain HTTP 401 has no such frame, so `@connectrpc/connect-web` throws `"missing trailer"` with code `Unknown` (because `grpc-status` is also absent from the headers).
### Evidence from Datadog
The log sequence confirms this:
| Timestamp | Message |
|---|---|
| 15:34:05.300 | **Token refresh failed, rejecting queued requests** |
| 15:34:05.301 | missing trailer |
| 15:34:05.302 | missing trailer |
| 15:34:06.449 | **Token refresh failed, rejecting queued requests** |
| 15:34:06.451 | missing trailer |
| 15:34:06.452 | missing trailer |
| ... | (repeats ~8 times in 9 seconds) |
Pattern: JWT expires → API calls get rejected by Envoy's JWT filter → plain 401 bypasses gRPC-web encoding → "missing trailer" → app tries token refresh → refresh fails → queued requests rejected → retry loop.
The specific log entry shows:
- **URL**: `grpcweb.mainnet.lava.xyz/credit/GetCreditSummary`
- **Code**: `Unknown` (no `grpc-status` header = response was not gRPC at all)
- **isAuthError**: `false` (the client can't detect it's an auth error because the gRPC error code is lost)
- **Component**: `ApiClient-Reactive`
### Why It Only Happens on Web
**Native** (iOS/Android): Uses native gRPC over HTTP/2. The app connects through a different transport that handles HTTP/2 trailers natively. Even if Envoy returns a 401, the native gRPC client can interpret the HTTP status code directly.
**Web**: Uses gRPC-web over HTTP/1.1. Trailers must be encoded as a special frame in the response body by Envoy's `grpc_web` filter. When this filter is bypassed (JWT rejection), the browser has no way to parse the response.
### The Fix
**Move the `grpc_web` filter BEFORE `JwtAuthentication`** in the filter chain at `envoy-configmap.yaml:70-152`:
```yaml
http_filters:
- name: envoy.filters.http.header_to_metadata
# ...
- name: envoy.filters.http.lua
# ...
- name: envoy.filters.http.cors
# ...
- name: envoy.filters.http.grpc_web # ← moved BEFORE JWT
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
- name: envoy.filters.http.JwtAuthentication
# ...
- name: envoy.filters.http.router
# ...
```
With this order, when the JWT filter rejects a request, the response flows back through `grpc_web → cors → lua → header_to_metadata`. The `grpc_web` filter will now properly encode the 401 rejection as a gRPC-web response with the trailer frame, and the browser client will receive a proper `UNAUTHENTICATED` gRPC error instead of "missing trailer".
### Secondary Benefit
This fix will also allow the web client to properly detect auth errors (`isAuthError` will become `true`), enabling the token refresh flow to work correctly rather than entering a retry loop.
### Other Scenarios This Fixes
Any Envoy local reply generated by the JWT filter will be affected. This includes:
- Expired JWT tokens
- Invalid JWT signatures
- Missing JWT on protected endpoints
- JWKS fetch failures (if `jwks-server` is down)
### `ExceptionUtils.kt` Note
The `ExceptionUtils.kt` correctly adds trailers to gRPC error responses. The issue is NOT with the backend services — they properly send trailers. The problem is that **Envoy's infrastructure-level rejection** (JWT filter) generates a response that bypasses the gRPC-web encoding entirely, before the request ever reaches the backend.