---
# System prepended metadata

title: missing-trailer-analysis.md

---

## Root Cause: Envoy Filter Chain Ordering

The "missing trailer" error is caused by **Envoy's HTTP filter chain ordering** in the `grpc_web_listener` configuration.

### The Problem

In `lava-k8s/helm/envoygateway/templates/envoy-configmap.yaml:70-152`, the filter order is:

```
1. header_to_metadata
2. lua
3. cors
4. JwtAuthentication    ← rejects here with 401
5. grpc_web             ← never processes the response
6. router
```

When the JWT filter (#4) rejects a request (expired token), it generates a **local reply** (HTTP 401). In Envoy, local replies only flow back through filters that already processed the request (filters 1-3). **The `grpc_web` filter (#5) never sees this response**, so the 401 is sent as a plain HTTP response — not encoded in gRPC-web format.

The gRPC-web protocol requires a **trailer frame** (flag byte `0x80`) in the response body. A plain HTTP 401 has no such frame, so `@connectrpc/connect-web` throws `"missing trailer"` with code `Unknown` (because `grpc-status` is also absent from the headers).

### Evidence from Datadog

The log sequence confirms this:

| Timestamp | Message |
|---|---|
| 15:34:05.300 | **Token refresh failed, rejecting queued requests** |
| 15:34:05.301 | missing trailer |
| 15:34:05.302 | missing trailer |
| 15:34:06.449 | **Token refresh failed, rejecting queued requests** |
| 15:34:06.451 | missing trailer |
| 15:34:06.452 | missing trailer |
| ... | (repeats ~8 times in 9 seconds) |

Pattern: JWT expires → API calls get rejected by Envoy's JWT filter → plain 401 bypasses gRPC-web encoding → "missing trailer" → app tries token refresh → refresh fails → queued requests rejected → retry loop.

The specific log entry shows:
- **URL**: `grpcweb.mainnet.lava.xyz/credit/GetCreditSummary`
- **Code**: `Unknown` (no `grpc-status` header = response was not gRPC at all)
- **isAuthError**: `false` (the client can't detect it's an auth error because the gRPC error code is lost)
- **Component**: `ApiClient-Reactive`

### Why It Only Happens on Web

**Native** (iOS/Android): Uses native gRPC over HTTP/2. The app connects through a different transport that handles HTTP/2 trailers natively. Even if Envoy returns a 401, the native gRPC client can interpret the HTTP status code directly.

**Web**: Uses gRPC-web over HTTP/1.1. Trailers must be encoded as a special frame in the response body by Envoy's `grpc_web` filter. When this filter is bypassed (JWT rejection), the browser has no way to parse the response.

### The Fix

**Move the `grpc_web` filter BEFORE `JwtAuthentication`** in the filter chain at `envoy-configmap.yaml:70-152`:

```yaml
http_filters:
  - name: envoy.filters.http.header_to_metadata
    # ...
  - name: envoy.filters.http.lua
    # ...
  - name: envoy.filters.http.cors
    # ...
  - name: envoy.filters.http.grpc_web                    # ← moved BEFORE JWT
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
  - name: envoy.filters.http.JwtAuthentication
    # ...
  - name: envoy.filters.http.router
    # ...
```

With this order, when the JWT filter rejects a request, the response flows back through `grpc_web → cors → lua → header_to_metadata`. The `grpc_web` filter will now properly encode the 401 rejection as a gRPC-web response with the trailer frame, and the browser client will receive a proper `UNAUTHENTICATED` gRPC error instead of "missing trailer".

### Secondary Benefit

This fix will also allow the web client to properly detect auth errors (`isAuthError` will become `true`), enabling the token refresh flow to work correctly rather than entering a retry loop.

### Other Scenarios This Fixes

Any Envoy local reply generated by the JWT filter will be affected. This includes:
- Expired JWT tokens
- Invalid JWT signatures
- Missing JWT on protected endpoints
- JWKS fetch failures (if `jwks-server` is down)

### `ExceptionUtils.kt` Note

The `ExceptionUtils.kt` correctly adds trailers to gRPC error responses. The issue is NOT with the backend services — they properly send trailers. The problem is that **Envoy's infrastructure-level rejection** (JWT filter) generates a response that bypasses the gRPC-web encoding entirely, before the request ever reaches the backend.
