# catalog pod memory investigation

Looking for clusterbot catalog initial FBC mem spike. Assessment of existing infrastructure per-pod.

custom cluster metric query

sum (container_memory_rss{namespace='openshift-marketplace',container="",pod!="",pod=~".*-catalog.*"}) by (pod)

my version

sum(container_memory_rss{pod='marketplace-operator-d4f794477-97tp5',namespace='openshift-marketplace'}) BY (namespace)

# clusterbot launch 4.10 (sqlite)

## redhat-marketplace 

- resource-request ~52MiB, spiking to 105MiB 1m30s after creation, then ~52MiB flat
- rss 85Mib, gradual growth with spikes to ~104MiB
- working-set ~53MiB, gradual growth to ~63MiB

## redhat-operators 

- resource-request 52.43MiB solid
- rss 56MiB, growth with spikes in 1h to 114MiB
- working-set 49Mib, growth with spikes in 1h to 79MiB

## community-operators 

- resource-request 52.43MiB solid
- rss 146MiB ~linear growth to 206MiB
- working-set 112Mib, ~linear growth to 144MiB

## certified-operators 

- resource-request 52.43MiB solid
- rss spiky with return to base state, still showing persistent growth 75 - 81
- working-set spiky with return to base state, still showing persistent growth 104 - 114

# clusterbot launch 4.11 (fbc)

## redhat-marketplace 

- resource-request - steady-state 53MiB;
- rss - initial 110MiB, dropping w/in 2min to 42MiB, semi-linear gain to 70Mib with a spike of almost 100MiB
- working-set - 119MiB initial, dropping w/in 2min to 85MiB, some creep to 100Mib over 20min

## redhat-operators 

- resource-request - rock steady at 52.43
- rss - 117MiB with occasional spikes to 235MiB, 184MiB. Good recovery to base state.
- working-set - steady around 400MiB+-10

## community-operators 

- resource-request - rock steady at 52.43
- rss - 95MiB, mostly-linear growth to 118MiB over 30mins
- working-set - 254MiB+-20MiB steady-state

## certified-operators 

- resource-request 52.43MiB solid
- rss 57-77MiB periodic growth
- working-set matching rss, but 181-191MiB

# clusterbot launch 4.12

## redhat-marketplace 

- resource-request rock steady at 52.43
- rss initial spike 78MiB, after 2min dropping to 45MiB, then incredibly jaggy but returning faithfully to 59MiB floor
- working-set mimicing rss, but ranging 110MiB to 94MiB, floor of 100MiB

## redhat-operators 

- resource-request rock steady at 52.43
- rss 105MiB, spiky showing gradual growth to 123MiB, ceiling of 168MiB
- working-set mimicing rss, spiky showing 377MiB gradual growth to 388Mib, ceiling 400MiB

## community-operators 

- resource-request rock steady at 52.43
- rss 217MiB initial, dropping in 1m to 85Mib, linear growth to floor of 103MiB, spike ceiling 155MiB
- working-set no initial spike, 245MiB growing linearly to 260MiB, spike ceiling 282MiB

## certified-operators 

- resource-request rock steady at 52.43
- rss occasional spikes to 126MiB, but faithfully returning to base 61MiB+-2
- working-set as rss, but base 184MiB+-3

# standalone docker

based on the fact that community-operators was the least-well-behaving catalog pod of the set, we assessed the impact of the no-precached versus precached docker images w.r.t. memory demand stability.

| variant | initial cpu | cpu after 10s | initial mem | mem after 10s |
|---------|-------------|---------------|-------------|---------------|
| uncached | 100+ | ~0 | 700MiB | 700MiB |
| precached | 14 | ~0 | 14MiB | 14MiB |

not only was the memory request higher in the uncached variant (by a factor of 50!) but it was not released for the uncached variant, which meant that it was a sunk cost over the lifetime of the container.

# catalog stats over revisions

Just to rule out the possibility that the 50x memory footprint might be the result of normal growth, here's a table of the official catalogs which demonstrate a universal diminishing trend.

| cat | ver | #package | #bundle |
|-----|-----|----------|---------|
| redhat-operator-index | 4.8 | 79 | 1717 |
| | 4.9 | 83 | 1238 |
| | 4.10 | 95 | 1236 |
| | 4.11 | 92 | 1064 |
| | 4.12 | 66 | 905 |
| certified-operator-index|4.8 |210 | 657 |
| | 4.9 | 143 | 375 |
| | 4.10 | 131 | 329 |
| | 4.11 | 106 | 283 |
| | 4.12 | 84 | 230 |
| redhat-marketplace-index|4.8 | 124 | 363 |
| | 4.9 | 56 | 120 |
| | 4.10 | 50 | 100 |
| | 4.11 | 42 | 88 |
| | 4.12 | 38 | 75 |
| community-operator-index|4.8 | 218 | 3286 |
| | 4.9 | 180 | 2041 |
| | 4.10 | 182 | 2025 |
| | 4.11 | 177 | 1957 |
| | 4.12 | 170 | 1838 |

{
    "encoding": {"color": {"field": "cat", "quantitative": "nominal"}},
    "data": {
        "values": [
            { "cat": "redhat-operator-index", "version": "4.08", "packages": 79, "bundles": 1717},
            { "cat": "redhat-operator-index", "version": "4.09", "packages": 83, "bundles": 1238},
            { "cat": "redhat-operator-index", "version": "4.10", "packages": 95, "bundles": 1236},
            { "cat": "redhat-operator-index", "version": "4.11", "packages": 92, "bundles": 1064},
            { "cat": "redhat-operator-index", "version": "4.12", "packages": 66, "bundles": 905},
            { "cat": "certified-operator-index", "version": "4.08", "packages": 210, "bundles": 657},
            { "cat": "certified-operator-index", "version": "4.09", "packages": 143, "bundles": 375},
            { "cat": "certified-operator-index", "version": "4.10", "packages": 131, "bundles": 329},
            { "cat": "certified-operator-index", "version": "4.11", "packages": 106, "bundles": 283},
            { "cat": "certified-operator-index", "version": "4.12", "packages": 84, "bundles": 230},
            { "cat": "redhat-marketplace-index", "version": "4.08", "packages": 124, "bundles": 363},
            { "cat": "redhat-marketplace-index", "version": "4.09", "packages": 56, "bundles": 120},
            { "cat": "redhat-marketplace-index", "version": "4.10", "packages": 50, "bundles": 100},
            { "cat": "redhat-marketplace-index", "version": "4.11", "packages": 42, "bundles": 88},
            { "cat": "redhat-marketplace-index", "version": "4.12", "packages": 38, "bundles": 75},
            { "cat": "community-operator-index", "version": "4.08", "packages": 218, "bundles": 3286},
            { "cat": "community-operator-index", "version": "4.09", "packages": 180, "bundles": 2041},
            { "cat": "community-operator-index", "version": "4.10", "packages": 182, "bundles": 2025},
            { "cat": "community-operator-index", "version": "4.11", "packages": 177, "bundles": 1957},
            { "cat": "community-operator-index", "version": "4.12", "packages": 170, "bundles": 1838}
        ]
    },
    "layer": [
        { "mark": "line", 
          "encoding": {
            "x": {"field": "version", "type": "nominal", "title": "version"},
            "y": {"field": "bundles", "type": "quantitative", "title": "bundles"}
          }
        },
        { "mark": "line", 
          "encoding": {
            "x": {"field": "version", "type": "nominal", "title": "version"},
            "y": {"field": "packages", "type": "quantitative", "title": "packages"}
          }
        }
    ]
}