# Ghost Module Documentation

*Created by Jimmy Lin*

## End to End Workflow Diagram


## 1. Introduction

The Ghost Module is a software emulator that simulate the behavior of a pluggable transceiver module without hardware. It loads a memory dump, exposes the module state machine (MSM) and data path state machine (DPSM), computes derived control bits and flags, monitors bit error rates (BER), and even runs test matrices to generate expected values.

### Why we need it?

The Ghost Module defines one reusable set of inputs and expected outputs—aligned with CMIS and customer requirements—so we can auto-generate expectations and compare them against FRT runs, eliminating full matrix rebuilds and enabling automation.

## 2. Installing & Running

### 2.1 What you download

- **Ghost_Module.zip** — a portable build. When extracted you will see:
- **GhostModule.exe** — the launcher (GUI runner).
- **_internal/** — bundled Python runtime and libraries (e.g., PySide6, NumPy, DLLs). Keep this folder next to the EXE.

No separate Python install is required. Just unzip and run.

### 2.2 Install (really just extract)

1. Right click → Extract All… to a writable path (e.g., C:\Tools\GhostModule).
2. Ensure the folder contains GhostModule.exe and the _internal/ directory.
3. (Optional) Place your Test Metrics file(s) and Memory Dump anywhere you like; they don't need to live next to the EXE.

### 2.3 Run

1. Double click GhostModule.exe. If SmartScreen appears, choose More info → Run anyway (publisher unsigned builds only).
2. In the GUI, select:
- Input (Test Metrics XLSX/CSV/TXT),
- Memory Dump,
- Output file path (TXT/CSV/XLSX),
- Customer profile and any advanced options (address columns, max conditions/stimuli).
3. Click Run. Progress and logs appear in the window; the output file is written to the path you chose.

## 3. Architecture Overview

<center>

##### Ghost Module Components & Responsibilities

</center>

Ghost Module file map (12 files, 4 blocks):

- **Red — State Machines**
- **Role**: Governs the main module state machine and the data path state machine; implements transition logic for each state.
- **Files**: msm.py, dpsm.py.

- **Brown — Memory**
- **Role**: Manages the memory image. page_reader.py loads/saves memory dumps and maps them to upper/lower page structures; utils.py provides helpers to read/write and update memory while the model runs.
- **Files**: utils.py, page_reader.py.

- **Blue — Controls, Signals & Flags**
- **Role**: Applies controls, computes flags, and derives signals that drive the state machines; includes BER computation/monitoring.
- **Files**: controls.py, customer_flags_logic.py, customer_signals_logic.py, ber_monitor.py.

- **Green — Front end**
- **Role**: User/automation entry points (GUI and CLI) plus the host that maps Test Metrics operations to engine calls.
- **Files**: GUI/ (or GUI.py), host.py (functions consumed by CLI/Test Metrics), generate_expected_values.py (CLI ).

- **Other tooling**
- main.py can be used for quick ad hoc testing—assemble combinations and run without a Test Metrics file.

:::info
Note: ber_monitor.py lives in the Blue block above.
:::

## 4. Ghost Module Files: What Each One Does

### 4.1 page_reader.py

#### Purpose
Owns the Ghost Module's memory image and provides structured access to Lower Page (0x00:00–7F) and Upper Pages (0x00/0x01…:80–FF). It can parse a textual dump into in-memory pages and format the current pages back to a dump for logging, repro, or diffing.

#### Memory model
Pages stores one lower page (bytes 0–127 of page 0) and a dict of 128-byte upper pages (including page 0's upper half). It enforces valid offsets (lower vs. upper regions) whenever bytes are added.

#### Core responsibilities
- **Parse dumps → pages**: parse_dump() recognizes page/table headers and hex rows, then writes bytes into the correct region via Pages.add_bytes().
- **Accessors**: get_lower_page(), get_upper_pages(), get_page(p) expose the in-memory views used by state machines and controls.
- **Serialize pages**: format_dump() and write_formatted_dump(path) reconstruct a human-readable dump of page 0 (lower & upper) and all other upper pages

#### Higher-level helpers
- **Module state**: get_module_state(State) reads the 3-bit module state code from lower page byte 3 and maps it to the State enum.
- **Module controls**: get_controls() decodes control bits from lower page byte 26.
- **Data-path**:
- get_data_path_status(DPState) reads 3-bit per-lane DP state codes from page 0x11 (two lanes per byte).
- get_control_set() extracts AppSelCode, DataPathID, and ExplicitControl (page 0x11, bytes 0xCE–0xD5).
- data_path_groups() groups lanes by DataPathID.
- **Random access**: get_byte(page, byte) reads any upper-region byte with bounds checking.

#### Collaborators / Used by
msm.py, dpsm.py, controls.py, and utils.py read from page_reader to decide transitions, compute derived signals/flags, and to snapshot memory before/after stimuli.

<div style="text-align: center;">
<img src="https://hackmd.io/_uploads/ry_yS4aqxg.png" width="300" />
</div>

### 4.2 msm.py

#### Purpose
Implements the module's high-level state machine and coordinates with the data-path SM. It reads the initial state from memory and writes back state changes as they occur.

#### State model
States include resetting, reset, mgmt_init, and the module states module_low_pwr, module_pwr_up, module_ready, module_pwr_dn, module_fault. Module states carry a 3-bit code (module_code) with a helper to map code→state.

#### How stepping works
step(controls, dpsm_controls, max_steps=4) updates derived signals, then loops transitions until steady state or max_steps is reached (4 is the longest needed path). On each transition it writes the new state to memory, refreshes derived signals, and finally triggers the DPSM step loop (up to 10 steps).

:::warning
For detailed state definitions, see [Module and Dp State Logics](https://hackmd.io/@Jimmy775533/Bkh1wET5xg)
:::


#### Memory updates
write_state_to_memory() encodes module states to lower page byte 3, bits 1–3 using update_pages_bits; it also sets a validity bit at byte 8, bit 0. Non-module internal states are not written.

#### Primary APIs
step(), write_state_to_memory(), plus the state handlers shown above.

#### Collaborators
Uses page_reader for page access, controls for inputs/derived-signal updates, dpsm for data-path coordination, and utils.update_pages_bits for memory writes.

### 4.3 dpsm.py

#### Purpose
Implements the Data-Path State Machine (DPSM). It tracks per-lane TX states, applies lane/group controls, advances states to steady, and writes each lane's 3-bit state code back to the memory.

#### State model
States: DPDeactivated, DPInit, DPDeinit, DPActivated, DPTxTurnOn, DPTxTurnOff, DPInitialized—each with a 3-bit code and helpers for code ↔ state and per-lane memory layout (2 lanes/byte, 4 bits per lane).

:::warning
For detailed state definitions, see [Module and Dp State Logics](https://hackmd.io/@Jimmy775533/Bkh1wET5xg)
:::

#### How stepping works
step(controls, msm, State, module_controls, max_steps=10) first updates derived signals using MSM readiness & low-power, then iterates until no lane changes (or max_steps). For each group it:
- applies group re-deinit (dp_re_deinit_s) to all lanes,
- applies RF input disable to all lanes but allows per-lane override via tx_squelch_disable, or
- performs normal lane-level control.

On any change it writes the lane state to memory, updates the group state, and (if anything changed) updates the Page 0x11 summary.

#### Primary APIs
step(...), write_state_to_memory(lane), plus the state handlers shown above.

#### Collaborators
Reads groups/status from page_reader, consumes controls from controls, considers module readiness from msm, and uses utils.update_pages_bits for in-place writes.

### 4.4 controls.py

#### Purpose
Central hub for inputs and derivations. It models external pins, reads memory-mapped controls, computes derived signals used by MSM/DPSM transitions, and writes derived flags (including Page-11 summaries) back to CMIS.

#### Components
- **ExternalControls** – In-memory model of hardware pins (e.g., low_pwr_request_hw, vcc_reset, per-lane RF/TX controls).
- **ModuleControls** – Reads control bits from memory (lower page byte 26) and derives reset_s, low_pwr_s, low_pwr_ex_s.
- **DataPathControls** – Groups lanes, reads DP control bytes (Page 0x10/0x11), computes per-lane/group signals with a customer strategy, and exposes summaries.

#### Derived-signals flow
1. Read control bytes / external pins →
2. Compute module signals (reset_s, low_pwr_s, low_pwr_ex_s) and DP signals per lane/group (e.g., dp_deinit_s, dp_tx_auto_squelch_disable_t, dp_re_deinit_s, dp_deactivated_s) using pluggable customer logic →
3. MSM/DPSM consume these to advance toward steady state.

#### Flags & memory updates
- update_derived_flags(...) builds per-customer bitmasks and writes lane flags to Page 0x11.
- page11_summary(...) ORs a byte range on Page 0x11 and sets Page 0, byte 4, bit 0 as the summary flag.
- Uses update_pages_bits(...) for safe in-place writes.

#### Primary APIs
- ModuleControls.update_derived_signals() – refresh module-level signals.
- DataPathControls.update_derived_signals(module_ready, low_pwr_s) – compute DP signals per group/lane.
- DataPathControls.update_derived_flags(lane: Optional[int] = None, customer: Optional[str] = None) – write flags for one lane or all lanes, using selected customer rules.
- DataPathControls.page11_summary(start_byte=134, end_byte=152) – recompute Page-11 summary flag.

#### Collaborators
Reads pages via page_reader.PageReader; consumed by msm.py and dpsm.py; writes via utils.update_pages_bits; customer behavior injected from customer_signals_logic and customer_flags_logic.

### 4.5 customer_flags_logic.py & customer_signals_logic.py

#### Purpose
Defines how raw lane control bits + module state are turned into derived signals used by MSM/DPSM transitions. It provides a baseline (Meta) strategy and customer variants (Google, Nvidia).

#### Data types
- **LaneBits**: raw inputs per lane (e.g., dp_deinit_t, dp_tx_disable_t, dp_tx_force_squelch_t).
- **LaneSignals**: computed outputs (dp_deinit_s, dp_re_deinit_s, dp_deactivated_s, passthrough of dp_tx_auto_squelch_disable_t).

#### Customer variations
- **Meta**: Baseline logic
- **Google**: same as baseline except 'force squelch' is ignored in dp_deactivated_s.
- **Nvidia**: currently mirrors Meta (hook for future differences).

#### API
- get_signals_strategy(name) → strategy object (defaults to meta).
- SignalsStrategy.compute_lane(lane_bits, *, module_ready, low_pwr_s) → LaneSignals.

### 4.6 customer_flags_logic.py

#### Purpose
Builds byte-level masks for Page 0x11 flag bytes, based on external controls and lane/group context. DataPathControls.update_derived_flags resolves a strategy and applies these masks to memory.

#### Baseline helper (Meta)
baseline_build_masks(ctx, lane) produces a dictionary of byte→mask (e.g., 147, 148, 150, 152 and RF/RX bytes 136–146) and iterates lanes, skipping deactivated ones.

Handles cases such as DUT2 TX lane disable/force squelch, DUT1 RF input disable, and DUT2 RF input disable (AC/DC).

#### Customer variations
- **Google**: focuses on dut2_tx_lane_disable; RF-disable avoids setting 140/142; force-squelch still drives 140/142/144/146 to all ones.
- **Nvidia**: currently mirrors baseline; message placeholder shows where to customize.

#### API
- get_strategy(name) → returns MetaStrategy, GoogleStrategy, or NvidiaStrategy (default: meta).
- DerivedFlagStrategy.build_masks(ctx, lane) → {byte_index: mask} for Page 0x11 updates.

#### How it's used (context)
DataPathControls.update_derived_flags(...) resolves the strategy, builds masks with a FlagsContext, and writes updated bytes to Page 0x11; then recomputes the Page-0 summary bit.

### 4.7 utils.py

#### Purpose
Provides update_pages_bits(...), a safe, single-entry utility to write one bit or a contiguous field into the Ghost Module memory map (lower or upper pages). It selects the correct byte-array region, applies a mask, and logs the before/after value.

#### Responsibilities
- Route writes to lower page (page 0, bytes 0–127) vs upper pages (bytes 128–255 of page 0 or any page >0).
- Update either a single bit (bit_index, value) or a field (bit_start, bit_length, field_value).
- Validate bounds and throw clear errors for bad arguments.

#### Key API

update_pages_bits(pages_reader, page=0, byte_index=128, *, 
                  bit_index=None, value=None, 
                  bit_start=None, bit_length=None, field_value=None)

#### Collaborators
Expects a PageReader (for memory access). Called by host/controls/MSM/DPSM paths whenever register bits/fields must be updated.

### 4.8 ber_monitor.py

#### Purpose
Computes per-lane BER status for DUT1 and DUT2 based on control states and DPSM readiness, storing results like dut1_ber[lane] / dut2_ber[lane] and exposing a simple reporting API. Uses BERValue enums (GOOD, INVALID).

#### Responsibilities
- Build lane groups from page_reader.data_path_groups() and read control bytes (e.g., deinit/disable/squelch, RX disable/squelch) from Page 0x10.
- Update BER for both DUTs using clear rules (e.g., RF input disable, TX disable/force squelch in a lane's group, RX output/squelch conditions, lane activation via DPSM).
- Provide helpers to print, reset, query per-lane, list invalid lanes, and get group status.

#### Key APIs
- update_ber_values(dpsm, controls, module_controls) — recompute all lanes (calls _update_dut1_ber / _update_dut2_ber).
- get_ber_report() — dict of dut{1|2}_ber_lane{n} → value.
- print_ber_status() — human-readable table.
- get_lane_ber(dut, lane) / get_invalid_lanes(dut) / get_group_ber_status(dut, group_id) / reset_ber_values().

#### Collaborators
Reads bytes via PageReader; uses ModuleControls (external RF/TX settings) and DPSM (lane activation state) as inputs.

### 4.9 host.py

#### Purpose
Coordinates high-level actions requested by the GUI/CLI with the engine. It updates CMIS page bits, steps the MSM/DPSM, refreshes derived flags, and (when provided) triggers BER recomputation and reporting.

#### Responsibilities
Provide one-shot recipes that map Test Metrics operations to concrete actions:
- bring module to ready or low-power,
- set data path to activated or initialized,
- per-lane and all-lane TX/RX controls (disable/enable, force-squelch, squelch-disable),
- RF input disable per DUT/lane,
- clear latch bytes on Page 0 and Page 0x11,
- fetch/print BER status.

Perform safe writes via utils.update_pages_bits(...), then drive the state machines and flag/BER updates in the correct order.

#### How a typical action runs (pipeline)
1. Write the requested control bit/field to the right page/byte (update_pages_bits).
2. Step the state machine(s): msm.step(...) and/or dpsm.step(...) until steady.
3. Update derived flags (controls.update_derived_flags(...)) and the Page-11 summary.
4. Refresh BER (ber_monitor.update_ber_values(...)) and optionally print a report.

#### Key APIs
- set_module_ready_state(...) / set_module_low_power_state(...) — drive module power sequencing.
- set_data_path_activated(...) / set_data_path_initialized(...) — drive DPSM to target states and update BER.
- **Per-lane controls**: tx_lane_disable(...), tx_force_squelch(...), tx_squelch_dis(...), rx_output_dis(...), rx_squelch_dis(...).
- **All-lane controls**: tx_allLane_disable(...), tx_squelch_disall(...).
- **DUT/global**: tx_auto_squelch_mode(...), dut_rf_input_dis(...), clear_latch(...).
- **Reporting helpers**: get_ber_status(...), print_external_controls_status(...).

#### Collaborators
- **MSM/DPSM**: msm.ModuleStateMachine, dpsm.DataPathStateMachine (state progression).
- **Controls**: ModuleControls, DataPathControls (derived signals & flags).
- **Utils**: update_pages_bits (bit/field writes).
- **BER**: BERMonitor (per-lane DUT1/DUT2 status).

### 4.10 generate_expected_values.py (CLI)

#### Purpose
Reads a Test Metrics file, simulates the Ghost Module once per row, and writes register/BER results in pre_post format—producing an output sheet you can compare with FRT.

#### What it does
- Parses inputs & config (paths, address range W–II by default, static columns A–V, max conds/stimuli).
- Loads a memory dump, then re-initializes a fresh system per row (PageReader, MSM, DPSM, Controls, BERMonitor).
- Executes Conditions then Stimuli by mapping spreadsheet commands to host.py actions (e.g., tx_lane_disable, setreadystate).
- Snapshots registers before and after (PreDataRead / PostDataRead) across the configured address columns.
- Detects BER columns to the right of the address block and writes BER as pre_post too.
- Copies headers & static columns verbatim, then writes results as tab-delimited text (.txt).

#### CLI (typical)

python generate_expected_values.py \
  --input test_matrix.xlsx \
  --output expected_values.xlsx \
  --dump memory_dump_lowpwr.hex \
  --start-col W --end-col II --static-end-col V \
  --max-conditions 10 --max-stimuli 10

#### Row execution flow
1. Build fresh system from dump →
2. Apply cond1…condN (via host) →
3. Snapshot PreDataRead →
4. Apply stimulus1…stimulusN →
5. Snapshot PostDataRead →
6. Write pre_post values for each addressed register + BER → next row.

#### Key conveniences
- Accepts .xlsx or CSV/TSV/TXT inputs.
- Strict, readable address parser for headers (page.byte.bits / pageBank.byte.bits).
- Separates command parsing, execution, memory reading, and BER writing for clarity and easy extension.

### 4.11 GUI.py

#### Purpose
PySide6 desktop app that collects paths/options for a Test Metrics run, then hands them to a background worker that invokes generate_expected_values (MatrixProcessor). It shows progress, logs, and provides quick access to the output.

#### Responsibilities
- **Theme & UX**: Dark/light themes, custom palettes and QSS, splash screen, and a PrettySpinBox with bold +/− buttons.
- **Inputs panel**: Pick Input file, Memory Dump, Output .txt, and Customer (sets ACTIVE_CUSTOMER). File pickers + "Open Output Folder/File."
- **Advanced panel**: Configure Address row, Columns (Testcase end, Start, End) with uppercase validation and image tooltips, and limits for Max conditions/stimuli.
- **Run panel & Log**: "Run/Cancel" controls, progress bar, status bar, and a read-only log viewer with "Clear Log."
- **Persistence**: Remembers last settings via QSettings.

#### How execution works
1. on_run gathers UI settings, saves them, and starts a ProcessorWorker thread.
2. The worker imports generate_expected_values, builds a MatrixConfig, and calls MatrixProcessor.process(), redirecting stdout/stderr to the GUI.
3. on_ok/on_err update the log, enable "Open Output File," and manage progress/controls.

## 5. End to End Workflow Diagram


## 6. Workflow (GUI → CLI → Pipeline)

### 1. Select inputs in the GUI
- Pick Input (Test Metrics) and Output files, plus any options (dump path, column range, customer profile, etc.).
- The GUI passes these settings to the CLI.

### 2. CLI parses the Test Metrics file
- Opens the input sheet and validates the format.
- Creates the output sheet with the same header (row 1).
- Copies the left-side columns (case metadata, cond1…cond8, stimulus1…stimulus4) and leaves the right-side register/value columns blank to be filled by the run.

### 3. Initialize the Ghost Module
- Build and reset the GM (memory image + state machines).

### 4. For each test case (row): apply Conditions left→right
- The CLI calls into host.py, which maps spreadsheet keywords to concrete actions.
- Example: tx_lane_disable, utils.update_page_bits() writes the registers/memory. Call msm.step() or dpsm.step() (depends on the action) to advance states. On every step, update_derived_signals first to ensure the state machines see the latest inputs.
- Repeat steps until steady/ready.
- Then update_derived_flags based on current state/signals.
- Finally ber_monitor.py updates the BER value.

:::warning
Customer-specific logic (Meta/Google/Nvidia/…) is chosen in the GUI.
:::

### 5. Take PreDataRead snapshot
- Capture the current register/field values after all conditions are applied.

### 6. Apply Stimuli (same mechanism as Conditions)
- Use host-mapped functions → update page bits → step SMs → update derived signals/flags → update BER.

### 7. Take PostDataRead snapshot
- Capture register/field values after all stimuli are applied.

### 8. Write results
- For each requested field, combine values as pre_post (e.g., 1_1) and write them to the output row.
- (Optional) also emit a memory dump for debugging/repro.

:::info
In the diagram, N is the number of rows in the Test Metrics file. The pipeline runs once per row, so the loop executes N× (i.e., N times).
:::

## 7. Where to start: adding new flag rules or functions

When you need a new flag rule (Page 11 masks) or want to expose a new operation in Test Metrics, start here.

### 7.1 Add or change derived flag rules

1. **Define strategy** in customer_flags_logic.py:
- Create a new class (e.g., AWSStrategy) implementing build_masks(ctx, lane).
- Reuse helpers like the baseline mask builders where possible.
2. **Register the strategy** in get_strategy() so DataPathControls.update_derived_flags() can find it.
3. **Wire selection**:
- Use the GUI Customer dropdown or set ACTIVE_CUSTOMER to pick your strategy at runtime.
4. **Validate bytes**:
- Confirm the Page 11 byte indices you modify; update page_reader constants if new bytes are introduced.

### 7.2 Add a new derived signal rule (affects transitions)

1. Implement a strategy in customer_signals_logic.py (e.g., AWSSignalsStrategy) and implement compute_lane(...).
2. Register it in get_signals_strategy() and ensure DataPathControls.update_derived_signals() passes the right module_ready/low_pwr inputs.

### 7.3 Add a new host action / CLI function

1. **host.py**: create a function that performs the operation by calling utils.update_pages_bits(...), then step the SMs and update flags → BER in the standard order.
2. **generate_expected_values.py**: map a new spreadsheet keyword (e.g., rf_input_dis:dut1,lane3) to the new host function.
3. **GUI (optional)**: add a control only if you want to expose it interactively.

### 7.4 If you change DPSM logic

1. **Update handlers**: review/modify the per state handlers in dpsm.py
2. **Keep writes in sync**: ensure write_state_to_memory() still encodes the correct 3 bit codes and validity bits for each lane.
3. **Signals alignment**: confirm DataPathControls.update_derived_signals() (and customer strategies in customer_signals_logic.py) produce the signals your new transitions expect.

### 7.5 Add or change parsing logic (file format, headers, commands)

If your spreadsheet or text file uses different formats—or you want new command syntax—start in generate_expected_values.py:

1. **New command/function keyword**: update CommandType enum; add parsing in CommandParser (e.g., identify_command, parse_* helper); implement the action in CommandExecutor.execute_command() and route to a new host function in host.py.
2. **Different header/address format**: extend AddressParser.parse_header_address() (and helpers split_pagebank, parse_bits) to recognize the new pattern; if addressing semantics change, adjust MemoryReader.extract_value() accordingly.
3. **Different column names/locations**: tweak MatrixProcessor.find_columns() and/or MatrixConfig (e.g., max_conditions, max_stimuli, start_col_label, end_col_label, static_end_col).
4. **Different file type/delimiter**: extend _load_input_any() to read your format.
5. **New Pre/Post switches** (e.g., PostDataRead(clearlatch) variants): extend _parse_postdataread_args() and _postdataread_wants_clearlatch().
6. **BER columns/labels**: edit BERResultWriter.parse_ber_header() and related snapshot/writing helpers.
7. **Output formatting**: adjust write_results() and BERResultWriter.write_pre_post_cells() (and ensure_txt_suffix/save_tsv if needed).

After parser changes, run a few rows through the GUI to verify: commands execute, pre/post snapshots line up, and BER writes to the expected columns.