# forwards

the high objective right now is block processor performance.

heavy operations during chain sync, especially processing of the large genesis, result in blocking all execution of the javascript thread hosting the block processor. the extension backend and entry to the rpc service share a single thread, so this is significant. after onboarding, users encounter confusing behavior and an extended waiting period before it's possible to connect to a frontend interface.

the 'offscreen' technique has been used previously to parallelize other heavy operations like transaction building within the rpc services implementation, by providing a way to launch worker threads. if the block processor were also capable of running in its own thread or launching worker threads when necessary, it wouldn't block the entire service worker every time it does something heavy, and it could parallelize tasks by launching workers.

so running the block processor partly or completely within the chrome runtime's 'offscreen' document is the accepted technique to improve performance and parallelize work.

### apis we use from chrome runtime

- chrome.runtime messaging -- this is outside of the present implementation
- chrome.storage storage read -- this is input to the present implementation
- (offscreen, to launch workers) --
- chrome.windows - spawning popup for custody

approaches:
- target standard dom
- require offscreen client to be provided (context, or constructor)

target standard dom could be:
- actual standard dom
- polyfilled offscreen

## offscreen

the 'offscreen' document is the chrome runtime's facility to support code that requires the standard browser dom api, and the features sought to achieve acceleration are within that standard browser dom api.

### previously

unfortunately, previous work using 'offscreen' has been painful. in `@penumbra-zone/services`, the chosen technique has been difficult to develop and debug properly, because it adds multiple layers of abstraction, messaging, and seralization/deserialization between the parent extension thread and its ultimately launched workers.

the implementation in `@penumbra-zone/services` is also implicitly dependent (in an undocumented way) upon code that is not present within the penumbra-zone web repo. any third-party consumer of the services package must independently implement significant parts of the feature. it's also the only part of the services package which contains code specific to the chrome runtime.

i think replicating that pattern in the block processor is a bad idea.

### don't repeat that

instead of digging deeper, it makes more sense to go the opposite direction: parts of the code which need to launch a worker should simply launch a worker in the standard way.

this implies chrome runtime consumers must run the entire block processor within the offscreen document. this will involve some minor changes to block-processor init for consumers, but it's not very significant. all init is already passed as parameters. database access is still available from within the offscreen document. the chrome runtime is still available, so control message handlers in any implementation will function the same.

thus, the technical goal is to target a standard browser DOM runtime.

i would advocate ultimately running the block processor within a worker, so the messaging code is never thread-blocked.

### simplify services

this is also an opportunity for simplification. with limited work, services can also execute within the offscreen document, and perhaps also within an independent worker.

again parts of the code which need to launch a worker should simply launch a worker. a complete implementation of the transaction build functionality can be provided within the package. complete testing and third party consumption will become possible.

most of the work required involves completing specification of the `HandlerContext` interface.

## the plan

0. relocate 'offscreen client' to outside the services package

offscreen client becomes a general-purpose 'launch worker' utility which manages lifecycle of the offscreen document only, and wraps provided implementations with the necessary i/o.

either create a 'context' package in penumbra-zone web, or require providers like prax to implement it.

then services and the block processor can use this tool from handler context or from constructor parameters.

1. create a package specifically for the block processor.

right now it's thrown in with `@penumbra-zone/query` when it's significant enough to be its own package.

2. eliminate `@penumbra-zone/query`.

the remainder of query is very thin wrappers to rpc methods. it's only consumed by `@penumbra-zone/services` and the block processor. this is a large diff but mostly consists of having callers just use the `PromiseClient`s directly.

a method similar to `PenumbraClient.service` replaces all of the 'context client's presently in `HandlerContext`, one for the remote fullnode and one for loopback services (ban cross-method calls).

at this point, services' `HandlerContext` looks like this:

// todo: use PenumbraClient, or just a services method? a map? just a transport?

const db: IndexedDbInterface = ctx.values.get(dbCtx);

// used for obtaining remote rpc clients.
// remote.origin is the remote GRPC endpoint
const remote: PenumbraClient = ctx.values.get(remoteCtx);
// alternatively:
const remote: Transport = ctx.values.get(remoteCtx);

// used for obtaining clients to other local services.
// should also replace same-service cross-method queries.
// local.origin refers to this host's `PenumbraProvider` origin (extension url)
const local: PenumbraClient = ctx.values.get(localCtx);
// alternatively:
const loopback: Transport = ctx.values.get(loopbackCtx);

// service-restricted context
const { sk }: { sk: SpendKey } = ctx.values.get(custodyCtx);
const { fvk, walletId } = { fvk: FullViewingKey, walletId: WalletId } = ctx.values.get(viewCtx);

the same `PenumbraClient` interface may be provided as a block processor parameter, for access to the full node.

interface QueryClientProps { // to be renamed
  viewServer: ViewServerInterface;
  db: PenumbraDb;
  numeraires: AssetId[];
  stakingAssetId: AssetId;
  genesisBlock: CompactBlock | undefined;

  // fullnode.origin is the remote GRPC endpoint
  fullnode: PenumbraClient;
}

3. relocate the database interface definition

this could be moved either to `@penumbra-zone/storage`, or a new shared package `@penumbra-zone/context` with minimal deps, describing all interfaces necessary to implement `HandlerContext` for services. relocated/collected interfaces simplify dependency relationships for current consumers services, block processor, and (later) their wasm utilites.

at this point, the components of block processor and services are very nearly able to run in independent documents or workers - database subscriptions are the last blocking feature.

everything is much more easily third-party consumeable, but can't be parallelized by launching individual components in workers until database subscriptions are refactored to work cross-document.

4. remove prax internal messaging types and logic from `@penumbra-zone/services` and target standard dom

at this point, services is third-party consumeable.

to improve more generally,

5. break up the wasm package into purpose-specific packages

- utilities for consumers
- utilities for services (building transactions)
- utilities for the block processor

this will support lower memory consumption of the wasm package, also a high priority issue. wasm will init faster.

6. consume the database interface in the wasm package

instead of defining an independent interface in parallel and performing independent access. this has been a source of critical bugs.

at this point all packages improved for third-party consumption. consumers just provide their database implementation, and hook up the context required by services (just parameters like keys and constructed clients to the remote node).

optionally,

7. `@penumbra-zone/storage` interface can be improved, and the implementation can be deleted or remain penumbra's provided implementation of the database interface.

8. refactor database subscriptions to work cross-document

this would enable block processor, services, etc to all run in their own workers. only init, and message routing would occur in the main service worker. thus nothing should ever block RPC or background work with heavy computation.

## overall benefits

- the block processor becomes a distinct module instead of weirdly interdependent with queriers and wasm in an unspecified way
- the services now use an abstracted interface to other modules so they are pluggable instead of hard dependencies on specific implementations
- wasm now consumes an interface to the database instead of defining its own

## Parallelizing the block processor workloads

1. identify the number of encrypted note payloads in a compact block
2. Launch X web workers (X physical cores), and each web workers x_i handles (N mod X) work, where N is the number of note payloads.
3. presently, wasm scan_block performs sct insertion when processing a decrypted payload. the sct position is used in nullifier derivation, so processing decrypted payloads can't happen in parallel, without some mechanism to share the sct.
4. Research feasibility of parallel hashing