# Final block verification by orchestrator

## Terminology
https://gist.github.com/frozeman/d52090f3b8c4b654cacc279d894df31c#terminology

## Purpose
Both sides (pandora and vanguard) should be able to assure synchronization of eth1 and eth2 layer.

## Actors
Pandora
Trusted Pandora
Vanguard
Trusted Vanguard
Validator
Orchestrator

## Extension / Modification
Current Pandora <-> Validator connectivity is handled by rpc `eth_getWork`. We should extend/change this endpoint to provide parentHash as a parameter to be more precise

## Block statuses

0 Pending
1 Verified
2 Invalid

CONSIDER `unknown` state -> do a research, but leave it for now

type Status string

const (
	Pending  Status = "Pending"
	Verified Status = "Verified"
	Invalid  Status = "Invalid"
	Skipped  Status = "Skipped"
)

## Fallback after orchestrator database loss
In my humble opinion we should use not only `pending` blocks/headers for verification. There can be a scenario where verification based on pending also should happen:

@reconnectOrchestrator @orchestrator @pandora @vanguard
@Given as an Orchestrator I want to continue my duties
  And Trusted Pandora notified orchestrator about pending blocks
  And Trusted Vanguard notified orchestrator about pending blocks
@When Orchestrator reconnects with empty database
  And Pandora is not having pending blocks anymore
  And Vanguard is not having pending blocks anymore
@Then Orchestrator should rebuild realm database based on cannonical chain

## Connectivity
In my opinion we should maintain only RPC as a server on Orchestrator side and 2 orchestrator clients for the following endpoints:

### Payload is used same to both `orc_confirmPanBlockHashes` and `orc_confirmVanBlockHashes`

type BlockHash struct {
	Slot uint64
	Hash common.Hash
}

type BlockStatus struct {
	BlockHash
	Status Status
}

### [DEPRECATED] Mocked responses:
In my opinion having particular hash to test different scenarios are handy, so I propose those particular hashes responses:

const (
	MockedHashInvalid = "0xc9a190eb52c18df5ffcb1d817214ecb08f025f8583805cd12064d30e3f9bd9d5"
	MockedHashPending = "0xa99c69a301564970956edd897ff0590f4c0f1031daa464ded655af65ad0906df"
)

Any other hash by default will send status `Verified`

**NOTE:**
this will be changed after providing serious implementation on orchestrator side (last step)

### orc_confirmPanBlockHashes

pan > orc : confirmBlockBatch: 'orc_confirmPanBlockHashes', params: [[{slot: 1, hash: blockHash},...]]
    > [{slot: 1, status: 'pending'}, {slot: 2, 'verified'}, {slot: 3, 'invalid'}, ...]

// ConfirmVanBlockHashes
func (api *PublicFilterAPI) ConfirmVanBlockHashes(
	ctx context.Context,
	request []*BlockHash,
)(response []*BlockStatus, err error) {
	if len(request) < 1 {
		err = fmt.Errorf("request has empty slice")

		return
	}

	response = make([]*BlockStatus, 0)

	for _, blockRequest := range request {
		status := Verified

		if MockedHashInvalid == blockRequest.Hash.String() {
			status = Invalid
		}

		if MockedHashPending == blockRequest.Hash.String() {
			status = Pending
		}

		response = append(response, &BlockStatus{
			BlockHash: BlockHash{
				Slot: blockRequest.Slot,
				Hash: blockRequest.Hash,
			},
			Status:    status,
		})
	}

	return
}

### orc_confirmVanBlockHashes

van > orc : confirmBlockBatch: 'orc_confirmVanBlockHashes', params: [[{slot: 1, hash: blockHash},...]]
    > [{slot: 1, status: 'pending'}, {slot: 2, 'verified'}, {slot: 3, 'invalid'}, ...]

// ConfirmPanBlockHashes
func (api *PublicFilterAPI) ConfirmPanBlockHashes(
	ctx context.Context,
	request []*BlockHash,
)(response []*BlockStatus, err error) {
	if len(request) < 1 {
		err = fmt.Errorf("request has empty slice")

		return
	}

	response = make([]*BlockStatus, 0)

	for _, blockRequest := range request {
		status := Verified

		if MockedHashInvalid == blockRequest.Hash.String() {
			status = Invalid
		}

		if MockedHashPending == blockRequest.Hash.String() {
			status = Pending
		}

		response = append(response, &BlockStatus{
			BlockHash: BlockHash{
				Slot: blockRequest.Slot,
				Hash: blockRequest.Hash,
			},
			Status:    status,
		})
	}

	return
}

### Nice to have
In my opinion if any new pair is confirmed and added to Orchestrator database we should notify Pandora and Vanguard instead of awaiting them to pull and flood us with bunch of redundant requests. It can happen in subscription as the following:

van > orc && pan > orc: new block confirmed
   (OVER PUSH, WSS/IPC; `orc_subscribe`; params; ['orc_newConfirmedPair']
   -> {... "result":"0xcd0c3e8af590364c09d0fa6a1210faf5"}}
   notification: {..., "method":"orc_newConfirmedPair","params":{"subscription":"0xcd0c3e8af590364c09d0fa6a1210faf5","result":{"pair":{
    "slot": num,
    "vanguardHash": common.Hash,
    "pandoraHashes": []common.Hash,
   }, <...>}}))

Representation of `pair` model

{
    "pair":{
        "slot": uint64,
        "vanguardHash": common.Hash,
        "pandoraHashes": []common.Hash,
}

## Block synchronization scenarios

### Orchestrator confirmation to Pandora

@confirmPandoraHash @orchestrator @pandora
@Given as an Pandora I want to insert pending block into my cannonical chain
  And Trusted Pandora notified orchestrator about pending blocks
  And Trusted Vanguard notified orchestrator about pending blocks
@When Orchestrator confirms a block
@Then Pandora inserts block into its cannonical chain

### Orchestrator decline to Pandora

@confirmPandoraHash @orchestrator @pandora
@Given as an Pandora I want to insert pending block into my cannonical chain
  And Trusted Pandora notified orchestrator about pending blocks
  And Trusted Vanguard notified orchestrator about pending blocks
@When Orchestrator decline a block
@Then Pandora removes block from pending queue

### Orchestrator pending to Pandora

@confirmPandoraHash @orchestrator @pandora
@Given as an Pandora I want to insert pending block into my cannonical chain
  And Trusted Pandora notified orchestrator about pending blocks
  And Trusted Vanguard notified orchestrator about pending blocks
@When Orchestrator confirms its pending
@Then Pandora inserts block again into pending queue

## Block production scenarios

### Pending parent discarded

@confirmPandoraHash @doNotProduceBeforeConfirmation @pendingParentDiscarded @pandora
@Given as a Trusted Pandora I want to insert pending block into my cannonical chain
  And Pandora notified orchestrator about pending blocks
  And In current slot Trusted Pandora create the block

@When Pandora prepares the execution block for signature
  And Orchestrator does not confirm pending parent

@Then Trusted Vanguard waits until pending block is confirmed
  And Trusted Vanguard did not resolve pending
  And Trusted Vanguard does not start block production

### Pending parent confirmed

@confirmPandoraHash @doNotProduceBeforeConfirmation @pendingParentAccepted @pandora
Given as a Trusted Pandora I want to insert pending block into my cannonical chain
  And Pandora notified orchestrator about pending blocks
  And In current slot Trusted Pandora create the block

When Pandora prepares the execution block for signature
  And Orchestrator does confirm pending parent

Then Pandora inserts pending parent into cannonical chain
  And Pandora returns work based on cannonical parent

### No pedning parent

@confirmPandoraHash @doNotProduceBeforeConfirmation @noPendingParent @pandora
Given as a Trusted Pandora I want to insert pending block into my cannonical chain
  And Pandora notified orchestrator about pending blocks
  And In current slot Trusted Pandora create the block

When Pandora prepares the execution block for signature
  And there is no pending parent

Then Pandora returns work based on cannonical parent

## Desired behavior of orchestrator

We cannot assume that Orchestrator, Vanguard and Pandora will start silmutanously each time they want to have symbiotic flow. Therefore:

* Orchestrator must be able to fetch not only pending
* Orchestrator should confirm or discard validity of any block from the past
* Orchestrator must be able to schedule and handle reorgs
* Orchestrator must be able to handle shards
* Orchestrator must be able to reconnect if any failure happen on Pandora or Vanguard
* Orchestrator must limit connectivity with untrusted parties