# EPF6 - Week 8 Updates

## tl;dr

- Wrote down [a task list](https://hackmd.io/@junsong/BkpzYgyugl) for SSZ-QL, which will be used for project management.
- Made our [specification document](https://hackmd.io/@junsong/rkAN9lIIxx) better with our mentors' valuable feedback.
- Tackling into SSZ variable-sized types (e.g., List), achieved some progress.

## Details

### Task List

Our mentor [Radek](https://github.com/rkapka) requested us to list up tasks to do, so that we can manage this project in Github. The task list will be a main description of Github Issues, and my PRs will probably link to this meta issue.

Bastin's [list](https://github.com/OffchainLabs/prysm/issues/12991) was great, and I love the structure of the list. Here's my [task list](https://hackmd.io/@junsong/BkpzYgyugl). I'm going to open a new issue for SSZ-QL, and fill up my task list.


I noted a couple of [questions](https://hackmd.io/@junsong/BkpzYgyugl#Questions) about the overall project structure and management, and Radek answered to my questions via Discord thread. It is much clear at this moment!

### Toward Perfect Specification...

Specification must be clear enough to understand and implement. This is challenging because all participants should reach to the common consensus by communicating again and again. Regarding that, I guess Radek's feedback was super helpful to make a better document. You might see his comments in [our document](https://hackmd.io/@junsong/rkAN9lIIxx)!

One of an agenda we discussed is about the endpoint itself. At first, the document only contains one single endpoint for querying `BeaconState`. (`/eth/v1/beacon/states/{state_id}/query`) He suggested to add one more endpoint for blocks (`/eth/v1/beacon/blocks/{block_id}/query`) so that our feature can cover more broad domain. Or we might choose to **consolidate** into one single endpoint, say `/eth/v1/beacon/query`. I agreed with his point, and as we are currently working on a generalized engine for any SSZ object, I'm quite confident to add one more endpoint. It's a low-hanging fruit provided we support the feature for `BeaconState`. For the consolidated endpoint, I would suggest to defer this discussion after the features are fully implemented because we might need to hear from our real users about the usability.

### SSZ: Variable-sized Types

In my last update, I was thrilled to share [my first PoC is done](https://hackmd.io/@junsong/S1UsoDuwel#Proof-of-Concept-completed-and-to-be-continued%E2%80%A6). It supports an arbitrary SSZ object with fixed size. However, it eventually should handle variable-sized types as most of the SSZ object used in the Beacon Chain are variable-sized.

Basically SSZ types like `List` and `Bitlist` are variable-sized. Also, for any `Container` type that **contains** a variable-sized type field, it also becomes variable-sized. `List` is one of the most commonly used data type. [We can see `attesting_indices` in `IndexedAttestationElectra` is a `List` of `uint64`](https://hackmd.io/@junsong/B1v3x4wwle#Annotated-process-with-a-concrete-example-Get-datatargetroot-from-IndexedAttestationElectra).

// Start with a pointer to empty object and calculate SSZ info of `IndexedAttestationElectra`.
info, err := sszquery.PreCalculateSSZInfo(&ethpb.IndexedAttestationElectra{})
require.NoError(t, err, "PreCalculateSSZInfo should not return an error")

I started working on dealing with those types. One fundamental question here: How can I calculate the offset and the length of variable-sized types? As my PoC supports a precalculation phase with an empty object like above, it is impossible to make a guess of the total byte lengths of an arbitrary variable-sized object.

func PopulateFromValue(sszInfo *sszInfo, value any) error {
	if sszInfo == nil {
		return fmt.Errorf("sszInfo is nil")
	}

	// If the type is not variable-sized, we don't need to fill in the info.
	if !sszInfo.isVariable {
		return nil
	}

	if value == nil {
		return fmt.Errorf("value is nil")
	}

	switch sszInfo.sszType {
	// In List case, we have to set the actual length of the list.
	case List:
		listInfo, err := sszInfo.ListInfo()
		if err != nil {
			return fmt.Errorf("get list info: %w", err)
		}

		val := reflect.ValueOf(value)
		if val.Kind() != reflect.Slice {
			return fmt.Errorf("expected slice for List type, got %v", val.Kind())
		}

		if err := listInfo.SetLength(uint64(val.Len())); err != nil {
			return fmt.Errorf("failed to set list length: %w", err)
		}

		return nil
	// In Container case, we need to recursively populate variable-sized fields.
	// Also, it is expected to read an actual offset from the marshalled data.
	case Container:
		marshalledData, err := value.(ssz.Marshaler).MarshalSSZ()
		if err != nil {
			return fmt.Errorf("failed to marshal value: %w", err)
		}

		for fieldName, fieldInfo := range sszInfo.containerInfo {
			childSszInfo := fieldInfo.sszInfo
			if childSszInfo == nil {
				return fmt.Errorf("sszInfo is nil for field %s", fieldName)
			}

			if !childSszInfo.isVariable {
				// Skip fixed-size fields.
				continue
			}

			if len(marshalledData) < int(fieldInfo.offset+4) {
				return fmt.Errorf("marshalled data is too short for field %s", fieldName)
			}

			// NOTE: The offset is always 4-byte sized.
			fieldInfo.actualOffset = bytesutil.FromBytes4(marshalledData[fieldInfo.offset : fieldInfo.offset+4])

			// Recursively populate variable-sized fields.
			fieldValue := dereferencePointer(value).FieldByName(fieldInfo.goFieldName)
			if err := PopulateFromValue(childSszInfo, fieldValue.Interface()); err != nil {
				return fmt.Errorf("populate from value for field %s: %w", fieldName, err)
			}
		}

		return nil
	default:
		return fmt.Errorf("unsupported SSZ type for variable size info: %s", sszInfo.sszType)
	}
}

My approach was to **populate the `sszInfo` with an actual value**. We MUST be able to set the length of `listInfo` after `sszInfo` consturction, so I added some getter/setter functions on several structs. `Container` case was quite tricky: The byte offset is not set at the precalculation phase, so it must be set after populating.

func TestRoundTripSszInfo(t *testing.T) {
	specs := []sszquery_testutil.TestSpec{
		getIndexedAttestationElectraSpec(t),
		getValidatorSpec(t),
		getBeaconBlockHeaderSpec(t),
	}

	for _, spec := range specs {
		sszquery_testutil.RunStructTest(t, spec)
	}
}

I also refactored my testing code to test any SSZ object like above. In the following week, I'm gonna keep adding the test `specs`.