# Delving into dynamic-ssz (`dynssz`)

{%preview https://github.com/pk910/dynamic-ssz %}

I found Philip ([pk910](https://github.com/pk910))'s [`dynamic-ssz`](https://github.com/pk910/dynamic-ssz) is in active development recently. This drew my attention as I'm currently working on [SSZ Query Language](https://github.com/eth-protocol-fellows/cohort-six/blob/master/projects/ssz-ql-with-merkle-proofs.md) for any **arbitrary** SSZ object. It's well-documented so that I could learn a lot from this project as well as the rationale for each parts.

## What problem does this project solve?

[`fastssz`](https://github.com/ferranbt/fastssz) is widely used in Go ecosystem regarding SSZ. Prysm also currently uses forked version of it ([OffchainLabs/fastssz](https://github.com/offchainlabs/fastssz/)). `dynssz` leverages on `fastssz` for static encoding/decoding, while also supports dynamic specifications and configurations.

One common usecase is for mainnet/minimal presets. Minimal presets often choose much smaller value for ease of testing. `MAX_COMMITTEES_PER_SLOT` reduces from `64` to `4` when it comes to minimal.

class Attestation(Container):
    # [Modified in Electra:EIP7549]
    aggregation_bits: Bitlist[MAX_VALIDATORS_PER_COMMITTEE * MAX_COMMITTEES_PER_SLOT]
    data: AttestationData
    signature: BLSSignature
    # [New in Electra:EIP7549]
    committee_bits: Bitvector[MAX_COMMITTEES_PER_SLOT]

Using `dynssz`, we can provide different spec values and encode **dynamically** with respect to the context. For marshalling `Attestation`, we **should** know the length of `committee_bits` which is `MAX_COMMITTEES_PER_SLOT`. `dynssz` supports custom Go struct tags like `dynssz-size`, so we can annotate `committee_bits` as `dynssz-size:"MAX_COMMITTEES_PER_SLOT"`.

## Key component: `TypeDescriptor`

// TypeDescriptor represents a cached, optimized descriptor for a type's SSZ encoding/decoding
type TypeDescriptor struct {
	Type                   reflect.Type
	Kind                   reflect.Kind         // Go kind of the type
	Size                   uint32               // SSZ size (-1 if dynamic)
	Len                    uint32               // Length of array/slice
	Limit                  uint64               // Limit of array/slice (ssz-max tag)
	ContainerDesc          *ContainerDescriptor // For structs
	ElemDesc               *TypeDescriptor      // For slices/arrays
	HashTreeRootWithMethod *reflect.Method      // Cached HashTreeRootWith method for performance
	SszType                SszType              // SSZ type of the type
	IsDynamic              bool                 // Whether this type is a dynamic type (or has nested dynamic types)
	HasLimit               bool                 // Whether this type has a limit (ssz-max tag)
	HasDynamicSize         bool                 // Whether this type uses dynamic spec size value that differs from the default
	HasDynamicMax          bool                 // Whether this type uses dynamic spec max value that differs from the default
	HasFastSSZMarshaler    bool                 // Whether the type implements fastssz.Marshaler
	HasFastSSZHasher       bool                 // Whether the type implements fastssz.HashRoot
	HasHashTreeRootWith    bool                 // Whether the type implements HashTreeRootWith
	IsPtr                  bool                 // Whether this is a pointer type
	IsByteArray            bool                 // Whether this is a byte array
	IsString               bool                 // Whether this is a string type
}

Each SSZ data structure can be represented as one instance of `TypeDescriptor`. It contains all information needed for dealing with SSZ and merklelization.

// buildTypeDescriptor computes a type descriptor for the given type
func (tc *TypeCache) buildTypeDescriptor(t reflect.Type, sizeHints []SszSizeHint, maxSizeHints []SszMaxSizeHint, typeHints []SszTypeHint) (*TypeDescriptor, error) {
    // so on...   
}

`buildTypeDescriptor` ([Link](https://github.com/pk910/dynamic-ssz/blob/2bc682ea7e2a39f7e0c37d83b8700954c79ba69a/typecache.go#L140)) builds the actual `TypeDescriptor` if the given `reflect.Type` was not "hot" in `TypeCache`.

	if sszType == SszUnspecifiedType {
		switch desc.Kind {
		// basic types
		case reflect.Bool:
			sszType = SszBoolType
		case reflect.Uint8:
			sszType = SszUint8Type
		case reflect.Uint16:
			sszType = SszUint16Type
		case reflect.Uint32:
			sszType = SszUint32Type
		case reflect.Uint64:
			sszType = SszUint64Type

		// complex types
		case reflect.Struct:
			sszType = SszContainerType
		case reflect.Array:
			sszType = SszVectorType
		case reflect.Slice:
			if len(sizeHints) > 0 && sizeHints[0].Size > 0 {
				sszType = SszVectorType
			} else {
				sszType = SszListType
			}
		case reflect.String:
			if len(sizeHints) > 0 && sizeHints[0].Size > 0 {
				sszType = SszVectorType
			} else {
				sszType = SszListType
			}
			desc.IsString = true

		// unsupported types
		default:
			return nil, fmt.Errorf("unsupported type kind: %v", t.Kind())
		}

		// special case for bitlists
		if sszType == SszListType && strings.Contains(t.Name(), "Bitlist") {
			sszType = SszBitlistType
		}
	}

	desc.SszType = sszType

It first determines `SszType` by the `Kind` of `reflect.Type`. Then, in accordance with `SszType`, it builds the descriptor recursively. For example, `Container` type will be handled like:

	case SszContainerType:
		err := tc.buildContainerDescriptor(desc, t)
		if err != nil {
			return nil, err
		}

To calculate the size or marshal/unmarshal an object, getting a type descriptor must precede. For example, `SizeSSZ` uses `sourceTypeDesc` and then passes it into `getSszValueSize`, which means it should 1) get from the type cache or 2) build a type descriptor from scratch.

func (d *DynSsz) SizeSSZ(source any) (int, error) {
	sourceType := reflect.TypeOf(source)
	sourceValue := reflect.ValueOf(source)

	sourceTypeDesc, err := d.typeCache.GetTypeDescriptor(sourceType, nil, nil, nil)
	if err != nil {
		return 0, err
	}

	size, err := d.getSszValueSize(sourceTypeDesc, sourceValue)
	if err != nil {
		return 0, err
	}

	return int(size), nil
}

## Some breakthroughs

### `uint256` Support

{%preview https://github.com/pk910/dynamic-ssz/blob/master/docs/api-reference.md#handling-large-integers-uint128uint256 %}

As Go doesn't support `uint128` and `uint256` in its native type system, `dynssz` handles them as a byte array. For example, `uint256` is represented as `[32]uint8 or [4]uint64`.

// detect some well-known and widely used types
if t.PkgPath() == "github.com/holiman/uint256" && t.Name() == "Int" {
	sszType = SszUint256Type
}

Also, there's a code for determining the type with its package name, which seems quite hacky but useful.

### Distinguish ambiguous type (e.g., `List` vs. `Bitlist`)

func (d *DynSsz) getSszTypeTag(field *reflect.StructField) ([]SszTypeHint, error) {
	// parse `ssz-type`
	sszTypeHints := []SszTypeHint{}

	if fieldSszTypeStr, fieldHasSszType := field.Tag.Lookup("ssz-type"); fieldHasSszType {
		for _, sszTypeStr := range strings.Split(fieldSszTypeStr, ",") {
			sszType := SszTypeHint{}

			switch sszTypeStr {
			case "?", "auto":
				sszType.Type = SszUnspecifiedType
			case "custom":
				sszType.Type = SszCustomType
			case "wrapper", "type-wrapper":
				sszType.Type = SszTypeWrapperType

			// basic types
			case "bool":
				sszType.Type = SszBoolType
			case "uint8":
				sszType.Type = SszUint8Type
			case "uint16":
				sszType.Type = SszUint16Type
			case "uint32":
				sszType.Type = SszUint32Type
			case "uint64":
				sszType.Type = SszUint64Type
			case "uint128":
				sszType.Type = SszUint128Type
			case "uint256":
				sszType.Type = SszUint256Type

			// complex types
			case "container":
				sszType.Type = SszContainerType
			case "list":
				sszType.Type = SszListType
			case "vector":
				sszType.Type = SszVectorType
			case "bitlist":
				sszType.Type = SszBitlistType
			case "bitvector":
				sszType.Type = SszBitvectorType

			default:
				return nil, fmt.Errorf("invalid ssz-type tag for '%v' field: %v", field.Name, sszTypeStr)
			}

			sszTypeHints = append(sszTypeHints, sszType)
		}
	}

	return sszTypeHints, nil
}

`dynssz` introduces a new Go Struct Tag `ssz-type` for removing ambiguity. In my past experience, it is quite hard, or to be clear, *impossible* to determine between `List` and `Bitlist` from the given struct definition in Prysm. I think this explicit tag can help, while we need to fix the `sszgen` code for supporting it.

## Key takeaways (for SSZ-QL implementation and Prysm context)

- Distinguish between `Uint8 ~ Uint64`, not making it as sole type `UintN`.
- For `uint256`, we don't use `holiman/uint256`. Maybe I need to whitelist field names that use `uint256`...?
- For distinguishing `Bitlist` with `List`, I think I might use the type name like:

// special case for bitlists
if sszType == SszListType && strings.Contains(t.Name(), "Bitlist") {
	sszType = SszBitlistType
}