# Enclave Forking

## High Level Overview
* Parent enclave
* SM will mark the parent’s EPM as read-only. (this will be replaced by the snapshot)
* SM will allocate a new read-only
* Child Enclave
* SM will create an EPM region with the same number of EPM as the parent and initialize the enclave with initial registers ($satp, etc)
* All pointers will still point to the parent process
* SM allocates metadata to mark the child enclave as child of parent enclave
* Each child will receive a read-only PMP entry for parent upon enclave switch
* Upon write access to the parent’s EPM pages, SM will get a access fault
* Store/AMO access fault -- Exception code 7
* SM will copy the page
* Note: SM may have to copy multiple pages (multiple forks) to other enclaves
* I.E. Parent enclave spawns forks multiple child enclaves and parent enclave edits a page -- must copy the page to each child it spawns
* Code can be handled in: `void machine_page_fault(uintptr_t *regs, uintptr_t dummy, uintptr_t mepc)`

## Code Modifications

/* enclave metadata */
struct enclave
{
    ...
    enclave_id eid; //enclave id
    ...
    enclave_id parent_eid; // parent eid 
    struct list children; //  list of children eids
}

* Each enclave will now have a parent eid, `parent_eid`
* Default value will be `-1`
* Any enclave that is forked will have its parent eid set.
* Each enclave will now have a list of its children enclaves, `children`
* This isn't completely necessary
* Might be useful to keep track of children eids if we wanted to implement `wait`
* Upon `context_switch_to_enclave`, we also want to flip the PMP regions of the parent enclave
* The PMP regions will grant `read` access to the child enclave, but NOT write.
* Ensures that the host still cannot access this read-only image.
* PMP region will be marked by `enclave[eid].regions`
* The SM is responsible for ensuring fork is correct, so the EID of the of parent enclave is guaranteed to be correct.

`void machine_page_fault(uintptr_t *regs, uintptr_t dummy, uintptr_t mepc)`

`void pmp_trap(uintptr_t* regs, uintptr_t mcause, uintptr_t mepc)`

* `int cpu_get_enclave_id()`
* Use this to find the enclave which caused the access fault
* In the parent, we need to keep track of all dirty pages since we called our snapshot, we can keep track of this through the access fault.
* Can find the `dram_base` of the parent enclave by doing `enclaves[parent_eid].dram_base`
* Check if the faulting address, `mepc`, is within the parent enclave's EPM and that it is a `write` access
* If it is in the parent's EPM, copy the page into the child's EPM.
* This can be done easily by keeping track of the EPM base of each enclave `runtime_pa_params.dram_base`

    offset = (mepc & PAGE_MASK) - runtime_pa_params.dram_base;
    memcpy(enclaves[cpu_get_enclave_id()].dram_base + offset, enclaves[parent_eid].dram_base + offset, PG_SIZE);

* If the `mepc` isn't in the parent's enclave then handle the trap normally

/*
    Creates a copy of the parent enclave (enclave that called fork) 
    and returns the child enclave's eid.  
*/

enclave_id enclave_fork(uintptr_t* regs, enclave_id eid, int load_parameters){
        
    enclave_id child_eid = -1; 
    
   // Copy registers (at time of fork), parameters (dram_base, runtime_base, runtime_entry, etc)
   // Create new enclave with new enclave eid
   child_eid = copy_enclave(regs, load_parameters);
   
   // Add parent enclave's PMP region to child enclave 
   // Set child enclave's parent eid to correct eid
   // Add child to parent enclave's children list
   register_child(child_eid); 
   
   return child_eid; 
        
}                

* We also set `$a0` register to be `0` for the child's process since the return value of the child process upon 'fork()' should be `0`

# Non-determinism (post-init measurement)
* After the child enclave is initialized and created, it might receive input parameters (i.e. in serverless computing, we can provision the enclave with some function)
* At the time of `fork()`, the parent enclave's state may have changed (i.e. stack and heap are modified) since its initial measurement
* We need to be able to attest some intermediete state of the enclave
* Can we use the initial measurement of the parent enclave to help speed up intermeidete hash?

## Solution 1: Selective Measuring
* The enclave application can designate a region which will not be measured.
* For enclave applications that are non-deterministic, we can mark sections in code that won't be included in the intermediete measurement
* This can be a simple ELF section, which must be marked and cannot be modified after the enclave application is initialized in the SM
* This can be implemented by using a bit in the PTE to mark a page to NOT be included in the measurement
* Initial measurement still contains the non-measured section
* Initial measurment ensures the non-measured sections are zero'd out
* Upon any intermediete measurement, marked pages will not be considered.
* This ensures that no adversary modifies the binary when it is initialized by the SM
* Cons:
* An adversary can exploit this by putting injected code in the non-measured pages.
* Adversary can inject code in pages marked as non-measured and create a valid attestation report.
* We can mitigate this with standard buffer overflow protections (i.e. canaries, bounds checking, etc)

## Solution 2: Measure Log between Host and Enclave

* Non-determinism is from outside the EPM (i.e. inputs given from the host)
* SM keeps track of logs sent and received from an external source (i.e. host, enclave, etc.)
* All messages recieved/sent from external source will be logged and included in measurement.
* What about concurrency, where communication between several external sources are in non-determinsitic order?
* We can use a Multi-Set Hash (https://people.csail.mit.edu/devadas/pubs/mhashes.pdf)
* Hashes do not depend on the order of the set
* Cons:
* This doesn't solve non-determinism from the program itself (i.e. calling on a RNG within an enclave)
* We assume ALL non-determinism is from communication