# Open Enclave SDK SIG-Architecture Meeting

**When:** weekly at 10am Tuesday, US West Coast [(convert to your local timezone)](https://www.thetimezoneconverter.com/?t=10:00&tz=PT%20%28Pacific%20Time%29)

Join Zoom Meeting (updated link on July 7)
https://zoom.us/j/95309871627?pwd=K1RmbmZtUUowNFhRbWFZRVN4R2VmUT09

Meeting ID: 953 0987 1627
Password: 208079

## Scope and Mission

This meeting is a place for technical discussions related to project architecture.

Rather than having informal architecture discussions during the Triage meeting, we are creating this meeting as a dedicated space for all interested parties and stakeholders to come together and drive consenus on difficult and often cross-domain questions. Broadly speaking, if it might relate to a [design document](https://github.com/openenclave/openenclave/tree/master/docs/DesignDocs), it's probably in scope for this meeting.

The Open Enclave project values being a friendly and inclusive space. This meeting is open to all community members. Everyone is asked to abide by the procedure documented below, and reminded that the community follows a [Code of Conduct](https://github.com/openenclave/openenclave/blob/master/docs/CodeOfConduct.md).

## Procedural Guidelines

The chair of the next meeting should attempt to **establish an agenda before the meeting**, but attendees may (and should) add to the agenda themselves in this shared document. hackmd.io is a collaborative editing tool, and everyone is welcome to edit this in real-time as well.

**At the start of the meeting**, the chair should ask for a volunteer to take minutes and roll call. Minutes should capture any decisions made and action items discussed. Minutes are stored in this document, and are public.

The chair should **get consensus on the agenda**, editing as appropriate. It is the chair's job to ensure that the meeting then follows the agenda, and that everyone is given an equal chance to speak.

Because the community is distributed by nature and this meeting may not include everyone every time, any decision must be followed up with an artefact (for example, a PR or an email to oesdk@ mailing list) to record the outcome.

## Agenda for the next meeting

## Meeting Notes
### 1 November 2022 10:00 AM PST
Chair: Radhika
Note taker: Mark

* Opens
* Dave: There is a hackathon going on this weekend at IETF where RATS people will be there will anyone else on this call be participating even if remotely? See the "Attestation and TLS" section at https://wiki.ietf.org/en/meeting/115/hackathon

* Interoperable attested TLS and design for Open Enclave SDK https://github.com/CCC-Attestation/meetings/blob/main/materials/ShanweiCen_Interoperable_ATLS.pdf (Shanwei presenting)
* Proposal has been reviewed in CCC Attestation SIG - recommendation is to move ahead. Some projects already have pull request (RATS-TLS) to adopt the concept.
* Existing RA-TLS libraries for Gramine, OE Attested TLS, and RATS-TLS are based on same whitepaper, but each implementation has implemented its own X.509 cert extension OIDs and evidence formats. They also implement their own cert generation and libraries.
* Thus, each project cannot interoperate with other projects. For example, Gramine cannot establish an RA-TLS connection with OE.
* Additionally session freshness is not addressed in whitepaper.
* Goal is interoperability (see slides), and to align with existing TLS and standards.
* Dave: is this proposal the same as: https://www.ietf.org/id/draft-fossati-tls-attestation-02.html#name-authors-addresses-2 ?
* Shanwei: it is not the same. Thomas' proposal is to extend TLS protocol to support new cert / evidenc types that are not X.509. Our RA-TLS proposal uses the same X.509 cert, just unify and standardize its evidence extension.
* Dave: we should work together with IETF. Who from Intel is participating?
* Shanwei: Ned Smith. Shanwei will check with Ned to make sure that we are aligned.
* Radhika: has somebody tried to use RA-TLS between different SDKs and found this to be a problem?
* Shanwei: No, but the CCC Attestation SIG has identified this issue and suggested that this is needed after a review of projects.
* Rathna: would all projects need to use the same CBOR Tag?
* Shanwei: Yes, projects would use the same OID and registered CBOR Tags
* OE support: Objective is to maintain backward compatibility with current OE Attested TLS support while adding support for Interoperable Attestated TLS (see slides)
* Rathna: there are multiple CBOR tags and data/serialization methods - would we need to have both generation and verification for all of these in OE?
* Shanwei: yes, you would need to support CBOR which has tags to identify how each element is serialized.
* Dave: in middle SGX/TDX Evidence and Endorsements example, why do you need two entries in the array rather than a single blob.
* Shanwei: to tell whether the claims buffer is OE serialized or CBOR serialized. We are using the CBOR tag as the format id.
* Dave: you could use a flag with existing format ids
* Shanwei: existing OE format IDs are OE specific. Preference is to use standardized CBOR tags and flags.
* Shanwei: Propose that OE add support for this solution:
* Radhika: do you want to make a design PR or provide the presentation on a public place.
* Shanwei: Can help to create a design PR, but will look for help
* Radhika: prefer Shanwei to do the design PR.
* Shanwei: Design PR created: https://github.com/openenclave/openenclave/pull/4653

### 16 August 2022 10:00 AM PST
Chair: Radhika

* Opens
* EDMM update
* Update on EDMM Kernel Status: code is integrated into the mainline: unlikely that we will have any changes to the Kernel API
* Update status on EMM source code repo: https://github.com/intel/sgx-emm
* Design Doc: https://github.com/intel/sgx-emm/blob/main/design_docs/SGX_EMM.md
* No recent API changes since the last approval of the interfaces
* Ming-Wei: Would like the mm_init function to return a status:
* ACTION: Haitao: review mm_init request and will reply in the PR. Can resolve this offline.
* Ming-Wei: all other APIs look good. If issues are discovered during OE design, then can raise an issue in the sgx-emm repo
* Can get agreement and merge the PR offline.
* Current plan is to release Intel SGX SDK with sgx-emm as a submodule in Q4.
* Enclave Common Loader APIs: see https://github.com/intel/linux-sgx/blob/edmm_v5/psw/enclave_common/sgx_enclave_common.h#L271
* Plan to add an additional "enclave_error" parameter to return an error code to functions enclave_modify and enclave_alloc. The parameter will return the currently defined error codes (which show as return values now). This is not in the code yet.
* Radhika: will enclave_common changes be backward compatible?
* Mark: Yes, it will be a minor version update.
* Ming-Wei: will Windows Enclave Common Loader be provided?
* Mark: Windows and Linux Enclave Common Loader are able to provide the same interface and abstract out the differences in the kernel interface with some minor exceptions (mainly error codes that may be returned).
* Action: Mark: provide documentation of differences in Windows and Linux Enclave Common Loader
* Action: Mark: look into open sourcing the Windows Enclave Common Loader
* Next steps for Open Enclave.
* Review EDMM implementation design. Future meeting.

### 12 July 2022 10:00 AM PST
Chair: Radhika

* Opens
* https://github.com/openenclave/openenclave/issues/4546: Review openssf scorecard badge
* We will discuss this at a later meeting.
* EDMM Update (Haitao/Mark)
* EDMM code for the kernel has been committed to the 'tip tree' - this is to be merged for the 5.20 release. (https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/sgx)
* Github Repo for EMM Source code: we will put common code in a separate repo. This can be pulled in as a submodule. Haitao is in the process of moving this code.
* The design document and driver interface will be placed in this repo.
* Enclave Common Loader APIs: see https://github.com/intel/linux-sgx/blob/edmm_v5/psw/enclave_common/sgx_enclave_common.h#L271
*

### 28 June 2022 10:00 AM PST
Chair: Mark
* Opens:
* Introduce https://github.com/openenclave/openenclave/issues/4546: this can be discussed in July 12th meeting.
* Dave: Open Source Security Foundation has a best practices for securing Open source SW. One element is a badge - the PR adds the scorecard badge.
* We would like to go through the "passing" sections of the scorecard form in the next meeting (July 14th)
* ACTION: review the form and fill in any answers - if you do not have edit permissions, then send Dave Thaler email for access
* EDMM Discussion:
* Github Repo for EMM Source code: Haitao is working through the process to get a repo for this code
* Haitao has added some comment to the Linux Driver Interface documentation.
* Review Enclave Common Loader APIs - supports both Linux and Windows (including EDMM)
* Small adjustments to EMM OCall APIs
* Update on other EDMM ACTION items from June 7th

### 7 June 2022 10:00 AM PST
Chair/Minutes: Mark
* EDMM update (Haitao Huang): review memory manager design with latest kernel interfaces. Haitao reviewed the EDMM design. The core of the design is a common memory manager which exposes a set of APIs to allocate/deallocate/modify memory. This design can be common to many SDKs. Upper layer components to manage heap, add threads, etc. can use the memory manager. There is an additional abstraction layer for OCalls and additional functions within sgx_enclave_common to abstract calls to the kernel (Linux/Windows)
* Existing design doc PR in OE: https://github.com/openenclave/openenclave/pull/3991
* Anand: is there an API to determine whether EDMM is available at run-time?
* Haitao: no, enclave assumes that EDMM is supported.
* Dave: are white boxes (see diagram) common?
* Haitao: they are just the API definitions (they are common, so they should be green)
* Haitao: changes since last review:
* added a new initialization API sgx_mm_init
* removed dependency on malloc. The EMM now uses a very simply malloc for internal allocation for bookkeeping related memory consumption.
* Kernel API has stabilized (discusions on LKML)
* Dave: does Windows support EDMM?
* Mark: yes, but we (Intel) need to abstract the APIs in the enclave common loader for Windows.
* Dave: this is a document about the APIs that OE can use for EDMM. Do not have any issues merging it.
* Radhika: do not have any problems merging it to OE since OE uses it.
* Ming-wei: would like a seperate repo for this library so that we can declare it as a sub-module (for the code). Documentation can also go there (see below notes)

* Current implementation of EDMM memory manager is: https://github.com/intel/linux-sgx/pull/838
* Haitao: Currently the code is in a branch of the Intel SDK repository located here: https://www.github.com/intel/linux-sgx/tree/edmm_v5/sdk/emm
* Dave: can put the design docs under the above location. Can use a doc-link to link the document into the OE SDK repo
* AGREED: we will merge the design document when it is approved (PR 3991). Eventually, we will put this document in the repository which hosts the code and then link to it from the OE repo.
* Options for release of code:
* OE builds itself: want a separate repo and import it as a submodule
* AGREED: preference is for this option.
* OE just uses binary: Intel releases with the SDK
* Radhika: concern about build flags.
* Ming-wei: this will be best because the Intel SDK will test it.
* Dave: would be simpler to consume as source-code for code within the enclave.
* SGX2 EDMM: https://github.com/openenclave/openenclave/pull/3639
* Need to move forward with Section 4 (EDMM Enabled Features) in the proposal.
* Mark: can some of this be common code between the SDKs?
* Haitao: can look at MystikOS code for mmap, mprotect to see if it is reusable.
* Ming-wei: current APIs may be sufficient for the SDK. We don't necessarily need to implement all features as long as the libOSs can provide these (ex. creating threads)
* Dave: using common functions such as mmap and mprotect would be good in the SDK since they are TEE agnostic. The initial implementation can just call the sgx specific functions for SGX and fail for other TEEs.
* Dave: this would be a good document to link to other EDMM component designs.
* EDMM Driver Interface: https://github.com/openenclave/openenclave/pull/3755
* Should this document be updated?

### 17 August 2021 10:00 AM PST
Chair: Anand
* EDMM follow up: SGX1 support in the proposed memory manager
* Changes to edger8r generated code (Deferred to v1.0)
* Are they breaking changes?
* Should host apps be able to load enclaves built with other versions?
* [Dave] My opinion: yes, don't assume that host and enclave are built by the same person/organization at the same time. If that is too hard, workaround is to make edger8r-generated host code compile into a static lib that can be redisted in source or binary form rather than redisting the EDL. (Today edger8r does not generate the code to build a static lib that for you, hence the problem.)

### 27 July 2021 10:00 AM PST
Chair: Manoj
Minutes: Radhika

* EDMM follow up
* Review Updates (https://github.com/openenclave/openenclave/pull/3991)
* Haitao:created an abstraction layer so the same edmm lib between intel sgx sdk and oe sdk
* Dave: the header with the APIs that both OE and Intel SDKs would implement will be included via a submodule in OE
* Dave: recommend that we prefix all typedefs with "sgx_"
* Anand: the mutex_type for oe is different - oe_mutex, however the header references sgx_mutex. Haito: this is not a problem since we are only using pointers to sgx_mutex.
* Anand: will this lib work with SGX1 also? Haitao: wont break anything, but will not do anything on SGX1
* Anand:
* Related comment: https://github.com/openenclave/openenclave/issues/2023#issuecomment-613316501
* IceLake machines support large amounts of EPC (100+ GBs). Can the APIs proposed for SGX2/EDMM also work (be made to work first) for IceLake SGX1 enclaves? https://github.com/openenclave/openenclave/pull/3991
* Anand: Can we unify the mmap interface that Mystikos has? Haito: will look into this
* Anand: Would be great if memory manager(mmap) supports SGX1 as well, that way we can test with current drivers
* Bo will review and provide feedback offline
* Next steps before kernel is ready.
* Haitao: we will start prototyping and going through scenarios before the kernel is upstreamed. Kernel is being upstreamed in Jan.
* Haitao: VDSO is already in current kernel and can be used with SGX1
* Haitao: Untrusted side signal handler needs to be changed to use a callback
* Ming-WEi: When we update the kernel we have to update the signal handler.
* Haito: 5.11 kernel has VDSO interface
* Ming-Wei: Please file an issue for the signal handler
* Haitao: Intel SDK when VDSO is detected, we switch to a callback instead of singal handler. WE have both.
*

### 25 May 2021 10:00 AM PST
Chair: Manoj
Minutes: Radhika
* OE SDK meetings will remain weekly
* If there are no agenda items by 10:00 AM PST, the chair will cancel the meeting
* If the chair cancels the meeting, they will continue to chair for the next meeting
* If the meeting is not canceled, we will go over TODO on SIG-Arch board
* Watch the [e-mail] mailing list for cancellations. Do not rely on the outlook invite to be canceled.

### 4 May 2021 10:00 AM PST
Chair: Rathna
Minutes: Radhika
* Add SGX Enclave Memory Manager (Initial EDMM Support) https://github.com/openenclave/openenclave/pull/3991 step through proposed APIs (Haitao, MarkS)
* Mark recommends adding references flows from EDMM kernel design doc to the PR above.
* Mark: also add a flow for EACCEPT copy.
* Dave: Will these APIs be provided by a lib in a separate repo? Where will that repo reside? What will the license be?
* Mark: will take questions, will need to build a common model and abstract it out for Intel SDK or OE SDK.
* Dave: if you put it in OE SDK, there are some style conventions.
* Ming-WEI: These APIs need to be implemented by typing into internal OE implementation
* Radhika: How will these APIs be used in OE
* Haitao/Ming-Wei/Mark: when yoou call malloc, it calls the heap manager which will call these APIs
* Ming-Wei: What is the plan on this proposal?
* Mark: Will get approval to and make it experimental
* Dave: dont need it to be experimental if you have a different lib in a different repo which is added as a submodule.
* Haitao: Will define an abstraction layer for a lib that OE and Intel SDK to call in to.
* Mark: Can implement page permissions first and then go on
* Haitao: Will be back in a week or two

* Time permitting: When can we declare v1.0? #3996
* Dave: Production services are using OE, so sohuld be v1.0
* Radhika: MAA is requesting breaking changes
* Radhika: v1.0 comes at a higher cost

### 27 April 2021 10:00 AM PST
Chair: Ming-Wei
Minutes: Radhika
* https://github.com/openenclave/openenclave/issues/3807 : What is the current long term support branch?
* For mbedtls, we take the latest LTS version, which is 2.16.9 (recently updated by Manoj). That has their latest bug fixes as mbedtls maintains fixes for 2.7 LTS, 2.16 LTS and 2.x. We do not take their dev head version (2.26.0) with breaking changes. Their next LTS version is looking to be 3.0.0, which was supposed to be Q3 of 2020 but has slipped out to sometime in 2021.
* Went over backlog

### 20 April 2021 10:00 AM PST
Chair: Radhika
Minutes: Mark

* https://github.com/openenclave/openenclave/issues/3953 Support release mode enclaves #3953
* Dave has suggested to add OE_ENCLAVE_FLAG_AUTODEBUG which sets the FLAG_DEBUG parameter to Debug=1 if set and Debug=0 if not set.
* Dave: if the user code is loading the enclave, then it can detect the value of Debug and load the enclave with OE_ENCLAVE_FLAG_DEBUG based on that value.
* Xuejun: agree.
* Xuejun: How do you select the crypto for the enclave:
* Ming-Wei: There is a link variable (OE_CRYPTO_LIB) which will cause you to link against OpenSSL

### 13 April 20201 10:00 AM PST
Chair: Francisco
Minutes: Radhika

* **0-based SGX-Enclaves**:
* https://github.com/openenclave/openenclave/pull/3898
* Francisco: Overload oe_Create_Enclave and use -1 for base address so you use only one API
* Action: Mark and Francisco to add comments
* Action: Mark and Francisco will make a PR on PSW to meet these requirements.
* MArk: Further requirements: two enclaves with 0 based addresses.Neither solution(from OE or Intel SGX SDK) meets this requirement. Need to consider this in the design.
* **Mechanism for adding extra enclave data**:
* https://github.com/openenclave/openenclave/pull/3921
* Will defer to next week
* **Enable SIGSEGV handling inside SGX Enclave**:
* https://github.com/openenclave/openenclave/issues/3923
* Will defer to next week
* Mark: What is the next step on this?
* Mark: Agreement that this should be implemented.
* Publish oecert and oecert_enc.signed to /opt/openenclave/bin:
* https://github.com/openenclave/openenclave/pull/3947
* Do we need formal key to sign this enclave?
* Yen: This is an enclave that we ship with the SDK
* This is a debug tool with an enclave in it. We want developers to be able to run this tool
* Want to publish this utility
* Whose enclave is it? We use a random key
* Yen Lee: Will document and add a printout for users to know that they should not use this for production purposes.
* Dave: In the future would like all VS extension/VSCode extension and shipping binaries and utilities to use an OE key and not a MSFT key.
* Dave: Need an issue to track to sign enclaves and tools using non MSFT key.
* Dave: Agrees with Ming-Wei's comment about location of published tool
* Dave: Are other binaries signed? NuGet Packages and APT packages are signed with what key? Rob to weigh in.
* https://github.com/openenclave/openenclave/issues/3953
* Dave: Can you call oe_create_enclave for a release signed enclaves
* Mark: If you build and sign an enclave with DEBUG=0, then can you call oe_create_enclave in debug mode?
* DEBUG=1 means both possibilities are OK
* DEBUG=0 means you cannot load the enclave in debug mode
* Dave: do we have a tool that dumps sigstruct?
* This is a reasonable request if we dont have such a tool

### 06 April 20201 10:00 AM PST
Chair: Manoj
Minutes: Mark

* **0-based SGX-Enclaves**:
* https://github.com/openenclave/openenclave/pull/3898
* Bo: Has addressed several comments from reviews. Legacy code is outside the scope of the PR.
* Windows: still requires exploration.
* Mark: The Windows CreateEnclave() API interprets a base address of zero as a request for the OS to choose the enclave base address.
* Mike: agree - probably need a change to Windows to support this feature.
* ACTION Bo: will state that this feature is Linux only
* Added future exploration - two way sandbox
* Anand: concern that marshalling data could be in the enclave ELRANGE.
* Bo: agree, this should be addressed in the design of the shared buffer.
* Mike: maybe the user could just mmap the ELRANGE pages as none to prevent marshalling structures from overlapping with ELRANGE.
* Can find minimum mmap address from the kernel and map pages above this address.
* Bo: looking for reviewers to approve the PR

* **Mechanism for adding extra enclave data**:
* https://github.com/openenclave/openenclave/pull/3921
* Mike: Would like to have full control over the enclave layout.
* When OE populates an enclave, it adds program segments, then heap, then threads which contain TCS, stack etc.
* Would like to add regions between the program text segment and the heap.
* oe_load_extra_enclave_data_hook() function allows the host to inject extra data
* Mark: suggested that constructs needed for EDMM (for the enclave to manage its memory layout) may perform some of the functions.
* Mike: want these changes to be as small as possible to prevent interference with future features. If there is a method of defining regions within an enclave, then we could probably use this instead.
* Anand: changes would be needed to oesign to support this feature:
* Mike: Yes, there were changes to oesign. ACTION: Mike will update the document with the changes which were made to oesign
* Anand: would this interfere with FIPS work?
* Ming-wei: no this will not interfere
* Mike: suggested that EEID could be backed out as there is not a user. The feature is experimental. ACTION: Mike will write an issue about this.
* Ming-Wei: this feature is a superset of the EEID feature. Mike: agree. This feature could be used instead.

* **Enable SIGSEGV handling inside SGX Enclave**:
* https://github.com/openenclave/openenclave/issues/3923

### 30 March 20201 10:00 AM PST
Chair: Anand

* **apkman demo**
* Demo/Usage Page: https://github.com/anakrish/apkman

* **0-based SGX-Enclaves**: Follow ups
* https://github.com/openenclave/openenclave/pull/3898

### 23 March 2021 10:00 AM PST
Chair: Francisco

* **Debugger Support for Multi-Module Enclaves**
* https://github.com/openenclave/openenclave/pull/3838
* Request folks to review this offline and post any feedback/approve.
* Anand has updated the PR based on feedback and requests Mark and Dave to get feedback

* **0-based SGX-Enclaves**
* https://github.com/openenclave/openenclave/pull/3898
* 0 based enclave is an advanced feature and not a default recommendation
* There are two separate features
* Do we want to expose the ability for an app developer to control start address? Is it insecure? why does start offset need to be part of identity
* Is there a recommendation for how to tell people how to set start address
* Bo: If it is a low address, Linux and Windows host would probably deny
* Bo: Set it relatively big.
* Simon: Capture in doc what the range of addresses should be.'
* Dave: flag should use 1 bit instead of 32 bits. Use bitfield instead of debugger flags.
* Mark has tried this on Linux
* Mark: On Windows, needs a closer look. Proposal may not work on Windows.
* Simon: Can be a Linux only feature.
* Anand: We use clang's speculative load hardening. To use speculative load hardening, we should use this feature. clang requires 2GB of memory to be mapped in a particularway. The lower 2GB needs to reserved.
* Simon: This should be rolled into documentation

* **apkman**: Package Manager and Toolbox for Enclave Development
* Should we go over this instead in a Sig Dev-Tools meeting?
* Usage Demo: https://github.com/anakrish/apkman

* **Next SIG-arch chair**
* Informal vote on designated chair for SIG-arch to update in docs.

### 16 March 2021 10:00 AM PST
Chair: Simon

* Support for FIPS Certified Crypto Module:
* https://github.com/openenclave/openenclave/pull/3874
Dave & Anand - No technical revisions required. Looks good.

* Close on Sealing APIs ([#3652](https://github.com/openenclave/openenclave/pull/3652))

Cedrick - Investigating build errors.

* EDMM support
Simon - Can we bring next steps of EDMM support to next sig arch meeting.

### 9 March 2021 10:00 AM PST
Chair: Mark
Minutes: Francisco
* Brainstorming: Process for upgrading tools and 3rd party libraries
* https://github.com/openenclave/openenclave/issues/3815
* Doc: https://github.com/openenclave/openenclave/issues/3815
* Notes:
* Alex: Will get back next week to see which flags that were listed are required. Main driving feature is co-routines. Also, stack-smash protection, C++20 support, control-flow check.
* Alex, Anand: Users should be able to use those flags on their own. We don't currently believe the flags need to be added to the project itself.
* Radhika: After building SDK with Clang8 could not build samples using Clang7. TODO: Need to check if we can make the statement "Minimum version of Clang required will be Clang10" (Clang10 or higher)
* Anand: C code should be OK, but C++ may not be.
* Radhika: During the last upgrade, it was the C++ code that had issues.
* Dave, Simon: Can we dictate that VS2017 is no longer supported and only support VS2019, since VS2019 supports Clang11? Especially, given that some extensions are not compatible with both versions?
* Direction: Move VS and VSCode Extensions to reference Clang11. Will note in end-of-March release that it will be the last one supporting Clang8, that they should look to Clang11. The subsequent release will target Clang11 (barring any issues discovered during the Clang11 investigations).

* Design: Debugger Support for Multi-Module Enclaves:
* PR: https://github.com/openenclave/openenclave/pull/3838
* Pending discussion: Attach scenario edge-case. Behavior when debugger attaches to an application while the runtime is in the process of populating the list of modules/notifyin the debugger:
https://github.com/openenclave/openenclave/pull/3838#discussion_r589910826
* Notes:
* Mark: Do not see an issue with this edge-case.
* Anand: Previously used approach is to allow reception of two notifications and ignore 2nd notification if it is a duplicate. Similarly, a debugger can ignore a notification for a module that no longer exists (was already removed).
* Support for FIPS Certified Crypto Module:
* https://github.com/openenclave/openenclave/pull/3874
* Notes:
* Ming-Wei:
* The thing that is being FIPS certified is the single shared library. Shared library will not extend to generic support for loading a shared library.
* Key restrictions being proposed is to inject a static library during enclave creation.
* Simon:
* What is the user experience during any failures when creating these enclaves?
* Anand:
* You can enforce some of these in the loader
* Radhika:
* The loader can have meaningful error message.
* Arthur:
* A tool could examine the binary to check to see if it would meet the criteria/restrictions set on these types of binaries.
* Dave:
* This information is untrusted. Can an attack be made here?
* Ming-Wei:
* OEHost Lib can handle this, the information is measured.
* Mark, Ming-Wei: Lots of work being done around relocation. Should we include ELF parser in the enclave?
* Simon, Radhika: Sig-Arch should have a more in-depth discussion about loader. We can separate this FIPS feature from the overall loader discussion (requirements, customer interest, development/maintenance cost). Anand will file an Issue for the longer term loader roadmap. Mark suggested that dynamic loading may benefit from taking a longer look at the loader.
*

### 2 March 2021 10:00 AM PST
Chair: Ming-Wei
Minutes: Manoj
* Design: Debugger Support for Multi-Module Enclaves:
* PR: https://github.com/openenclave/openenclave/pull/3838
* Doc:
https://github.com/openenclave/openenclave/blob/67bb39f9bc0b05ba8332e2a76bdaa66e94dc82b5/docs/DesignDocs/DebuggerSupportForMultiModuleEnclaves.md
* Mark: Will update the review comments on document day after.
* Close on Sealing APIs ([#3652](https://github.com/openenclave/openenclave/pull/3652))
* Simon: Seal.h header file should not be in bits. Move it to include folder.
* Dave: Preference to have less number of header files to be included manually by developer.
* Cedrick: Don't include it in common header file like enclave.h. As user has to manually opt into this plugin , hence it is acceptable for developer to include seal.h mnaully in code. Agreed to implement this way.
* Simon: Dont use OE_FAILURE. It is very generic and cannot map which module or context thrown this error. Use OE_UNSPPORTED.
* Cedrik: Will make final changes and will submit the PR. Will file a bug for doxygen related error.
* Brainstorming: Process for upgrading tools and 3rd party libraries
* https://github.com/openenclave/openenclave/issues/3815
* Doc:
https://github.com/openenclave/openenclave/issues/3815
* Anand : We are seeing issues on OE SDK due to outdated libraries. Want to use this topic to come up with a process on how to update 3rd party libraries regularly.
* Dave: Use release cycles as a check point to check the versions of 3rd party libraries and dependencies. As part of the next release cycle planning, brainstrom about possible library updates.
* Anand: Do we have a preference to which clang version we need to update to for building SDK. Each ubuntu version has its own default clang version.
* Dave: Is it possible to say minimum version or higher. Mention in support doc clang 8 or higher.
* Simon: If it is only single version, pick the higher version
* Anaand: Ubuntu 16 dont support clang 11
* Simon: Can we package the clang 11 and ship it irrespective of ubuntu supportability?
* Anand: Spectre mitigation clang 7 has hidden flag for it. The clang 7 may not have all the security flags that clang 8 support. Users can have a work around it but tiresome.
* Simon: To bypass all this struggle, provide a build environment in the container format.
* Dave: How vscode support falls in in comaprision to container build environments? In addition to vscode support, we can provide container build environment.
* Radhika: Regarding the libraries and dependencies output, need to formalize the prioritization strategy.
* Simon: It can be driven by customer requirements. CCF is an example for libcxx issue.
* Anand & Radhika: Start of the release, identify the libraries due for an update at start of release and make it part of release tasks.
* Dave: Functionality related updates can have medium priority compared to the update required for security fixe. Security fixes related update should take highest priority.
* Anand: Keep performance related aspect also in view while deciding priority.
* Radhika: regarding the clang tooling should we move to lowest common denominator?
* Simon: It comes to a question of how prescriptive should we ? Do we need to mention all the details of security flags.
* Radhika: Do we need to mention the customers to use the minimum version of clang to build sdk and samples.
* Simon/Mingwei: Backward compatibility issues
* Anand: Can we stick to the clang version of vscode? But vscod ehas slow release cycles, how it will affect it?
* Radhika: Can we have different export targets for different clang versions?
* Radhika: Look at visual studio and see what is the version they are using. If it is clang 11, we should update our tooling to clang 11. If it is clang 10 or lower, we need to brainstrom onhow to address it?
* Simon: Regarding the tooling, are partner teams okay with we updating to different versions? Different teams requirements might be different? Example MystikOS VS CCF requirement?
* Ananad: How can user find out the library versions for dependencies? Should we publish a table of versions for dependencies.
* Mingwei: Ye, we should have a document to capture this information.

### 23 Feb 2021 10:00 AM PST

Chair: Manoj
Minutes: Radhika

* Design: Debugger Support for Multi-Module Enclaves:
* PR: https://github.com/openenclave/openenclave/pull/3838
* Doc:
https://github.com/openenclave/openenclave/blob/67bb39f9bc0b05ba8332e2a76bdaa66e94dc82b5/docs/DesignDocs/DebuggerSupportForMultiModuleEnclaves.md
* Austin pointed out that the debugger attach scneario is not covered. Also crash dump is not covered
* Mark: How does stack stitching work in this scenario? Mark will review the PR and provide feedback
* Anand: Multi modules dont introduce new complexity for debugging scenarios, it works as it would in the general scenario
* Simon: Dont see scenario where you are adding a module in the middle
* Mike: you can unload from the middle
* Dave: Is singly linked list the right data structure? Is doubly linked list better? This is a suggestion for Anand to think about
* Anand: In production enclaves, debugging is not supported. In debug enclaves is performance important? Think it is not significant
* Francisco: Would not optimize for this now since it is not going to have a significant performance impact.
* Anand: Dont want to break backward compatibility.
* Mark: Are you modifying the contract for moudule load?
* Anand: No, but we introduce new contracts for secondary modules.
* Anand: In this design we are addressing two different modules
* exe depends on dll, we know this at init time
* dlopen, dont know this at init time
* For PE and ELF enclaves is there a preference for int3 vs hidden breakpoints?
* Austen: Prefer hidden breakpoints
* Action: Anand will update the doc with this feedback
* Dave: The design can move ahead since we have unified Windows and Intel.
* We will wait till Mark provides feedback and merge design after this feedback is given
* Mark: Does this design take into dynamic thread creation? Do we want to take care of that?
* Anand: Prefer not to add it. Since focusing on single dynamic loading of crypto module right now.
* Simon: Is thread creation design fully baked and how much work would it?
* Mark: Did it for Intel SDK and will look through it.
* Brainstorming: Process for upgrading tools and 3rd party libraries
* https://github.com/openenclave/openenclave/issues/3815

* Close on Sealing APIs ([#3652](https://github.com/openenclave/openenclave/pull/3652))
* Cedric: Did anyone have a chance to go over it? No, so will go over it in the meeting
* Action: Add comments to say use macros to initialize structs rather than manually initilialize it
* Simon: Does the impl use the seal setting IV?
* Cedric: If IV is not specified, it will be random
* Simon: This sounds fine
* Simon: oe_seal_setting_tee_agnostic_max: In runtime, if enum value is less than a certain value, error out. Also document this.
* Simon/Dave: Use a reserved range instead. 64K numbers, as lower range or higher range.
* Dave: in oe_seal_setting_t change int type to enum
* Cedric This enum is extensible by third party
* Action: Simon/Dave: Please review so we can close in the next week
* Thanks Cedric!
### 16 Feb 2021 10:00 AM PST

Chair: Haitao
Minutes: Francisco

* Design: Debugger Support for Multi-Module Enclaves:
* PR: https://github.com/openenclave/openenclave/pull/3838
* Doc: https://github.com/openenclave/openenclave/blob/67bb39f9bc0b05ba8332e2a76bdaa66e94dc82b5/docs/DesignDocs/DebuggerSupportForMultiModuleEnclaves.md
* Problem intro:
* Native debuggers do not know how to load symbols. OE SDK runtime invokes functions to help debugger locate and debug enclave.
* Linux uses hidden breakpoints for **oe_debug_notify_enclave_created**() and **oe_debug_notify_enclave_terminated**()
* Windows does not use hidden breakpoints, uses RaiseException Windows API inside the two functions above.
* Dave: Why do OSes use diff approach? Anand: Will offline research this.
* For multi-module enclaves, we still need the same info path of module's ELF module and memory region where the enclave was loaded, and there are **oe_debug_notify_module_loaded** and **oe_debug_notify_module_unloaded** APIs called by the SDK runtimes.
* When running on Windows the enclave cannot call RaiseException inside the enclave, so instead the enclave can use int3 to notify load and unload.
* Dave: If module is OS independent, this means what you use to solve the Windows case will also be the solution for Linux case.
* Haitao: Why not use an ocall? Anand: Want to try to avoid adding frames to the call stack. There are some cases (initialization) where you can't make ocalls.
* Dave: What is specific for multi-modules? There isn't anything obvious that this is for multi-modules. Anand: Each API is called for each module, not for each enclave.
* Anand: Note that the debugger contract version number is increasing.
* Francisco and Dave questions on which APIs are called for single vs multi module enclaves. Reply by Anand:
* Single module enclave: Only the "enclave" APIs are called
* Multi-module enclave: The "enclave" APis are called for the "primary", then each subsequent module (secondary modules) have the "module" APIs.
* Anand: Primary module has TCS, secondary modules only need symbols.
* Dave and Simon: Can you merge/converge the APIs? Anand: There are considerations regarding backward compatibility.
* Cedric: Question on registers used in the assembly for int3 - why not RCX instead of RBX. Anand: We can change to RCX (or any other) if that's a better.
* Cedric: Does enclave need to know if it is being debugged when it executes int3? Anand: If int3 is executed when not being debugged the enclave will crash. Francisco and Haitao: You can read that (is enclave currently actively being debugged) from the TCS.
* Cedric: Could you use NOP (no op instruction) as placeholder instead of INT3, and have the debugger change it in real-time depending on whether it is being debugged or not? Francisco: Can E Debug Write instruction make this modification on the page or would that fault if the page protections don't allow it? Anand: Will follow up to make sure. Francisco: Windows versions let you call WriteProcessMemory() to modify RX pages (debuggers can use this in Windows to set break points, for example), so the answer is yes, this is possible to do. Anand: Will follow up with debugger team on general approach of using NOP and modifying to INT3.

* Brainstorming: Process for upgrading tools and 3rd party libraries
* https://github.com/openenclave/openenclave/issues/3815
* Anand: No proposal, but brainstorming.
* Simon: Normal procedure is to look at the MSFT list of approved 3rd party libraries, but no process to regularly check and upgrade anything.
* Will follow up next week on this topic.

### 9 Feb 2021 10:00 AM PST
Chair: Anand
Minutes: Ming-Wei

* EDMM fault flow: https://github.com/openenclave/openenclave/pull/3755
* Haitao exlains the fault handling flow that was newly added to the doc.
* Simon: Since kernel v5.11 already supports the vDSO inferface, is there plan to switch to the new interface? Does implementation need to support both the old and new interfaces (signal handling and vDSO).
* Mark: Using vDSO is more efficient for SGX2 EDMM support. Hopefully we can convince kernel people to adopt this implementation. Signal handling implementation is still required by SGX1 out-of-tree driver and therefore we need to keep. Enclave common loader could abtract the detail and inform what interface should be used.
* Simon: Want to point out that some work requires to be done on the OE (for vDSI support).
* Table in the Fault Handling in Kernel section
* Mark: Kernel will know what action to take based on the page states (PFEC, PTE, etc)
* Enclave Handling of Faults
* Simon: Seems that the enclave/SDK needs to keep track of the state of dynamic allocated pages. What are components that are sharable across different SDKs?
* Mark: <to be added>
* Debugger Support
* Anand: Will a debug access to a EPC page that is not yet allocated cause failure?
* Haitao: The EDBGRD should return errors.
* Next step
* Mark: Will ask Intel kernel team to take a look on the doc. Also, we would like people from other SGX projects to review the docs and see if there is any gap.
* Simon: Let's send an invitation to the people mentioned above.
* Haitao & Mark will do the reachout. Estimate 3-4 weeks to keep the PR open and allows people to comment on the doc.

* Sealing API (https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652]
* Brainstorming: Process for upgrading tools and 3rd party libraries https://github.com/openenclave/openenclave/issues/3815

### 2 Feb 2021 10:00 AM PST
Chair: Cedric
Minutes: Mark

* Mark: Suggest that we make the afternoon (5:30 PM PST meeting) an ad-hoc rather than a regular weekly meeting. Looking for feedback to see if the regular meeting is needed. It will be cancelled today.
* Continuation of [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652] (https://github.com/openenclave/openenclave/pull/3652))
* Cedric: has changed the oe_seal API: See the header file in the PR.
* Specify what plug-in to use
* Pass in an array of settings (number specified in "settings_count" parameter): each setting has
* header with name and size
* two names devined: "setting policy" and "additional context"
* SGX specific settings
* Dave: Each setting is called out individually. Cedric: because each setting can be configured seperately
* Cedric: have configured default setting values, also configure defaults based on the key name.
* Simon: where is the plug-in registration done?
* Cedric: Cannot register the plug-in implicitely, but can add plug-in definitions in an array in a specific ELF section.
* Simon: can you create a global constructor which would register the array. Can see how attestation does registration of plug-ins.
* Dave: OE supports C++ global contstructors, but the order that global constructors are called is not ensured.
* Simon: plug-ins that are separate libraries may be specifically registered by the user.
* Dave: some concerns: oe_register_seal_plugin should not be under the sgx directory:
* Cedric: other TEEs could have their own register. Can choose at compile time.
* Dave: another way to do this is to have the code be agnostic.
* Simon: can create a TEE agnostic version (use void*) and cast the specific version to the agnostic version, then you can give it a specifically typed pointer.
* Simon: solution addresses all the design concerns. Expect some feedback from code review.
* Simon: would like to see tests too.
* Dave: for public APIs, would also need sample code. For small APIs, this can be a code snippit, but for large APIs (like this), it should be a sample application.
* Cedric: will check-in the original document

### 26 Jan 2021 10:00 AM PST
Chair: Simon
Minutes: Radhika

* [Haitao/Mark] Continuation from last week: Review SGX2-EDMM interface and flows in next week's meeting. See https://github.com/openenclave/openenclave/pull/3755
* The enclave does not want to keep protection mapping for all the pages it has allocated, it just caches the permissions after mprotect for the current page and after eaccept, it can forget it. If the application does not make losts of changes, this is OK.
* MS: With the ocall to mprotect, the kernel can ignore the request, once. If the enclave does an eaccept and there is nothing to accept, there is a page fault.
* MS: If you do a EMODPR, the PR bit state, it could impact the enclave's ability to do another operation on the page. The kernel should tell the enclave if it actually modified the privelege in which case the enclave should do an EACCEPT.
* HT: If you do an EMODPR, you should always do an EACCEPT
* *Action MS/HT*: Clarify what shoould happen after an EMODPR? Alwyas do an EACCEPT/

* Next steps for review:
* MS: Flows are great!
* MS: Outline fault handing that the kernel may and may not see. Outline what the enclave should do for fault handling
* Outline a table.
* SL:
* Comments from folks should be resolved
* After fault handling tabulation, what does approving this mean?
* MS: Need feedback from LKML
* MS: Define Enclave common loader interfaces
* MS: Then define lower level functions - trusted mprotect and memmap
* AK: Can you also describe how sgx_tcs_t is validated for tcs allocation?
* AK: Think about debugger workflows
* Continuation of [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652] (https://github.com/openenclave/openenclave/pull/3652))
* Cedric: Will take inputs from Simon and implement a patch. Thanks Cedric!
* Dave: Simon's refinement of using init structures(instead of args), is extensible and flexible. Suggest moving ahead with Simon's refinement
* Cedric: Please leave in comments what you would like entropy mask to be named.
* Simon: Dont call it entropy.
* Cedric: It will be a TEE agnostic paramater.

* Move to prow
* SIG-CI will send out instructions
### 19 Jan 2021 10:00 AM PST
Chair: Mingwei
Minutes: Francisco

* [Haitao/Mark] Would like to review SGX2-EDMM interface and flows in next week's meeting. See https://github.com/openenclave/openenclave/pull/3755
* Concern that the proposal is geared towards Linux and OE does not want to have support for a feature that is Linux only or Windows only if both can be supported.
* Mark: The design should apply to Windows as well, but Windows support is not (well) documented. If/when Windows documentation details its support for EDMM we can take a closer look to make sure there are no gaps.
* Kernel can decide to implement EAUG directly with MAP_POPULATE or do EUAG on a #PF. Doesn't change the API, but may affect performance.
* Mark: We can make a table for how to handle page faults that happen inside an enclave and how to handle each case.
* Anand: Can kernel deliver fault to a different thread? Mark: VDSO interface lets kernel call-back into the same thread inside the enclave can handle the fault. The exit info in the SSA lets the enclave check. Haitao: This callback mechanism is preferable to a signal handler.
* Simon: What is IPI? Haitao: Inter-Processor Interrupt
* Anand: What happens if debugger tries to read a page that hasn't been EACCEPTed yet (using EDBGRD)? Mark: Depends on state of the page. Mark will investigate for next time.
* Ran out of time. We will cover TCS allocation next time.
* Continuation of [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* Not covered today
* [Dave] (not urgent but want to queue up for discussion at some meeting, ok to move to attestation meeting) feasibility of host-verify style attestation from kernel mode code
* Not covered today, moved to SIG-Attestation meeting.

### 12 Jan 2021 10:00 AM PST
Chair: Radhika
Minutes: Mark

* Continuation of [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* Note: Shanwei suggests moving to next week, as Cedric won't be available for this meeting.
* [Dave/Anand] use of intrinsics in enclave C/C++ libraries
* Some intrinsics only need headers: https://github.com/openenclave/openenclave/issues/3795 Other intrinsics also need a lib: https://github.com/openenclave/openenclave/issues/1766
* Dave: Other projects may want to use these headers. Question about pulling in the Apache licensed headers.
* Simon: can we include the header files from the host?
* Anand: this is not an option because we do not pass in the system headers and we are compiling on Windows.
* Dave: was able to copy the header and C files and use them. This can be a workaround for the application right now.
* Proposal is to include Apache licensed headers or MIT Licensed headers and compiler RT in the OE SDK package:
* Anand: Intel SDK ships with a version of the compiler RT: see https://gcc.gnu.org/onlinedocs/gccint/Integer-library-routines.html#Integer-library-routines
https://github.com/intel/linux-sgx/tree/master/sdk/compiler-rt
* Dave: and the Intel files are MIT licensed
* Plan will be to provide specific intrinsics (Math, ...) as needed with the OE SDK Package.
* [Haitao/Mark] Would like to review SGX2-EDMM interface and flows in next week's meeting. See https://github.com/openenclave/openenclave/pull/3755
* Added to next week's discussion.
* [Christoph] Making EEID non-Experimental
https://github.com/openenclave/openenclave/issues/3793
* Bo: concerns about the security design - how to limit the potential risk by converting to fixed length and location of EEID pages. Have proposed fixed length and location as an alternative.
* Christoph: have a draft proposal for this.
* Bo: requests that the Intel team review the security (no feedback from Intel or the community yet).
* Will likely require changes.
* Christoph: it can be experimental, but want it included in the SDK package.
* Bo: to move something to non-experimental will result in it being included in the package.
* Need to assess the ROI to determine if this can be prioritized.
* Christoph: this is needed for cloud sealing service.
* Simon: this is tested in CI/CD. Can the package be published internally for the customer to use while the team gets the feature ready.
* Christoph: this would be fine
* Radhika: this could be a significant effort to create the package.
* Radhika: experimental code can be removed (it may not be made non-experimental in the future), the customer needs to understnd this.
* Bo: experimental feature may not have enough security review. We should discourage the use in a production enclave.
* Proposal for supporting deep copy of out parameters https://github.com/openenclave/openenclave/issues/3699
* ACTION: Dave to review
* Others should review offline too.

### 5 Jan 2021 10:00 AM PST
Chair: ???
Minutes: Simon
* [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* Still waiting on Cedric to update the PR with the content from the WIP document.
* Cedric not in attendence, so defer to next week for when he can join.

* Simon asked if Haitao could present his proposal on the kernel interface to the SGX libraries (common loader) for EDMM support
* [Mark] I'll work with Haitao on this, but it won't be next week.
* Just want to make sure that the thin interface to the kernel is sufficient for building the other runtime pieces using EDMM as described in [Mark's comments](https://github.com/openenclave/openenclave/pull/3639/files#r511011452).
* [Anand] Do the kernel changes include changes to vdso the OE runtime would need to accommodate?
* [Mark] No, they don't.
* [Dave] Does Haitao's proposal change the OE public API?
* [Mark] No, it's only for the common loader dependency, but OE should consider how it's choice to expose public APIs for EDMM (as in [#3639](https://github.com/openenclave/openenclave/pull/3639)) would need to use it.

### 15 Dec 2020 10:00 AM PST
Chair: Radhika
Minutes: Ming-Wei
* [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652](https://github.com/openenclave/openenclave/pull/3652))
- Cedric: The WIP document is used for the discussing purpose. Added a section for comparing the two options and would like to see if Simon and Dave agree on the assessment.
- Shanwei: Suggest update PR with the content in the WIP document so that people can review.
- Cedric: Will update the PR.
- Radhika: Since Simon is on vacation, are you willing to wait until the new year or want to make progess now.
- Cedirc: ok to wait.
* Radhika: This is the last sig-arch meeting this year. We will resume the meetings next year.
* See you and happy holidays!

### 8 Dec 2020 5:30 PM PST
(cancelled)

### 8 Dec 2020 10:00 AM PST
Chair: Cedric
Minutes: Francisco

* Init-Time Configuration: https://github.com/openenclave/openenclave/pull/3726
* Bo: Already merged (discussed in SiG attestation). Agreed to not make changes to existing OE get evidence API.
* [Sealing](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) ([PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* Cedric: Since Dave and Simon are not here, they can follow up offline.

### 1 Dec 2020 10:00 AM PST
Chair: Ming-Wei
Minutes: Francisco
* Sealing (https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) [PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* oe_seal() API
* Cedric proposes a
* set of parameters that are common for all TEEs (Cedric says entropy is one such param)
and
* another set that are not
* Simon wants to prove that there is such a thing as a set of generic parameters that are indeed TEE-agnostic.
* Option 1
* seal_my_data() API takes no custom parameter and it passes in all defaults to oe_seal()
* Dave: The OE seal handler can decide which sealer to use, maybe use weak symbols and whatever the linker decides gets picked.
* Simon, Cedric, Dave: There needs to be a runtime mechanism to register a sealer.
* Simon: Proposes get rid of settings and let OE ignore all these parameters and letting opaque out params. The plugin handles everything.
* Dave: There is value in having TLV settings.
* Dave: Three types of settings - Settings supported by OE SDK, settings common by all TEEs (TEE neutral), SGX specific.
* Option 2
* Cedric's original proposal
* Sealing is picked at build time not run time
* Precludes the OE, in the future if needed, to do anything.
* entropy is not hard-coded parameter, but an octect string.
* Simon: People that understand KDF understand Label and Context. Instead of concatenating into an octect string named 'entropy', you can just pass them as two params.
* Cedric: There are many ways to combine label and context. Would prefer to leave the exact mechanism to combine these into an octect string to the developer.
* Dave: Should not expose seal keys to developers other than for backward compatibility.
* Dave:
* Decision 1
* Do we want to have an API with a UUID that can be muxed
or
* Do we want approach that applications link directly to the library
* Decision 2
* Need for 1-step seal or 2-step seal
* Simon: Decision 2 is not independent of Decision 1

* Init-Time Configuration: https://github.com/openenclave/openenclave/pull/3726

### 24 Nov 2020 5:30 PM PST
Chair: Anand
Minutes: Radhika
* Protected Code Loader support in OE - Xiangping will update on the status https://github.com/openenclave/openenclave/pull/3684
* does dynamic linking have impact on oesgn tool. oesign will need to sign the combination of sos and enclave apps
* Simon's prototype sign tool figures out dependencies and signs them. No user interface change. only main enclave will have oeeinfo data structure
* Xiangping: Can static an dynamic linking exist in parallel? Anand: Yes
* Xiangping: current PCL is designed for static link. Can coexist.
* Proposal: In first proposal would like to have PCL to support static libs. Single enclave image. PCL is a separate section.
* In second proposal, would like PCL to support dynamic link.
* There is a section for decryption. Could be a separate dso file and remain unencrypted for the PCL. Multiple shared libs, PCL lib will be in plain text and other shared libs will be encrypted.
* Dave: Can PCL not be statically linked? Can go into ELF file. PCL has to be plain text and go into enclave ELF file.
* Anand: Yes, PCL can be statically linked.
* Xiangping: User can either choose based on sections(static) or based on modules(dynamic)
* Cedric : what is the intersection between PCL and DB(dynamic binding)
* Anand
* want to unify the architecture
* If we had DB, then one of the modules could be encrypted and PCL could decrypt that
* If we use DB and have FIPS crypto, then the decryption of PCL will not use this FIPS crypto.
* Anand Questions:
* Does PCL have its own copy of decryption algorithm?
* YOu dont use malloc or any C lib?
* Xiangping: No
* Dave:
* Is decryption algo processor specific?
* Xiangping: No. All TEEs can use the same algorithm
* Cedric: We can certify PCL.
* Dave: No, need to be able swap out the algorithms.
* Radhika: There is no scenario that needs both at this time
* Dave: there may be scenarios with libOS
* Radhika: Suggest experimental feature with #ifdefs around it.
* Dave: Agree with experimental feature
* Xiangping: Will investigate DB to see if PCL can work
* Mark Shanahan: Have you been able to work around the relocation table being relative relocations?
* Xiangping: Will investigate this further
* MArk Shanahan: Intel loader will load ELF file in its given formtat. OE takes the relocation table and changes it all to be PCrelative. The trusted part of the loader goes to the table. The PCL encyrpts the whole table.
* Conclusion: Go with option 1 and make it experimental. Revisit with DB merged in and determine interop with DB feature
* All meetings after Dec 15th will be cancelled. TDB: If Dec 15th meetings are cancelled(will determine on Dec 1st)

### 24 Nov 2020 10:00 AM PST
Chair: Anand
Minutes: Radhika

* Discuss [2 design options of sealing APIs](https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_options.md) [PR#3652](https://github.com/openenclave/openenclave/pull/3652))
* Cedric proposes an API based on Simon's feedback - Option 1
* Major question - what UUIDs for sealers are available to the enclave? Can you enumerate the sealers at runtime or buildtime?
* Dave: we dont need to enumerate them via API. We dont have that for attestation. We can just document in OE SDK.
* At build time, developer makes choice of sealer.
* Mark Shanahan: In attestation, evidence is sent to a verifier and the verifier needs to pick the UUID.
* Dave: Verifiers should be able to support multiple algos and the attester can pick one.
* Mark: For encryption, you may want to seal something for aother enclave. The other enclave cannot unseal the data.
* Mark: If an enclave supported two different sealing algorithms, then how do you choose?
* Cedric: Sealer and unsealer provided by same ISV. link time choice of UUID
* Dave strongly prefers option 1
* Cedric: Settings contains entropy
* Option2
* Cedric: Option 2 is simpler.
* Dave: From an API perspective it is not OK to have a TEE agnostic API for a TEE specific structure. Change key_info to void *
* Dave: oe_initialize_seal_key and oe_seal are inconsistent since entropy is taken as a param in one and not in the other
* Dave: expect sample for option 2 to be bigger.
* Cedric: In option 1,we need to decide seal key params for each TEE, option 2 is easier since we know how to code against a specific TEE
* Dave: Option 1 supports things that option 2. opt_params can be used to pass things that are not in seal_key_settings
* Dave: Option 1, I can specify which plugin I want by specifying UUID. How do I do this in Option2
* Cedric: Developer uses weak symbols(see PR), no documentation is needed. The soure code is explicit about the implementation
* Dave: please update sample code for caller in each option
* Dave: Thanks for writing this up and for answering the questions.
* Anand: We will revisit this next week
* Deep Copy of Out Parameters: https://github.com/openenclave/openenclave/pull/3699
* Ming-WEi's updates: without shared memory, required nested ecall. SGX in OE does not support nested calls either
* This design does not work without shared memory.
* Dave: What would the edger8r code be for OPTEE?
* Ming-Wei: This is experimental currently. edger8r will generate performant code, so it can be used on SGX
* Anand: We can support ocalls on OPTEE in edger8r
* Ming-Wei: ocalls are fine, ecalls will not work on both OPTEE and SGX. Making an ecall inside an ocall wrapper
* Anand: OPTEE can be implemented by making the call in the same direction. ocalls make two ocalls, one to get the size of the buffer and the other to give the buffer that was allocated
* Ming-Wei: Will update doc to add a command line argument for oeedger8r.
* Anand: If you have variable length objects. How does the user free them?
* Ming-Wei : Caller frees nested structs since they have apriori knowkedge of struct. oeedger8r will free code for Callee at the time when the callee returns.
* Dave: Another impl, allocate a great contiguous buffer for the big nested stuct and free later
* Ming-WEi: In the future, we can have better handling
* Dave: outstanding issue does not block anything(how to implement on OPTEE), but at this time, this is experimental, so that is OK.
* Ming-Wei: In the future, can generate different code based on platform edl added.
* Dave: this is an interesting thought
* Anand: PR looks good. We can iterate further on this
* Ming-Wei: Switchless + deepcopy out param marshalling OK
* Anand: OPTEE does not support switchless.
* Dave: Update design doc to say something about switchless
* Init-Time Configuration: https://github.com/openenclave/openenclave/pull/3726
* Move to non experimental for openssl
* Samples for review: https://github.com/openenclave/openenclave/pull/3750
* THanks for reviweing Dave!

### 17 Nov 2020 5:30 PM PST
Chair: Simon Leet
Minutes: Radhika

* PCL with dynamic linking
* Xiangping: Was investigating dynamic linking for PCL
* Intel SDK has disabled dynamic linking due to sec concerns
* Simon: what are these sec concerns
* Mark: Concept of dynamic binding was complicated therefore disabled.
* Simon: Dynamic loading is much more complicated since you have to know before all measurements of all modules
* Mark: If we have a protected PCL and we port it to OE SDK, would it be helpful?
* Mark: Dynamic binding + protected PCL is much more work.
* Mark: Current priority: Getting feature parity
* Simon: Timeline: v0.14 for dynamic binding
* Mark: If code loader were to protect ISV code, then it reaches out to unencrypted runtime, there is a side channel attack surface
* Mark: Side channels would be using page faults.
* Mark: in PCL in Intel SDK, you dont know what components, layout. If you do know these, then it is easier
* Simon: We would always know the runtime libs - libC etc and rough layout of it. So it does not really change surface
* Mark: If the lib is in the clear, it is a lot easier to do things.
* Simon: Issue with PCL is that it requires the loader to understand PCL in a specific way.
* Anand:
* Can dynamic binding be used to implement PCL
* Can PCL be extended to support multiple modules?
* Simon: Intel will be adding PCL while MSFT is adding dynamic binding which causes issues
* Will these two ever converge?
* Simon: Intel is thiking about EDMM with dynamic loading and that will change the loader as well
* Simon: How will EDMM + dynamic linking + PCL interact?
* Ming-Wei: How is statically encrypted binary different from a encrypted dynamically linked binary? Is there a doc to discuss the security difference here?
* Mark: When everything is encryped much harder to figure out loader. When crypto is the clear, it is safe to see where the code is being accessed and how.
* Ming-Wei: LibC being encrypted or not does not seem to change sec considerations
* Radhika: Can PCL and Dynamic binding exist side by side without an intersection
* Anand: Can one be an experimental feature. Since one of the features is not finalized, it is hard to base it on any of the features.
* Mark: Xiangping have yoou invetsigated how much work it is to leverage dynamic binding for PCL?
* Xianping: Investigating and will come up with proposal.
* Simon: PCL is more ready, so maybe it can come first?
* Xianping: What about EDMM?
* Simon: Will dynamic loading work with PCL in EDMM?
* Mark: will do a comparative analysis of PCL by itself vs PCL built over dynamic binding. Action: Xianping Action: Simon will respond to questions over email?
* Mark: The code to get the key to decrypt the module should be in unexcrypted module which makes this difficult in TEE-agnostic manner.
* Dave: Would like APIs to be multi TEE, or we decide if the API is specific to SGX. Only decrypt after the attest on other TEEs.
### 17 Nov 2020 10:00 AM PST

* Meeting is cancelled - many participants have other engagements

### 10 Nov 2020 5:30 PM PST
Chair: Dave Thaler
Minutes: Mark Shanahan

* Simon to discuss dynamic binding of modules and how they relate to PCL.
* Shanwie: is this specific to SGX.
* Simon: we don't have a way for OP-TEE to do this right now. This is more relevant to SGX.
* Dave: an OP-TEE enclave can call into another enclave so it does not actually need this if the shared library is instead another enclave
* Demo - See branch https://github.com/openenclave/openenclave/tree/feature/dynamic_binding and try the demo
* OE SGX Loading - mostly the same as the Intel SGX loader. Exceptions are:
* OE loads ELF file then patch the elf file with patched global vars
* Changes to support dynamic loading:
* Perform dependency walk - assumes that the primary enclave file is a process independent exe (same as a .so). This contains the metadata.
* .conf file params are ignored in shared libraries, only the main executable parameters are used
* Dependency binaries must be in the same path - simple way to find the corect binary files.
* Files are loaded in their link order. Mark: may need to be initialized in the opposite order of load (the dependencies may need to be initialized before the modules that depend on them)
* Fixed number of modules that can be loaded (32 right now).
* Support 4 types of relocations:
* May not support arbitrary relocations
* Intel supports a larger set of relocations - would like to know why (some do not show up such as GOT relocations)?
* Rounding out the design is to address constraints:
* Change relocations to all relative relocations in the load - don't need ELF parsing code in the enclave. Instead just relocate to the base address inside the enclave.
* Intel trusted loader has ELF parsing code.
* Performing the ELF parsing will make the solution more adaptable in the future to features such as dynamic code loading using EDMM.
* Dave: some questions about how we can do attestation if the enclave does dynamic loading (that can take place in the SIG-Attestation meeting...)
* Anand: are there any build time restrictions on what the .so modules can contain?
* If A (the primary enclave executable) is providing libc functions and B is dynamically loaded, then how will it know about the functions in A?
* Simon:
* Enclave Layout - see slide
* Debugging Considerations - need to notify the debugger for each module
* Protected Code Loader (PCL)
* Question: does the design for the PCL get simpler if we have dynamic loading? Instead of specifying custom sections, can just encrypt specific sections of a dynamic linked module.
* Does it make the PCL simpler? If so, then should we wait for this feature?
* Simon will put the slides to openenclave/publictalks.

### 10 Nov 2020 10:00 AM PST
Chair: Cedric
Minutes: Radhika
* Proposal for supporting deep copy of out parameters
https://github.com/openenclave/openenclave/pull/3699
Ming-Wei: Do we suport where shared memory is not available and go with a design? Or do we
Anand: Swithcless calls not supported on SGX
Ming-Wei: Use case: currently outparams passed as user_check so OE cant make any checks.
Anand: Size of outparams not being known before hand results in a double call pattern. This feature eliminates this.
Dave: You can use TEE specific features when they are not.
Dave: This deepcopy is important to various features.
Dave: Would like optimizations when shared memory is there.
Dave: Would like APIs to be TEE agnostic. Two approaches
1. Design to be TEE agnostic. Can implement for SGX
2. Design to be TEE agnostic and be shared memory implementation.
Anand: Would like something that is easier to get feedback.
Dave: See problem with freeing.
Simon: Interface is no specialization of choice.
Is it OK for implementation to give different implementations on different TEEs?
Dave: Is it annotatabe?
Ming-Wei: Is not about annotation, how you generated code. edger8r does not generated code
Anand: Irrespective of this feature, ecall and ocall dispatching implicitly uses shared memory and user cannot opt out of it.
Simon: Make explicit what decisions you are making.
Dave: edger8r-> generate code(if (sharedmemory) -> go one way, if not -> dont go this way)
Dave: OPTEE impl can come later
Simon: Can do SGX implementation now and do OPTEE as an add on. We are not giving a new annotation type.
Radhika: Users have request for feature for security reasons and performance is a consideration
Cedric: Go with SGX, if bidirectionally another set of calls.
Dave: bidirectional is solved by the doc
Ming-Wei: on ocalls ->ecall with nested ocall. ocall->ecall
Simon:
Anand: if we are making an ecall. the enclave returns a pointer. The pointer lives in host memory. Is there a special API to free this memory?
Lets say callee does malloc, after memcpy, nested pointers will be freed.
Cedric: A single struct might have multiple buffers.
Anand: Call individual frees for all buffers?
Dave: suggests call foo and free_foo, some deep struct? Then have a special free API.
Simon: Be explicit about how this would be used and freed.
Ming-Wei: If we agree that we can create SGX implementation first, will update PR with updated design for this direction.
Anand: midl has precedent for this problem. Would like to take a look at it.
Simon: explicit alloc and free-> abstract it out. dont assume it will be malloc
Anand: Can zigzag calling be implemented as two calls with state beign cached locally?
Simon: Just have design
Conclusion:
1. We have agreement that we will go with shared memory model
2. Memory freeing per buffer basis is fine ->No consensus
3. Made a suggestion that have an explicit free method(not malloc and free)
4. Ming-Wei will go and flesh out nested structs for the shared memory model and we can iterate on memory freeing there.

* If time permits, design discussion around Cedric’s PR to Add Sealing APIs [#3652](https://github.com/openenclave/openenclave/pull/3652)
https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI_%20draft2.mdg
Simon and Dave: Consider that you cannot pass a key around.
Cedric: oe_seal_key_info is a TEE specific structure which takes all the params.
Dave: seal key, entropy and policy are not TEE specific
Cedric: Data blob can also be used for oe_get_seal_key
Dave: wants get_seal_key to be deprecated
simon: Need not be structs, they are data blobs. They are not exposed as such
Simon: See attestation and create_Enclave to see how APIs are TEE agnostic.

### 9 Nov 2020 9:00 AM PST - Cancelled
Chair: Ming-Wei
Minutes: Radhika
Design discussion around Cedric’s PR to Add Sealing APIs [#3652](https://github.com/openenclave/openenclave/pull/3652)
https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI.md

### 3 Nov 2020 10:00 AM PST
Chair: Ming-Wei
Minutes: Radhika
* Design discussion around Cedric’s PR to Add Sealing APIs [#3652](https://github.com/openenclave/openenclave/pull/3652)
https://github.com/binxing/openenclave/blob/wip/sealing/docs/DesignDocs/SealingAPI.md
- Simon: Crypto suite is more than an algorithm
- An implementation in oe_seal can make choices.
- How can we make this extensible?
- There can be two implementations
- Use 0 IV so compatibility with Intel
- You use random IVs
- Cedric
- Sealing is not a communication protocol like TLS
- Dave
- Sealing is like a high latency protocol
- Cedric
- Sealing and unsealing should be done from the same enclave
- ISV does sealing and unsealing
- A new algo might want to use due to weakness in old algo
- oe_seal can embed version in blob. oe_unseal will use version
- we need not offer choices to developer
- Dave
- Enclave might never get the key
- Are all the params needed to derive the key in key info?
- Simon
- Seal and unseal is not determined by algo
- Seal and unseal is a narrow stack in SGX
- Dave
- Where you are passing in keyinfo, you should be able to pass in all information that you would need.
- For agility - Put all the information you need to identify the crypto into keyinfo blob
- Cedric
- Not saying that crypto agility is not important. oe_seal need not offer choices on algorithm.
- Dave
- SGX seal need not to offer agility
- Other TEEs put information into keyinfo, put all params into keyinfo
- Simon
- User choices are needed for sealing for example on OPTEE
- Cedric
- SDK offers an optimial choice, build time option rather than runtime. Statically link a lib.
- We dont need to offer choices as params.
- ISVs decide what lib to statically link against
- Once the choice is made, ISV will stick with the choice
- At build time implementation can be switched out.
- Simon
- The unit of agility is not the crypto algo
- Cedric
- Ofcourse there are unlimited ways to encrypt data, but we need not offer them as runtime choices
- Library linked in decides the crypto choices
- Dave
- Proposed conclusion: cryptoagility IS important and supportable via choice of the plugin you link with. The "key info" blob structure can then be specific to the library, and opaque to OE
- Cedric
- Is not like attestation.
- We need interoperability with Intel SDK for unseal
- Simon
- Can have UUID discriminator to tell OE which lib to call out to
- Cedric
- How do we switch between vendors?
- How to enable TEE specific features?
- Dave : Keyinfo can pass in entropy and pass it into plugin
- You can not always have defaults, seal key has other usages. get_seal_key_by_policy is insufficuent? Dave agrees.
- Simon: disagrees with this. The question is why do we need a key at all? We should only have sealing. We should remove both get_seal_key and get_seal_key_by_policy
- Cedric: Need get_seal_key needed for keyinfo?
- Dave: No.
- Cedric: How do you write key specific keyinfo?
- Dave: Need differentiation between TEE specific and lib specific
- Keyinfo blob can be lib agnostic and/or TEE agnostic
- Simon: You can write cross TEE API by using oe_seal with no params and the TEE implements
- Simon: If there are universal params for sealing, we can discuss what those are
- Cedric
- Is creating keyinfo a dev problem?
- Dave
- Need not ever call an API to populate keyinfo
- Simon
- In general, on fence about getkey API. Agree get_seal_key is not a good API. Should we just say this is an oe_sgx API? And not use this for cross_plat
- Cedric
- would like user friendly API
- Large variety of identities across TEEs, as long as I can produce information on the type of seal key I am getting.
- Not looking at solving 100% of the problems. With a user friendly API for 90% is a good goal. Would not like to over engineer.
- Conclusion(Simon, Ming-Wei and Dave): Thanks so much Cedric! Appreciate all the writeup and thought!

* SGX 2 EDMM Continue Discussion https://github.com/openenclave/openenclave/pull/3639
* Proposal for supporting deep copy of out parameters
https://github.com/openenclave/openenclave/pull/3699

### 27 Oct 2020 5:30 PM PST
Chair: Xiangping
Minutes:

* add Proposal for Protected Code Loader support in OE https://github.com/openenclave/openenclave/pull/3684 (Xiangping) - continue discussion from Oct 20th
* Anand: how is the liboepcl.a using standard libraries and plugable allocators?
* Xiangping: PCL library is not using malloc. The decryption of the code is done in place. It is using memset/memcopy - need to check whether these are intrinsicly supplied by the compiler.
* Xiangping: the PCL library is using the OpenSSL library.
* Simon: this could pose a problem for FIPS certification since all crypto within the enclave would need to be FIPS certified.
* Dave: there would be equivalent certifications for other countries - the crypto should be pluggable.
* Dave: the design should be forward looking so that FIPS certified crypto can be loaded.
* Simon: Question of whether OpenSSL has its own allocators - there can be only one allocator within the enclave (one heap).
* Simon: There is no point in encrypting the OE core code - how can we ensure that it is not part of the encrypted code. Need to consider the API surface.
* How can we ensure that they do not become encrypted and are available to the PCL library?
* ACTION Xiangpin: will respond in the PR
* Xiangping: encryption is done by a tool "oeencrypt.." which encrypts specific sections within the encalve elf image.
* Relocations on the ELF image are completed after decryption
* Simon: Does PCL code to do relocations before it runs?
* Shanwei: the design document explains when relocations are done.
* Simon: would like a generalization to the API surface on how the enclave gets the decryption key?
* Some TEEs may want to use a different model to get the key
* Simon has added his comments to the PR
* Radhika: there can be some simplification to the design if multi-module loading is used.
* Simon: we would not need to disect a single elf binary if multiple modules are used.
* Simon: Can discuss what the OE team had planned in a future SIG-Arch meeting.
* Xiangping: will look into this.
* Simon: what is the minimum that we can implement now? Is this an SGX specific implementation?
* Xiangping: will review and respond to comments in the PR.

### 27 Oct 2020 10:00 AM PST - Cancelled
* Topics copied to Nov 4 Meeting

### 20 Oct 2020 5:30 PM PST
Chair: Simon
Minutes: Mark

* add Proposal for Protected Code Loader support in OE https://github.com/openenclave/openenclave/pull/3684 (Xiangping)
* Feature is implemented in SGX SDK - some ISVs want their enclave binary protected on the disk and when loading.
* The feature consists of:
* Encryption Tool: Encrypting the enclave image - there is a sample where the user functionality to be encrypted is placed in a library.
* There is a tool to encrypt the library image (.a file) before the enclave is signed
* At load time, the encrypted library is decrypted before relocations are performed within the enclave (at initialization)
* A Protected Code Loader (PCL) table is defined as a separate section in the ELF file
* Some ELF sections are left unencrypted - .bss, .tbss, etc.
* Sections containing IP information - .text, .data, .rodata are encrypted in the elf file
* OE trusted runtime must decrypt sections.
* Decryption key must be provisioned and then sealed on the platform - it is provided to the OE trusted runtime to be unsealed and then used to decrypt the encrypted code sections
* Modifications to OE run-time library - see PR
* Debuggin considerations - breakpoints can only be added after PCL is done decrypting enclave code.
* Questions:
* Anand: how is the liboepcl.a using standard libraries and plugable allocators? How can we ensure that they do not become encrypted and are available to the PCL library?
* ACTION Xiangpin: will respond in the PR

* Anand: have you thought about encrypting multiple modules
* Alvin: how does the application get the key to decrypt the encrypted code:
* Xiangping: there is a utility enclave which performs remote attestation to get the encryption key and then seals the encryption key for the real enclave to unseal. The unsealed encryption key can then be used to decrypt the code.
* The sample code will demonstrate this action for the user (note: this sample may not actually use remote attestation, but just seal a key provided in plaintext).
* Simon: a sample demonstrating the use would be very useful.
* Simon: the sealing/unsealing may not may to OP-TEE very well
* Dave: this functionality is very useful in OP-TEE. We need to figure out the right layer to abstract out for TEEs. Looking for Hernan to provide input.
* Simon: only API change is a type (oe enclave setting pcl enum value) and data struture for passing in the sealed encryption key blob.
* Dave: is the enum value specific to the structure of the sealed blob? Is this the same kind of structure that could be used in other TEEs? (maybe the TEE would know the blob structure - it is a blob to everything but the TEE code.)
* Simon: concern that user needs to build an entirely different infrastructure (seperate utility enclave to provision the app with the decryption key and then seal the blob).
* Simon: how would this design change if the enclave could dynamically load modules?

### 20 Oct 2020 10:00 AM PST
Chair: Anand
Minutes: Radhika

* SGX 2 EDMM Continue Discussion https://github.com/openenclave/openenclave/pull/3639
* Christoph: Would like to start discussion
* Simon: Has added comments and there are two opens we need to discuss and these are recorded on the PR
* APIs provided in oecore-paging
* Some other component will figure out how to use that memory
* Two different models
* Pluggable allocators interface with the EDMM
* This is specifically for libOS
* Christoph: PR was with SGXLKL in mind.
* Simon: Was there an expectation that SGXLKL that there would be a memory partition?
* Christoph: Yes
* Simon: How does this interact with the pluggable allocator
* Anand:
* Pluggable Allocator: Init gives a range of memory it should manage
* Simon: Does SGX-LKL just use OE's memory management?
* Christoph: SGX-LKL gives 5 pages to OE and the rest for itself and the app.
* Mark: mmap anonymous is supported inside an enclave. LKL may not want to get the mem from the heap. On Windows virtual alloc(instead of mmap)
* Mark: we dont have linux kernel APIs, we can make some assumptions. Have Windows APIs. ECL abstracts the different OSs supported into a common API. Enclave needs to know what pages are allocated and how they will be used. Heap manager can now expand and contract with an sbreak. Adding a thread->add pages, add protections etc. This model hides the lower level details.
* Simon: Would like to make progress on the high level questions. POC: allocator says I need more memory and more pages are allocated.
* Mark: high level functionalities can be built after the low level functions are decided upon.
* Simon: Intersection with switchless calls?
* Do we want people to control where the memory is allocated from? Might be importnat for libOS. Mechanism is different from runtime.
* Action items
* Mark will add a short writeup for primitives should be added to the OE SDK
* Dave to propose what this will be in OPTEE. Will mmap work? Hernan is te right person to discuss this.
* Design discussion around Cedric’s PR to Add Sealing APIs [#3652](https://github.com/openenclave/openenclave/pull/3652)
* https://github.com/binxing/openenclave/blob/dev/sealing/docs/DesignDocs/SealingAPI.md
* Made TEE agnostic API
* Cedric presented his design doc
* Proposing a design where Intel SDK seals data and OE SDK unseals.
* Dave: oeseal takes keyinfo but oeunseal does not?
* Does every TEE need to encode the algo into the blob?
* Simon: This can be implementation for sealing.
* Dave: Put this in the design doc. AR Cedric
* Dave: Is it a requirement to unseal something sealed on a different TEE? Simon: Anti feature
* Dave: Sometimes you get a handle to a key, but not keyinfo since this in HW.
* Simon: oeseal API gets keyinfo, but needs to add a third variant for SGX
* Dave: Get key info by policy is necessary would be multi TEE. keyinfo is a generic blob, not specific to a specific TEE. Think Cedric's APIs are perfectly fine if GetKeyInfoByPolicy.
* Dave: Can we use GetKeyByPolicy to get sealing keys.
* Action: Comment on the design doc.
* Get Quorom for Sig Arch Meeting today evening: 6

### 13 Oct 2020 5:30 PM PST
Chair: Xiangping
Minutes: Mark

* Discuss FXSAVE/XSAVE and other state handling differences beween OE SDK and Intel SDK.
* e.g. XSAVE state is not correctly persisted between enclave transitions in OE SDK [#3644](https://github.com/openenclave/openenclave/issues/3644)
* Issue #1520 outlines what the OE SDK wants to do in order to preserve context - proposal is to not preserve XSAVE state across OCalls.
* Haitao: main concern is that the state must be cleared on exit (EReturn or OCall) from the enclave.
* Bo: Intel SGX SDK clears upper bits of registers using vzeroupper for performance (see https://software.intel.com/content/www/us/en/develop/articles/avoiding-avx-sse-transition-penalties.html )
* Simon: would like some guidance from Intel on this use
* Simon: not sure why Intel preserves XSAVE state across enclave calls (before ECalls and OCalls)
* ACTION Haitao and Juan: will review this and comment in the issue.
* Design discussion around Cedric's PR to Add Sealing APIs [#3652](https://github.com/openenclave/openenclave/pull/3652)
* Design proposal for PThread Support: see Xiangquan's initial document here: https://github.com/xiangquanliu/openenclave/blob/dev/xiangqua/pthread/docs/DesignDocs/libc_pthread_proposal.md
* Xiangquan reviewed why Pthread APIs are needed - they are used by MKL-DNN which builds with OpenMP which uses PThread
* Reviewed the proposed APIs:
* oe_thread_create() - this is an internal API
* Simon: does this depend on EDMM?
* Xianquan: no, if there are no TCSs available, then the function returns an error.
* Simon: does this box us into any behavior? Will it automatically work when EDMM is present. Would like to understand how this will work when EDMM is present. Looking for this to be in the design (also the oe_thread_create API should not change). Mark/Xiangquan - agree.
* Simon: the EDMM design does not need to be designed, but the user experience should be defined - should not require any change on the user's part if they are running on SGX2 vs. SGX1.
* Bo: why do you have the oe_thread_create() API? Why not just have pthread_create()?
* Simon: the architectural layering has oe_thread_create apis as part of OE core. pthread_create() is part of libc libraries (a higher layer).
* Xiangping: suggest that Xiangquan submit document as a PR so that comments can be inserted into the document.
* pthread_create() - this is the external API
* Only supports 'joinable' threads
* Dave: Must pass NULL for pthread_attr_t structure
* Simon: does OpenMP pass a non-NULL parameter? Would like this requirement explicitly defined.
* Juan: might need to modify OpenMP code to pass NULL
* Simon: may need to dig into the attributes - should not just use default if the user passes in attributes, but some may not be possible within an enclave.
* makes an OCall to create a new OS thread (will select an existing TCS)
* if the thread is created successfuly, then it will make an ECall to the user defined hook function (start function) provided to pthread_create()
* oe_thread_join / pthread_join
* Bo: which layer manages synchronization events?
* Xianquan: the host
* Bo: concern that the host can wakeup the thread and attack the control flow.
* Mark: oe_thread_join() design should use the trusted synchronization objects which are provided by OE. These keep synchronization objects inside the enclave but only use untrusted to suspend/wake the thread. If a thread is woken erroneously, then it just ocalls to sleep again.
* OE github issue https://github.com/openenclave/openenclave/issues/3597
* Detached thread has better performance in Windows
* Simon: we can modify oe_thread_create() to support additional functionality. The reason that this API was created was so that one API could be used on both Linux and Windows. A good proposal would work for Linux and Windows.
* Revisit tcrypto APIs https://github.com/openenclave/openenclave/issues/3610

### 13 Oct 2020 10:00 PST
Chair: Radhika
Minutes: Dave

* Encoding for enclave path in oe_create_enclave
* https://github.com/openenclave/openenclave/issues/3565 (but Anand to create separate issue specific to oe_create_enclave)
* Anand was going to do experiments with edger8r for the new issue and report back on the other issue, defer that until Anand is present
* 3565 not urgent, can be backlogged until VS can change?
* Wait until new issue is filed
* SGX2 EDMM design proposal
* https://github.com/openenclave/openenclave/pull/3639
* Dave put this on the raise visibility across SIG-Arch
* Dave's questions are around public APIs and whether can be more general than only SGX2, to apply to any other TEEs that might sypport dynamic memory management
* ACC team is not resourcing anything for this area before end of year
* Mark: Windows supports EDMM. Linux needs kernel changes. Design doc is a good start but missing some details. Haitao and Mark will look into it. Limited resourcing for this quarter.
* Simon: do need to talk about EDMM-host interaction, start sketching it out now. Proposed APIs might be premature. E.g., what about just doing on-demand paging instead of explicit API. (Simon will add comments on the PR)
* Simon: Might try to assume that Linux host model will not be too far off the Windows host model
* Simon: Should there be an enclave-side page management using pre-reserved memory like Intel did for OpenVino support? This may allow use with additional TEEs.
* Hernan also has knowledge of Linux/OP-TEE dynamic memory management
* Two topics: (a) host kernel interaction, (b) how does the enclave do memory management at runtime - prereserved pool, on-demand, explicit APIs, etc. & to what extent enclave app has control or does OE SDK do some on apps behalf
* Maybe have a meeting dedicated to just this topic with all the interested people. (Radhika can create a 9am meeting next week if Christoph, haitao, Cedric, etc. can make that time)
* Review of project board
* https://github.com/openenclave/openenclave/issues/3518 heap page flags
* Related to discussion above of PR #3639 but 3518 is specific to SGX1
* Having executable heap pages can be dangerous from a security perspective
* https://github.com/openenclave/openenclave/issues/3560 vbs
* Closing as question answered, but might reopen (and backlog) if a feature request is submitted
* https://github.com/openenclave/openenclave/issues/3551 relocation type...
* dropped from sig-arch
* https://github.com/openenclave/openenclave/issues/3597: related to meeting later today
* https://github.com/openenclave/openenclave/issues/3610 tcrypto apis
* Conclusion was to supply as a separate library. (Intel to come back later if they want it to be part of PE)
* Probably ok to close after confirming and meeting later today
* https://github.com/openenclave/openenclave/issues/3644 XSAVE
* Simon to contribute a writeup on this topic
* Keeping it on the board for next week
* Radhika nominates Anand to chair next week :)

## Archived notes for Jan 2020 - Oct 6, 2020 can be found below
https://hackmd.io/G7OAaHoyQK6wHn1CHMY-DQ