# Open Enclave SDK SIG-Testing Meeting

**When:** weekly at 5:30pm Tuesday, US West Coast [(convert to your local timezone)](https://www.thetimezoneconverter.com/?t=17:30&tz=PT%20%28Pacific%20Time%29)

*(note the half hour offset start time; this is to suppor the PRC team joining in the morning)*

Join Zoom Meeting
https://zoom.us/j/99740211738?pwd=azU5VTFRNmtQY2hXeVZRbTlvQ3dodz09

Meeting ID: 442 805 832
Password: 010203

## Scope and Mission

This meeting is a place for technical discussions related to project testing.

The initial scope of this meeting includes anything related to test cases, driver testing, the CI/CD system, or third-party (downstream) test integration. We can clarify the scope over time; if this is too broad, we can split into subgroup meetings.

The Open Enclave project values being a friendly and inclusive space. This meeting is open to all community members. Everyone is asked to abide by the procedure documented below, and reminded that the community follows a [Code of Conduct](https://github.com/openenclave/openenclave/blob/master/docs/CodeOfConduct.md).

## Procedural Guidelines

The chair of the next meeting should attempt to **establish an agenda before the meeting**, but attendees may (and should) add to the agenda themselves in this shared document. hackmd.io is a collaborative editing tool, and everyone is welcome to edit this in real-time as well.

**At the start of the meeting**, the chair should ask for a volunteer to take minutes and roll call. Minutes should capture any decisions made and action items discussed. Minutes are stored in this document, and are public.

The chair should **get consensus on the agenda**, editing as appropriate. It is the chair's job to ensure that the meeting then follows the agenda, and that everyone is given an equal chance to speak.

Because the community is distributed by nature and this meeting may not include everyone every time, any decision must be followed up with an artefact (for example, a PR or an email to oesdk@ mailing list) to record the outcome.

## Agenda for the next meeting

### 22 September 2020
Chair: Radhika
Minute Taker: Ming-Wei

* Proposal to switch this meeting timeslot to a OE SIG-Arch meeting every other week (MarkS):
* Sept 22nd would be OE SIG-Testing. OE SIG-Testing would be every two weeks.
* Sept 29th: OE SIG-Testing would be cancelled and OE SIG-Arch meeting can be held in the timeslot
* Radhika: Will send out notifications to folks who are interested in joining the discussion.
* Ming-Wei: This may cause problems that people need to wait for two weeks for discussing architecture-related topics.
* Radhika: Will check with Dave to see if we can make this as a hybrid meeting.
* Question on support of Relocation types: Issue https://github.com/openenclave/openenclave/issues/3551 Relocation type R_X86_64_GLOB_DAT is not supported?
* Xiangping: This type is needed by PCL.
* Simon: What is the use case of R_X86_64_GLOB_DAT in the PCL?
* Xinagping: PCL uses weak symbols.
* Simon: Whether if R_X86_64_GLOB_DAT is relocated does not impact the functionality of PCL.
* Xinagping: Currently, this issue may affect digesting/signing. Require further investigation.
* Simon: We're planning to support R_X86_64_GLOB_DAT for multi-module support. We'd like to know what is the timeline that the support is also required by PCL.
* Mark: It apears that Intel PCL implementation does handle R_X86_64_GLOB_DAT, but need to check whether it really affects the functionality of PCL.
* Anand: What's the action item that can unblock the PCL porting?
* Xiangping: We have workround already.
* Mark: Can Xiangping submit a PR?
* Xiangping: Will submit a PR.
* OpenSSL test support: PR https://github.com/openenclave/openenclave/pull/3571 add OpenSSL digest test
* This is the first test. There will be other following test, will submit them one by one.
* Enclave lost error schedule.
* Radhika: We may not be able to support this by v0.12, but will support in the master once it's ready. Will that be ok?
* Mark: Will bring it up to the folks.

### 8 September 2020
Chair: Radhika
Minute Taker: Radhika

Meeing cancelled as there are no topics to discuss.

### 1 September 2020
Chair: Anand
Minute Taker: Radhika

**Agenda**
* Issue https://github.com/openenclave/openenclave/issues/3387 (Supports JIT compilation and PCL), Xiangping is doing a POC for PCL in OE, and can introduce how we deal with RWX/RX pages when loading enclave
* Xiangping: If machine supports EDMM, trts reverts back certain page permissions
* Ming-Wei: Is PCL in INtel SDK based on EDMM? Xiangping: No
* Xiangping: Currently working on porting PCL. without EDMM, with PCL, pages have RWX pages today.
* Simon: The SGX loader has to introduce a memory section that it loads as RWX pages. What controls this today? Is it an aspect of the application or is it explicit config as part of enclave creation.
* Juan and Xiangping: Additional tool to encrypt certain sections(adds W to those sections) before signing process.
* Simon: This should be available for free in OE loader.
* Juan: We create two named sections - code that needs to encrypted and data to be encrypted. These sections tell the tool which sections need permissions to change.
* Simon: If you have an SGX1 implementation, you will have RWX pages for the PCL loader.
* Simon: What is the problem you are seeing? Xiangping: Still investigating it.
* Anand: Does the PCL impl support debugging?
* Xiangping: Yes
* Anand: Have a POC for debugging multi module enclaves, might be useful in this context.
* Mark: Intersection between PCL and multi module loading interesting
* Can make code to be protected a module and encrypt that alone
* Simon: Have questions about sections and segments. Would like an example
* https://download.01.org/intel-sgx/sgx-linux/2.11/docs/Intel_SGX_Developer_Reference_Linux_2.11_Open_Source.pdf
* Sample - https://github.com/intel/linux-sgx/tree/master/SampleCode/SampleEnclavePCL
* Dave: Request that this information be put in a design doc and review in SIG-Arch?
* Xiangping is working on it and set up a meeting for the proposal and we can set up a special session to review it. Thanks!

* Issue https://github.com/openenclave/openenclave/issues/3449 (RSA key generation failure with OE OpenSSL), questions/discussions on register the rdrand engine for the APIs need to access random (Ming-Wei, Lily )
* Simon: You only need to only do this on initialization of the enclave. One API OK?
* Dave: Why cant it be default ang just work?
* Simon: We are using non default impl of RAND and should we not surface this to user? Users should know what crypto algos are getting used.
* Ming-Wei: rdtsc: Specifically, rdtsc not supported by SGX
* Simon: rtdsc incurs 3 calls to host. Terribly inefficient.
* Dave: Why doesn't the default that Intel does not work for OE?
* Simon: Intel changed default implementation of openssl rd_rand, effectively redistributing a fork of OpenSSL.
* Dave: rand_bytes changes could be upstreamed to OpenSSL
* Mark: OE solution is to leave it broken and then force everyone to provide a replacement.
* Simon: Currently implementation is the MVP. We could additionally hard code the default registration of our rdrand.
* Simon: The question is "Do we want to be transparent about this defaulting?"
* Simon: The predictive resistence is not the same as that of the default.
* Mark: Better does not always defined (?)
* Dave: config file controls what rand_bytes implementation is chosen. OpenSSL already supports that.
* Mark: Either force everyone to use one that works within the enclave or they could implement their own.
* Simon: APIs are built around its own implementation of rand_bytes. We have no plans to make the default implementations work within the enclave.
* Simon: What is the short term solution to unblock Open SSL use for AEs?
* Lily: Is the registration method only for SGX or for OP-TEE?
* Simon: Currently Open SSL is only for SGX. Plan is that OE might have to switch to a FIPS certified crypto module when available. The registration function when it happens, will work across OTEEs. Microsoft might contribute a FIPS certificed crypto (or it might not).
* Simon: Is it OK for short term to do explicit registration?
* Lily: Agree
* Simon: Would there be parameretization for initialization?
* Mingwei: We can only configure the source of entropy.
* Dave: We we have to plan enclave capable OpenSSL as part of OE SDK or outside it?
* Mingwei: Yes, our own binary distribution.
* Dave: Long term registration ought to be optional.
* Simon: Initialization currently is quite trivial. It could get complicated based on discussion.
* Dave: Discuss later.
* Simon: There was an attempt to use default implementation of rand_bytes. Callouts and rand_bytes need more security review.
* Timelines:
* Mark/Lily: We have near term workarounds. Currently using master branch. Will continue to use this for experimentation.
* v0.11: OpenSSL experimental
* Simon: Basic (non experimental) Open SSL v.12 October.
* Need it to be non experimental by October?
* Mark: Not holding up any releases in October. Good for now.

## Minutes from previous meetings
### 25 August 2020
Chair: Radhika
Minute Taker: Radhika

**Agenda**
* Yan has submitted a PR for edger8r. Thanks!
* Ming-Wei: Intel openssl expects 970 mitigations. In our impl, we have 8800 mitigations. We are 10x higher. We inserted many more lfences than Intel expects
* Ming-Wei has sent an email
* Ming-Wei has used CF mitigation
* Jian will take a look. Thanks!
* Note: OE only currently only inserts lfence before return instruction and indirect branches.

### 18 August 2020
Chair: Radhika
Minute Taker: Ming-Wei

**Agenda**
* Edger8r EDL test options
* Yan has run the Intel SDK EDL tests using OEedger8r, no bugs
* Using the older version (OCAML) of oeedger8r, the result was fine. The result of using C++\-based implementation is different.
* Yan has some questions on error message checking of Edger8r, it seems some errors are ignored, Yan is opening an issue for this.
* https://github.com/openenclave/openenclave/issues/3409
* tcmalloc test
* Xiaonan has questions on how we can contribute our tcmalloc tests to OE, it seems the tcmalloc PR is still not merged
* Xianon: Issue 3216 is not merged yet.
* Anand: We cannot make it to the v0.10 release. We can submit a PR to make it as an experimental feature.
* Shunda: Is there place that we can put tests for tcmalloc?
* Anand: Currently we don't have allocator-specific tests (only general ones), any tests are welcomed. We would like to make it as an experimental feature first along with the tests. We will move it out of experimental when we decide which allocator to be the OE default.
* OpenSSL LVI mitigation
* SGX-SGX repo checks in a modified OpenSSL perl script (https://github.com/intel/intel-sgx-ssl/blob/master/openssl_source/x86_64-xlate.pl) to generate the assembly files.
* SGX-SSL repo has a patch for assembly files (generated by the modified perl script): https://github.com/intel/intel-sgx-ssl/blob/master/openssl_source/lvi.openssl1d.asm.patch
* After applying the patch, SGX-SSL uses a preparation script to preprocess those pathed assembly files. The preprocessing logic is scanning the assembly files and searching for byte code that requires LVI mitigation. The .title directive is used by the script to skip the byte code that are constant value (e.g., table). *Note*: The preparation script is not released yet.
* With the above process, SGX-SSL repo only checks in the files that require LVI mitigation (https://github.com/intel/intel-sgx-ssl/tree/master/openssl_source/Linux).
* The .title directive is not used by assembler.
* Ming-Wei: It would be great if Intel can remove the .title directives (not supported by clang) in the checked-in files, so that OE can use them directly on both Windows and Linux.
* Mark: Intel will submit a PR to the SGX-SSL repo.

### 11 August 2020
Chair: Anand
Minute Taker: Radhika

**Agenda**
* Follow-up discussion of OpenSSL LVI Mitigation Support (Ming-wei, Jing) - related to PR for Open SSL port: https://github.com/openenclave/openenclave/pull/3335
* Ming-Wei Current problem: patches cannot be reused in clang
* Jing He will submit a script that they use to generate LVI mitigations for gcc.
* Anand suggests passing the assembly mitigated files directly to assembler. We have a way forward with existing patches in the repo
* Intel updates the sgx-ssl repo every time there is an openssl release also takes into account the Intel SDK release

* Follow-up to July 14 ACTION (Mark): Intel will publish a list of combinations of driver and SW Packages. Please see the Pull Request here: https://github.com/intel/SGXDataCenterAttestationPrimitives/pull/126/files
* Thanks Mark! (Mark: actually, thanks to Haitao!)

* Anand: Was looking at certain characteristics of sgx-ssl
https://github.com/intel/intel-sgx-ssl/blob/master/Linux/package/docs/Intel(R)%20Software%20Guard%20Extensions%20SSL%20Library%20Linux%20Developer%20Guide.pdf
https://github.com/intel/intel-sgx-ssl/tree/master/Linux/sgx/libsgx_tsgxssl
* Anand: Init section is handled differently in sgx-ssl than in open-ssl
* Anand: Robustness to multiple threads
* Anand: Going forward how do we ensure that we are implementing
* Mark: tpthread was added before the Intel SDK supported this. That is why this was there.
* Mark: have to go through the functionality to see if it exists
* Look at where it is used in openssl and see if it is used in a secure way
* Radhika: How did you identify the gaps and decide what POSIX APIs to implement?
* Jing-He: did an analysis to see what APIs would be supported
* https://github.com/intel/intel-sgx-ssl/blob/master/Linux/package/docs/Intel(R)%20Software%20Guard%20Extensions%20SSL%20Library%20Linux%20Developer%20Guide.pdf 5 Appendix lists supported functionality
*
* Radhika: Here is Ming-Wei's work
* Here is the list of unsupported tests https://github.com/openenclave/openenclave/blob/master/tests/openssl/tests.unsupported
* Here is the list of supported tests
https://github.com/openenclave/openenclave/blob/master/tests/openssl/tests.supported
* Shunda: Will look into porting Intel test cases.

## Minutes from previous meetings

### 4 August 2020
Chair: Brett
Minute Taker: Mark

**Agenda**
* Edger8r EDL test options:
* Run the EDL test in intel SDK for OE SDK to increase the coverage
* ACTION: Shunda: will run the Intel test case on the OE Edger8r.
* Anand: It is best to use the published OE Edger8r from OE release rather then build it from the repository.
* Anand: OE does not support the allow list or certain .dll functionality.
* Anand: OE supports Ocall deep copy.
* Shunda: Intel test cases test this.
* Review the differences between EDL test of OE SDK and intel SDK and add new tests
* Shunda: have reviewed the test cases from OE and from Intel SDK - there are about 80% overlap. Will recommend test cases to port.
* Testing of OpenSSL:
* We have a PR for Open SSL port: https://github.com/openenclave/openenclave/pull/3335
* Ming-wei: have a PR for OpenSSL 1.1.1. Are currently useing the test case from OpenSSL. Does Intel have any specific test cases for OpenSSL?
* Jing: Intel has SGX SSL port: https://github.com/intel/intel-sgx-ssl
* We have a test app: https://github.com/intel/intel-sgx-ssl/tree/master/Linux/sgx/test_app
* Ming-wei: request is to point OE team to Intel SGX SSL to unit/fuctional tests.
* Ming-wei: How are the assembly files with LVI mitigations in the Intel SGX SSL generated:
* Jing: we use the Perl script and then perform some manual changes. Some instructions were provided as byte-codes. Most of these needed to be manually changed.
* Simon: when OpenSSL updates and the assembly file is changed then is the script run again and then the manual update done again?
* Jing - yes, we must perform the manual update again.
* Ming-wei: How can OE get teh LVI mitigations?
* Current port does not modify any of the OpenSSL repo. If OE will make a copy of the Intel modified files for OE OpenSSL, then this will be the first modifications.
* Simon: need to see how complicated it is to do perl script modifications to fully automate the LVI mitigations. ACTION: Ming-wei will create an issue to discuss this process.
* Simon: if Oe supports the OpenSSL compatible surface, then the user can adopt the Intel SGX SSL to get LVI mitigations. These are not directly portable because of the Intel shim layers.
* Ming-wei: have TLS test that is planned to be ported.

## Minutes from previous meetings

### 28 July 2020
Chair: Brett McLaren
Minute Taker: Radhika

**Agenda**
* Tests for edger8r
* Brett: Discussion on the future of CI. Every compnay's tests are run before merge(PR merge). Current one is not going to work. A single Jenkins file cannot be run against different infrastructures due to variations in infrastructure.
* Stewart: Each company will have their own infrastructure and Jenkins file. Valid statement we should do it now.There is no business requirement that they should be identical Jenkins files. THere should be coordination on new items being added and items being removed.
* Brett: Validation strategies should stay the same.Attention needs to be paid to who has access to run the CI
* Stewart: We should define infrastructure such that is impervious to bad code
* We had an option of whitelisting a group of people. Whitelists get stale.
* Secret provisioning, Issue #3325: Example for github CI for PRs - did not find recommendation for secret provisioning.
* Action: Brett: Can you add best practices/recommendation to #3325
* Anand: Are we moving to a system where we dont have bors r+ and instead manually merge
* Brett: No manual merge, but no bors r+;something different (prow possibly).
* Anand: Should we invalidate approvals if someone changes the PR after it is approved?
* Alvin: Agree with Anand.
* Brett: We can unapprove it.
* Radhika to set up meeting for SIG-Testing so contributors from Europe can join

### 21 July 2020
Chair: Brett McLaren
Minute Taker: Radhika

**Agenda**
Discussion of the Future of CI
1. Automation
* Prow
* Pros/Cons
* Workload balancer for CI instances
* MVP
* Automated labelling
* Owners for code repos/trees
* Labelling for trusted code to build
* Might need more than 1 "test all command"
* Other options from prow?
* Manual, many pull request validations
* another merge bot for distributed ci
* Backend
* Post Submit server rebuilds
* How to manage containers and versioning for organization
* Infrastracture as code
* Onboarding other repos?
* Self Service
* Gate Keeper of the above
* Anything else we want to accomplish via automation that is not testing
* How to enable Developers of the SDK outside of CI, what can we do, what would people like to see
* Containerized Dev environments
* How can we get people to use it?
* How can we make this public?
* Testing
* Testing Matrix, for or against?
* Containerization vs host, do we need/want both?
* Do we still need SGX1 Machines?
* What about things that are not easily supported Rhel 8 containers etc
* Status Contexts
* If a job fails and logs are not available, does this gate the PR?
* How to align the developer and CI contribution experience
* CI "Limbo"
* Where to document all this?
* Onboarding of other repos
* Altered PR process
* Chatops
* labels
* closing/opening PR's
* alerting of code owners
* retrying tests etc.
2. Decoupling build infrastracture
* Test-infra
* CI Images versus Dev Images
* CI images support of non standard dependency installation?
* Dependency changes, where do they belong how do we handle
3. Metrics
* Monitoring of CCC, Azure, Intel instances
* How do we monitor Intel Instances?
* Prometheus is being looked at for Azure and CCC, no decision to be made yet.
* Grafana
* Code Coverage
4. Security
* Secret rotation
* How is everyone's infrastracture going to be configured for ports?
* Can this be up to the "owners" of the infrastracture?
* How to deal with securty concerns/reporting
5. Roll out strategy
* Phase 1, dogfooding
* Phase 2, oeedgr8r
* Phase 3, onboard another external ci (CCC split)
* Phase 4, openenclave repo
* Phase 5, Intel Inboarding

* Action: Brett will write up an issue and then we can get more eyes on it from people who will onboard to this CI system.

* [Fragment test](https://github.com/openenclave/openenclave/issues/3262)
* Should the test be modified to use fordblks/uordblks instead?
* Amaury found that the fragment tests use dlmalloc APIs. Can the PRC team use generic APIs so this works with snmalloc as well?
* Shunda: we will take a look. Assign to Xiaonan.
* Should there be at least an internal API exposing the total/in-use amounts, which this test can target, and which effectively becomes part of the pluggable allocator API?
* Anand: Will address in follow up meetings.

* https://github.com/openenclave/openenclave/issues/3284 samples/remote_attestation and tests/attestation_xxx failed on pre-production platform
* Yen Lee is the maintainer, but not here
* Shunda: Can contribute code to ensure that the CA is not hard-coded. Scenario: Dev uses a sandbox and has their own CAs. The failure happens in verification. The change needed is to sample code and change the TCB check code.
* Simon: Master branch will have CAs since that is secure. A branch can be set up and the tests can be modified that way
* Shunda: the suggestion is to change the sample code. Requirement is for users to get their CA to pass
* Simon: The CA value and TCB value are hardcoded in .c and .h files. One way forward is to issue guidance to users on how to recompile the code to test against these CA and TCB values. Updating the readme files in samples and attestation-would that be helpful? Shunda: Yes
* Simon: PRC team can issue a PR to the change in the attestation sample README file. Shunda: Yes, that is possible.
* Shunda: Will submit PR for README.
* The issue can be assigned to Yan Xue

### 14 July 2020
Chair: Anand
Minute Taker: Radhika

**Agenda**
* Discussion of Future Tests (Shunda):
1. Stress Test
a. memory management
* Shunda would like to expand on tests they have already added to OE.
b. multi-threads
i. Provide operation type (create enclave, do ecall and ocall, memory management, etc. for extend) interfaces to user
ii. Provide memory management type(malloc, calloc, realloc, etc.) interface to user
iii. Provide memory size interface to user
iv. Provide thread count interface to user
2. tcmalloc Test (Note: OE uses snmalloc)
a. Add tests for tcmalloc after tcmalloc (or snmalloc) merged into OE
Anand:
- OE will ship with liboesnmalloc.a in v0.10. There are two tests
that lock down the pluggability. [tests/snmalloc](https://github.com/openenclave/openenclave/tree/master/tests/snmalloc) and a [plugable_allocator sample](https://github.com/openenclave/openenclave/tree/master/samples/pluggable_allocator)
- Anand went over sample and perf improvements
- There is an unpublished port of tcmalloc to OE. [openenclave/gperftools branch: openenclave-port-2.7](https://github.com/openenclave/gperftools/tree/openenclave-port-2.7). Also a PR to lock down the pluggability:
[tcmalloc support for OE SDK](https://github.com/openenclave/openenclave/pull/3216). Intended for v0.11
* Anand: Do you have generic allocator tests for tcalloc that can be used for snmalloc? What were the tcmalloc tests you had?
* Xiaonan: Testing is similar to dlmalloc
* Radhika: Concern about testing multiple allocators will nX cost of testing infrastructure. We can add full allocator tests to nightly tests.
* Brett: For testing of allocators - use 2-3(1604, 1804 and Windows) configuration and test the allocator for each PR - this should catch 90% of the issues. We should run atleast one config in a container.
3. Edger8r Test
a. Add edger function tests after deepcopy ocall is supported.
Anand: oeedger8r should currently support deepcopy ocall.
Needs tests to lockdown.
b. Add edger function tests after Allow lists are supported
Anand: oeedger8r does not support allow lists since an ocall
is not allowed to make ecalls back to the enclave (reentrant ecall).
* Shunda : we can add some deep copy tests
4. Intrinsic Test
a. Check if there is intrinsic tests in OE, if not, we can add some test
* Shunda: These are compiler instrinsics. Intel SDK has a few tests.
* Xiaonan is reviewing and will evaluate if this is needed
* Anand: we dont have mm intrinsic header files. Will follow up with Simon
5. Other test/gaps?

* libUnwind
* edger8r
* Brett: We dont support CentOS etc. How often does a bug exist on CentOS when it does not occur on another operating system?
* Mark: Usually the issues are compiler and tool chain issues
* Shunda: Agree with Mark
* Anand: Does Intel CI test the latest version only or do you tests against older drivers?
* Shunda: Yes, but also have regression tests for stable releases
* Radhika: Can Intel publish a matrix of supported combinations of drivers and packages
* Mark: Will look into this.

### 7 July 2020
Chair: Alvin
Minute Taker: Radhika

**Agenda**
* https://github.com/openenclave/openenclave/issues/3219 (oesign: succeeded in signing enclave with a key file in wrong format)
* This issue is related to the PR below
* Simon: openssl does not reject this particular file
* Suggestion: Change the format of the file so openssl does not accept it.
* https://github.com/openenclave/openenclave/pull/3224 (add 3 oesign key cases and fix oesign digest usage typo)
* Adds 3 cases
* Simon: Are the tests testing crypto or oesign? The crypto lib passes certain cases, since oesign just passes the keys etc down to the underlying crypto. If oesign does not check for something it should, we should fix it.
* Equivalence classing: Test every possible input, even though output is the same. Test a class. Example order 3 is one class. Invalid format of keys is a class. Then look for something valuable-maybe config files. Config files have gaps in testing.
* Xiang Ping: oesign digest usage has a typo
* Anand: Are these tests for a certain scenario? Do you have equivalent tests in Intel?
* Anand: Is there process improvements that can be made? So we can discuss what should be tested next?
* Simon: Doing some test planning might be more efficient, so we identify gaps before hand.
* Mark: Good idea. Based on what we already have which can easily be ported.
* Shunda: Will list and discuss in the next meeting
* Mark: If we have a list, anyone who wants to contribute can pick something up.
* Simon: Code coverage numbers are getting lower
* PRs will start getting lower
* We will need to look at PRs coming in to verify they are adding sufficient tests
* Ming-Wei: Need to budget time for someone to invetsigate where code coverage is falling short.
* Shunda confirmed that 3rd party libs are included in code coverage numbers.
* Anand: of the 3rd party libs, libUnwind is not well tested, would appreciate tests for this.
* Simon: Intel had additional testing of libC, this is an interesting test gap.
* Mark: We add a proprietary test suite which may not be possible to be shared.

### 23 June 2020
Chair: Radhika
Minute Taker: Radhika

**Agenda**
* PR https://github.com/openenclave/openenclave/pull/3172/files
* Issue https://github.com/openenclave/openenclave/issues/3099
* (child_process_destroy test occasionally fails in CI)
* Jing has added sleep() into test code to walkaround the issue.
* Action: Jing will root cause this. Thanks!
* Action: Reevaluate enabling the test next week
* Issue https://github.com/openenclave/oeedger8r-cpp/issues/28 (Trusted memory address as ECALL in parameter, expected error result)
* We still think that we need to add check for outside enclave pointer in edger8r for SGX. SGX hardware doesn’t have the access limitation of accessing from enclave to host. Without the check of outside pointer, there might be security issues.
* Anand: It is not clear to me if you think there is a security issue with oeedger8r generated code or if you think there is a usability issue with the generated code. We think we don't have any security issue. We just marshal differently since OE SDK supports multiple TEEs. While SGX allows enclave to access host memory, OP TEE doesn't. The generated code therefore does not make an assumption that it can access host memory. Pointer inputs are completely serialized on the host side before being sent over. The generated code doesn't see the original pointers; but enclave side runtime copies buffer before handing it off to generated code.
I have aded a PR in OE repository that locks down passing enclave memory pointer back into the enclave as in param.
https://github.com/openenclave/openenclave/pull/3176
* Action: Shunda, request you to review PR 3176
* Issue https://github.com/openenclave/openenclave/issues/3141 (OESDK: Enclaves cannot access host memory by default)
* SGX hardware doesn’t have the protection for accessing host memory from enclave.
* This follows from the discussions of 3176. Lets continue discussing this next week if there are concerns with 3176
* Anand: Host is expected to fully serialize the buffers and the trusted runtime copies the serialized buffer in one memcpy before invoking the generated code.
* How do we notify users that upstream attestation is not working?
* Brett: Options: Email blast, badge, gitter notifications
* Mark: Poll the service before running tests, that would be the first thing that appears in log. A simple test that grabs a cert and checks that is is valid would be a simple way to check if the service is running.
* Brett: How simple can we make this? This is a good idea.
* Mark: This should be relatively simple
* Action: The open Enclave team will investigate this and come back when we have a design
* Brett: this affects not just contributors, also users who are running attestation samples, ctest etc

### 16 June, 2020
Chair: Ming-Wei
Minute Taker: Radhika

**Agenda**
* PR https://github.com/openenclave/oeedger8r-cpp/pull/27 (add tests for trust memory address and NULL as ECALL parameter) and its relevant issue https://github.com/openenclave/oeedger8r-cpp/issues/28 (Trusted memory
address as ECALL in parameter, expected error result)
* Anand: In the OE SDK, we copy all the input into a single buffer and pass it into an enclave on an ecall so the enclave cannot access host memory by default. This is so we can work with SGX and Trustzone
* This is different from Intel SDK
* If you print the address in the entry to the function in the enclave, the address is different in host side and enclave side.
* user_check implies, it is fully users's responsibility
* In both SDKs, the host cannot access enclave memory
* In Intel SDK, the enclave SDK can access host memory
* In OE SDK, the enclave cannot access host memory
* Please see https://github.com/openenclave/openenclave/issues/3141 for differences in the two SDKs.
* Issue https://github.com/openenclave/openenclave/issues/3099 (child_process_destroy test occasionally fails in CI)
* We are trying to reproduce the failure, How to setup an environment of "ACC1804 gcc Debug LVI e2e which seems to be the only platform with the problem?
* Brett provided information on how to get started. PRC team use ansible to create environment
* --repeat-until-fail <n>` where _n_ is the number of iterations to halt at if no failures are found.
* Use -v for verbose output
* PR https://github.com/openenclave/openenclave/pull/3106 (add 3 more oesign cases on config check)
* Action: Simon to review the PR
* Thanks a lot to Xiangping for rbasing on Simon's new signing test framework!
* Discussion on removing e2e testing from daily builds
* Brett: scripts are checked in and CI breaks. So we do need e2e testing.
* Short term: What we can do is implement a process till CI team tests and reviews it.
* Long term: If anything tests the scripts folder, then e2e test should be run. Otherwise, dont run
* retry solution was implemented, but we are hitting a corner case
* Anand: Did the PRC team have a complete end2end test for developer experience? Shunda: Not at this time.
* Short-term COnclusion. Keep e2d testing.
* Ming-Wei can we reduce the number of configs in e2e? The rest can be in nightly build.
* Brett: This seems perfectly fine.
* Action: Rob or Brett can reduce the number of configs
* Discussion on standardizing the process of tracking/maintaing repos (under the openclave github organization) other than openenclave
* Today we have oeedger8r, openenclave-site
* There is a plan to move all our 3rdparty dependencies (as a fork) under the openenclave organization
* How do we keep track of issues and PRs in all these repos?
* How do we test these repos?
* Brett: We can have a bot which pings people.
* Conclusion: Bot is a great idea!
* Ming-Wei: Suggests an organizational level bot.
* Anand: We can continue discussions this.

### 9 June, 2020
Chair: Anand
Minute Taker: Radhika

**Agenda**
* https://github.com/openenclave/openenclave/pull/3099 (child_process_destroy test occasionally fails in CI)
* Discuss how to reproduce the issue and whether it is caused by our test
* We have disabled this test since it fails sporadically in CI
* Shunda: This happens only with LVI builds
* Radhika: To repro sporadic failures, we can loop tests
* Ming-Wei: bors try run did not run tests with LVI mitigations. So probably does not test with LVI mitidations
* Simon: All LVI tests have an LVI suffixes.
* Simon: repeat tests until fail params for ctest
* `--repeat-until-fail <n>` where _n_ is the number of iterations to halt at if no failures are found.
* Use -v for verbose output
* https://github.com/openenclave/oeedger8r-cpp/issues/22 New issue about oeedger8r test log:
* Yan has questions about the test logs
Anand: This was an issue with the "virtual enclave loader" not gracefully
handling virtual-enclave shared library extensions (.so | .dll)
Have a PR for it: https://github.com/openenclave/oeedger8r-cpp/pull/23
Anand: Will add tests for deepcopy ocalls
* https://github.com/openenclave/oeedger8r-cpp/pull/21 OEEdger8r PR (add tests for structure that contains pointer) :
* Yan want to confirm the permission and next steps
Anand: Currently CI is run automatically for each PR/each push to a PR.
Also we require that each branch be up-to-date with master.
Anand: Will manually merge PRs and prioritize PRC team's PRs
* When you
* https://github.com/openenclave/openenclave/pull/3106 OESign PR (add 2 more oesign cases on config check):
* Xiangping want to discuss the next step
* Simon: https://github.com/openenclave/openenclave/pull/3121 makes adding tests simpler. Please see description of PR for guidance on extending current oesign tests.
* Simon: Feel free to change how tests are written
* Anand: Can we add code coverage for oesign? Yes
* Xiangping: Intel SDK has some 2-step signing tests
* Simon: Please feel free to add tests when the digest signing PR comes in. Note that the functionality is a little different from Intel SDK.
* Anand: Does Intel have fuzz testing for edger8r and sign tool? Shunda: No.
* Chair for next meeting: Ming-Wei. Thanks!

## Minutes from previous meetings
### 2 June, 2020
Chair: Anand
Minute Taker: Radhika

**Agenda**
* https://github.com/openenclave/openenclave/pull/3085 (Add OE stress test of ecall)
* Need to clarify the test point.
* Anand: 26 seconds for each CI run.
* Action: Radhika to review it and we can merge it.
* Brett: I would like to see a plan on how to aggregate data in the future
* Ming-Wei: This should be wrapped in a cmake. And instead of hardcoding values for number of runs, it could be a param
* Anand: We can access as is right nowbut eventually need to factor in Ming-Wei's suggestion above
* https://github.com/openenclave/oeedger8r-cpp/pull/4 (add tests for deep copying where the size attribute is correctly used)
* Is there any CI test needed for the oeedger8r-cpp repo? Can we merge the PR?
* Thanks Xue Yan for adding this test
* Action: Radhika to give PRC team permissions for bors try
* Action: Anand, once Brett has CI going, we can merge this PR

### 26 May, 2020
Chair: Anand
Minute Taker:

**Agenda**
* https://github.com/openenclave/openenclave/issues/3070 (New test design: OE SDK stress test)
* How to integrate stress tests into CI? Is there any guideline for stress test?
* These are long running and might crash
* Nightly builds run libC tests
* Shunda: Make 1000s of ecalls and ocalls runs, not for performance, planning to use it to find corner cases where these would fail. PRC team will provide more details on what the test would be in the issue.
* Brett: Suggest adding manual test and then figure out how to put this in CI/CD.
* Shunda?: manual tests exist ... how do we run it and see the logs?
* Aeva: separate proposal (in the future). Can CI run performance regression tests?
* Aeva and Brett: Graphing of failures would be useful. Currently we have the dashboard and badges. Can improve on this.
* Aeva: Can the PRC team run the stress tests and report tests and we link against the webpage.
*
* Shunda: we run a multi-day test suite before release
* Radhika: maybe this could be part of our nightly job
* xzhao: We can control the time the test is run(via param). No big difference in problems found in test running 1/2 day vs 2 days
* Ming-Wei: Would be useful to see an instance of the test and also see a log.
* Shunda: Will share the test and report.
* Next steps: AR : PRC team will add proposal to issue 3070.
* https://github.com/openenclave/openenclave/pull/2887
Switchless test for one tcs.
This required a change to the SDK to carefully order ecalls and switchless worker thread creation during enclave creation. Both the source code fix and the test have been merged into master via PR https://github.com/openenclave/openenclave/pull/3065. #2887 can now be closed. Thanks Yanxue.
* Thanks to PRC team for iterating on this
* Anand made a source code fix and the test went in along with the fix.
* https://github.com/openenclave/openenclave/pull/3034 The child-process test is now reenabled in master branch and passes CI. It fails however in user machine since driver may not necessarily be up to date. How do we handle this?
* Looks like there is potential sporadic failure?
* In 1.33 driver, the behavior has changed.
* Test seen to fail sporadically
* Shunda: Suggests that PRC team can take a look. Will confirm with Intel driver owner.
https://oe-jenkins-dev.westeurope.cloudapp.azure.com/blue/organizations/jenkins/pipelines%2FAzure-Linux/detail/Azure-Linux/488/pipeline
* C++ edger8r repository: https://github.com/openenclave/oeedger8r-cpp
* Benefits
* C++ instead of Ocaml
* Low barrier to contribute
* Easier to implement new features (e.g: variable-sized outputs)
* Tooling - Ocaml not well supported on Windows. Earlier workflow used to require about 20-30 minutes to build the OCaml compiler. C++ edger8r can be compiled in a matter of seconds.
* Feature-complete in terms of code-generation
* All existing tests are incorporated. Thorough testing in terms of code-generation.
* Uses a mock host/enclave environment to execute and test edger8r generated code without depending on OE SDK/SGX. "Virtual Mode" of execution as opposed to "Simulation Mode".
* Needs more work. Not expected to be high development cost.
* Incorporate front-end checks earlier provided by the shared OCaml front-end.
* Simple functionality like untrusted-dir/trusted-dir.
* How can we test more thoroughly?
* Anand: Requests more testing around edl and edger8r. Would like to leverage any insights and tests that Intel has for this.
* Shunda: Will look into moving more tests to OE SDK.
* Anand: Our new C++ edger8r has no warnings that Intel front end generates. More tests that Intel supports will definitely help. Our old edger8r used Intel's parser and front end. There are other low hanging fruit related to Intel SDK's frontend.
* Yan Xue: If we add more tests which repo should we add it to?
* Ming-Wei: Thinks the tests should be added to openenclave https://github.con/openenclave/openenclave repo.Anand agreed.
* Xiangping: In the future we will not need ocaml for openenclave.

### 19 May, 2020
Chair: Ming-Wei
Minute Taker: Mark

**Agenda**

* https://github.com/openenclave/openenclave/pull/3046 (2 oesign cases: 1. sign w/ empty config 2. sign with config containing neg vals)
* We closed PR #3020 and submit #3046, so it is the same with #3020, need review and BORS try.
* The bors try has passed - Jordan will sign off, it can be merged.
* https://github.com/openenclave/openenclave/pull/2887 (Add switchless test for one TCS)
* The case timeout is because a failure on switchless enclave worker-thread code before calling our test code.
* Anand will investigate. Xue Yan can help
* https://github.com/openenclave/openenclave/pull/3034 - this failure is due to a change in the driver which now permits a forked child to make an ecall to the enclave. The logic in the test needs to be inverted to expect a child Ecall to succeed.
* This requires the 1.33 version of the driver.
* Jing will test with the new driver.
* Anand: team is considering converting Edger8r from OCAML to another implementation. Looking for more tests for the Edger8r to ensure that any changes can be tested.
* Shunda: team will look into porting more Edger8r tests to OE.

### 12 May, 2020

Chair: Radhika
Minute Taker: Radhika

**Agenda**
* https://github.com/openenclave/openenclave/pull/2965 Add switchless worker threads sleep and resume test
* [Radhikaj] Submitted a bors r+ request
* https://github.com/openenclave/openenclave/pull/2887 Add switchless test for one TCS
* [Radhikaj]Here are the logs: The following tests FAILED: 18 - tests/switchless_one_tcs (Timeout) Errors while running CTest. Here is the config it is failing on : Sim 1804 clang-7 SGX1-FLC Release and Sim 1804 clang-7 SGX1 Release.
* [Anand] Cast to volatile in the while loop so compiler does not optimize to a forever loop
* [Anand] Will submit a PR to the author's branch to speed this along
* https://github.com/openenclave/openenclave/pull/2774 Add fragment tests
* [Radhikaj] Submitted a bors r+ request to merge it
* https://github.com/openenclave/openenclave/issues/3011 Sample project remote_attestation failed in execution on a SGX FLC platform
* [Radhikaj] From the log it looks like the remote attestation sample passed
* [Radhikaj] Remote sttestation does not work outside of Azure VMs. This is currently being worked on by ShanWei and YenLee
* A question from Shunda: DeepCopy is not supported in Ocall, any plans for it? We are reviewing OE DeepCopy implementation and testing and want to add more tests for DeepCopy.
* [Anand] Was not prioritized as there was no immediate scenario
* [Ming-Wei] Can be implemented if there is a scenario
* [Shunda] There was a bug found and fixed in Intel deep copy scenario. Was trying to follow up to see if there was a similar issue in OE SDK
* [Mark] Is this a scenario that is widely used? We should prioritize if it is widely used.
* [Shunda] It is not widely used AFAIK.
* [Mark] If someone explicitly asked for it, will let us know.
* To get logs
This is the issue that describes the lack of logs: https://github.com/openenclave/openenclave/issues/2878
It describes workaround to access the pipelines https://oe-jenkins-dev.westeurope.cloudapp.azure.com/job/pipelines/ and look up the log via job number

### 5 May, 2020

Chair: Radhika
Minute Taker: Radhika

**Agenda**

* https://github.com/openenclave/openenclave/pull/2965 add switchless worker threads sleep and resume test
* Anand signed off, Jordan has comments pending resolution
* Action: Yan Xue: Please resolve comments
* https://github.com/openenclave/openenclave/pull/2888 add enclave loading testing in simulation mode
* Has two sign offs. Needs DCO and squash
* Action:JingHe: Please resolve conflicts and then we will merge. The conflict is only in fromatting
* https://github.com/openenclave/openenclave/pull/2887 add switchless test one TCS
* Changing this to static volatile status_e switchless_func_status would prevent the hang in release mode
* Action: yanxue Can you incorporate the volatile change requested by @anakrish? we can then merge this PR
* https://github.com/openenclave/openenclave/pull/2774
* The fix is to disable the test if USE_DEBUG_MALLOC is true/on.
* Action: Radhika: Suggest commit and Yanxue can incorporate: In tests/CMakelists.txt if USE_DEBUG_MALLOC is true, skip this test
* Action Anakrish: Will run the test in simulation mode to see if this happens
* Dave: what happens when dlmalloc in non debug mode fails? Would it get a nullptr? Or can it be a grabage pointer
* Anand: Within the enclave dlmalloc returns null when out of memory. We already have a test for this.
* Dave: Linux behavior(default), malloc returns a garbage pointer and not null in OOM condition
* Mark: The test does test for fragmentation. If the enclave properly configures a heap range, then the malloc should always point to a valid address in that heap range.

* Anand: Would like tool to see memory paging cost
* Action: Anand open an issue in openenclave repo to discuss options
* Mark: There are some methods in Linux to check for how much paging has occured etc.
### 28 April, 2020

Chair: Radhika
Minute Taker: Radhika

**Agenda**

* https://github.com/openenclave/openenclave/pull/2939 (One more child process case) Need feedback and next steps
* Approved
* Needs DCO
* Address one comment
* https://github.com/openenclave/openenclave/pull/2888 (Add enclave loading testing in simulation mode)- BORS success when Jing tried, but fail when merging, seems network issue, need retry?
* CI fails. Logs attached to issue
* FAILED: tests/sim-mode/enc/CMakeFiles/enc.dir/props.c.obj
* https://github.com/openenclave/openenclave/issues/2920 (Pthread and oethread tests timing out) Need to discuss the next steps
* https://github.com/openenclave/openenclave/pull/2887 (add switchless test for one TCS) BORS build failure, similar to 2888
* Failure is not similar to 2888. The specific test this added is timing out
* 17 - tests/switchless_one_tcs (Timeout)
* Code coverage: Priority for OE devops? Perhaps OE SDK could use codecov.io? e.g. https://codecov.io/gh/microsoft/CCF (douglasmaciver)
* CD: Should OE SDK invest in automated release-packages testing? (douglasmaciver)
* Pulled from SIG-Testing board: https://github.com/openenclave/openenclave/issues/2798
* Merge enforcement alternatives
* Intel for feedback?
* Pulled from SIG-Testing board: https://github.com/openenclave/openenclave/issues/2347
* Cross platform build/test scripting

**Minutes**

* https://github.com/openenclave/openenclave/pull/2939 (One more child process case) Need feedback and next steps
* Signed off
* Needs DCO
* Address one comment
* Action: JingHe can you DCO the change and address the comment?
* Anand: These are great tests
* https://github.com/openenclave/openenclave/pull/2888 (Add enclave loading testing in simulation mode)- BORS success when Jing tried, but fail when merging, seems network issue, need retry?
* CI fails. Logs attached to issue
* FAILED: tests/sim-mode/enc/CMakeFiles/enc.dir/props.c.obj
* MingWei: Suggested a fix. Thanks!
* https://github.com/openenclave/openenclave/issues/2920 (Pthread and oethread tests timing out) Need to discuss the next steps
* ErrnoTests in same and different enclaves
* Jordan:: to repro run the test a bunch of times in a row. Compiled it with LVI and a release mode build. Action Jordan: Please add details to issue.
* Action: XueYan, please try to repro it.
* https://github.com/openenclave/openenclave/pull/2887 (add switchless test for one TCS) BORS build failure, similar to 2888
* Failure is not similar to 2888. The specific test this added is timing out
* 17 - tests/switchless_one_tcs (Timeout)
* Anand: has a fix in enclave termination for switchless manager. Expects test will pass. And has set up a bors try again. Action: Anand to follow up
* Code coverage: Priority for OE devops? Perhaps OE SDK could use codecov.io? e.g. https://codecov.io/gh/microsoft/CCF (douglasmaciver)
* Ming-Wei: oe-sdK team has a prototype. oe-sdk exports code coverage report. CI/CD team needs to take that report and integrate it into existing pipeline.
* Anand: Is our code coverage for end user
* Ming-Wei: We can have code coverage for end user and for OE SDK
* CD: Should OE SDK invest in automated release-packages testing? (douglasmaciver)
* Douglas: The idea is to test packages that we release in CI
* Radhika: We have this for everything except for 2019.
* Douglas: Will add 2019
* Pulled from SIG-Testing board: https://github.com/openenclave/openenclave/issues/2798
* Merge enforcement alternatives
* Radhika: Can Intel provide feedback on above issue?
* Action: Mark to follow up with David Tian
* Douglas: Still investigating this.
* Pulled from SIG-Testing board: https://github.com/openenclave/openenclave/issues/2347
* Cross platform build/test scripting
* Jordan: was trying to fix issue in a sample and had issues with bash related to file pathing etc. Bash on Windows fails in unexpected ways. This is not urgent since nothing is broken currently. No immediate benefit.
* Douglas: Also advocate for Python. If there is an opportunity or reason to change, we can revisit this.

## Minutes from previous meetings
### 21 April, 2020

Chair: Radhika
Minute Taker: Mark and Aeva

**Agenda**

* PR #2888 https://github.com/openenclave/openenclave/pull/2888 (Sim-mode test, no feedback yet)
* PR #2887 https://github.com/openenclave/openenclave/pull/2887 (“one TCS” Switchless test, no feedback yet)
* PR #2774 https://github.com/openenclave/openenclave/pull/2774 (Fragment tests, BORS test build fail?)
* Issue #2889 https://github.com/openenclave/openenclave/issues/2889 (Unexpected return of “one TCS” Switchless ecall)
* Issue #2798 -- https://github.com/openenclave/openenclave/issues/2798 (Merge enforcement alternatives)
* Issue #2878 -- https://github.com/openenclave/openenclave/issues/2878 (Some jenkins builds don't have links to the build logs for each pipeline)

**Minutes**
* PR #2888 https://github.com/openenclave/openenclave/pull/2888
* Sim-mode test, no feedback yet - Anand to review
* PR #2887 https://github.com/openenclave/openenclave/pull/2887
* “one TCS” Switchless test, no feedback yet - Anand has added comments, please review
* PR #2774 https://github.com/openenclave/openenclave/pull/2774
* Fragment tests, BORS test build fail? - Radhika the test was timing out. Anand to review the test failure. Radhika will assign the PR to Anand
* Issue #2889 https://github.com/openenclave/openenclave/issues/2889 (Unexpected return of “one TCS” Switchless ecall)
* Issue #2798 -- https://github.com/openenclave/openenclave/issues/2798
* Mark to remind Intel CI/CD team to review the issue.
* Issue #2878 -- https://github.com/openenclave/openenclave/issues/2878
* Some jenkins builds don't have links to the build logs for each pipeline)
* Radhika - there is a workaround documented in the issue
* Jordan: This is a bug in Jenkins, probably will not be fixed in the near term. When a PR fails for this case, put a link to the #2878 issue. Aeva: added comment on the issue to this effect.
* Jordan will look in Jenkins open issues - if he finds a fix, he will post details in the issue.
* Radhika - send this issue to the distribution list
* Dave: Put a note in the Contributing.md documentation, near BORS dashboard link
* Issue #2890 (issue is closed) -
* Anand: The log looks like the test had not been run.
* Jordan: the test was run- the results are presented differently.
* Discussion of how we track items and set agenda
* Simon: we have a SIG-Testing project board: Do we want to track action items on the project board?
* Aeva: We don't have a SIG-Testing Chair (yet... pending ratification of Governance docs update - https://github.com/openenclave/openenclave/pull/2778 )
* Jordan: anyone who has an issue can add it to the agenda. It would be nice if the chair went to the board and pulled the items from be board.

### 14 April, 2020

Chair: Mark
Minute Taker: Radhika

**Agenda**

* Issue #2798 -- https://github.com/openenclave/openenclave/issues/2798
* Issue #2805 Jenkins GitHub multi-status checks -- https://github.com/openenclave/openenclave/issues/2805
* Issue #2746 add spinlock/multithread/errno check in same/different enclave https://github.com/openenclave/openenclave/pull/2746
* Issue #2774 add fragment tests https://github.com/openenclave/openenclave/pull/2774
* Issue #2773 Add test cases about child thread https://github.com/openenclave/openenclave/pull/2733
* Adding more tests for Switchless ECall Feature
* Samples test is timing out again... Fixed in PR #2861?

**Minutes**
* Issue #2805
* In a nutshell- leverage different Jenkins servers and spread out load to enable scalability
* What would change as a contributor
* Brett: We are polling on PRs and automatically kicking off builds. This is temporary
* Simon: If you poll on PRs and kick off testing- this will overload the system since folks update PRs a lot.
* Brett: In the short term - the system runs tests and merging is left to the committers
* Brett: In the long term - bors will not be used as it has limitations
* The main advantage is that there are multiple Jenkins backend servers which share the load
* Jordan: Can this splitting between multiple servers be independent from front end?
* Brett: Yes. This parallelizes jobs.
* Brett: Bors is a bottleneck
* Dave: bors try can be parallelized. Bors r+ should be serialized.
* Simon: bors should batch and run tests and merge them together
* Next step: Action: Brett : Split up backend servers. Keep control mechanism as bors.
* Permissions management is separate
* Issue #2746
* Anand: currently this is a docker login issue. We will continue to try to merge. No further action required from contributor
* Issue #2773
* Jordan: This has a build failure. Action: JingHe can you fix the build issue
* Issue #2774
* Jordan: tests/memory is timing out. Action: XueYan: Can you please take a look at this?
* Switchless ecalls
* Shunda: Would like to add more tests.
* Anand: there are a few tests for switchless tests.
* PR #2861
* Hope to fix samples test timing out
* Dave recommends that we split samples tests into multiple tests

### 7 April, 2020

Chair: Aeva
Minute Taker: Radhika

**Agenda**
* add fragment tests <https://github.com/openenclave/openenclave/pull/2774>
* add spinlock/multithread/errno check in same/different enclave <https://github.com/openenclave/openenclave/pull/2746>
* Looks like Intel is setting up additional OE CI. Any blockers/anything to discuss? +talk about automated comments on OE PRs

**Minutes**
* PR #2774
* Jordan: Test needs to be reproducible, test uses random, need ability to repro failures. See comment: https://github.com/openenclave/openenclave/pull/2774#discussion_r405109269. Print seed on each run and to repro, take seed as input. Action: Jordan will write a snippet
* XueYan: Found a leak bug, opened #2776. Action to Radhika: Plan this for v0.10.
* Aeva: Test is testing heap frangmentation and leaks
* Radhika: We can remove the check in the end for heap checks and merge the PR. When #2776 is fixed, the heap check can be enabled
* Action to Jordan/Anand: What does dlmalloc return when malloc fails? If it returns null, then the test works as expected
* PR #2746
* PR has been approved, current CI failure. Action for XueYan: Investigate failure
* Intel setting up additional OE CI:
* David Tian is working on a POC for Intel CI integrated with OE CI.
* PR #2709
* https://github.com/openenclave/openenclave/pull/2709
* This PR is going to be dropped and HeJing will add to an existing test instead
* Issue #2798
* Discussion for community to weigh in on pros and cons of alternatives to bors, gather goals / requirements for CI systems
* Aeva: Use #2798 to document requirements for "Intel CI integrated with OE CI" item above

### Tuesday, March 31
**Chair:** Aeva & Mark (co-chairs)
**Topics:**
* Welcome everyone to the new meeting
* ask for topics
* Topics:
* add test cases about child thread https://github.com/openenclave/openenclave/pull/2733
* append SDK testing https://github.com/openenclave/openenclave/pull/2709
* Change heap-pages of tests to a value appropriate for snmalloc https://github.com/openenclave/openenclave/pull/2730
* Separating HW CI/CD from simulation CI/CD? (is there an issue open already to track?)

**Minutes**
* Current process: 2 approvers and no explicit block
* add test cases about child thread https://github.com/openenclave/openenclave/pull/2733
* Test expects that ecall to succeed.
* OE returns an error
* We should open another issue to investigate, the test itself will not succeed
* We should discuss what the expected behavior should be
* OE should not segfault, should have graceful handling
* Action to Hejing (sp?) - open issue describing expected behaviour
* Action to Anand: work on debugging & fixing current behavior
* Fix the issue related to segfault and then merge the test PR #2733
* append SDK testing https://github.com/openenclave/openenclave/pull/2709
* enclave types and flags are described here: https://github.com/openenclave/openenclave/pull/2709#discussion_r398971576
* Action to Hejing: look into flags and design the test matrix
* We can further discuss the simulate and debug flags for OP-TEE: Action to Dave: please comment against Simon's comment above
* https://github.com/openenclave/openenclave/pull/2730
* Bumping up size of enclaves to allow for snmalloc
* Anand will try to reduce it to 2048 heap pages and run tests
* Simon points out that #2709 adds enclaves with small heap sizes which has an intersection with PR #2730
* Anand recommends: All enclaves should have a min size of 2048
* Windows does not do EPC paging, if a VM has a small EPC size then a max size of 2048 would be problematic. Available EPC is a static number for a VM.
* Action to Jordan: Open an issue to document the limitations on EPC size for CI/CD to run successfully
* how do we delegate "bors try" permissions? action CGC: discuss / define policy for this
* item for next meeting: separating HW CI/CD tests from simulator CI/CD tests to match Aeva's diagram presented to the CCC