---
# System prepended metadata

title: 'Vessels: Efficient and Scalable DNN Prediction on Trusted Processors'

---

Vessels: Efficient and Scalable DNN Prediction on Trusted Processors
===

## Introduction

* Deep learning is popular and prevalent
  * Object, voice recognition, image classification
  * Medical diagnosis, face recognition, autonomous driving
  * DNN
  * Training -> prediction

* Stealing and manipulation attacks on deep learning
  * Training data, model, architecture in the form of data
  * Data breaches, Trojan attack
  * Data at rest and in motion are protected by cryptography
  * Difficult to protect data in use (during prediction).

* Protecting DNN prediction using Intel SGX
  * SGX can protect the confidentiality and integrity of data during DNN prediction
  * EPC limit of 128MB
  * Linux: page swapping, Windows: can't run
  * One page swapping takes XXX cycles [CK: Cite]
  * DNN prediction requires large memory
    * DenseNet model is XXX MB and requires XXX MB memory for prediction
  * EPC is shared by all enclaves in the physical machine
  * Scalability issue with many prediction requests in parallel
    * Instant DNN prediction in the cloud (e.g., Siri, Alexa, [CK: Use the examples in the Neurosurgeon paper])
    * EPC thrashing

* Existing work and their problems
  * Fig. 1: Performance of existing DNN prediction with SGX
    * AlexNet (?)
    * Native, Eleos, TF-Trusted [CK: Use these as the baselines instead of Darknet-SGX in evaluation?]
    * Y-axis: Avg. response time (per request) [CK: instead of throughput?]
    * X-axis: Number of simultaneous requests
  * Eleos: We ported Darknet (a DL framework in C++) to run on Eleos
    * Replaces SGX page swapping with more efficient page swapping inside the enclave
    * Slow because not designed for DNN
      * DNN prediction requires to read and write many floating point numbers
        at a very high frequency
      * Frequent page swapping and address translation for each floating point 
        number
  * TF-Trusted: TensorFlow Lite on SCONE
    * SCONE
      * Reduces the number of context switching for system calls
      * Does not address page swapping, which is more significant to the performance of DNN prediction
    * TensorFlow Lite
      * Quantization: integer-only prediction for embedded devices
      * Changes the DNN model: affects model accuracy and doesn't support layers that require floating point operations
    * Slow because XXX
  * Both do not address the scalability issue

* In this paper, we focus on the memory usage of DNN prediction with SGX
  * Systematic study: found memory footprint of DNN prediction can be significantly reduced
    * [CK: Turn the bullet points O1-4 to regular sentences]
    
* A new system for efficient and scalable DNN prediction with SGX
  * Vessel: an optimized enclave for efficient and scalable DNN prediction
  * Reduces peak enclave memory without modification to DNN architecture or model
  * This reduces expensive SGX page swapping significantly
  * Brings the performance of protected DNN prediction with SGX to that of unprotected prediction
  * Carefully schedules to launch vessels for parallel prediction requests to avoid EPC threshing, hence minimizes the impact on scalability

* Key techniques of the system
  * Shared memory across layers
  * On-demand weight loading
  * EPC-aware scheduling: based on enclave size estimation before execution

* Experimental results

* Contributions
