# Django ownCloud

This is our very draft plan to implement a FOSS online file storage service using Django.

@ellaeon started the project, joined by @M157q, we still need a Frontend Developer, though.

### The problem with ownCloud/nextcloud

Following may be biased, not thoroughly investigated.

#### Apps lack general architectural conventions

- No general, common class for model, view, controller. For example, many apps need to build their own view
#### Apps don't reuse/extend each others' components and capabilities

- With an Music app that can parse ID3 tags, no ID3 information is shown in the Files app
- Apps don't inherit and override one another's components, instead re-creating their own solutions
- Apps are indepent, instead of enhancing each other. For example, a photos app can't override Files app's previewer.
- Apps can't be registered to open a specific filetype, nor can them add filetype icons, etc.

#### Apps work together badly

- Clicking on a FLAC file in File browser prompts the download window instead of bringing you to the Music app

#### Speed leaves a lot to be desired

- Doesn't pre-compute thumbnails
- No cache on client side (localStorage or IndexedDB)
- No backend application multithreading, many extensions are not thread-safe

#### Impossible to support other protocols such as websocket

This is due to the nature of PHP and php-fpm/FastCGI/mod_php.

## Naming

- Feng Yuan 豐原
- Fo Tiao Qiang 佛跳牆
- Hun Dun 餛飩
- Gu Rou Tang 骨肉湯
- Tam Sui 淡水
- Kong Ngiug 焢肉（客家話）

## Comments

[django-comments-xtd](https://github.com/danirus/django-comments-xtd)

## File tree

### Tree
[django-polymorphic-tree](https://github.com/django-polymorphic/django-polymorphic-tree)

* doesn't support django 1.10 yet
* [django-rest-framework + django-polymorphic ModelSerialization](http://stackoverflow.com/questions/19976202/django-rest-framework-django-polymorphic-modelserialization)
* [Django Rest Framework general model serializer](https://blog.hipwerk.com/django-rest-framework-general-model-serializer/)

### ~query across models~
Use [django-model-utils](https://github.com/carljm/django-model-utils)'s [`InheritanceManager`](https://django-model-utils.readthedocs.io/en/latest/managers.html#inheritancemanager)

[Example on SO](http://stackoverflow.com/questions/14992693/multiple-models-one-common-primary-key-django)

**NO, polymorphic-tree already has this feature**

### Storage

https://github.com/chibisov/django-proxy-storage
Store metadata in DB or MongoDB

#### Extensions

https://github.com/jazzband/django-queued-storage
> This storage backend enables having a local and a remote storage backend. It will save any file locally and queue a task to transfer it somewhere else using Celery.

Future: create an extension with this feature

## Permission

object level permission: https://django-guardian.readthedocs.io/en/stable/userguide/assign.html

- need custom checker for inheriting permissions from parent directory

### Ownership

- Only the owner can change permissions
- exception: owner can decide to allow sharees to invite others
- sharees invite others -> can only have the same permission as the inviter
- Transfer ownership
- complex, I guess simply sharing with others is enough -> implement in the future
> M157q: Maybe we can have `owner`, `editor`, `viewer` permissions like Google Drive? The creator of objects become `owner` of them as default, and `owner` can do anything like set the permission of other people for obejcts he/she owned.
> pellaeon: yeah, I'd also like permission to be extensible, i.e. plugins can define their own permission. this should be possible with `django-guardian`

- move entire subtree to receiver's home dir
### Inheritance model
- Owner can do anything
- New objects do no associate any permission
- permission affects the entire subtree, until overridden by subnode's perm

### Performance and other considerations
- pre-fetch and cache all permissions traversing from node to root?
- key-value cache for each node?
- perhaps we only need to cache the ones with permission different from its parent
- Is it better to link permission with UNIX file owner and permissions?
- permission checker need to check 2 places
- this would be great for external applications to share a data directory. eg. FTP server can be set up serving the directory
- this means we'll need system account for each registered user (and group)
- file operations won't need to check DB, check filesystem instead, would this improve performance?

## Activity stream

https://github.com/justquick/django-activity-stream

## Notifications

[django-socketio](https://github.com/stephenmcd/django-socketio)

* only support socketio 0.6, but now it's in 1.4.5
* someone suggested against using socketio: http://stackoverflow.com/a/38558531

django-channels

* no socketio -> not needed

> M157q: I just found [tschellenbach/Stream-Framework](https://github.com/tschellenbach/Stream-Framework)
in [Django Packages : Notification](https://djangopackages.org/grids/g/notification/)
> + Stream Framework is a Python library, which allows you to build newsfeed and notification systems using Cassandra and/or Redis.
>
> It looks like we can use this Django app to build both activity stream and notifications.

> pellaeon: nice, this should be an even better solution :+1:

## IndexedDB Sync

https://github.com/mc706/offline_sync_demo

- no good support

## WebDAV

Quick intro to WebDAV: http://www.cs.unibo.it/~fabio/webdav/WebDAV.pdf

### Server
https://github.com/meteozond/djangodav

https://github.com/mar10/wsgidav
(not usable)

We're going to develop our own WebDAV lib: `django-rest-framework-webdav`

#### django-rest-framework-webdav

- WebDAV main rfc: http://www.webdav.org/specs/rfc4918.html
- SabreDAV additionally [supports](https://github.com/fruux/sabre-dav/blob/master/lib/DAV/Xml/Response/MultiStatus.php#L15) synctokens: http://tools.ietf.org/html/rfc6578#section-6.4

#### XML lib
django-rest-framework-xml only supports simple XML. not enough, so we're going to create our own new XML Renderer.

##### Sabre

Sabre implements base XML functions in [`Sabre\Xml`](https://github.com/fruux/sabre-xml/blob/master/lib/Element/Base.php), WebDAV elements in [`Sabre\DAV\Xml`](https://github.com/fruux/sabre-dav/tree/master/lib/DAV/Xml). We can copy its class structures.

Sabre-xml has no schema validation.

##### lxml

`lxml.etree`: is a generic API for XML and HTML handling. It aims for ElementTree compatibility and supports the entire XML infoset. It is well suited for both mixed content and data centric XML. Its generality makes it the best choice for most applications.
`lxml.objectify`: a specialized API for XML data handling in a Python object syntax. It provides a very natural way to deal with data fields stored in a structurally well defined XML format. Data is automatically converted to Python data types and can be manipulated with normal Python operators. Look at the examples in the objectify documentation to see what it feels like to use it.

ref: http://lxml.de/FAQ.html#what-is-the-difference-between-lxml-etree-and-lxml-objectify

`lxml.objectify`: You can plug additional data classes into objectify that will be used in exactly the same way as the predefined types.

--

lxml supports three schema languages: DTD, Relax NG and XML Schema.

- WebDAV XSD: https://github.com/lookfirst/sardine/blob/master/webdav.xsd (outdated?)
- No WebDAV DTD found.

### Client

owncloud: evert/davclient.js

### Spec Drafts and discussions

- CardDAV, CalDAV sharing: https://github.com/evert/webdav-sharing
- Note: [not for push notifications](https://github.com/pimutils/vdirsyncer/issues/171)
- [webdav PUSH based on HTTP long-polling discussion, 2014](http://lists.calconnect.org/pipermail/tc-push-l/2014-April/000025.html)
- [Apple's CalDAV push notif](https://github.com/apple/ccs-calendarserver/blob/master/doc/Extensions/caldav-pubsubdiscovery.txt)
### Reference

https://github.com/pimutils/vdirsyncer

## Settings & plugin system

Needs a system for all plugins to manage settings:

- CRUD frontend, backend
- Should provide simple inheritable classes and registers, similar to django.contrib.admin

## REST framework

http://www.devsplanet.com/question/35881201

Recursion fields:
http://stackoverflow.com/questions/21112302/how-to-serialize-hierarchical-relationship-in-django-rest

---

# Frontend

### Ideas
+ Maybe copy from https://github.com/sitcon-tw/roboconf (Semantic UI)
+ Use Vue.js
+ https://fe-driver.github.io/vue-beauty/#!/components/menu
+ https://luojilab.github.io/radon-ui/#!/common/audio
+ http://element.eleme.io/#/component/message
+ webpack ? https://github.com/owais/django-webpack-loader
+ gettext https://www.npmjs.com/package/django-gettext-vue-loader
+ Show Spinner when loading an Async Vue.js Component http://codepen.io/sirlancelot/pen/NNdyWJ

## Actual

### brainstorm

### Dynamic Vue async-components
- .vue files generated by Django template
- aren't they required to be built by webpack?
- entry point not in extension app, but in Fengyuan
- can webpack build js file with no entry point?
- yes, with [Normal chunk](http://webpack.github.io/docs/code-splitting.html#normal-chunk)
- async-components: any Vue.component() initiated object can be passed
- `resolve` is a generic callback, not vue specific. Anyway, object passed into `resolve` will be used as definition or options object
- use webpack component:
<code><pre>Vue.component('async-webpack-example', function (resolve) {
// This special require syntax will instruct Webpack to
// automatically split your built code into bundles which
// are loaded over Ajax requests.
require(['./my-async-component'], resolve)
})</pre></code>
- use vue-loader with webpack to load .vue files directly
- CSS in .vue seems to be loading fine
- imgs converted to webpack import, handled by webpack, should work
- what if extensions' .vue require global modules (like state modules), if these are built into bundle, don't they interfere with already loaded bundles?
- <del>note that extensions' build system (webpack) has no idea about parent modules</del>
- no, using `CommonsChunkPlugin` can extract common modules into common chunk. [ref](http://webpack.github.io/docs/code-splitting.html#commons-chunk)

### Allow only static HTML to be loaded
- much easier
- we can `$.get` the `template` string from server
- No CSS ?

> pel: no, this feels overly complicated for now

## Directory upload

Firefox 50+ supports the [HTML5 directory upload spec](https://wicg.github.io/directory-upload/proposal.html)

Chrome supports unofficial directory upload feature, used by Google Drive. Standardized version currently not supported. [stackoverflow question](http://superuser.com/questions/909112/does-firefox-support-folder-upload)

### JS libs

- flow.js
- unknown support for HTML5 dir upload
- chunked
- resumable.js
- unknown support for HTML5 dir upload
- chunked
- [uppie](https://github.com/silverwind/uppie)
- supports HTML5 dir upload
- XHR2

### Q: how does google drive handle this?
- chunked ?

### Notes
- HTML5 file picker may specify MIME type, might be useful

## Parse.com

Parse 基本上提供了一套 server + client SDK ，讓你快速做 app ，資料、檔案儲存、通知、物件 ACL 什麼幾乎都搞定了，甚至還有 cloud function （其實就是 RPC）

如果我們有 Parse 的話，再加上它的 client SDK, 基本上 app 和 desktop client 都做完了。

但問題是 Parse server 是 nodejs... 後端要接 MongoDB。 Parse server 和 client 使用 [REST API](http://parseplatform.github.io/docs/rest/guide/) 溝通。

https://github.com/cereigido/django-parsesync/ 可以將 django model 同步到 parse server

不過照這樣看起來，我們可能需要把 Parse REST API 用 django 整個重新實作一次

--

## Roadmap

1. Files tree
a. data directory scanner, should be compatible with ownCloud
b. [permission](https://github.com/dwatkinsweb/django-acl)
2. Basic frontend
3. django-rest-framework-webdav