# Keeep

A tool to inventory physical items. Designed with a focus on home inventory.

# Introduction

## Key features
All data (images, metadata, structure) is synced continusiouly with a third party service like Dropbox / Google Drive / Onedrive.

This adds a great deal of complication, but the reward is that users can easily copy & paste, drag and drop, search, and otherwise access from anywhere. It also means that intially, I don't need to create a CRUD interface.

For version 1, users will have the option to not sync with any cloud provider, and host everything on my server.

## Design Philosphy
Keeep is designed with the following tenets foremost in mind.

### Replicate the Real World

Careful consideration has been given to the data model to ensure that it can accurately reflect a real-world network. For instance, IP addresses are assigned not to devices, but to specific interfaces attached to a device, and an interface may have multiple IP addresses assigned to it.

### Serve as a "Source of Truth"

NetBox intends to represent the _desired_ state of a network versus its _operational_ state. As such, automated import of live network state is strongly discouraged. All data created in NetBox should first be vetted by a human to ensure its integrity. NetBox can then be used to populate monitoring and provisioning systems with a high degree of confidence.

### Keep it Simple

When given a choice between a relatively simple [80% solution](https://en.wikipedia.org/wiki/Pareto_principle) and a much more complex complete solution, the former will typically be favored. This ensures a lean codebase with a low learning curve.

### Terminology
`Item` is the foundational object for the app. It essentially represents a "folder" on a filesystem.

An item can have an unlimited number of children (subfolders).

An item can also contain an abritrary about of metadata. This occurs by reading a `info.yml` file in the directory.

`Filesystem` is an abstract term referring to either a local filesystem, or a remote filesystem provider. It is the raw implementation of a particular `Inventory`.

`Provider`: Ex: Dropbox, Google Drive, OneDrive

`Inventory`: A user can have access to multiple independent inventories. E.g., "Home" is a particular Dropbox folder they own. "Mom's house" is a Google Drive folder shared with Mom. "Warehouse" is a OneDrive folder owned by the company they work for.

`Account`: Represents a subscriber. Contrasted to `User` below.

`User`: Accounts can create or remove unlimited children "users" who have specific permissions (e.x., for particular inventories).

# Development

## Inventory Models

### `Inventory`
- `name`: String (Nickname to show users. E.g. "Home", "Mom's house", "Warehouse")
- `slug`: String (URL Safe: app.tld/user/slug)
- `account`: Foreign Key (`Account`)
- `provider`: ForeignKey(`Provider`)
- `token`: String (to authenticate with provider)

### `Item`
Caches data about a "folder" on a filesystem

- `name`: String (mirrors the FS filename)
- `inventory`: Foreign Key (`Inventory`)
- `parent`: Foreign Key (`Item`)
- `guid`: Guid64 (URL safe Base 64)
- `provider_item`: Foreign Key (`ProviderItem`)
- `info`: JsonData (Read from `info.yml`)
<!-- - `attachments`: ManyToManyField (`Attachment`) -->

#### Properties
- `permalink() -> String` (BASEURL + `/guid`)
- `parent() -> Item` Returns the parent item
- `children() -> [Item]` Returns the direct children as an array of `Item`s
- `path() -> [Item]` Returns the full path to this item
- `images() -> [Attachment]` Returns array of related attachments filtered by `type=image`

### `Attachment`
Mirrors a "file" on a filesystem
- `name`: String (mirrors the FS filename)
- `parent`: Foreign Key (`Item`)

#### Methods:
`attributes() -> [Attribute]` Returns all attributes
`filetype()`: String (`image`, `pdf`, `doc`, `other`)

### `Attributes`
Mirrors file attributes on a filesystem.
- `attachment`: ForeignKey (`Attachment`)
- `name`: String
- `value`: String

-------------

## Provider Models

For processing results from a remote provider.
### `Provider`
- `name`: String (ex, "Dropbox")
- `active`: Bool

This contains a list of providers that the superuser has whitelited as installed and available for use.

### `ProviderItem` (rename to `ProviderCache`)
- `inventory`: ForeignKey (`Inventory`)
- `api_version`: String
- `datetime`: Datetime
- `data`: JsonData
- `superceded_by`: ForeignKey(`ProviderItem`)

-------------

## Authentication Models

### `Account`
For now, inherit native Django properties. Roadmap:
- `username`: String (can be changed by user)
- `password`: Hash
- `email`: Email
- `payment_ref`: String (ex: for Stripe payments)
- `subscription`: ForeignKey (`Subscription`)

`username` must not equal any reserved keywords (see `views.py`)

### `User` extends `Account`
- `parent_account`: Foregin Key (`account`)

### `Token`
(consult Netbox)
- `token`: Hex
- `expires`: Datetime

Methods:
`getPermissions()` -- reads permissions from `Permission`

### `Permission`
(consult Netbox)
- `name`: String (ex: can_edit_item)
- `value`: Bool
- `data`: Arbitrary Object Reference (ex: `Item`)

Do we use this model internally to unlock paid features, as well as to allow accounts to regulate child users?

### `Subscription`
- `name`: String
- `description`: String
- `price`: Float

## `Classes`
### `Provider`
- `name`: String (ex, "Dropbox")

#### Methods
##### `await crawl()`
Crawls the API using credentials from the parent object's `token` attribute. API will return a list of entries, which we save (for future troubleshooting & faster queries) as a new `ProviderItem` object.

For each entry, we execute the following decision tree:
- If the entry is a folder, call this method again recursively. `await crawl(entry)`
- If the entry appears to be new (ID does not match any `ProviderItem.data.id`):
- Create new `Item`: `create_item(entry)`
- Else (the entry already exists):
- If this is a duplicate GUID:
- Determine whether this entry should be considered the original
- If yes, then assign a new GUID to the other one
- If no, then assign a new GUID to this one
- Add an entry to the event log indicating that this took place. Eventually users will see this in their dashboard or event log.
- If our previous cache was out of date ()
- Mark the old cache as "superceded" by this one
- Update entry (`update_item(entry)`)

###### Detecting modified files

Items/fold

New parent
New name (but that would be detected as a new item.... do we move it over somehow?)

Attachments/Files

New parent??
New filename
Changed hash

Item is deleted / removed

##### `create_item(entry)`
Takes an API result `entry` and uses it to create a matching `Item` object.
Will raise an exception if the `Item` already exists.

##### `update_item(entry, report_only=False)`
Takes an API result `entry` and uses it to find the existing `Item` object. If any fields are out of date, update them.

If `report_only` is supplied, then return a list of updates this function was prepared to make.

## URLs

### Inventory

| Path | View | Comment |
| -------- | -------- | --- |
| / | KeeepHome | Description of the app with signup links. If users are signed in, redirect to `/username` |
| `/<username>` | UserHome | User's dashboard / home screen |
| `/<username>/<inventory-slug>/` | ItemList(Root) | Item List (Inventory root) |
| `/<username>/<inventory-slug>/<guid>` | ItemDetail(guid) | Item Detail View |
| `/q/<guid>` | ItemDetail(guid) | Item Detail View |

https://keeep.app/allenellis/friendship -- Inventory Root [ this is a problem because it requires unique inventory names per account, which is permeable ]
https://keeep.app/q/GJVwtn/optional-item-slug-here -- URLs for each item. Permaurls

## Ramblings

I've got to figure out the business logic here.

### Dropbox primary
In this case there is no local copy of the user's data. All "filesystem" interactions are actually Dropbox API calls.

### Fatal errors
All fatal errors should be saved to an error log and ping Allen

### Metadata examples
The following fields could be auto identified by us and made more prominent somehow.
- `notes`: String
- `serial`: String, Serial Number
- `price`: Float, Price, including currency
- `quantity`: Int, Quantity
- `model`: String, Model
- `manufacturer`: String, Manufacturer.
- `url`: URL
- `tags`: [String]
- `receipt`: Image or PDF?
- `manual`: Image or PDF?

#### Possible future metadata?
- Support for depreciation

### Permissions
Future things to be carved into permissions:

- `can_[edit/read]_tokens`
- `can_[edit/read]_item`, including all children unless overridden further down
- `can_[edit_read]_subscription`
- `is_admin` (marks all privileges as 'yes')

### Preferences

### Item Stacks
A concept to allow for the concept of quantites.

If an item's name ends in ` (stack)`, then the following assumptions will be made:
- All direct children of this item will be represented as members of the stack, affecting a "quantity" integer of their parent.
- The user is free to rename child items as they see fit - it won't affect this relationship.
- Each child object will inherit all metadata and images of their parent, unless they override them.

To be clear, the definition of whether or not an item is actually a member of a stack, is simply whether its direct parent's item name ends in ` (stack)`.

##### Example stack:
- Main Warehouse Shelves
- Blackmagic HDMI to SDI converter (stack)
- 1
- 2
- ABC
- DEF
- BLKMG01
- BLKMG02

I could easily split this stack... example - we're setting up another warehouse and want to leave half of them there. Cut & paste operation from my local machine

- Backup Warehouse Shelves
- Blackmagic HDMI to SDI converter (stack)
- DEF
- BLKMG01
- BLKMG02

Each child item could have its own `info.yml` file to ovverride variables like "serial number", etc. The UI could have th ability to show draw this out as a table, showing children items as rows and columns for each piece of metadata identified in the parent item. Let people quickly update serial numbers and such.

#### Stacks of stacks
Does that break anything? When does it make sense?

#### Uses cases to consider for stacks:

- White Tie Warehouse - serialized items (Blackmagic Converters)
- White Tie Warehouse - nonserialized items (50' Edison cable)
- Kitty litter remaining
- Pills in bottle
- Groceries: # of pot pies remaining

#### Cases when this is not useful

- "Assorted bag of screws" - just keep as a single item
- "Box of light bulbs" - light bubls will come and go, do you really intend to keep an inventory up to date? If not, just indicate that this is where extra light bulbs belong, and be done with it.

### "Share" links
These would proably be an implmentation or extension of permissions. When a user chooses to geerate a "share" link for an item, the following would occur:
- New token is created with read-only access to this item
- The user is shown a confirmation screen including:
- A link to a perma url that uses that specific token (app.tld/user/public/shareid)
- The ability to set an expiration time for that token

This I guess would require creating a new model linking these new "share" URLs against the token they are valid with.

### Local only, no Dropbox
Allen doesn't want this. I don't trust my server with my inventory particularly. So the only reason to think about this would be
1. Getting a proof of concept up and running sooner
2. Future subscribers might desire this

In this event, we could still replicate the idea that the FS is trusted over the database.

Data and queries can come from the database, querying the `Item` class. It can operate like it's the primary source for read operations at least.

For starters I'll have a manually triggerable "sync" option that will read data from the FS and update the data in the database accordingly. In the future this will be run in the background periodically.

It turns out this is actually much harder because I can't trust FS edits because of potential duplicate GUIDs. So the only way "local only, no Dropbox" works is if I have a feature-complete CRUD GUI, which is a ways out.

##### Migrating between filesystem types
Not directly supported. Direct users to create a new account, copy & paste their files over.

To do this, we'll have concepts like a "user" owns multiple "inventories", and each of those have to stick with a particular driver.

##### Transfering ownership
If I sell an item to a friend, then I need a UI in-app to do this.

After I sell it to him, the item needs to be copied (?) into their account, and then removed from ours. Lots of details to think about here:
- Prevent users from spamming each other with transfer requests
- Wait until users accept the item before finalizing the transfer

##### CRUD edits
Initally, the application will be view only. All changes must be done via the filesystem.

Eventually, when we implement (for v1), any implementations of the `save()` method for any of my models that relate to the filesystem, need to be modified to follow this basic algorithm:

- Write updates to the filesystem
- But first, check to see if the filesystem updated before we were ready. If so, either auto merge or fail.
- Read from filesystem, compare to our desired state
- If data still doesn't match, try again (?) or fatal error
- If data matches, save the update to the database

> Should I version the data in my DB? Rather than overwrite it? Something built-in to my database provider that can handle this?
>

##### What to do about duplicate GUIDs?
(This is only an issue with remote providers. If we are using a FS that we control, then no problem)

Because we will save GUIDs as a field in `info.yml`, we need to deal with the eventuality that someone will duplicate a folder at some point.

The general plan is: the folder that was modified earlier is allowed to keep its old GUID. Whichever folder is newer is assumed to be a duplicate, and will have a new GUID assigned (that means we'll edit its `info.yml` file)

----

Unfortunately, duplicating a folder retains the original modified by date. So what else can we look at to determine which was the original?
- The old one would have its old path still intact. So do we have that somewhere handy to cross-reference against?
- The new one might have a new name (Like "Folder name (copy)")
- Maybe Dropbox has some extra metadata to help us out, but I'd rather be provider-agnostic with this solution so we aren't reinventing the wheel & making new tests with each provider

An older idea which I no longer like, was to always refer to the Dropbox GUID, but this would be different per provider, and then we either have very long URLs, or else have to make a mapping between my shorter GUIDs and Dropbox's - which is another point of failure (what if that gets out of sync?)

##### What to do about missing GUIDs?
If a user is silly enough to start nuking `info.yml` files, then that's their problem. We'll auto generate new ones just like it's a new file.

If we want to get really fancy, we could notice that a file used to have `info.yml` and notify the user with a notice.

### Search
Longer term (v2?) integrate search functionality. An advanced filtering tool could allow you to filter based on any particular metadata found.

### Input methods
People ingesting data via barcode, CSV, voice, etc?

Alexa endpoint. "Alexa, add this item to my inventory. Ok, where is it?"

Mobile app with camera support.

### Loaning functionality
Loan an item, track when it is expected to be returned.

## Crazy ideas
Support for Amazon ordering, track your Amazon order
"Subscriptions". I've run out of kitty litter.

Interface anything you buy and auto add to inventory

## Interested Parties
- Insurance comapny
- Me who needs something - do I own it?
- Me who shares things I have
- Me who needs to make sure I don't run out of things
- Me who needs to make purchasing decisions
- Me that needs information about the item (price I paid, is it under warranty)
- Warehouses?
- Grocery scanning app?

# Assorted Notes
Don't allow people to register with certain usernames
(`q`, `admin`, `accounts`, consider others)

# Plan of attack

## 1. Get proof of concept up and running
*Time estimate: 20-40 hours*
- Implement all of the models and fill them with barebones data.
- Create all needed methods, even if they return placeholder data. Mark with `#todo`
- Create the views
- Link with Dropbox API
- Write `crawl()` method

# Todo
- Beef up security at AWS, whitelist our IPs
- Get AWS security keys out of the configuration file
- Files that I upload to S3 need to be tagged or otherwise associated with inventories on my system
- When attachments are deleted, the corresponding file should be deleted from S3