# (DRAFT) Design: File-type resolution in Napari
Nathan Clack, Eric Perlman, 2021 Sep 3

## Problem
Napari has a pluggable interface for accessing persisted data. Data are stored in different file formats; one or more plugins may provide the ability to interact with these formats. This document concerns how Napari resolves which plugin to invoke when given a filename or uniform resource identifier (URI). Specifically, Readers will be considered.

There are two main sub-problems:
1. Determining the filetype given a string.
2. Determining which plugin to invoke given the filetype.

#### Concerns

1. Access to the resource via local or remote storage may be missing, limited, or slow.
2. The number of plugins under consideration may be large.
3. The number of file types under consideration may be large.
4. New file types and plugins may be added dynamically.
5. Users may prefer to use one plugin over another for a particular file type.
6. Users may have unique preferences about how file names or URI's are mapped to file type.
7. Sometimes file type can not be determined based on file name or URI alone.
8. The plugin interface is changing soon to support a way of declaring capabilities.

#### Requirements

1. The problem should be solvable in constant time with respect to the number of plugins.
2. The problem should be solvable in constant time with respect to the number of file formats.
3. Accessing the resource to determine or validate the format should be done as a last resort.

#### Current approach
The current approach is to serially iterate through installed reader plugins calling `napari_get_reader()`. Plugins are expected to return `None` as quickly as possible to reject incompatible files.

As a result the current approach is burdened with performance problems. It scales linearly in the number of plugins. Also, the current approach assumes an undue degree of trust in plugin authors; the current design affords plugins the ability to take an arbitrary amount of time.

To manipulate which reader a user prefers for a format, the user is required to reorder plugins[^1] so their preference is found first. This forces a requirement on Napari to maintain a stable order for the plugin list [^2]. This seems hard for the user to maintain.

[^1]: Verify this. I'm not sure how to actually do this.
[^2]: Is this really a maintained/documented invariant?

## Background

[Mime types](https://datatracker.ietf.org/doc/html/rfc2046) are a common standard for uniquely identifying file types. Plugins and users should be able to add new mime-types as long as they are unique. Operating systems usually have a facility for describing known mime types. These can be accessed using [`mimetypes`](https://docs.python.org/3/library/mimetypes.html), a standard python library.

## Design

A two-step process is used to resolve the Reader to use for a given resource.
1. Map the resource to a mime type.
2. Then map the mime type to a Reader.

#### Determining the filetype given a string representing resource name[^3]

File type detection should occur in stages, attempting to match types using the cheapest method available and progressing to more expensive methods until a match is found.

A resource name can be categorized as a URI if it is prefixed by a valid `scheme` (See [`RFC3986 Appendix A`](https://datatracker.ietf.org/doc/html/rfc3986#appendix-A)). Otherwise it is a file name.

If the resource name is a URI,

1. Attempt to look up the `scheme` in a dictionary mapping registered scheme's to mime types.
2. If that fails and the scheme is `http` or `https` attempt a `HEAD` request, and use the `Content-Type` returned in the header of the response as the mime-type, if possible.
3. Failing that, depending on the nature of the error[^4], attempt a `GET` request and attempt to use the `Content-Type` in the response header. Use a `Range` request to attempt to limit the `GET` to the first few bytes of the resource; the first 64 bytes may be used for file-type validation later on.
4. If the `GET` succeeded in returning the first few bytes of content, but did not have `Content-Type` in the response, attempt to match the first few bytes against a registry of filetype signatures. See, for example, the [`filemagic`][] library.
5. If none of the methods above succeed, give up.

If the resource name is a file name,

1. Attempt to look up the extension[^5] in a dictionary matching extensions to mime-type.
2. If that fails or there is no extension for the file name, attempt to read the first few bytes of the file and attempt to match the first few bytes against a registry of filetype signatures. See, for example, the [`filemagic`] library.
3. If none of the methods above succeed, give up.

As detailed above, this requires the application to maintain two dictionaries:
1. URI scheme -> mime type
2. file extension -> mime type

Additionally, it may be convenient to extend this scheme to allow users or plugins to register a set of regular expressions that can be matched against the resource name to force the mime type.

If the mime type cannot be determined, then the application could attempt to ask the user for input.

[^3]: the filename or URI.
[^4]: for example, move to the next step if the error is `405 - Method not allowed`. Don't move forward if the error is a `404 - Not found`. Error types may not be reliable, so maybe always move to the next step.
[^5]: The substring starting just after the last `.` character in the string and extending to the end of the filename.

[`filemagic`]: https://filemagic.readthedocs.io/en/latest/guide.html

#### Determining which plugin to invoke given the filetype

The application should maintain a dictionary mapping mime types to a collection of plugins that can service that type.

When a Reader is requested for a resource, its mime type is inferred (see above). That type is resolved to a collection of plugins that yield compatible Readers. The application will serially invoke each Reader in a defined order until one succeeds. The application should ensure a Reader fails if it belongs to a disabled plugin. If no readers are available for the mime type, the application should suggest compatible un-installed plugins present in the plugin index.

The order that determines the sequence in which Reader's are tried should be controllable by the user. Many approaches may work. For example, the plugins could be stored together with a score determining the user's preference so the one with the highest score is selected.

Plugins should declare the mime types they can handle. On installation of a plugin, the application should append a reference to the plugin to the corresponding mime type entries.

## Backwards compatibility

Currently, plugins cannot declare their mime types. Napari could ship with a predefined set of associations for known plugins with this limitation.

## Limitations/Concerns

- [ ] Mime types for zarr, n5?
- [ ] What does it look like when someone registers a new type?
- [ ] How does someone register a new scheme?
- [ ] What does it look like when a plugin author encounters the problem of registering a new file type?

Reader's are serially tried until one succeeds. Validation is probably part of the "try." As a result, errors from strict Readers may be hidden. Invalid files will invoke each of the Readers registered for a file type.

## NOTES/TODO

- [ ] This is more about mapping to Readers (and maybe Writers). Plugins may provide Readers. Revise to make wording more consistent. Narrow to readers in intro.
- [ ] Address non-http/https URIs. Should remote content be moved into a seperate doc?