###### tags: `R4MP`

# Data Source

Capture having a separate data source that can be shown in grid.

- Where to define in config. Layers? Alt section?
- Has similarity to layers. an id, a type, possibly a url, fields, attribute loader.
- No position in layer stack
- Do we auto-legend everything if no legend config provided?
- Does a "cosmetic" data source make sense? Don't think so.
- Legends do need to bind via id, so would make sense to have them be layerish
- Formats to support?
- ArcGIS table endpoint
- JSON file (what schema? proposing array of attribute objects. additional format for efficiency: header array, data array)
- GeoJSON file (we strip off the geometry)
- CSV? (what schema? first row attribute name, every other row values)
- Legend support
- Should be able to have entry
- Click opens grid, same as layer
- Reduced other options (no visibility checks, no settings, only menus like remove/reload/open table)
- Details support
- hide symbology/zoomies
- Disable highlighting
- Would only be accessible via grid details button, or an api call / event to open details. Identify will not consider data layers
- Grid support
- hide symbology/zoomies
- support data source in merge grid
- When to download
- On load?
- On table open
- How to support a request for detail prior to table open (via api)
- Non arcserver source would need to trigger the full download first
- ArcServer source could in theory do a query, utilize quick cache
- How to store in memory
- Inherit from `CommonLayer`
- Will gain the loading process for free
- Might be a problem without an esri layer inside
- Also you don't "add" to map
- However you do open grid with layer payload, details button uses layer uid
- Inherit from `LayerInstance`
- Will keep all the interfaces the same, but lets us re-write the data base to avoid all the esri stuff
- `DataLayer`
- Debate what to move to a share and what to just duplicate
- Make an even more abstract common layer that shares items between data stuff and layer stuff
- `LayerInstance` -> `CommonLayer` -> `MapLayer` / `DataLayer`
- what goes in here?
- [See below](#Common-Layer-Split)
- Separate class starting with `DataInstance`
- No ties to `LayerInstance`
- Likely introduces a lot of type "ifs" everywhere, including event payloads
- Can we abstract what is currently in `AttribLayer` into that, thus having shared code base
- Filters
- Grid only? No real SQL engine available without a layer
- ArcServer source could support a permanent filter
- Attribute Loader
- Can we add a "direct injection" to this? What are file layers doing?

## Abstraction Pontifications

### What We Gots Now

- Attrib load from server. Should work for Arc Table
- Get Tabular Attributes. Should work for all variants
- GeoJSON parser. Should work for converting that format to an attribute-friendly format
- quick cache lookup / getGraphic pipeline?
- is there easy way to access this outside of `AttribLayer`?
- analyse `AttribLayer.getGraphic()` to see if it can be extracted to something else and not layer-tied
- attribute loader
- quick cache
- this.isFile
- can this be put as part of the attribute loader? or have a new status (might be 3 tier: file layer, server layer, server table)
- alternately, since file layers override getGraphic, this never gets called right?
- if we need server fetch for single:
- url (not for file layer)
- field list
- this.renderer <- only for requesting style, only applies to the 2 layer types
- for file layer it's so different we should probably leave as is
- make more sense to just abstract the attribute parts? Do we really need sharing of the Graphic construction given it will never have geom or style?
- quick cache is only used in attrib-layer. it's protected but currently that is not only pointless, it causes dullard initiations in file layer to pacify typescript.
- suggest push quick cache inside attribute loader, or bundle those two together in an attribute manager type thing
- we can initialize quickcache with a stupid geometry, as long as it's not point it will be fine in datatable land
- the attribute loader and quick cache is initialized in `loadLayerMetaData` (part of `onLoadActions`)

### Common Layer Split

Conclusion: majority of CommonLayer moves to MapLayer, I don't think it's worth the split

Things in Common Layer

timers: {
draw: number | undefined;
load: number | undefined;
};

protected origRampConfig: RampLayerConfig;

protected loadDefProm: DefPromise; // a deferred promise that resolves when layer is fully ready and safe to use. for convenience of caller
protected loadPromFulfilled: boolean; // a boolean to track whether the promise has fulfilled or not

Things in Map Layer

_serverVisibility: boolean | undefined;
_scaleSet: ScaleSet;
_mouseTolerance: number;
_touchTolerance: number;
_drawOrder: Array<DrawOrder>;
// used to manage debouncing when applying filter updates against a layer. Private! but needs to be seen by FCs.
_lastFilterUpdate = '';
protected layerTree: TreeNode;

esriWatches: Array<__esri.WatchHandle>;
protected viewDefProm: DefPromise; // a deferred promise that resolves when a layer view has been created on the map. helps bridge the view handler with the layer load handler

### Extra AttLoader & QuickCache Ideas

Make a wrapper class. It exposes properties for both properties for internal objects. But also has a getter that does the undefined test and throws an error.

This makes object appear as always defined. Can have an update method on both that accepts the baseclass.

{
  get attLoader(): Baseclass {
    if private then return private
    else throw error
  }
  
  updateAttLoader(Baseclass) {
    private = pram
  }
  
  ^ same for cache
  
}

### Additional Secret Schemes

For making shared routines work nicely across different layer inheritance trees.

things in table generator

layer instance

- dataFormat
- uid
- oidField
- fields

attrib layer

- attribs (AttribSource)
- specifically, is calling the getAttribs(), a promise, and the result of this method would get set into the object (so can be done internally)
- esriFields (could be moved to layer instance)
- renderer. Note this is being actively used to generate the table icons, so we may need to make a simple renderer on DataLayers

other

how is merge grid using this method? i guess it's calling the public get tablular data endpoitn

If we make secret interface, can just pass the layer

if we dont make the interface, and move esriFields to common, params become
- layer as layerinstance
- attribs
- renderer

not the end of the world? making esri fields common can also mean our esri fields processor can be generic and just take a layer instance

update: making fake renderer proving a pain, need to do all sorts of nonsense.
could make parameter renderer or SVG string

or add getIcon to layer instance, have it return '' or error for bad layers
^ i like this idea
^ details uses the method quite a bit. this makes sense.

## THE PLON

### Config

- new variant of layer config
- new layer types
- `data-esri-table`
- `data-geojson`
- `data-csv`
- `data-json` (`data-json-keyvalue`, `data-json-array`)
- `id`, `url`, `name`, `nameField`, `rawData`, `expectedLoadTime`, `maxLoadTime`, `controls` (reduced default?), `disabledControls`, `fieldMetadata`, `caching`, `fixtures`

### Layer Classes

- New class `DataLayer` that implements `LayerInstance`
- Copy relevant stuff from `CommonLayer` (only a small subset)
- Subclass as needed for the specifics of our layer types
- Larger submodules
- Upfront load of file based sources. Re-use file layer calls
- Attribute bulk load and individual lookup for ArcGIS sources
- New boolean on Instance Interface, `.mapLayer`; true if normie, false if data
- Reasons: config alignment, event alignment

### Layer Store

- Does store impact map stack position?
- Possibly second internal array that returns things on same endpoint; related to that other issue

### Map API

- adding the layer should not attempt to add to the map

### Attrib and Quick Cache

- move shared stuff into attribute module, passing in specific parameters when needed

### Grid

- extra logic around the zoomies and symbol fields. hide if data layer
- alternately, have a special symbol for data stuff that helps indicate it's not layer bound. Database icon? Table icon?
- look into hiding zoom column unless it's a merge grid with some spatial some not?
- Should not apply grid filters back to layer (as it doesn't exist), nor import filters from layer.
- If stand alone grid, the sync to map should be disabled
- If merge grid, the sync should skip over data layers

### Details

- same type of logic around the zoomies and symbol

### Legend

- limit stuff as appropriate
- no settings
- no checkboxes. no symbol stack (or common fake simple one with above icon)
- does not particpate in re-order
- main click opens data table. context menu should do have that option

### Wizard

- add support?
- might be "wordy" to have the different file types, considering they will have similar extensions / endpoints
- no colour picker. no tooltip field

### Tiered Implementation

Following things can be built by students after main PR

- legend support
- wizard support
- grid support
- detail support

----

## Filtering on ArcGIS Server Table

Sample code that proved it worked (for getting OIDs)

[This](https://developers.arcgis.com/javascript/latest/sample-code/sandbox/?sample=featurelayer-query-basic) sandbox sample.

Add these imports

"esri/rest/support/Query",
"esri/rest/query"

, Query, RestQ

and then this code somewhere in the body

console.log('fun starts now');
var q = new Query();
q.returnGeometry = false;
q.where = "siteFullName_en LIKE '%Muskeg River%'";
var danbo = 'https://section917.canadacentral.cloudapp.azure.com/arcgis/rest/services/TestData/Oilsands/MapServer/3';

RestQ.executeForIds(danbo, q).then(myIds => {
  console.log('done thanks', myIds)
});