# Arches Upload Errors

### Record of 2023-09-22 upload:

#### done:
- Cleared space for new uploads from the `/data` directory.
- Kept three backups per month on the days (10th, 20th, 30th) of each month and deleted the rest.
- Uploaded the data to server in the `/data/import+export/import/20230922_RS`
- Started to import the data to arches and recorded the errors in txt files in the same directory
- Added Discolouration to Evidence collection
- Added Qubba to Interpetation collection
- Added Hafir to Interpetation collection
- Added Palace to Interpetation collection
- Corrected the CNES / Airbus entry in the Image Type collection to CNES Airbus
- Added Enclosures as an alternative to the Enclosure entry in the Evidence thesaurus
- Added Complex Enclosures as an alternative to the Enclosure entry in the Evidence thesaurus
- Added rectangular to the Evidence shape collection
- Removed all special characters from Climate zones collection
#### to do:
- export data archesjson form and try to check where does the indexing break
- Change the value in the resource instance type to the format arches accepts
- change the values from the alias to the preferred label (Building to Structure) - Done
- Parse the rest of the errors
- Change "Complex Enclosure" value to "Complex enclosure" - Done
- Change "Without dry season-Hot summer (t)" to "Without dry season Hot summer (t)" - Done
- Add missing value to the "Name Type" collection - Done
- Add Elias Michaut to the Actor model - Done
- Change "Stone Mound" to "Stone mound" - Done
- Change "Complex Enclosure" to "Complex enclosure" - Done

#### problems faced:
1. Error in the reindexing of the database using `python manage.py es index_database` and got the following error while reindexing the `Administrative division` resource model. Might be due to a dependencies

Indexing concepts
/opt/projects/ENV/lib/python3.8/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
  warnings.warn(message, category=ElasticsearchWarning)
/opt/projects/ENV/lib/python3.8/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Specifying types in bulk requests is deprecated.
  warnings.warn(message, category=ElasticsearchWarning)
/opt/projects/ENV/lib/python3.8/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Specifying types in count requests is deprecated.
  warnings.warn(message, category=ElasticsearchWarning)
Status: Passed, In Database: 1399, Indexed: 1399, Took: 0 seconds
Indexing resource type 'Information'
Status: Passed, Resource Type: Information, In Database: 1, Indexed: 1, Took: 0 seconds
Indexing resource type 'Grid'
Status: Passed, Resource Type: Grid, In Database: 0, Indexed: 0, Took: 0 seconds
Indexing resource type 'Example'
Status: Passed, Resource Type: Example, In Database: 0, Indexed: 0, Took: 0 seconds
Indexing resource type 'Administrative division'
0% [███   ] 100% | ETA: 00:00:00 | Item ID: Resource object (cc48ec51-867b-4dd9-94fe-7683eaac07e3)Traceback (most recent call last):
  File "manage.py", line 30, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/opt/projects/arches/arches/management/commands/es.py", line 134, in handle
    self.index_database(
  File "/opt/projects/arches/arches/management/commands/es.py", line 172, in index_database
    index_database_util.index_db(clear_index=clear_index, batch_size=batch_size, quiet=quiet)
  File "/opt/projects/arches/arches/app/utils/index_database.py", line 30, in index_db
    index_resources(clear_index=clear_index, batch_size=batch_size, quiet=quiet)
  File "/opt/projects/arches/arches/app/utils/index_database.py", line 55, in index_resources
    index_resources_by_type(resource_types, clear_index=clear_index, batch_size=batch_size, quiet=quiet)
  File "/opt/projects/arches/arches/app/utils/index_database.py", line 97, in index_resources_by_type
    document, terms = resource.get_documents_to_index(
  File "/opt/projects/arches/arches/app/models/resource.py", line 297, in get_documents_to_index
    datatype_instance.append_to_document(document, nodevalue, nodeid, tile)
  File "/opt/projects/arches/arches/app/datatypes/concept_types.py", line 69, in append_to_document
    date_range = self.get_concept_dates(value.concept)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 188, in __get__
    raise self.RelatedObjectDoesNotExist(
arches.app.models.models.RelatedObjectDoesNotExist: Value has no concept.

2. Error in reindexing the actor model using the command `python manage.py es index_resources_by_type -rt ccbd1537-ac5e-11e6-84a5-026d961c88e6` got the following error.

Indexing resource type 'Actor'
/opt/projects/ENV/lib/python3.8/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
  warnings.warn(message, category=ElasticsearchWarning)
Traceback (most recent call last):
  File "manage.py", line 30, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/opt/projects/arches/arches/management/commands/es.py", line 150, in handle
    index_database_util.index_resources_by_type(
  File "/opt/projects/arches/arches/app/utils/index_database.py", line 97, in index_resources_by_type
    document, terms = resource.get_documents_to_index(
  File "/opt/projects/arches/arches/app/models/resource.py", line 297, in get_documents_to_index
    datatype_instance.append_to_document(document, nodevalue, nodeid, tile)
  File "/opt/projects/arches/arches/app/datatypes/concept_types.py", line 69, in append_to_document
    date_range = self.get_concept_dates(value.concept)
  File "/opt/projects/ENV/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 188, in __get__
    raise self.RelatedObjectDoesNotExist(
arches.app.models.models.RelatedObjectDoesNotExist: Value has no concept.

The error is caused by a broken link between the resources stored in the database and the collections. For each node of the type `concept` or `concept-list` arches associate a collection of concepts. Each concept represent a particular value and can have multiple labels to represent this value. For example a the collection `name type` can have a concept `primary` as a preferred label for English. Primary could be referred to by an alternative label `main` or `inital`, or it could be refereed to by the french `primaire`. Each collection, concept, and label has its own UUID. Arches does not store the label (actual string displayed to the user), instead it stores a UUID that refers to a label associated with a concept within the collection . If the link between the resource model and the collection was broken for any reason ( could be modifying the collection or the concepts) then the UUID stored in the resources will resolve to a label that does not exist and Arches will throw an error when reindexing the database.

This is the problem we faced in reindexing the Actor resources and most likely will face with the rest of the resource models too. Here is how this error was resolved:
- Export the resources into JSON
- Find the broken node
- Attach the concept to the node
- Update the UUID of the labels in the resources
- Import the data again into Arches
- reindex the resource model

3. Error parsing the geometry of a polygon because of a repeated point

Exception detail: ('3 document(s) failed to index.', [{'index': {'_index': 'maeasam_project_resources', '_type': '_doc', '_id': '0fdb09d2-e17c-40cd-bf08-715c325c1dae', 'status': 400, 'er[68/1862]
ype': 'mapper_parsing_exception', 'reason': 'failed to parse field [geometries.geom.features.geometry] of type [geo_shape]', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'Provide
d shape has duplicate consecutive coordinates at: POINT (27.958561103754 14.6878576172873)'}}, 'data': {'displaydescription': '<Site information>, Structure, , , High, No, Bare rock or soil, Dese
rt hot, ', 'resourceinstanceid': '0fdb09d2-e17c-40cd-bf08-715c325c1dae', 'graph_id': '30026cdf-7ae7-11ec-ba7b-fa163e7d893d', 'map_popup': 'Undefined', 'displayname': '<MAEASaM  id>', 'root_ontolo
gy_class': 'http://www.cidoc-crm.org/cidoc-crm/E27_Site', 'legacyid': 'RS_SDN_0004689', 'tiles': [<TileModel: TileModel object (f8f52d6e-1296-427e-9c35-14388a591fa0)>, <TileModel: TileModel objec
t (3658d3b8-e0af-485f-9d41-220e6fe08ca6)>, <TileModel: TileModel object (1d316163-c84d-4038-aeca-96ced4d54c67)>, <TileModel: TileModel object (763f03b3-4251-4293-8f7b-9da00eedffa9)>, <TileModel:
TileModel object (2d002c38-d933-4b3b-a8e8-be8a670573ca)>, <TileModel: TileModel object (feb90a6f-c73b-4b95-a52f-fa8a19a14f9e)>, <TileModel: TileModel object (8770e249-ac98-4349-870e-1f832eba0a84)
>], 'permissions': {'users_without_read_perm': [], 'users_without_edit_perm': [], 'users_without_delete_perm': [], 'users_with_no_access': []}, 'strings': [{'string': 'High', 'nodegroup_id': UUID
('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Desert hot', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'No', 'no
degroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Bare rock or soil', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': Fals
e}, {'string': 'Structure', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': '12241', 'nodegroup_id': UUID('5f9996d4-3521-11ee-88b3-fa163ebfff9e'),
'provisional': False}, {'string': '146199', 'nodegroup_id': UUID('5f9996d4-3521-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Historic map check', 'nodegroup_id': UUID('00458e54-35
d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Visual inspection', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Bing', 'nodegr
oup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': '10 possible tumuli', 'nodegroup_id': UUID('9ea23dc4-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': False},
 {'string': 'square meter', 'nodegroup_id': UUID('cc4fee56-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Area', 'nodegroup_id': UUID('cc4fee56-35e2-11ee-88b3-fa163ebfff9e'), '
provisional': False}], 'dates': [{'date': 20221118, 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'nodeid': '4baca436-35df-11ee-88b3-fa163ebfff9e', 'provisional': False}, {'date':
 20221118, 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'nodeid': '8cc083b2-35de-11ee-88b3-fa163ebfff9e', 'provisional': False}], 'domains': [{'label': 'High', 'conceptid': UUID(
'e0a4c3e8-66ff-4a9b-93fc-9c1b2500bf3a'), 'valueid': '1c28e674-caaf-4e5d-9af8-ea450a31ebdd', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'Desert
hot', 'conceptid': UUID('5bffbc75-b2c2-4177-a2a9-94bf357fc592'), 'valueid': '85367f48-e360-4758-8899-d3f369bee2bf', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': Fa
lse}, {'label': 'No', 'conceptid': UUID('7f41eea0-7a80-4463-a75b-bb0ff8a740ac'), 'valueid': 'c948337d-c999-47f3-9c91-ce95ee50d477', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), '
provisional': False}, {'label': 'Bare rock or soil', 'conceptid': UUID('7161ad7a-7ed4-43f5-b354-3287b3835da2'), 'valueid': 'd19b6dee-65c9-4fac-ad48-f0cde5878a0e', 'nodegroup_id': UUID('cb5432cc-3
516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'Structure', 'conceptid': UUID('30e18a63-67b5-4def-9ab0-5b3773ba22ff'), 'valueid': 'ab4d5b0a-a074-439d-8a0c-0a48b704729d', 'nodegrou
p_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'Historic map check', 'conceptid': UUID('cd3a80a1-2033-4c3d-a265-22166233174f'), 'valueid': 'b91f820f-8aff-48
e8-a73b-598914d01994', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'Visual inspection', 'conceptid': UUID('254a24fc-e9ad-44be-bf78-5cf376d56dce'
), 'valueid': 'f5e9d2d4-6226-42ce-a00d-0ce48c5e83e3', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'Bing', 'conceptid': UUID('1c7dbdb8-fa69-4bde-
88e9-0e1d9c27fb8e'), 'valueid': 'f8cce6dd-657f-46c5-bed3-143863b410c1', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'square meter', 'conceptid':
 UUID('41619316-c843-44e3-b095-c2be673e5c50'), 'valueid': 'a28d2f57-5a88-4076-8521-fc145ea99255', 'nodegroup_id': UUID('cc4fee56-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'label': 'A
rea', 'conceptid': UUID('a612e7b3-c4d1-40a7-a761-0c44e0bffa8f'), 'valueid': '9ef10992-99c8-4dc0-985a-7b38321ac59b', 'nodegroup_id': UUID('cc4fee56-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': Fa
lse}], 'geometries': [{'geom': {'type': 'FeatureCollection', 'features': [{'id': 'a27aec21-646c-446c-b987-c844dd00f892', 'type': 'Feature', 'geometry': {'type': 'MultiPolygon', 'coordinates': [[[
[27.958561103754, 14.6878576172873], [27.958561103754, 14.6878576172873], [27.9586976827355, 14.6881407223606], [27.9587952391509, 14.6882971040533], [27.9590098632646, 14.688351028749], [27.9590
349492, 14.688351028749], [27.9590126505908, 14.6881056712761], [27.9587729405416, 14.6879924292727], [27.9587729405416, 14.6879924292727], [27.9587229819165, 14.6878860638044], [27.9586289261683
, 14.6877804595134], [27.9585701413257, 14.687795081649], [27.958561103754, 14.6878576172873], [27.958561103754, 14.6878576172873]]]]}, 'properties': {}}]}, 'nodegroup_id': UUID('e4a6a8a2-3522-11
ee-88b3-fa163ebfff9e'), 'provisional': False, 'tileid': UUID('3658d3b8-e0af-485f-9d41-220e6fe08ca6')}], 'points': [{'point': {'lon': 27.958798026476998, 'lat': 14.688065744131201}, 'nodegroup_id'
: UUID('e4a6a8a2-3522-11ee-88b3-fa163ebfff9e'), 'provisional': False}], 'numbers': [{'number': 1376.59, 'nodegroup_id': UUID('cc4fee56-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': False}], 'date
_ranges': [], 'ids': [{'id': 'bd01a336-e7e5-4b6c-a4b9-a97766b55bfe', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'id': '443219f0-ccbc-4a82-b38b-f079ad5d5
a60', 'nodegroup_id': UUID('5995abfe-35e1-11ee-88b3-fa163ebfff9e'), 'provisional': False}], 'provisional_resource': 'false'}}}, {'index': {'_index': 'maeasam_project_resources', '_type': '_doc',
'_id': 'c9209e6e-92ac-4e5a-9d15-b8b9ed56c88a', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse field [geometries.geom.features.geometry] of type [geo_shape
]', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'Provided shape has duplicate consecutive coordinates at: POINT (27.9053201379833 14.6894082773918)'}}, 'data': {'displaydescript
ion': '<Site information>, Structure, , , Medium, No, Bare rock or soil, Desert hot, ', 'resourceinstanceid': 'c9209e6e-92ac-4e5a-9d15-b8b9ed56c88a', 'graph_id': '30026cdf-7ae7-11ec-ba7b-fa163e7d
893d', 'map_popup': 'Undefined', 'displayname': '<MAEASaM  id>', 'root_ontology_class': 'http://www.cidoc-crm.org/cidoc-crm/E27_Site', 'legacyid': 'RS_SDN_0004709', 'tiles': [<TileModel: TileMode
l object (869f1609-6f2f-4dc3-9e0b-fbbd4ae631d5)>, <TileModel: TileModel object (a2bf50bc-49df-4ad1-abce-5e6742e84d25)>, <TileModel: TileModel object (240b987c-78cf-4d16-a258-3becae5afdcd)>, <Tile
Model: TileModel object (6cf874f6-c4d1-4bcc-afc1-f6237a9beb36)>, <TileModel: TileModel object (46411d5d-94a3-4504-b01c-5663d51a2092)>, <TileModel: TileModel object (003a10b5-cc7c-47fb-be8e-1f9971
7aaf89)>, <TileModel: TileModel object (15f9d38f-0e32-4eb4-9b4c-270381ae35ed)>], 'permissions': {'users_without_read_perm': [], 'users_without_edit_perm': [], 'users_without_delete_perm': [], 'us
ers_with_no_access': []}, 'strings': [{'string': 'Medium', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Desert hot', 'nodegroup_id': UUID('cb54
32cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'No', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Bare rock or soil', 'n
odegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Structure', 'nodegroup_id': UUID('cb5432cc-3516-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'s
tring': '12241', 'nodegroup_id': UUID('5f9996d4-3521-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': '147079', 'nodegroup_id': UUID('5f9996d4-3521-11ee-88b3-fa163ebfff9e'), 'provision
al': False}, {'string': 'Historic map check', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Visual inspection', 'nodegroup_id': UUID('00458e54-3
5d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'Bing', 'nodegroup_id': UUID('00458e54-35d0-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': '12 possible tumuli but har
d to distinguish', 'nodegroup_id': UUID('9ea23dc4-35e2-11ee-88b3-fa163ebfff9e'), 'provisional': False}, {'string': 'square meter',

To fix this one we need to build a parser that checks for the duplicate points and delete any duplicate occurrence of the points

---
### Paragraph for the report:
Uploading the remote sensing data to Arches:
- matching data with thesaurus and collections
- date format mismatch

Index the data:
- broken links between labels uuid in data and collections
- missing collections in resource models which has data in it

---
### Errors by the column in the remote sensing data:
#### Evidence:
- replace `Building` with `Structure` *arches failed to recognize the alternative label in the concept*
#### Evidence Shape:
- add `Rectangular` to the concept
-
---
### Potential errors by data type :
- Concept:
- concept dos not exist
- value does not exist in the list
- new value
example of the error:`2023-09-22 05:43:21,605 arches.app.utils.data_management.resources.formats.csvfile ERROR concept error, Building on line 2 - This is an invalid concept prefLabel, or an incomplete UUID. Unable to save.` which means that the `Building` value does not exist in the target concept and therefore arches cannot find it
- list of values instead of one value

- Concept - list:
- concept dos not exist
- value does not exist in the list
- new value

- Date:
- text date instead of numerical
- Wrong format
`2023-09-22 05:43:21,606 arches.app.utils.data_management.resources.formats.csvfile ERROR datatype: DateDataType value: 01/02/2023 on line 2 - time data '01/02/2023' does not match format '%Y-%m-%d' or is not a prefLabel in the given collection.`

- EDTF:
- Wrong format

- Geojson:
- max number of nodes per polygon
- two nodes or more are closer to each other then the tolerance
- clipped string (sometimes excel or other tools clip the geojson string and you get EOF(end of line) error)

### other requirements:
- required:
- empty or missing data
example error `2023-09-22 05:43:21,606 arches.app.utils.data_management.resources.formats.csvfile WARNING The Evidence node is required and must be populated in order to populate the Site information, Evidence, Evidence shape, Interpetation, Interpretation certainty, Ground truthed, Land use land cover, Climate zone, Relationship to other site nodes. This data was not imported.`

### Record of 2023-09-26 upload:

#### done:
- Fixed the date format for Survey date in Sudan CSV
- Fixed the date format for Image date in Sudan CSV
- Removed "\" from all CNES Airbus entries in the Sudan CSV
- Change the values from the alias to the preferred label (Building to Structure)
- Change "Complex Enclosure" value to "Complex enclosure"
- Replace "Complex Structure" to :Complex structure"
- Change "Without dry season-Hot summer (t)" to "Without dry season Hot summer"
- Change "Dry Winter-Warm Summer (t)" to "Dry WinterWarm Summer"
- Added all the correct Grid Level 1 ID formats
- Added missing values to the "Name Type" collection
- Added Elias Michaut to the Actor Resource Model
- Corrected all the incorrect "Measurement units" format
- Corrected all the date formats

#### to do:
- Add the missing dates for the Google Earth survey for Sudan for the 2023-02 and 2023-03 date range

### Record of 2023-09-27 and 2023-09-27 upload:

#### done:
- Created a python scripte to modify and correct errors in the upload CSV fore being uploaded to Arches
- Re-indexed the Actor and Administative resource models
- Changed requirement options in the Remote Sensing model to eas upload process
- Removed the "unkown" from the evidence attribute and changed it to standing remains(Elias has confirmed this change as it affected 5 of his entries)
- Modified the Climate Zone thesauris to make the options clearer and more user friendly and modifying sections that would have resulted in Arches upload errors
- Found a structural mistake in the MAEASaM collection that has resulted in duplication of the MAEASaM thesauris. We believe that somebody accidently made the MAEASaM thesauris a collection instead of the specific thesaurus that they were working on
- The duplicate collection has been removed
- Modified the mapping file to find the `MAEASaM ID` in the data csv

<<<<<<< 
        {
            "arches_nodeid": "2734f10e-3516-11ee-88b3-fa163ebfff9e",
            "arches_node_name": "MAEASaM  id",
            "file_field_name": "MAEASaM  id",
            "data_type": "string",
            "export": true
        },
=======
        {
            "arches_nodeid": "2734f10e-3516-11ee-88b3-fa163ebfff9e",
            "arches_node_name": "MAEASaM  id",
            "file_field_name": "MAEASaM  ID",
            "data_type": "string",
            "export": true
        },
>>>>>>> 

### Errors by the column in the remote sensing data:
#### Evidence
- Change "Building" to "Structure" as Arches is not reading the alternative lables during upload
- Change "Unknown" to "Standing remains" as unkown has been removed from the thesauris
#### Image Type
- Changed "CNES / Airbus" to "CNES Airbus" as the "/" was cuasing errors in uploading
#### Climate Zone
- Changed "Dry Winter-Hot Summer (t)" to "Temperate Dry Winter Hot summer" to reflect modifications made to the Climate Zone thesauris to improve user friendliness
#### Surveyor Name
- Changed "Ed Burnett" to "Ed Burnett, Edward Burnett" to match the instance as it appears in the Actor model to the upload
#### Measurement unit
- Changed "m2" to "square meter" to reflect the unit used in the thesaurus

#### problems faced:
- An incorrect prefab label or broken UUID for entries that are correct and have complete UUIDs

#### to do:
- Go through all the thesaurus entries and collections to double check if they are correct after the deletion of the duplicate MAEASaM collection
- Determine why the prefab label error is occuring for correct entries
- Recreate the threat type thesaurus and make it a collection

added conflict