# AMIGOS data processing Currently, the AMIGOS stations deployed in Antarctica depend on some scripts and configuration here to operate and are described below. For more details see the [taberna repository](https://bitbucket.org/nsidc/taberna). ## Short Burst Data Most data from the AMIGOS stations is recieved via the Short Burst Data (SBD) service. The process is as follows 1. AMIGOS station sends a short (<2000 byte) SBD message to the Iridium SBD service 1. The Iridium SBD service forwards that message embedded in an email (to wallinb@nsidc.org currently) 1. The SBD message is appended to a logfile `/share/apps/amigos/<env>/incoming/sbd/<station>_sbd.txt` (see procmail config: [procmailrc](https://bitbucket.org/nsidc/taberna/src/master/procmail/.procmailrc) and processing script [parse_amigos_sbd_emails.py](https://bitbucket.org/nsidc/taberna/src/master/procmail/parse_amigos_sbd_emails.py)) 1. The script `cli/taberna/schedule.py` runs the `taberna.run.ingest_sbd()` routine hourly to ingest this logfile, inserting it into the database, exporting csv version, sending a daily digest email, etc... ## Uploads Large data files such as DTS, camera images, and TPS are uploaded to `restricted_ftp` under e.g. `/disks/restricted_ftp/<station>`. They are rotated every half hour to `/share/apps/amigos/<env>/incoming/uploads` and `/share/apps/amigos/<env>/archive/uploads` where they are ingested by `taberna.run.ingest_uploads()` hourly. See the [crontab](https://bitbucket.org/nsidc/taberna/src/master/crontab/crontab.kryos) and [rotate_incoming.sh](https://bitbucket.org/nsidc/taberna/src/master/crontab/rotate_incoming.sh) for relevant commands. ## Processed data FTP access Processed data is stored at `/share/apps/amigos/prod/processed` and rotated to `/disks/sidads_ftp/pub/incoming/wallinb/itgc_tarsan/amigos/processed` every half hour via cron on kryos (see [crontab](https://bitbucket.org/nsidc/taberna/src/master/crontab/crontab.kryos) and [rotate_processed.sh](https://bitbucket.org/nsidc/taberna/src/master/crontab/rotate_processed.sh)). ## Migrating away from kryos Suffice to say it is not ideal to have the dependencies described above on kryos. An improvement would be to handle the SBD messages within the [taberna](https://bitbucket.org/nsidc/taberna) system, but that requires some non-trivial work 1. Setup an email server and email address for this purpose 1. Change the email associated with the IMEI numbers of the stations for the [SBD service](https://www.iridium.com/services/iridium-sbd/) or 1. Setup to use the SBD DirectIP protocol. [See developer guide](https://usermanual.wiki/Pdf/Iridium20Short20Burst20Data20Service20Developers20Guide20v30.896763731/html)