# Progress with the 1.3m DFOT data archive! - Data available from 2012 - 2022 (June) - Prior to 2016: 2012,13,14,15 sparse and irregular data! 2023B --- - The data from February and March 2023 has been processed and ready for the archive. - Need to get the data for January as well as April until today. - Also enable the daily processing of the data and enable it to be downloaded ASAP. --- - We have started from 2022A, and moved retrogressively. - Header updates: astrometry correction, filter information and observatory information. - Files renamed to our own observatory name convention. - All the data being organized in a logical structure in the archive. --- ### 2022A: - Problems with some data: date is not entered correctly (year is absurd in some files!) - Good news: Only a few dates with such data. - January to March processed, April and May with some problems. - To be renamed yet. --- ### 2021: - 2021A and 2021B processed completely. 43,000 fits files and ~300 failed in astrometry etc. - Updated the files with the corrected airmass values for all the files ( had to use another set of equations (code modified by Neha). - Thumbnails generated for every science frame, for a quick view! - Searchable through sky positions through the archive. --- ### 2020: - 2020A has been processed completely. (~14000 files processed) - Major issues with 2020B, various issues with files (~23,000 files processed)/ ~5000 files separated for re-inspection. - Have to look for them folder by folder, time consuming task. --- ### 2019 and 2018: - 2019A started this week, we expect 2019A and 2019B to be finished by 10-12 September. - Will move on to 2018, hope to finish it too, before 2022B observations start. - No major issues so far. --- ### 2017 and before: - Previous archive team had done some work on this. Some dates in 2017 and 2016 are left - All the files need to be reprocessed, but not much time consuming. (Majority of updates done alreay) - I have prepared a color coded google sheet for this. --- ### 2017 and before (New updates): - All the previous data (2012-15) has been prepared now. - Only 2018 remains now (another 10-15 days) - Verifying the 2016 and 2017 final data (a week at most) - Expect 1.3m DFOT data to be finished before December 15, 2022! --- ### New updates during YAM: - Data storage at 1.3m PC (use RAID system?) - Common user interface for the telescopes? - Open source alternatives (SIRIL/N.I.N.A among others.) - can be tested at 1.3m initially. --- ### Other updates: - Prepared a short set of guidelines for the name convention to be followed while observing it from this session onwards. - Running the codes on immediately obtained data. Automate the process, with plenty of checks in between. A Bottleneck! - Log for the 1.04 m ST. - Cross match the data with the DFOT observation log, very helpful so far. --- ### A photometry pipeline with the following features: - Written purely in PYTHON, eliminating the need for IRAF based tasks for cleaning. - For photometry , PHOTUTILS is being used, eliminating the need for DAOPHOT and much faster. - Alignment of images, performing astrometry and also **calibrating our instrumental magnitudes** --- - Querying magnitudes directly from the table of sources. This kind of automation is only possible through PYTHON currently. - The idea is to implement this for the entire ARIES archive ~around a million data frames and multiple million sources in them!