SES MSc 2024 - Model and DADCP Data - Access, Format, Processing

<h1> SES MSc 2024 - Model and ADCP Data - Access, Format, Processing </h1> [toc] # Getting set up with PyCharm ## Install PyCharm can be installed directly from https://www.jetbrains.com/pycharm/download/?section=windows (you'll want to scroll down to 'Community Edition' rather than professional). It is also shipped with Anaconda, in case you already have this installed. :::info If you want to use PyCharm, it's best to bypass any mention of creating a 'conda' environment, or 'conda install' commands. PyCharm can create a virtual environment for us (instead of Anaconda). We will use this environment and use `pip install ...` commands instead of `conda install ...`. ::: ## Create a new project (and virtual environment) First thing to do is to create a new project from the PyCharm home screen or file menu. In the pop-up, select 'Custom Environment' and 'Generate New' options. Name the project as you wish in the very first box. Open a terminal (using the '>\_' icon). Type `pip list` and you will see the packages that are installed in your new virtual environment (which PyCharm just created). It should contain `pip` and may also include `setuptools` (perhaps `wheel` as well). ```shell= (venv) PS C:\Users\{USERNAME}\PycharmProjects\pythonProject> pip list Package Version ---------- ------- pip 23.2.1 setuptools 68.2.0 wheel 0.41.2 ``` :::warning **If using PyCharm as shipped with Anaconda**, you might find issues running `ocwm` for the first time. This is caused when `setuptools` is missing. If `setuptools` is missing, install using `pip install setuptools`. Re-run `pip list` to check that it is now installed. ::: ## Download and install OCMW If you have not done so already, download OCWM from here: https://git.ecdf.ed.ac.uk/uoe-open-coastal-modelling/ocmw (you can use Git clone if preferred). In Windows explorer, unzip the download, and copy the path to OCMW to clipboard. Go back to PyCharm and type the following in the terminal (pasting over the path to wherever you unzipped to): ```shell= cd "C:\Users\{USERNAM}\Downloads\ocmw-main" ``` Check the contents of this directory (typing `ls`); you should see the following: ```shell= (venv) PS C:\Users\{USERNAM}\Downloads\ocmw-main> ls Directory: C:\Users\{USERNAM}\Downloads\ocmw-main Mode LastWriteTime Length Name ---- ------------- ------ ---- da---l 30/05/2024 10:44 apps da---l 30/05/2024 10:44 docs da---l 30/05/2024 10:44 examples da---l 30/05/2024 10:44 images da---l 30/05/2024 11:24 ocmw da---l 30/05/2024 11:00 ocmw.egg-info -a---l 30/05/2024 10:44 186 .gitignore -a---l 30/05/2024 10:44 798 .readthedocs.yaml -a---l 30/05/2024 10:44 237 environment.yml -a---l 30/05/2024 10:44 2053 Licence-BSD-3.txt -a---l 30/05/2024 10:44 2478 README.md -a---l 30/05/2024 10:44 101 requirements.txt -a---l 30/05/2024 10:44 794 setup.py ``` If you don't, go a folder deeper into `ocmw` (typing `cd ocmw`) -- you need to be in the folder that contains `setup.py`. Once you've found `setup.py`, install `ocmw` by typing: ```shell= pip install -e . ``` Typing `pip list` will show that `ocmw` is now installed. Also check [the full documentation](https://ocmw.readthedocs.io/en/latest/index.html) to be clear on the steps above. ## Using the example Python scripts You'll want to copy the three files from `...\msc_metocean\MSc 2024\Python_Scripts` to your new PyCharm project workspace, so that you can use and edit this (whilst others will do the same on their local systems - try not to edit the reference scripts). An example of a reasons to make copies of these is that line 1 in `redapt_adcp_locations_utmz30.txt` contains an absolute file path: ```= Data Path = P:/MSc 2024/Example_Data_Sets/OTM/obm_hr/RESULTS ``` This script will only work if your `msc_metocean` drive is mapped to `P:/` on your system (by coincidence). You'll want to change this path (probably just the letter `P:\`) to whatever is relevant on your machine. Note that you do not need to `cd` into this drive - this `.txt` file is really just a set of arguments that go into a Python function, i.e. Python knows it should look in this location for the `.slf` TELEMAC files, because you specified this argument. Once you have successfully run the command... ```python= python3 ./otm_extract_by_location.py -r loc -f redapt_adcp_location_utmz30.txt ``` ... the output files will appear in `P:/MSc 2024/Example_Data_Sets/OTM/obm_hr/EXTRACT` (with `P:/` changed accordinly). This is the shared folder location, so you will have just reproduced these example extracts (see the timestamps to see if you have succssfully recreated new files). Once you get using this script for your own purposes (i.e. saving locally and editing the argument parameters in `redapt_adcp_locations_utmz30.txt`), you will most likely be saving extracts 'next to' result files which you have saved locally. The extract files are `.mat` files which you can open with Matlab and/or Python for your post-processing. ## Possible errors after successful install ### Error 1 - `python3` alias command :::warning If you have issues running this in the terminal: ```python= python3 ./otm_extract_by_location.py -r loc -f redapt_adcp_location_utmz30.txt ``` either form PyCharm or Conda, try swapping `python3` with `python`. ::: ### Error 2 - bad space characters :::warning You may also have issues if you copy the above command from PowerPoint (the space characters are not recognisable in the terminal). This issue will return a message stating the file `redapt_adcp_location_utmz30.txt` doesn't exist (or something similar). Try typing the command out in full instead to fix this issue. ::: ### Error 3 - not finding data files :::warning The first argument in `redapt_adcp_locations_utmz30.txt` is an absolute path - you must change the letter `P:/` to whichever drive you have mapped `msc_metocean` to. ::: <br> <br> --- # Using Jupyter Notebook from your PyCharm project (virtual environment) If you've gone through the steps above to set up PyCharm, you might want to run Python in interpretive mode by using Jupyter Notebooks. It will save you some headaches by re-using the virtual environment you've already set up (and forgetting about Conda). From your PyCharm project, open a terminal (`>_` icon), type the following: ```shell= pip install notebook jupyter notebook ``` You'll see a browser window pop up, if you create a new notebook, you'll see this appear in your PyCharm project once you save it (`.ipynb` file). You can check that your virtual enviornment is set up by running `import ocmw` in a cell, then running. # Managing datetime ## For replacing characters, e.g. `/` for `-` In Matlab... ```matlab= >> date_string = "2024/05/28 10:00:00" date_string = "2024/05/28 10:00:00" >> strrep(date_string, "/", "-") ans = "2024-05-28 10:00:00" ``` In Python... ```python= >>> date_string_with_slash = "2024/05/28 10:00:00" >>> date_string_with_hyphen = date_string_with_slash.replace("/", "-") '2024-05-28 10:00:00' ``` ## Converting to datetime objects Some time series data sources will be recorded alongside timestamps recorded as strings, as above. This is clear and explicit (providing you know the time zone), and does not require an epoch, as date numbers do. But, you cannot do any arithmetic operations on a string - this is where `datetime` objects come in. To use the `datetime` object in Python, we can first check what *type* of object we are dealing with... ```python= >>> type(date_string_with_slash) <class 'str'> >>> type(date_string_with_hyphen) <class 'str'> ``` To convert a string to `datetime`, we need to first introduce (`import`) the standard Python library for processing `datetime`, and we can then use the `datetime.datetime.strptime` function (this refers to a *library* `datetime`, that contains a *module* `datetime`, that contains a *function* `strptime`) ```python= >>> import datetime >>> datetime_obj = datetime.datetime.strptime(date_string_with_slash, "%Y/%m/%d %H:%M:%S") >>> type(datetime_obj) <class 'datetime.datetime'> >>> datetime_obj = datetime.datetime.strptime(date_string_with_hyphen, "%Y-%m-%d %H:%M:%S") >>> type(datetime_obj) <class 'datetime.datetime'> ``` Clearly, we are doing the same thing twice here with `..._with_hyphen` and `..._with_slash`. We can just chose one and run with it. An alternative and tidier way to code this would be to introduce the `strptime` function from the `datetime` module within the `datetime` library (notice the different import line below, and that the function call can be writen more concisely as `strptime({args})`, instead of `datetime.datetime.strptime({args})`)... ```python= >>> from datetime.datetime import strptime >>> datetime_obj = strptime(date_string_with_hyphen, "%Y-%m-%d %H:%M:%S") >>> type(datetime_obj) <class 'datetime.datetime'> ``` We can now do operations on dates and times: ```python= >>> datetime_start = strptime("2024-05-28 10:00", "%Y-%m-%d %H:%M") >>> datetime_end = strptime("2024-05-28 12:00", "%Y-%m-%d %H:%M") >>> datetime_end - datetime_start datetime.timedelta(seconds=7200) ``` ## Formatting from different styles of datetime strings We can handle whatever format we want because of the second argument in the function used above. For example, a special flag `%b` can be used for month abbreviation... ```python= >>> datetime_usa = strptime("2024 28 May 12:00", "%Y %d %b %H:%M") >>> datetime_usa datetime.datetime(2024, 5, 28, 12, 0) >>> datetime_uk = strptime("28-May-2024 12:00", "%d-%b-%Y %H:%M") >>> datetime_uk datetime.datetime(2024, 5, 28, 12, 0) ```