Transfer data to and from NeLS using web browser, FileZilla, and command line

<center><img src="https://i.imgur.com/sP7cyyX.png" alt="drawing" width="200"/></center> # Transfer data to and from NeLS using web browser, FileZilla, and command line <Transferring and organising data in NeLS> <html> <div style="background-color: #f2f3f4"> In this hands-on exercise, we will log in to the **NeLS Portal** and upload some files to our storage area in NeLS. We will do this in three different ways: via the web portal; with FileZilla - a software dedicated to comfortable data transfer to and from remote servers; and also using the terminal window and the secure copy (`scp`) command. Main goal of this session is to get everyone familiar with the first two ways of connecting to NeLS, while the third approach (connecting to NeLS via terminal) is optional and intended mainly for the participants with some prior command line experience and with preference to use this type of connection to NeLS. You can access the **NeLS portal** at [https://nels.bioinfo.no/](https://nels.bioinfo.no/). </div> </html> ## Access the NeLS portal <html> <div style="background-color: #f2f3f4"> **NeLS** - the Norwegian e-Infrastructure for Life Sciences, is an infrastructure for analysis, sharing and storage of high throughput genomics data. Access to the infrastructure is mainly through a web portal with an easy-to-use graphical interface that allows a user to manage data in their project(s). Everyone with FEIDE credentials can access the portal using FEIDE. Those future users of NeLS who do not have FEIDE can get access by contacting the **ELIXIR Norway helpdesk**. </div> </html> **Task:** Open [NeLS portal](https://nels.bioinfo.no/) in a new tab/window of your web browser. If you are not already logged in NeLS (with FEIDE or NeLS Idp), you need to provide your username and password to log in. Please do so now. :::success :+1: That was the end of this section. Please remember to type "**done nels access**" in the chat so that we know about your progress. ::: ## Set up your local computer <html> <div style="background-color: #f2f3f4"> To successfully learn transfering files to and from NeLS, we will first create a folder on your local computer which will contain all the data necessary for the following exercises. Afterwards, we will download the elixir logo used at the top of this document and store it in the new folder. Later on we will download some additional data into the new folder. Note: Names of files, folders, commands, and other functionality elements will be highlighted through the exercises in the following way: `file_name_example.txt`. User specific (and thus variable) character strings as for example user_name will be written between `<` and `>` as `<user_name>` to indicate the meaning. </div> </html> <br/> I. Create a specific folder dedicated to this workshop on your local computer: **Windows users:** **Create a new folder** inside `/c/Users/<user_name>/` and call the new folder `elixir`: To create the folder, open File Explorer. Expand `This PC` in the left-hand side menu (if need be) and click on `OS (C:)`. Afterwards, double-click on the `Users` folder and double-click on the folder with your `<user_name>`. Right-click on the empty surface area (not on a folder name) of the window and select `New > Folder`, screenshot is shown in Figure 1. Type `elixir` as the name of the new folder. <center> <img src="https://i.imgur.com/xesdr6o.jpg" width="800" /> **Figure 1:** Win10, File Explorer - creating a new folder. </center> **OsX users:** **Create a new folder** inside `/Users/<user_name>/` and call the new folder `elixir`: To create a new folder open Finder and go to Macintosh HD. Double click the `Users` (Brukere) folder and double click on the folder with your `<user_name>`. Right click the empty surface area and select `New Folder` (Ny Mappe). Type `elixir` as the name of the new folder. <center> <img src="https://i.imgur.com/r36g87r.jpg" width="800" /> **Figure 2:** OsX, Finder - creating a new folder. </center> **Linux users:** **Create a new folder** called `elixir` with the following command (or consult details individually, if needed). ``` mkdir /home/<user_name>/elixir ``` From now on, we will refer to the elixir folder simply as `elixir` regardless of where exactly the folder is located on your computer. <br/> II. **Download a file** into the `elixir` folder: The following link will get you to [Elixir logo](https://i.imgur.com/sP7cyyX.png). Open the link in a new tab/window of your web browser. Right-click on the logo with your mouse/touchpad and save the image as `logo_to_transfer.png` in the `elixir` folder. <br/> :::success :+1: That was the end of this section. Please remember to type "**done setup**" in the chat so that we know about your progress. ::: <br/> ## Data transfer via the NeLS portal using web browser <html> <div style="background-color: #f2f3f4"> The NeLS portal provides functionality to upload the data from a local computer to NeLS and for basic manipulation of files and folders (copy, rename, move, delete). Brief overview of the available functionality is highlighted and described in Figure 3. In the following exercises, we will learn to use this functionality. </div> </html> <center> <img style="border:1px solid silver;" src="https://i.imgur.com/3LGBzO7.png" width="800" /> </center> **Figure 3:** Screenshot of the Personal Area in the NeLS portal, highlighted blocks: 1. Reload, Upload File, Add New Folder; 2. (De)select all/some items; 3. File and folder manipulation; 4. Download/Rename/Delete; 5. Projects Area - the same functionality available there as in the Personal Area. <br/> I. **Upload** the file `logo_to_transfer.png` to your Personal area in NeLS: Click on the button `Upload` in the top left corner of your web browser NeLS window - in Figure 3, block 1. <br/> II. **Rename** the file `logo_to_transfer.png` to `transferred_logo.png` using renaming icon for the file - in Figure 3, block 3. <br/> III. **Create a new folder** by clicking on the button `+ New Folder` (block 1). Name the new folder `uploaded_workshop_2022` and move the file `transferred_logo.png` into the new folder. To move a file to another location, one has to select the file using checkbox (block 4) and use the `Cut` button (block 2). Afterwards, one has to descend into the destination folder and press the `Paste` button located right next to the `Cut` button (block 2). <br/> IV. **Download** compressed folder `uploaded_workshop_2022` into the `elixir` folder on your local computer and inspect the content of the compressed archive. For the download, use a downloading icon that will show up for the folder nearby the renaming icon. <br/> V. **Delete** the folder `uploaded_workshop_2022` from the NeLS Personal Area. <br/> VI. **Delete** the compressed folder `uploaded_workshop_2022` from your local computer. <br/> :::success :+1: That was the end of this section. Please remember to type "**done nels transfer**" in the chat so that we know about your progress. ::: ## Set up and use FileZilla for data transfer to and from NeLS <html> <div style="background-color: #f2f3f4"> In the following part, we will set up FileZilla on your local computer to access NeLS portal with your credentials. Afterwards we will transfer some data through the established connection. This approach is better suited to transfer bigger files to NeLS than using the web interface introduced earlier. In order for FileZilla to establish the connection to NeLS, we have to collect some connection details in the NeLS portal and provide that pieces of information to FileZilla. Figures 4 and 5 show where the connection details can be obtained in NeLS. One of the connection details is your **ssh key**. Such a key is used for user's authentication as an alternative to a password. In a very simplified way, we can consider the ssh key to be your very long password stored in a text file. Therefore, we have to treat the key file as a piece of sensitive information and be careful about what can and cannot be done with the file. First, when you download the file onto your local computer, please pay attention to place the file into a location which is not backed up somewhere on a cloud. This is covered, if you followed the previous instructions about where to create the `elixir` folder. Second, you should check and possibly modify access permissions of the key file set to other users on your computer. We will provide a step-by-step guide to get you smoothly through the task. Figures 6 and 7 show FileZilla screenshots with filled details for the connection to NeLS and with established connection to NeLS, respectively. </div> </html> <br/> <center> <img style="border:1px solid silver;" src="https://i.imgur.com/i2Qm7eY.png" width="600"/> Figure 4: NeLS portal - Navigation to menu with connection details. <br/> <img style="border:1px solid silver;" src="https://i.imgur.com/wnU7QCp.png" width="600" /> Figure 5: NeLS portal - Connection Details: 1. username, 2. host, 3. key. </center> <br/> I. Make sure you have [FileZilla](https://filezilla-project.org/download.php) downloaded and installed on your local computer. <br/> II. Open `Connection Details` pop-up window in the NeLS portal: Hover with the hand cursor over your name in the top right corner until the menu with `Connection Details` shows on the screen (Figure 4). Click on the `Connection Details`. <br/> III. You will have to provide `Host` (1.) and `Username` (2.) to FileZilla in exactly the same form as they are typed in the `Connection Details` pop-up window in Figure 5. Keep the window open so that you can copy and paste them later on. <br/> IV. **Download** the SSH Key (3.) to the `elixir` folder on your computer. <br/> V. **Check and modify access permissions** of the downloaded key file: **Windows users:** Open File Explorer and navigate into the `elixir` folder. Right-click on the key file and select the option `Properties` at the bottom of the option list. Select submenu `Security` in the newly opened window - highlighted in red in Figure 6. If anyone else than you, `SYSTEM`, and `Administrators` has granted access to the key file, select such a user and remove all their access rights via clicking on `Edit...` button and denying everything. <center> <img src="https://i.imgur.com/85SnBQB.jpg" width="600" /> Figure 6: Security submenu in file properties. </center> **OsX users:** Open Finder and navigate into the `elixir` folder. Right-click on the folder. Select the option called `Open Terminal Here...` and confirm. Then type the following command in the terminal window and press enter: ``` $ chmod 600 <ssh_key> ``` `<ssh_key>` stands for the name of your ssh key file. This command will limit the access to the file exclusively to you. To simplify typing the whole `<ssh_key>` file name, type first 4 characters of the file name and then press `Tab` key on your keyboard - the rest of the file name should be auto-completed for you. You can close the terminal window with typing `exit` and pressing `Enter`. **Linux users:** Open file manager and navigate into the `elixir` folder. Right-click on the area showing content of the folder (but not directly on any of the files in there). Select the option called `Open Terminal Here...` and confirm. Then type the following command in the terminal window and press enter: ``` $ chmod 600 <ssh_key> ``` `<ssh_key>` stands for the name of your ssh key file. This command will limit the access to the file exclusively to you. To simplify typing the whole `<ssh_key>` file name, type first 4 characters of the file name and then press `Tab` key on your keyboard - the rest of the file name should be auto-completed for you. You can close the terminal window with typing `exit` and pressing `Enter`. <br/> VI. Run FileZilla. In the top menu, click on `File` and then on `Site Manager...`. Click on the `New Site` button and call the new site `NeLS`. Perform the following changes to the form: 1. Change `Protocol` to `SFTP - SSH File Transfer Protocol`. 2. Fill in `Host` from the NeLS Connection Details. 3. Change `Logon Type` to `Key File`. 4. Fill in `Username` from the NeLS Connection Details. 5. Browse for the Key File, allow all file types to show in the pop-up window and select the SSH Key File downloaded from the NeLS portal to the `elixir` folder. The Site Manager Form will at the end look similar to the screenshot in Figure 6. When ready, press the `Connect` button. When the connection to NeLS is successfully established, the content of the local computer is shown in the left-hand side window and the content of NeLS is shown in the right-hand side window as in Figure 7. <br/> <center> <img style="border:1px solid silver;" src="https://i.imgur.com/YmUrRrB.png" width="600" /> Figure 6: FileZilla New Site Manager Form - filled, info from the NeLS Connection Details in 1.,2.,3. <br/> <img style="border:1px solid silver;" src="https://i.imgur.com/64SjNE4.png" width="800" /> Figure 7: FileZilla, when connection to NeLS is established. </center> :::success :+1: That was the end of this section. Please remember to type "**done filezilla setup**" in the chat so that we know about your progress. ::: <br/> VIII. **Copy the file** `logo_to_transfer.png` into the `Personal` folder with FileZilla drag-and-dropping the file: Navigate in FileZilla (using either keyboard arrows or mouse/touchpad) to the `elixir` folder on your computer - in the left-hand side FileZilla window. Navigate in FileZilla to the `Personal` folder in NeLS - in the right-hand side window. Drag and drop the file from the local folder to the NeLS location. <br/> IX. Do the following in FileZilla: **Rename the file** `logo_to_transfer.png` in the `Personal` folder to `transferred_logo.png`. **Create a new folder** in `Personal` with mouse right-click. Call the new folder `uploaded_workshop_2021`. **Move** `transferred_logo.png` to the new folder. <br/> X. **Transfer** the `uploaded_workshop_2021` folder to your local computer with FileZilla. <br/> XI. **Remove** the `uploaded_workshop_2021` folder from `Personal` folder in NeLS. <br/> XII. **Thought experiment:** Is it possible to transfer data between `Personal` and `Projects` folders using FileZilla? <br/> XIII. When you want to connect FileZilla to NeLS again, you can hover with your cursor over the left-most icon (servers) right below the top menu and select `NeLS` connection when it shows up - as shown in Figure 7. <br/> XIV. **Remove** the `uploaded_workshop_2021` folder from your local computer (with FileZilla or otherwise). <br/> :::success :+1: That was the end of this section. Please remember to type "**done filezilla transfer**" in the chat so that we know about your progress. ::: ## Use terminal to connect to NeLS (optional) <html> <div style="background-color: #f2f3f4"> This section of the exercises is optional and aimed mainly on the participants interested in using command line for data transfer as well as having some prior experience with command line. Further content of the workshop does require the content of this section - FileZilla in combination with the [NeLS web portal](https://nels.bioinfo.no/) can be used for all the needed data transfer tasks. In the beginning of this section, we will use several standard commands in a terminal window to list directory content, copy and delete files on your local computer. Later we will use the `ssh` command to connect to the NeLS server and to navigate around the directory structure on the server. At the end, we will use the `scp` command to copy a file from a local computer to the NeLS server and back. If you are interested in deeper understanding of commands standardly used in terminal, we suggest, for example, reading through [software carpentry materials on bash](http://swcarpentry.github.io/shell-novice/) or attending one of their workshops which covers the topic. In the following text, `<user_name>` refers to your user name on the local computer, while `<user>` refers to the NeLS user ID. You can see your NeLS user ID in the connection details in the web interface of the [NeLS portal](https://nels.bioinfo.no/). </div> </html> <br/> I. **Windows users only:** please make sure you have [Git for Windows](https://git-scm.com/download/win) downloaded and installed on your local computer. The reason for installing Git is that the package contains a linux-like terminal called Git Bash which we will use. <br/> II. **Open a terminal window** in the `elixir` folder: **Windows users:** Open File Explorer and navigate into the `elixir` folder. Right-click with a mouse/touchpad on the area showing content of the folder (but not directly on any of the files in there). Select the option called `Run Git Bash Here...` and confirm. **OsX users:** Open Finder and navigate into the `elixir` folder. Right-click on the folder. Select the option called `Open Terminal Here...` and confirm. **Linux users:** Open file manager and navigate into the `elixir` folder. Right-click with a mouse/touchpad on the area showing content of the folder (but not directly on any of the files in there). Select the option called `Open Terminal Here...` and confirm. <br/> III. **List content** of the `elixir` folder with command `ls`: ``` $ ls logo_to_transfer.png <user>@nelstor0.cbu.uib.no.key ``` <br/> IV. **Print name of current/working directory** with `pwd` command. The output of the command will be either `/c/Users/<user_name>/elixir` (Win), or `/Users/<user_name>/elixir` (OsX), or `/home/<user_name>/elixir` (linux). ``` $ pwd /home/<user_name>/elixir ``` <br/> V. **Copy** `logo_to_transfer.png` to `logo_to_transfer_2.png`: ``` $ cp logo_to_transfer.png logo_to_transfer_2.png ``` <br/> VI. List again the content of the `elixir` folder, you can see the new file added to the output. ``` $ ls logo_to_transfer.png <user>@nelstor0.cbu.uib.no.key logo_to_transfer_2.png ``` <br/> VII. **Maintenance tip:** typing command `clear` and pressing enter will clear your terminal window and put the active line with a cursor at the top of the window. <br/> VIII. **Workshop sync:** That was the end of this section of the practical. Please type "**done command-line local**" in the chat. <br/> IX. Now we will use the `ssh` command to **access content of the NeLS server** through a terminal window. Typing the following command into the command line will initiate connection with NeLS server: ``` ssh -i <user>@nelstor0.cbu.uib.no.key <user>@nelstor0.cbu.uib.no ``` If this is your first connection to NeLS, you have to type `yes` and press enter to confirm that you want to connect to that server. Afterwards, NeLS welcome screen will show up as in Figure 8. <br/> <center> <img style="border:1px solid silver;" src="https://i.imgur.com/ERfZG6w.png" width="800" /> **Figure 8:** NeLS welcome screen. </center> <br/> X. **Print working directory** with `pwd` command. As you are connected to the NeLS server in this terminal window, the output of the command will be a path to your NeLS home directory. ``` $ pwd /elixir-chr/nels/users/<user> ``` <br/> XI. **List content** of your home directory with `ls` command. You can use parameter `-F` to easily see which records are folders (character `/` will show up after their names) and which are files (nothing added at the end). The output will be two folders called `Personal` and `Projects` - the same two folders you have previously seen in FileZilla as well as via the web interface of the NeLS portal. ``` $ ls -F Personal/ Projects/ ``` <br/> XII. **List content of sub-folder** `Personal`. There are several ways of doing this, we will show two of them. One can either descend to the sub-folder, list its content and go back to the home folder, or stay in the home directory and specify the sub-folder as an input to the `ls` command. The first approach: ``` $ cd Personal $ ls first.txt second.txt $ cd .. ``` The second approach: ``` $ ls Personal first.txt second.txt ``` <br/> XIII. Open another Terminal/GitBash window in the `elixir` folder. <br/> XIV. In the new window run the following `scp` command that will copy `logo_to_transfer.png` to your Personal area in NeLS. The `\`s at the end of the lines split the command to multiple lines for easier readability of the command - you can skip them and write the whole command (without `>` characters) into one line, substituting `<user>` with your NeLS user ID. ``` $ scp \ > -i <user>@nelstor0.cbu.uib.no.key \ > logo_to_transfer.png \ > <user>@nelstor0.cbu.uib.no:/elixir-chr/nels/users/<user>/Personal ``` <br/> XV. When `scp` command finishes the data transfer, go to the first terminal window and list content of the Personal area with `ls` command. If the window got inactive, close it, open another terminal window in the `elixir` folder and connect to NeLS via `ssh` again. Can you see the newly transferred file? <br/> XVI. **Rename file** `logo_to_transfer.png` to `transferred_logo.png` with `mv` command in the `ssh` terminal window: ``` $ cd Personal $ mv logo_to_transfer.png transferred_logo.png $ ls first.txt second.txt transferred_logo.png ``` <br/> XVII. **Create folder** `uploaded_workshop_2021` with `mkdir` command and **move** `transferred_logo.png` into the new folder with `mv` command in the `ssh` terminal window. ``` $ mkdir uploaded_workshop_2021 $ mv transferred_logo.png uploaded_workshop_2021 $ ls -F first.txt second.txt uploaded_workshop_2021/ ``` <br/> XVIII. **Download folder** `uploaded_workshop_2021` from Personal area in NeLS back to your local computer in the `scp` terminal window. Parameter `-r` in the `scp` command stands for 'recursively' and indicates that all the content of the folder `uploaded_workshop_2021` will be downloaded into the working directory (`.` in the last line of the command) on your local computer. ``` $ scp \ > -r \ > -i <user>@nelstor0.cbu.uib.no.key \ > <user>@nelstor0.cbu.uib.no:/elixir-chr/nels/users/<user>/Personal/uploaded_workshop_2021 \ > . ``` <br/> XIX. **Delete file** `transferred_logo.png` and folder `uploaded_workshop_2021` from NeLS Personal area using `rm` command in the `ssh` window. Parameter `-d` in the second `rm` command indicates that we are removing an empty directory. ``` $ cd uploaded_workshop_2021 $ rm transferred_logo.png $ cd .. $ ls -F first.txt second.txt uploaded_workshop_2021/ $ rm -d uploaded_workshop_2021 $ ls first.txt second.txt ``` <br/> XX. To stop the `ssh` command connecting to the NeLS server, we type command `exit`. ``` $ exit Connection to nelstor0.cbu.uib.no closed. ``` <br/> :::success :+1: That was the end of this section. Please remember to type "**done command-line transfer**" in the chat so that we know about your progress. :::

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.