Try   HackMD

User Guide - NTU BDSRC HPC

  • If you encounter any problem, please contact ntubdsrc@ntu.edu.tw
  • Change your password as soon as you log into the system.(Check out the User Setup section)

Overview

What is a terminal?

Terminal is a text-based interface allowing us to interact with the operating system. We can perform any task without using the graphical interface, that colorful desktop you usually see on the screen.
Different OS provides diffrerent name for terminal, you can find it with the following names:

  • macOS: terminal.app
  • windows: PowerShell
  • Linux: Terminal (right click to open if you are accessing through the desktop)

How to Access the computing resource?

There are 7 servers you can access in total. You can access all the HPC (High Performance Computing) server via the IP address 140.112.176.245. With this address, you will connect to master server by default. To access other servers, you need to specify which port you want to connect to.

  • Local machine: The computers you have on hand.
  • Remote machine: The servers you access via the Internet.

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

SSH Connection

SSH is a remote access protocal that allows you to connect any remote decive with terminal. The problem with this method is that it does not provide graphical interface like Desktop in Windows.

# In your terminal: 
## SSH Syntax
ssh <username>@140.112.176.245 -p <port number>

## If the terminal keeps yielding some weird failing messages that doe not affect the connection
ssh -o LogLevel=ERROR <username>@140.112.176.245 -p <port number>

## access server master by default
ssh <username>@140.112.176.245

## access node01 (port 2022)
ssh <username>@140.112.176.245 -p 2022

HPC SSH Port Number

Machine master node01 node02 node03 gpu01 gpu02 gpu03
Port - 2022 2023 2024 2025 2026 2027

HPC Premium Server Port Number (Application Required)

Machine gpu04
Port 2030

To end the SSH connection, you can type exit or use ctrl-D or ctrl-C

You can connect to other server "from master" simply by typing down the server's name.

# Example
ssh gpu01

NoMachine Remote Desktop

Installation

Please go to https://www.nomachine.com/download to download NoMachine for your local machine.

NoMachine app installed on your local machine is called NoMachine Client.

Using NoMachine

Open NoMachine app on your local machine, and click Add

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

Add connection:

  • Name: call it whatever you like
  • Host: 140.112.176.245
  • Port: 4000 (for connecting master. Other servers require different ports)
    Image Not Showing Possible Reasons
    • The image was uploaded to a note which you don't have access to
    • The note which the image was originally uploaded to has been deleted
    Learn More →

Ports available for NoMachine

master node03 gpu01 gpu02
4000 4002 4001 4003

Servers not listed in the above table do not have remote desktop access.

Log in with your account:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

Create a virtual desktop:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

Configure the resolution of the virtual desktop:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

And then you are good to go:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

To upload any file to the server via NoMachine, you can simply drag the desired files to NoMachine's window.

Configure the resolution or other settings

Click the upper-right corner:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

Then you can configure the desktop:

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

  • Your desktop will be killed if you do not log in for 7 days.
  • Each NoMachine server can accommodate a maximum of 10 sessions across all users simultaneously. If you cannot log in, Please use other servers or try again later.

User Setup

Change Password

Use passwd command and enter the current password to change it.

Only change your password from "master"

Some Utilities

There are several softwares you can launch on the terminal of the server.

# Launching MATLAB matlab # Stata17/IC 2GB memory limited xstata # Stata17/MP has more memory capacity xstata-mp # Stata in batch mode nohup stata-mp -b do [do-filename] & # Python python3 <filename>.py # Conda conda # Julia (only available in gpu01 and gpu02) julia

Use Jupyter Notebook on Remote Machine

Step 1: Run Jupyter server

jupyter-notebook

Step 2: open this URL (often it's http://locolhost:8888/tree) using Google Chrome

Image Not Showing Possible Reasons
  • The image was uploaded to a note which you don't have access to
  • The note which the image was originally uploaded to has been deleted
Learn More →

Use Remote Jupyter Notebook on Local Machine

# On "Remote" Server jupyter notebook --no-browser --port=8888 # On "Local" machine ssh -L 8888:localhost:8888 <username>@140.112.176.245 -p <port_number>

Then you can open the browser on your local machine, and type in the URL:http://localhost:8888, you will be able to edit the notebook from your locak machine.

Virtual Environment

Conda

Conda is a package and environment manager that allows you to isolate the environments of different projects.

# initial conda for shell conda init # create environment conda create --name <your project name> # activate the environment conda activate <your project name> # Deactivate the current active conda environment conda deactivate

Others

You can also use built-in venv or thrid-party package virtualenv to create virtual environments.
Please refer to https://python.land/virtual-environments/virtualenv

File Access Control

Your files might be visible to others. To prevent this, you need to set up access control. However, if your data isn't sensitive, you can choose to ignore this.

3 types of Permission: r(read), w(write), x(execute)

Checking File Permissions

# Example ls -l test.txt # Output -rw-rw-r-- 1 ryan ryan 0 九 2 14:58 test.txt # or you can use `getfacl` getfacl <filename>

For example a file's permission is -rw-rw-r--, you can interpret it as:

  • Is it a directory: - (it will be d if it's a directory)
  • Owner: rw- (you can read and write)
  • Group: rw- (people in your group can read and write)
  • Others: r-- (people outside of your group can only read)

rw- can be translated into 110, which is 6 in decimal value. So the permission of the example above can be represented as 660.

If you want to set the permission as rwxr-x---, you can simply type:

chmod 750 <filename>

Modify File Permission

# Syntax chmod <setting> <target name> # Example: add execute permission for yourself. chmod u+x <filename> # Example: Take away other's rwx permission chmod o-rwx <filename>

Modify Directory Permissoin

# Syntax chmod -R <your setting> <dir name>

u: you
o: others
g: users in your group
+: add
-: remove
=: assign

Group Control

You might want to share files and directories with your project members. For convenience, you can ask us to create a group for your project by submitting the application https://forms.gle/6Ro9YciS68czZi5E9

Group Basics
# show all the groups you are in groups # You might want to switch bewteen the groups # when you creare a new file newgrp <group name> # Type exit when finishing your operation for that group exit

If you are the owner of a group, you can add/delete the group member at your own will:

# add user gpasswd -a <username> <group name> # delete user gpasswd -d <username> <group name>

Further info on File Permission

Utilize GPU

For some package on python like Tensorflow, you can designate the GPU you want to run on.

# show all GPUs on this machine nvidia-smi # run on GPU 0 export CUDA_VISIBLE_DEVICES=0 # run on GPU 0, GPU 1 and GPU 2 export CUDA_VISIBLE_DEVICES=0,1,2
  • Do not add space in assigning the variable. It can cause trouble by typing:=0, 1, 2
  • Export the variable everytime before you run the python script.

VS Code Remote Editing

You might not want to transfer your file to the remote server everytime you modify your codes in your local machine. Therefore, directly editing the files on the remote server through VS code can be super convenient.

  1. Install extension "Remote Development", "Remote - SSH" and "Remote Explorer"

    Screenshot 2024-09-02 at 15.54.29

  2. Open a remote window

    Screenshot 2024-09-02 at 15.56.02

  3. Connect to Host

    Screenshot 2024-09-02 at 15.58.27

  4. Add new SSH host > Configure SSH Hosts

    Screenshot 2024-09-02 at 16.05.54

  5. Choose which file you want to save the configuration(you can choose the first one that appears)

    Screenshot 2024-09-02 at 16.02.31

  6. Edit the configuration

# Format Host <name whatever you like> HostNmae 140.112.176.245 User <your user name> Port <port number>

Screenshot 2024-09-02 at 16.10.01

  1. Establish Connection
    Screenshot 2024-09-02 at 16.15.00

Rstudio

You can also connect to Rstudio directly using your local browser with the following URL:
master: http://140.112.176.245:8787
node01: http://140.112.176.245:8788
gpu03: http://140.112.176.245:8789
gpu02: http://140.112.176.245:8790

Screenshot 2024-07-28 at 18.00.46

Basic Linux Guide

https://www.geeksforgeeks.org/linux-tutorial/