HackMD
  • Prime
    Prime  Full-text search on all paid plans
    Search anywhere and reach everything in a Workspace with Prime plan.
    Got it
      • Create new note
      • Create a note from template
    • Prime  Full-text search on all paid plans
      Prime  Full-text search on all paid plans
      Search anywhere and reach everything in a Workspace with Prime plan.
      Got it
      • Options
      • Versions and GitHub Sync
      • Transfer ownership
      • Delete this note
      • Template
      • Save as template
      • Insert from template
      • Export
      • Dropbox
      • Google Drive
      • Gist
      • Import
      • Dropbox
      • Google Drive
      • Gist
      • Clipboard
      • Download
      • Markdown
      • HTML
      • Raw HTML
      • Sharing Link copied
      • /edit
      • View mode
        • Edit mode
        • View mode
        • Book mode
        • Slide mode
        Edit mode View mode Book mode Slide mode
      • Note Permission
      • Read
        • Owners
        • Signed-in users
        • Everyone
        Owners Signed-in users Everyone
      • Write
        • Owners
        • Signed-in users
        • Everyone
        Owners Signed-in users Everyone
      • More (Comment, Invitee)
      • Publishing
        Everyone on the web can find and read all notes of this public team.
        After the note is published, everyone on the web can find and read this note.
        See all published notes on profile page.
      • Commenting Enable
        Disabled Forbidden Owners Signed-in users Everyone
      • Permission
        • Forbidden
        • Owners
        • Signed-in users
        • Everyone
      • Invitee
      • No invitee
    Menu Sharing Create Help
    Create Create new note Create a note from template
    Menu
    Options
    Versions and GitHub Sync Transfer ownership Delete this note
    Export
    Dropbox Google Drive Gist
    Import
    Dropbox Google Drive Gist Clipboard
    Download
    Markdown HTML Raw HTML
    Back
    Sharing
    Sharing Link copied
    /edit
    View mode
    • Edit mode
    • View mode
    • Book mode
    • Slide mode
    Edit mode View mode Book mode Slide mode
    Note Permission
    Read
    Owners
    • Owners
    • Signed-in users
    • Everyone
    Owners Signed-in users Everyone
    Write
    Owners
    • Owners
    • Signed-in users
    • Everyone
    Owners Signed-in users Everyone
    More (Comment, Invitee)
    Publishing
    Everyone on the web can find and read all notes of this public team.
    After the note is published, everyone on the web can find and read this note.
    See all published notes on profile page.
    More (Comment, Invitee)
    Commenting Enable
    Disabled Forbidden Owners Signed-in users Everyone
    Permission
    Owners
    • Forbidden
    • Owners
    • Signed-in users
    • Everyone
    Invitee
    No invitee
       owned this note    owned this note      
    Published Linked with GitHub
    Like BookmarkBookmarked
    Subscribed
    • Any changes
      Be notified of any changes
    • Mention me
      Be notified of mention me
    • Unsubscribe
    Subscribe
    # Delft RaspberryPi HPC workshop part 2 notes, 19 November 2021 # Workshop 2: Parallel Computing with a RaspberryPi cluster :::info First part of this workshop can be found here: [Workshop 1: Building a RaspberryPi cluster](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA) ::: ![](https://i.imgur.com/EK4Vqkx.jpg) Notes prepared by: Dennis Palagin, Jose Urra Llanusa, Jerry de Vos Special thanks go to Kees Lemmens and Joffrey Wallaart for their excellent [MPI course](https://www.tudelft.nl/cse/education/courses/mpi-programming) we used as an inspiration for slides. **Event description: https://www.eventbrite.co.uk/e/delft-open-hardware-hack-and-playhpc-with-a-raspberry-pi-cluster-ii-tickets-189735513027** <iframe src="https://docs.google.com/presentation/d/17s51k8n0BmiMxXaJ5BMUUV-pSX6Ze_TH/embed?start=false&loop=false&delayms=3000" frameborder="0" width="960" height="749" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe> # Recap from the [previous workshop](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA) 1. The [previous workshop](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA) ended with us having a functional RaspberryPi cluster, which should be ready to go. Login to the controller node and test slurm to make sure it works. Run the `sinfo` command and you should get the following output (in this example, for the RaspberryPi 3 with a ClusterHAT and 4 Pi Zero as nodes): ``` PARTITION AVAIL TIMELIMIT NODES STATE NODELIST mycluster* up infinite 5 idle <cluster-name>,p[1-4] ``` :::warning <details> <summary>Got an error (e.g. STATE = "down")?</summary> If the nodes are reporting as being "down", e.g. after a reboot of the node, then a simple re-initialisation will most likely help: ``` sudo scontrol update nodename=p1 state=idle ``` </details> ::: 2. The `scontrol show nodes` command gives you the information about available nodes in your cluster: ``` NodeName=cnat Arch=armv7l CoresPerSocket=1 CPUAlloc=0 CPUTot=2 CPULoad=0.08 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=172.19.181.254 NodeHostName=cnat Version=18.08 OS=Linux 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 RealMemory=1 AllocMem=0 FreeMem=176 Sockets=2 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=2 Owner=N/A MCS_label=N/A Partitions=mycluster BootTime=2021-10-14T09:13:12 SlurmdStartTime=2021-10-14T09:13:26 CfgTRES=cpu=2,mem=1M,billing=2 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=p1 Arch=armv6l CoresPerSocket=1 CPUAlloc=0 CPUTot=1 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=172.19.181.1 NodeHostName=p1 Version=18.08 OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020 RealMemory=1 AllocMem=0 FreeMem=137 Sockets=1 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=mycluster BootTime=2021-10-14T09:15:42 SlurmdStartTime=2021-10-14T09:16:38 CfgTRES=cpu=1,mem=1M,billing=1 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=p2 Arch=armv6l CoresPerSocket=1 CPUAlloc=0 CPUTot=1 CPULoad=0.06 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=172.19.181.2 NodeHostName=p2 Version=18.08 OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020 RealMemory=1 AllocMem=0 FreeMem=141 Sockets=1 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=mycluster BootTime=2021-10-14T09:15:43 SlurmdStartTime=2021-10-14T09:16:43 CfgTRES=cpu=1,mem=1M,billing=1 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=p3 Arch=armv6l CoresPerSocket=1 CPUAlloc=0 CPUTot=1 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=172.19.181.3 NodeHostName=p3 Version=18.08 OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020 RealMemory=1 AllocMem=0 FreeMem=140 Sockets=1 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=mycluster BootTime=2021-10-14T09:15:44 SlurmdStartTime=2021-10-14T09:16:41 CfgTRES=cpu=1,mem=1M,billing=1 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=p4 Arch=armv6l CoresPerSocket=1 CPUAlloc=0 CPUTot=1 CPULoad=0.02 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=172.19.181.4 NodeHostName=p4 Version=18.08 OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020 RealMemory=1 AllocMem=0 FreeMem=144 Sockets=1 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=mycluster BootTime=2021-10-14T09:15:45 SlurmdStartTime=2021-10-14T09:16:40 CfgTRES=cpu=1,mem=1M,billing=1 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s ``` Similarly, `scontrol show partitions` displays which partitions you configured: ``` PartitionName=mycluster AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=YES QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=cnat,p1,p2,p3,p4 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=6 TotalNodes=5 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED ``` 3. You can print the hostname on all of the nodes with the following command: ``` srun --nodes=5 hostname ``` Which should produce a similar result to the following: ``` <cluster-name> p1 p2 p3 p4 ``` To see how many concurrent tasks you can run, you can use the following command: ``` srun --ntasks=6 hostname ``` 4. Which should produce a similar result to the following: ``` <cluster-name> <cluster-name> p1 p2 p3 p4 ``` 5. Main Slurm commands to remember are: `squeue` — view scheduled jobs When you start running longer and longer jobs, it is useful to check their status. To do this, run the `squeue` command. By default, it displays all jobs submitted by all users, and their states: ``` $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 609 glmdev 24.sub.s pi R 10:16 1 node2 ``` Most of this info is pretty self-explanatory. The only thing I'll note is the ST column, which is the state of the job. R means that the job is running. `scancel` — cancel a scheduled job Once a job has been scheduled, it can be cancelled using the `scancel` command: ``` $ scancel 609 ``` (where 609 is the JOBID that you want to cancel). Note that you can only cancel jobs started by your user. `sbatch` — schedule a batch script `sbatch` is really the meat and potatoes of the Slurm scheduler. It's what we use most often when we want to schedule a job to run on the cluster. This command takes a number of flags and configuration, as well as a shell file. That shell file is then executed using whatever requested resources (nodes/cores/etc) are made available to it. # Running jobs in parallel ## Part 1: Let's go through the tutorials by Garrett Mills: As a warm-up, let's just follow parts 2 and 3 of the original tutorial and select a couple of nice examples: [Building a Raspberry Pi cluster tutorial Part 2 by Garrett Mills](https://glmdev.medium.com/building-a-raspberry-pi-cluster-aaa8d1f3d2ca) [Building a Raspberry Pi cluster tutorial Part 3 by Garrett Mills](https://glmdev.medium.com/building-a-raspberry-pi-cluster-f5f2446702e8) Special thank goes to the createor of the original tutorials [Garrett Mills](https://glmdev.medium.com/). **6. Example 1: Hello, World!** Our job begins with the definition of a batch file. This batch file is usually a bash script that runs our job, however it looks a bit different. We’ll create the file `helloworld.sh`: ``` #!/bin/bash #SBATCH --ntasks=6 #SBATCH --partition=mycluster cd $SLURM_SUBMIT_DIR echo "Hello, World!" >> helloworld.txt echo "The following nodes are reporting for duty:" >> helloworld.txt srun hostname >> helloworld.txt echo "Have a great day!" >> helloworld.txt ``` The file begins with a "[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix))". This is required, as it tells Slurm how to execute your job. This is followed by a number of flags that take the following form: ``` #SBATCH <flag> ``` These flags are simply any parameters that can be passed to the sbatch command. Here, we specify the number of CPUs and the partition of the cluster that we want to use. The `cd $SLURM_SUBMIT_DIR` guarantees that our job is running in whatever directory it was submitted from. Now, we can tell Slurm to schedule and run our job: ``` $ sbatch helloworld.sh Submitted batch job 639 ``` Since our job is very simple, it should be done basically immediately. If everything has gone according to plan, we should see the `helloworld.txt` file that we created: ``` Hello, World! The following nodes are reporting for duty: cnat cnat p3 p1 p2 p4 Have a great day! ``` You'll notice that the job doesn't output anything to the shell, which makes sense. If you had a job running for 4 hours, it's not very useful to have to have a terminal open the whole time to get output. Instead, Slurm outputs standard error and standard out to a file in the format `slurm-XXX.out` where `XXX` is the Job's ID number. **7. Example 2: Data processing with R** For our first project on the cluster, we're going to do some statistics! Data processing is a big part of what HPC clusters are used for. So, we're going to build a simple R program that generates some random values following a normal distribution, then creates a histogram and graph of those values and outputs them to an image file. Then, we're going to create a script to generate 50 of them using the scheduler. **Goal:** Output 50 randomly-generated normal distribution graphs and histograms to a folder, as images using R. **Setup:** Before we can start building our project, we need to install R. For those unfamiliar, [R is a programming language for statistical programming](https://www.r-project.org/). This makes it very well-suited for our task. Install R on each node with the following command: ``` sudo apt install r-base -y ``` When the installation completes, you should be able to use R on any of the nodes: ``` pi@node2 ~$ R --version R version 3.3.3 (2017-03-06) -- "Another Canoe" Copyright (C) 2017 The R Foundation for Statistical Computing Platform: arm-unknown-linux-gnueabihf (32-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under the terms of the GNU General Public License versions 2 or 3. For more information about these matters see http://www.gnu.org/licenses/. ``` **Theory:** We need to run a large number of relatively small jobs. So, what we will do is create a script that, when executed, runs the sbatch command to schedule the same job over and over again. We will use a Slurm job array to do this. Basically, we give the scheduler a script to run, and tell it an array of numbers to run said script, it will run the script once for each number in the array, and the script can access its index during each job. This is how we will generate 50 random normal curves. **The R program:** Before we can schedule our program, we need to write a quick R script to generate the normal data-sets. So, we'll create the file `generate.R`: ``` arg = commandArgs(TRUE) samples = rep(NA, 100000) for ( i in 1:100000 ){ samples[i] = mean(rexp(40, 0.2)) } jpeg(paste('plots/', arg, '.jpg', sep="")) hist(samples, main="", prob=T, color="darkred") lines(density(samples), col="darkblue", lwd=3) dev.off() ``` **Submissions script:** Now that we have our R program, we will create a submission script to run our jobs. Create the file `R_submit.sh`: ``` #!/bin/bash #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --partition=mycluster cd $SLURM_SUBMIT_DIR mkdir plots R --vanilla -f generate.R --args "plot$SLURM_ARRAY_TASK_ID" ``` Here, we tell Slurm to run the job on 1 node, with 1 core on whatever partition you specified. Then, we change directories to the current folder where we will submit the job from. Then we run the R program. The name of the plot file is set to `plot$SLURM_ARRAY_TASK_ID`. This will name the image file after whatever index of the array we tell Slurm to run our job against. For example, when this job is run for the 23rd time, it will output the file: `plots/plot23.jpg`. **Run the job:** We now have everything we need to run our job. From the login node, you can run the job like so: ``` $ sbatch --array=[1-50] R_submit.sh Submitted batch job 910 ``` Now, if we run `squeue`, we should see something like so: ![](https://i.imgur.com/O2kCjl4.png) :::info Tip: open a separate terminal window and make it "monitor" the queue with the `watch squeue` command. ::: When the jobs complete, the `plots` folder should have 50 of our randomly generated histograms, looking similar to this: ![](https://i.imgur.com/fYoDeEo.jpg) **8. Example 3: Hello, World! in C with MPI** In this part, we're going to set up OpenMPI, and take a look at running some jobs in parallel using MPI to make use of the multiple cluster nodes. OpenMPI is an open-source implementation of the Message Passing Interface concept. An MPI is a software that connects processes running across multiple computers and allows them to communicate as they run. This is what allows a single script to run a job spread across multiple cluster nodes. **Setup:** Install OpenMPI on every node: ``` $ sudo apt install openmpi-bin openmpi-common libopenmpi3 libopenmpi-dev -y ``` **C code:** We're going to create a C program that creates an MPI cluster with the resources SLURM allocates to our job. Then, it's going to call a simple print command on each process. Create the file `hello_mpi.c` with the following contents: ``` #include <stdio.h> #include <mpi.h> int main(int argc, char** argv){ int node; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &node); printf("Hello World from Node %d!\n", node); MPI_Finalize(); } ``` Here, we include the `mpi.h` library provided by OpenMPI. Then, in the main function, we initialize the MPI cluster, get the number of the node that the current process will be running on, print a message, and close the MPI cluster. We need to compile our C program to run it on the cluster. However, unlike with a normal C program, we won't just use `gcc` like you might expect. Instead, OpenMPI provides a compiler that will automatically link the MPI libraries: ``` $ mpicc hello_mpi.c ``` The `a.out` file is the compiled program that will be run by the cluster. **Submission script:** Now, we will create the submission script that runs our program on the cluster. Create the file `sub_mpi.sh`: ``` #!/bin/bash #SBATCH --ntasks=6 cd $SLURM_SUBMIT_DIR # Print the node that starts the process echo "Master node: $(hostname)" # Run our program using OpenMPI. # OpenMPI will automatically discover resources from SLURM. mpirun a.out ``` **Run the job:** Run the job by submitting it to SLURM: ``` $ sbatch sub_mpi.sh Submitted batch job 1211 ``` If we have everything working properly, this should create an MPI cluster with 6 nodes. Assuming this works, we should see some output in our `slurm-XXX.out` file: ``` Master node: cnat Hello World from CPU 1! Hello World from CPU 0! Hello World from CPU 3! Hello World from CPU 5! Hello World from CPU 4! Hello World from CPU 2! ``` **9. Example 4: Python with MPI: mpi4py** We're going to throw together a quick Python job that uses OpenMPI. To interface with OpenMPI in Python, we're going to be using a fantastic library called [`mpi4py`](https://mpi4py.github.io/usrman/). For our demo, we're going to use one of the demo programs in the `mpi4py` repo. We're going to calculate the value of pi (the number) in parallel. Before we can write our script, we need to install a few libraries. Namely, we will install the `mpi4py` library, and `numpy`. NumPy is a package that contains many useful structures and operations used for scientific computing in Python. We can install these libraries through `pip` on every node: ``` python3 -m pip install numpy mpi4py ``` **Create the Python program** As mentioned above, we're going to use one of the demo programs provided in the `mpi4py` repo. However, because we'll be running it through the scheduler, we need to modify it to not require any user input. Create the file `calculate_pi.py`: ``` from mpi4py import MPI from math import pi as PI from numpy import array def comp_pi(n, myrank=0, nprocs=1): h = 1.0 / n s = 0.0 for i in range(myrank + 1, n + 1, nprocs): x = h * (i - 0.5) s += 4.0 / (1.0 + x**2) return s * h def prn_pi(pi, PI): message = "pi is approximately %.16f, error is %.16f" print (message % (pi, abs(pi - PI))) comm = MPI.COMM_WORLD nprocs = comm.Get_size() myrank = comm.Get_rank() n = array(0, dtype=int) pi = array(0, dtype=float) mypi = array(0, dtype=float) if myrank == 0: _n = 20 # Enter the number of intervals n.fill(_n) comm.Bcast([n, MPI.INT], root=0) _mypi = comp_pi(n, myrank, nprocs) mypi.fill(_mypi) comm.Reduce([mypi, MPI.DOUBLE], [pi, MPI.DOUBLE], op=MPI.SUM, root=0) if myrank == 0: prn_pi(pi, PI) ``` This program will split the work of computing our approximation of pi out to however many processes we provide it. Then, it will print the computed value of pi, as well as the error from the stored value of pi. **Create and submit the job** We can run our job using the scheduler. We will request some number of cores from the cluster, and SLURM will pre-configure the MPI environment with those cores. Then, we just run our Python program using OpenMPI. Let's create the submission file `sub_calc_pi.sh`: ``` #!/bin/bash #SBATCH --ntasks=6 cd $SLURM_SUBMIT_DIR mpiexec -n 6 python3 calculate_pi.py ``` Here, we use the `--ntasks` flag, which requests a specific number of cores total. Because we are using MPI, we can have cores across machines. Therefore, we can just request the number of cores that we want. In this case, we ask for 6 cores. To run the actual program, we use `mpiexec` and tell it we have 6 cores. We tell OpenMPI to execute our Python program using the version of Python we installed. Note that you can adjust the number of cores to be higher/lower as you want. Just make sure you change the `mpiexec -n ##` flag to match. Finally, we can run the job: ``` $ sbatch sub_calc_pi.sh Submitted batch job 1215 ``` The calculation should only take a couple seconds on the cluster. When the job completes (remember, you can monitor it with `squeue`), we should see some output in the `slurm-####.out` file: ``` $ cat slurm-1215.out pi is approximately 3.1418009868930934, error is 0.0002083333033003 ``` ## Part 2: Specific application examples: ### Computational materials science example: Surface diffusion energy barriers using the Nudged Elastic Band (NEB) method This excercise is based on the ASE tutrial: https://wiki.fysik.dtu.dk/ase/tutorials/neb/diffusion.html Learn more about `ASE` here: https://wiki.fysik.dtu.dk/ase/about.html More tutorials can be found here: https://wiki.fysik.dtu.dk/ase/tutorials/tutorials.html **Theory** The Nudged Elastic Band (NEB) method is a technique for finding transition paths (and corresponding energy barriers) between given initial and final states of the atomic (molecular) system. The method involves constructing a "chain" of "replicas" or "images" of the system and relaxing their geometries in a certain way. ![](https://i.imgur.com/muJkNjB.png) **Figure**: Pictorial representation of a reaction path computed with NEB. Taken from https://www.scm.com/doc/AMS/Tasks/NEB.html At the beginning of a NEB calculation, the geometry of the initial and final systems are optimized to minimize their energy. Then, a rough approximation of the reaction path is built: a set of images is created by performing a linear interpolation between the initial and final systems. Optionally, an intermediate system can be provided, in which case the interpolation is performed between the initial and intermediate systems, and then between the intermediate and final systems. Finally, a reaction path is found by performing a simultaneous optimization of all the images. In the NEB method the images are not independent from each other. The force on each image depend on its neighboring images: at each step the forces parallel to the reaction path are eliminated and a so-called spring force is added that tries to keep each image in the middle between its neighbors. This does not let images slide to the initial or final reaction state and ensures that they are evenly distributed along the reaction path. #### 10. First, install the prerequisites: `ase` and `gpaw` modules: ``` python3 -m pip install --upgrade --user ase sudo apt install libxc-dev python3 -m pip install gpaw ``` :::info Tip: install `ase` on your laptop, and you will be able to access the `ase gui` command. This will let you use the graphical atoms/molecules editor and a way to easily visualize the results of your calculations. As it is written in python, it is easily installed on any operating system. ::: #### 11. Second, set up the initial and final states of the NEB calculation: ![](https://i.imgur.com/Pcxc3aA.png)![](https://i.imgur.com/oqqg1WD.png) For this, we will be using the following python script: ``` from ase.build import fcc100, add_adsorbate from ase.constraints import FixAtoms from ase.calculators.emt import EMT from ase.optimize import QuasiNewton # 2x2-Al(001) surface with 3 layers and an # Au atom adsorbed in a hollow site: slab = fcc100('Al', size=(2, 2, 3)) add_adsorbate(slab, 'Au', 1.7, 'hollow') slab.center(axis=2, vacuum=4.0) # Make sure the structure is correct: # view(slab) # Fix second and third layers: mask = [atom.tag > 1 for atom in slab] # print(mask) slab.set_constraint(FixAtoms(mask=mask)) # Use EMT potential: slab.calc = EMT() # Initial state: qn = QuasiNewton(slab, trajectory='initial.traj') qn.run(fmax=0.05) # Final state: slab[-1].x += slab.get_cell()[0, 0] / 2 qn = QuasiNewton(slab, trajectory='final.traj') qn.run(fmax=0.05) ``` Run this script, with: ``` python3 generate_traj.py ``` This script produces two files, "inital.traj" and "final.traj", which we will use in the actual NEB calculation. #### 12. Finally, run the parallel calculation: Instead of having one process do the calculations for all three internal images in turn, it will be faster to have three processes do one image each. In order to be able to run python with MPI you need a special parallel python interpreter, for example `gpaw`. ``` from ase.io import read from ase.constraints import FixAtoms from ase.calculators.emt import EMT from ase.neb import NEB from ase.optimize import BFGS from ase.parallel import world from gpaw import GPAW initial = read('initial.traj') final = read('final.traj') constraint = FixAtoms(mask=[atom.tag > 1 for atom in initial]) images = [initial] j = world.rank * 3 // world.size # my image number for i in range(3): image = initial.copy() if i == j: image.calc = EMT() image.set_constraint(constraint) images.append(image) images.append(final) neb = NEB(images, parallel=True) neb.interpolate() qn = BFGS(neb, trajectory='neb.traj') qn.run(fmax=0.05) ``` This example can then be run with `mpiexec -np 6 python3 gpaw_neb.py`. The submission script will look as follows: ``` #!/bin/bash #SBATCH --ntasks=6 #SBATCH --partition=mycluster cd $SLURM_SUBMIT_DIR mpiexec -np 6 python3 gpaw_neb.py ``` Now, simply submit your job to the queue: ``` sbatch sub_ase.sh ``` Your slurm output file will log the energies of the optimization iterations and will look something like this: ``` Step Time Energy fmax BFGS: 0 13:23:21 4.219952 3.5208 BFGS: 1 13:23:22 3.937039 2.1765 BFGS: 2 13:23:23 3.719814 0.4351 BFGS: 3 13:23:24 3.709652 0.2301 BFGS: 4 13:23:26 3.708879 0.2441 BFGS: 5 13:23:27 3.706088 0.2577 BFGS: 6 13:23:28 3.698532 0.2134 BFGS: 7 13:23:29 3.692121 0.2462 BFGS: 8 13:23:30 3.692274 0.1873 BFGS: 9 13:23:31 3.693484 0.1727 BFGS: 10 13:23:32 3.692659 0.1514 BFGS: 11 13:23:33 3.690809 0.0736 BFGS: 12 13:23:34 3.690202 0.0708 BFGS: 13 13:23:35 3.690382 0.0782 BFGS: 14 13:23:36 3.690426 0.1034 BFGS: 15 13:23:37 3.689890 0.0998 BFGS: 16 13:23:39 3.689029 0.0543 BFGS: 17 13:23:40 3.688737 0.0289 ``` :::info Tip: you can see the slurm output file being written in real time with the `tail -f slurm-xxx.out` command. ::: The program will also generate the file called `neb.traj`, containing the final coordinates trajectory. To visualize the results, you can create a series of plots, that show the progression of the NEB relaxation, directly at the command line: ``` ase nebplot --share-x --share-y neb.traj ``` This will generate a PDF file called `nebplots.pdf`, containing the images looking like this: ![](https://i.imgur.com/sdj0dZ3.png)![](https://i.imgur.com/2xme5C4.png) # Troubleshooting ## This will not work in our current setup ``` $ sbatch --nodes=3 --ntasks-per-node=2 sub_mpi.sh ``` This is because we are requesting more cores than what we actually have. We have 3 nodes, but we can only run one task per node, because the raspberrypi zeroes have only one. Pizero processor: Broadcom BCM2835,1GHz ARM11 Single-core processor,512MB RAM. Raspberry pi 3B (The controller node): Quad core Cortex-A72 (ARM v8). Quad-core means four cores. # Questions: Yehor: How to configure slurm task file, so it takes one node (with multiple CPU) and uses all CPU on it, but only on this node? Dennis: ``` #!/bin/bash #SBATCH --nodes=1 #SBATCH --ntasks-per-node=N ``` Martijn: Can slurm help to first run a number of tasks (N2), and based on the outcome start another job to run on other number (N2) tasks? First helpful links (Yehor): https://stackoverflow.com/questions/40004378/slurm-how-to-qsub-a-task-when-another-task-is-finished https://stackoverflow.com/questions/46427148/how-to-hold-up-a-script-until-a-slurm-job-start-with-srun-is-completely-finish

    Import from clipboard

    Advanced permission required

    Your current role can only read. Ask the system administrator to acquire write and comment permission.

    This team is disabled

    Sorry, this team is disabled. You can't edit this note.

    This note is locked

    Sorry, only owner can edit this note.

    Reach the limit

    Sorry, you've reached the max length this note can be.
    Please reduce the content or divide it to more notes, thank you!

    Import from Gist

    Import from Snippet

    or

    Export to Snippet

    Are you sure?

    Do you really want to delete this note?
    All users will lost their connection.

    Create a note from template

    Create a note from template

    Oops...
    This template is not available.


    Upgrade

    All
    • All
    • Team
    No template found.

    Create custom template


    Upgrade

    Delete template

    Do you really want to delete this template?

    This page need refresh

    You have an incompatible client version.
    Refresh to update.
    New version available!
    See releases notes here
    Refresh to enjoy new features.
    Your user state has changed.
    Refresh to load new user state.

    Sign in

    Forgot password

    or

    By clicking below, you agree to our terms of service.

    Sign in via Facebook Sign in via Twitter Sign in via GitHub Sign in via Dropbox Sign in via Google

    New to HackMD? Sign up

    Help

    • English
    • 中文
    • Français
    • Deutsch
    • 日本語
    • Español
    • Català
    • Ελληνικά
    • Português
    • italiano
    • Türkçe
    • Русский
    • Nederlands
    • hrvatski jezik
    • język polski
    • Українська
    • हिन्दी
    • svenska
    • Esperanto
    • dansk

    Documents

    Tutorials

    Book Mode Tutorial

    Slide Mode Tutorial

    YAML Metadata

    Contacts

    Facebook

    Twitter

    Feedback

    Send us email

    Resources

    Releases

    Pricing

    Blog

    Policy

    Terms

    Privacy

    Cheatsheet

    Syntax Example Reference
    # Header Header 基本排版
    - Unordered List
    • Unordered List
    1. Ordered List
    1. Ordered List
    - [ ] Todo List
    • Todo List
    > Blockquote
    Blockquote
    **Bold font** Bold font
    *Italics font* Italics font
    ~~Strikethrough~~ Strikethrough
    19^th^ 19th
    H~2~O H2O
    ++Inserted text++ Inserted text
    ==Marked text== Marked text
    [link text](https:// "title") Link
    ![image alt](https:// "title") Image
    `Code` Code 在筆記中貼入程式碼
    ```javascript
    var i = 0;
    ```
    var i = 0;
    :smile: :smile: Emoji list
    {%youtube youtube_id %} Externals
    $L^aT_eX$ LaTeX
    :::info
    This is a alert area.
    :::

    This is a alert area.

    Versions

    Versions and GitHub Sync

    Sign in to link this note to GitHub Learn more
    This note is not linked with GitHub Learn more
     
    Add badge Pull Push GitHub Link Settings
    Upgrade now

    Version named by    

    More Less
    • Edit
    • Delete

    Note content is identical to the latest version.
    Compare with
      Choose a version
      No search result
      Version not found

    Feedback

    Submission failed, please try again

    Thanks for your support.

    On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?

    Please give us some advice and help us improve HackMD.

     

    Thanks for your feedback

    Remove version name

    Do you want to remove this version name and description?

    Transfer ownership

    Transfer to
      Warning: is a public team. If you transfer note to this team, everyone on the web can find and read this note.

        Link with GitHub

        Please authorize HackMD on GitHub

        Please sign in to GitHub and install the HackMD app on your GitHub repo. Learn more

         Sign in to GitHub

        HackMD links with GitHub through a GitHub App. You can choose which repo to install our App.

        Push the note to GitHub Push to GitHub Pull a file from GitHub

          Authorize again
         

        Choose which file to push to

        Select repo
        Refresh Authorize more repos
        Select branch
        Select file
        Select branch
        Choose version(s) to push
        • Save a new version and push
        • Choose from existing versions
        Available push count

        Upgrade

        Pull from GitHub

         
        File from GitHub
        File from HackMD

        GitHub Link Settings

        File linked

        Linked by
        File path
        Last synced branch
        Available push count

        Upgrade

        Danger Zone

        Unlink
        You will no longer receive notification when GitHub file changes after unlink.

        Syncing

        Push failed

        Push successfully