CSI Tutorial Session: using DeePMD-kit, LAMMPS, QE, and DP-GEN

CSI Tutorial Session: using DeePMD-kit, LAMMPS, QE, and DP-GEN === #### Sept. 27 2019 @ Princeton CSI center ##### Tutors: Hsin-Yu Ko (Cornell CHEM), Marcos F. Calegari (Princeton CHEM), Linfeng Zhang (Princeton PACM) # Aim and plan The aim of this 4-hour hands-on session is to introduce to you how to use DeePMD-kit, LAMMPS, QE, and DP-GEN, at the basic level, and introduce you some technical details about DeePMD-kit and QE at an advanced level. First, we will use the example of a single methane molecule and go through the whole DP-GEN process that 1) uses DeePMD-kit for training; 2) uses LAMMPS to explore the configuration space; and 3) uses QE to do ab initio calculations and prepare the training data. The outout will be a uniformly accurate Deep Potential model for the methane molecule (~1.5h). Second, we will talk in more details about the GPU implementation of DeePMD-kit and its interface with LAMMPS (~0.5-1h), as well as the cpmd code in QE and the GPU implementation of the exact exchange part (1.5~2h). This part will be more technical and since it requires much more computational resourses, the students may not be able to run all the benchmark examples by themselves during the hands-on session. # Learning outcomes Once this tutorial is completed, students will be able to: + get familar with the basic input and output files, the code structure, and the features of DeePMD-kit, LAMMPS, QE, and DP-GEN + go one step deeper into the DeePMD-kit and QE codes, know basics of how they run on big CPU/GPU clusters + write a simple DP-GEN input file and use it to generate models for other systems + (advanced level) contribute to DeePMD-kit, QE, and DP-GEN to better serve their own research purpose and, if possible, the molecular simulation community. # Quick start We will use the `traverse` machine to do this tutorial. 1. Make sure you have an account on `traverse`. Suppose your username is $USER, you should be able to log on traverse by typing ``` $$ ssh $USER@traverse.princeton.edu ``` 2. Creat a link to Traverse scratch GPFS, go to SCR and creat a workspace with a name $workspace: ``` $$ ln -s /scratch/gpfs/$USER SCR $$ cd SCR $$ mkdir -p $workspace $$ cd $workspace ``` 3. Copy the tutorial materials to the workspace and tar extract it **(Or we put it online and download?)**: ``` $$ cp /home/linfengz/SCR/research/dpgen-example/092719_csi_handson_tutorial.tar . $$ tar xvf 092719_csi_handson_tutorial.tar $$ cd 092719_csi_handson_tutorial ``` 4. Set up general environment and modify the machine setup file `machine.json` with your own information ``` $$ module load anaconda3/2019.3 $$ module load rh/devtoolset/7 $$ module load cudnn/cuda-9.2/7.6.1 $$ module load cudatoolkit/9.2 $$ alias quedir=' squeue -u $USER -o "%.18i %.9P %.2t %Z"' $$ sed -i 's|xxxxxxx|'$PWD'/tmp|g' machine.json $$ sed -i 's|linfengzzz|'$USER'|g' machine.json ``` 5. Set up the conda python enviroment prepared for this session. We prepared an execusible script `set_conda.sh` for you, so you may do the following and take a look what is inside of it: ``` $$ ./set_conda.sh ``` 6. Set up ssh keys so that you can ssh traverse from traverse without the need to provide the password. We prepared an execusible script `set_sshkey.sh` for you: ``` $$ ./set_sshkey.sh ``` 7. Finally, you should be able to run the `dpgen` code by typing ``` $$ nohup ./run.sh & ``` It will be running on the background and automatically submit jobs, collect results, etc. More details will be explained below or during the session. # More resources - **DeePMD-kit and DP-GEN**: - **papers**: See [DeePMD-PRL][1] for the original version of Deep Potential Molecular Dynamics. See ([DeepPot-SE-NIPS][2]) for the smooth version, which has gained more popularity. See [DeePCG-JCP][3] about how to develop coarse-grained models based on a similar approach. See [DPGEN-PRM][4] about the active learning procedure for the generation of reliable Deep Potential models. - **codes**: All codes are available on Github (https://github.com/deepmodeling). They are open source under the `GNU LGPLv3.0` liscense. In particular, the devel branch of the [deepmd-kit][5] and the [dpgen][6] repos are extremely active. - **Q&A, discussion, and more information:** we recently lauched a website (http://bbs.deepmd.org/), hoping to better serve the community that uses deep learning tools for molecular simulation. It is still young. Any feedbacks are appreciated. We encourage the students to register and ask questions on the Q&A page. - **QE and CPMD**: - **TODO.** [1]: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.143001 [2]:http://papers.nips.cc/paper/7696-end-to-end-symmetry-preserving-inter-atomic-potential-energy-model-for-finite-and-extended-systems [3]:https://aip.scitation.org/doi/full/10.1063/1.5027645 [4]:https://journals.aps.org/prmaterials/abstract/10.1103/PhysRevMaterials.3.023804 [5]:https://github.com/deepmodeling/deepmd-kit/tree/devel [6]:https://github.com/deepmodeling/dpgen/tree/devel # The DP-GEN example We will explain step-by-step the input and output files and results of the `dpgen` code. We refer to additional slides for the second part of the tutorial. ## General Introduction `dpgen` does a series of iterations. In each iteration, there are three steps, `00.train`, `01.model_devi` and ` 02.fp`. + 00.train: `dpgen` will train several (default 4) models based on exisiting training data. The only difference between these models is the random seed for the initialization of network parameters. **DeePMD-kit will be used for this step.** + 01.model_devi : model_devi for model deviation. `dpgen` will use models obtained from 00.train and run MD to explore the configuration space. For different structures, larger deviation indicates worse accuracy of the models. Using this criterion, a few structures will be selected and put into next stage `02.fp` for accurate calculations based on first-principle methods. **LAMMPS will be used for this step.** + 02.fp : Selected structures will be calculated by first-principle methods. `dpgen` will obtain some new data and put them together with initial data and data generated in previous iterations. After that a new training will be set up and `dpgen` will enter the next iteration. **QE will be used for this step.** ## Setups ### Initial data for training and MD We prepared two sets of data to kick off the DP-GEN process: + A set of initial training data. We prepared it here: `/home/linfengz/SCR/research/dpgen-example/CH4.init.data/02.md/sys-0004-0001/deepmd` + A set of initial snapshots for the exploration stage. We prepared it here: `/home/linfengz/SCR/research/dpgen-example/CH4.init.data/01.scale_pert/sys-0004-0001/scale-1.000/` The snapshots for training and MD come from a direct MD simulation of CH4 at 50 K. ### DP-GEN Input files You take a look by typing `dpgen run -h`. The following text and more informationshould appear on the screen: ``` usage: dpgen run [-h] PARAM MACHINE positional arguments: PARAM parameter file, json/yaml format MACHINE machine file, json/yaml format optional arguments: -h, --help show this help message and exit ``` In the following parts, we will show you how to write `param.json` and `machine.json`. ## Writing `param.json` ```json { "type_map": [ "H", "C" ], "mass_map": [ 1, 12 ], "init_data_prefix": "/home/linfengz/SCR/research/dpgen-example/", "init_data_sys": [ "CH4.init.data/02.md/sys-0004-0001/deepmd" ], "init_batch_size": [ 8 ], "sys_configs": [ [ "/home/linfengz/SCR/research/dpgen-example/CH4.init.data/01.scale_pert/sys-0004-0001/scale-1.000/00000*/POSCAR" ], [ "/home/linfengz/SCR/research/dpgen-example/CH4.init.data/01.scale_pert/sys-0004-0001/scale-1.000/00001*/POSCAR" ] ], "sys_batch_size": [ 8, 8 ], "_comment": " that's all ", "numb_models": 4, "train_param": "input.json", "default_training_param": { "_comment": "that's all", "use_smooth": true, "sel_a": [ 16, 4 ], "rcut_smth": 0.5, "rcut": 5, "filter_neuron": [ 10, 20, 40 ], "filter_resnet_dt": false, "n_axis_neuron": 12, "n_neuron": [ 120, 120, 120 ], "resnet_dt": true, "coord_norm": true, "type_fitting_net": false, "systems": [], "set_prefix": "set", "stop_batch": 40000, "batch_size": 1, "start_lr": 0.001, "decay_steps": 200, "decay_rate": 0.95, "seed": 0, "start_pref_e": 0.02, "limit_pref_e": 2, "start_pref_f": 1000, "limit_pref_f": 1, "start_pref_v": 0, "limit_pref_v": 0, "disp_file": "lcurve.out", "disp_freq": 1000, "numb_test": 4, "save_freq": 1000, "save_ckpt": "model.ckpt", "load_ckpt": "model.ckpt", "disp_training": true, "time_training": true, "profiling": false, "profiling_file": "timeline.json" }, "model_devi_dt": 0.002, "model_devi_skip": 0, "model_devi_f_trust_lo": 0.012, "model_devi_f_trust_hi": 0.10, "model_devi_clean_traj": false, "model_devi_jobs": [ { "sys_idx": [ 0 ], "temps": [ 50 ], "press": [ 1 ], "trj_freq": 10, "nsteps": 1000, "ensemble": "nvt", "_idx": "00" }, { "sys_idx": [ 1 ], "temps": [ 50 ], "press": [ 1 ], "trj_freq": 10, "nsteps": 3000, "ensemble": "nvt", "_idx": "01" } ], "fp_style": "pwscf", "shuffle_poscar": false, "fp_task_max": 30, "fp_task_min": 5, "fp_pp_path": "/home/linfengz/SCR/research/dpgen-example/test_qe/CH4.init.data/pp/", "fp_pp_files": [ "H.tm.pbe.UPF", "C.tm.pbe.UPF" ], "user_fp_params": { "_comment": "pseudo_dir will be added automatically", "control": { "calculation": "scf", "restart_mode": "from_scratch", "outdir": "./OUT", "tprnfor": true, "tstress": true, "disk_io": "none" }, "_comment": "ibrav, nat and ntyp will be added automatically", "system": { "ecutwfc": 110, "ts_vdw_econv_thr": 1e-8, "nosym": true }, "electrons": { "conv_thr": 1e-8 }, "kspacing": 100 }, "_comment": " that's all " } ``` The following table gives explicit descriptions on keys in `PARAM`. The bold notation of key (such aas **type_map**) means that it's a necessary key. |Key | Type | Example | Discription | | :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| | *#Basics* | **type_map** | List of string | ["H", "C"] | Atom types | **mass_map** | List of float | [1, 12] | Standard atom weights. | *#Data* | init_data_prefix | String | "/path/to/dpgen-example/" | Prefix of initial data directories | ***init_data_sys*** | List of string|["CH4.POSCAR.01x01x01/.../deepmd"] |Directories of initial data. You may use either absolute or relative path here. | **init_batch_size** | String of integer | [8] | Each number is the batch_size of corresponding system for training in `init_data_sys`. One recommended rule for setting the `sys_batch_size` and `init_batch_size` is that `batch_size` mutiply number of atoms ot the stucture should be larger than 32. If set to `auto`, batch size will be 32 divided by number of atoms. | | sys_configs_prefix | String | "/path/to/dpgen-example/" | Prefix of `sys_configs` | **sys_configs** | List of list of string | [ ["/path/to/00000*/POSCAR"], ["/path/to/00001*/POSCAR"] ] | Containing directories of structures to be explored in iterations.Wildcard characters are supported here. | | **sys_batch_size** | List of integer | [8, 8] | Each number is the batch_size for training of corresponding system in `sys_configs`. If set to `auto`, batch size will be 32 divided by number of atoms. | | *#Training* | **numb_models** | Integer | 4 (recommend) | Number of models to be trained in `00.train`. | | **default_training_param** | Dict | { ... "use_smooth": true, "sel_a": [16, 4], "rcut_smth": 0.5, "rcut": 5, "filter_neuron": [10, 20, 40], ... } | Training parameters for `deepmd-kit` in `00.train`. You can find instructions from here: (https://github.com/deepmodeling/deepmd-kit).. We commonly let `stop_batch` = 200 * `decay_steps`. | | *#Exploration* | **model_devi_dt** | Float | 0.002 (recommend) | Timestep for MD | | **model_devi_skip** | Integer | 0 | Number of structures skipped for fp in each MD | **model_devi_f_trust_lo** | Float | 0.012 | Lower bound of forces for the selection. | **model_devi_f_trust_hi** | Float | 0.1 | Upper bound of forces for the selection | **model_devi_clean_traj** | Boolean | False | Deciding whether to clean traj folders in MD since they are too large. | | **model_devi_jobs** | [ { "sys_idx": [0], "temps": [100], "press": [1], "trj_freq": 10, "nsteps": 1000, "ensembles": "nvt" }, ... ] | List of dict | Settings for exploration in `01.model_devi`. Each dict in the list corresponds to one iteration. The index of `model_devi_jobs` exactly accord with index of iterations | | **model_devi_jobs["sys_idx"]** | List of integer | [0] | Systems to be selected as the initial structure of MD and be explored. The index corresponds exactly to the `sys_configs`. | | **model_devi_jobs["temps"]** | List of integer | [50, 300] | Temperature (**K**) in MD | **model_devi_jobs["press"]** | List of integer | [1,10] | Pressure (**Bar**) in MD | **model_devi_jobs["trj_freq"]** | Integer | 10 | Frequecy of trajectory saved in MD. | | **model_devi_jobs["nsteps"]** | Integer | 3000 | Running steps of MD. | | **model_devi_jobs["ensembles"]** | String | "nvt" | Determining which ensemble used in MD, **options** include “npt” and “nvt”. | | model_devi_jobs["neidelay"] | Integer | "10" | delay building until this many steps since last build | | model_devi_jobs["taut"] | Float | "0.1" | Coupling time of thermostat (fs) | | model_devi_jobs["taup"] | Float | "0.5" | Coupling time of barostat (fs) | *#Labeling* | **fp_style** | string | "pwscf" | Software for First Principles. **Options** include “vasp”, “pwscf”， “cp2k”， and “gaussian” up to now. | | **fp_task_max** | Integer | 20 | Maximum of structures to be calculated in `02.fp` of each iteration. | | **fp_task_min** | Integer | 5 | Minimum of structures to calculate in `02.fp` of each iteration. | | *fp_style == pwscf* | **fp_pp_path** | String | "/home/linfengz/.../CH4.init.data/pp/" | Directory of psuedo-potential file to be used for 02.fp exists. | | **fp_pp_files** | List of string | ["H.tm.pbe.UPF", "C.tm.pbe.UPF"] | Psuedo-potential file to be used for 02.fp. Note that the order of elements should correspond to the order in `type_map`. | |**user_fp_params** | json dicts | | User-specific input ## Writing `machine.json` When switching into a new machine, you may modifying the `machine.json`, according to your own settings. We've provided an example set on `slurm` as follows. When switching into a new machine, you may modifying the `MACHINE`, according to the actual circumstance. Once you have finished, the `MACHINE` can be re-used for any DP-GEN tasks without any extra efforts. An example for `MACHINE` is: ```json { "train": [ { "machine": { "machine_type": "slurm", "hostname": "localhost", "port": 22, "username": "linfengz", "work_path": "/home/linfengz/SCR/research/dpgen-example/test2/tmp" }, "resources": { "numb_node": 1, "numb_gpu": 1, "task_per_node": 4, "partition": "all", "exclude_list": [], "mem_limit": 8, "source_list": [ "/home/linfengz/SCR/research/dpgen-example/test_qe/train.env" ], "module_list": [], "time_limit": "23:0:0" }, "deepmd_path": "/home/linfengz/SCR/local/deepmd-kit-v0.12.6" } ], "model_devi": [ { "machine": { "machine_type": "slurm", "hostname": "localhost", "port": 22, "username": "linfengz", "work_path": "/home/linfengz/SCR/research/dpgen-example/test2/tmp" }, "resources": { "numb_node": 1, "numb_gpu": 1, "task_per_node": 2, "partition": "all", "exclude_list": [], "mem_limit": 8, "source_list": [ "/home/linfengz/SCR/research/dpgen-example/test_qe/lmp.env" ], "module_list": [], "time_limit": "23:0:0" }, "command": "lmp_serial", "group_size": 5 } ], "fp": [ { "machine": { "machine_type": "slurm", "hostname": "localhost", "port": 22, "username": "linfengz", "work_path": "/home/linfengz/SCR/research/dpgen-example/test2/tmp" }, "resources": { "task_per_node": 28, "numb_gpu": 0, "exclude_list": [], "mem_limit": 32, "with_mpi": false, "source_list": [ "/home/linfengz/SCR/research/dpgen-example/test_qe/qe.env" ], "module_list": [], "time_limit": "2:0:0", "partition": "all", "_comment": "that's All" }, "command": "srun /home/mandrade/QE-SVN/bin/pw.x < input", "group_size": 10 } ] } ``` The following table illustrates which key is needed for the three types of machine: `train`,`model_devi` and `fp`. Each of them is a list of dicts. Each dict can be considered as an independent environmnet for calculation. |Key | `train` | `model_devi` | `fp` | | :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| | machine | NEED | NEED | NEED | resources | NEED | NEED | NEED | deepmd_path | NEED | | command | |NEED | NEED | | group_size | | NEED | NEED | The following table gives explicit descriptions on keys in machine.json. |Key | Type | Example | Discription | | :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| |deepmd_path | String |"/home/linfengz/SCR/local/deepmd-kit-v0.12.6" | Installed directory of DeepMD-Kit 0.x, which should contain `bin lib include`. | machine | Dict | | Settings of the machine for TASK. | resources | Dict | | Resources needed for calculation. | # Keys in resources | numb_node | Integer | 1 | Node count required for the job | task_per_node | Integer | 4 | Number of CPU cores required | `numb_gpu` | Integer | 4 | Number of GPUs required | source_list | List of string | "....../qe.env" | Environment needed for certain job. For example, if "env" is in the list, 'source env' will be written in the script. | module_list | List of string | [ "Intel/2018", "Anaconda3"] | For example, If "Intel/2018" is in the list, "module load Intel/2018" will be written in the script. | time_limit | String (time format) | 23:00:00 | Maximal time permitted for the job |mem_limit | Interger | 16 | Maximal memory permitted to apply for the job. | with_mpi | Boolean | true | Deciding whether to use mpi for calculation. If it's true and machine type is Slurm, "srun" will be prefixed to `command` in the script. | # End of resources | command | String | "lmp_serial" | Executable path of software, such as `lmp_serial`, `lmp_mpi` and `pw.x`, etc. | group_size | Integer | 5 | DP-GEN will put these jobs together in one submitting script. ## Start Generator ### Basics You've started the DP-GEN process by doing `nohup ./run.sh &`. Let's continue: ```bash cat run.sh ``` We can see the following command ``` dpgen run param.json machine.json >log.out 2>&1 & ``` `dpgen` identifies the current stage by a record file, `record.dpgen`, which will be created and upgraded by codes. Each line contains two number: the first is index of iteration, and the second ,ranging from 0 to 9 ,records which stage in each iteration is currently running. 0,1,2 correspond to `make_train`, `run_train`, `post_train`. `dpgen` will write scripts in `make_train`, run tasks by specific machine in `run_train` and collect results in `post_train`. The record for `model_devi` and `fp` stage follows similar rules. In your current path, you may see a file `record.dpgen` we've introduced before. If the process of `dpgen` unfortunately stops by some reason, `dpgen` will automatically recover the main process by `record.dpgen`. You may also change it manually for your purpose, such as removing the last iterations and recovering from one checkpoint. You will also see a series of folders `iter.*`. They contain our main results that `dpgen` generates. In each folder, there're 3 sub-folders `00.train`, `01.model_devi` and `02.fp`, corresponding with 3 stages previously introduced. | Index of iterations | Stage in each iteration | Process | :--------------- | :--------------------- | :-------------------------------------- | | 0 | 0 | make_train | 0 | 1 | run_train | 0 | 2 | post_train | 0 | 3 | make_model_devi | 0 | 4 | run_model_devi | 0 | 5 | post_model_devi | 0 | 6 | make_fp | 0 | 7 | run_fp | 0 | 8 | post_fp ### Details of each iteration We've provided results from a successful runing in `92719_csi_handson_tutorial/ch4_previous_run`. Next we will use it to demonstrate details of `dpgen`. #### 00.train ```bash cd 92719_csi_handson_tutorial/ch4_previous_run cd iter.000000/00.train ls ``` You'll see ``` 000 002 data.init graph.000.pb graph.002.pb 001 003 data.iters graph.001.pb graph.003.pb ``` Here `graph.00x.pb` , linked to `00x/frozen.pb`, is the model `deepmd-kit` generates. Enter one folder, you will find: `frozen_model.pb input.json lcurve.out` + `input.json` is the settings for `deepmd-kit` for current task. + `lcurve.out` records the training accuracy of energies and forces. Then `cd 000` By `head -n 2 lcurve.out && tail -n 2 lcurve.out` You will see ``` # batch l2_tst l2_trn l2_e_tst l2_e_trn l2_f_tst l2_f_trn lr 0 6.12e+01 6.41e+01 6.21e+00 6.20e+00 1.93e+00 2.02e+00 1.0e-03 39000 1.26e-02 1.24e-02 8.56e-05 5.52e-05 1.24e-02 1.22e-02 4.5e-08 40000 1.26e-02 1.44e-02 7.44e-05 8.52e-05 1.24e-02 1.42e-02 3.5e-08 ``` The total number of batches here corresponds to our settings of `stop_batch_size` in `param.json`. Let's step forward. #### 01.model_devi Go back and enter `../../iter.000000/01.model_devi`. You will see ten `task.*` folders ranging from `task.000.000000` to `task.000.000009`. You may randomly select one of them, like `task.000.000006` and enter it. After`ls`, you will see ``` conf.lmp input.lammps model_devi.log model_devi.out ``` + `conf.lmp` serves as the initial point of MD. + `input.lammps` is the input file for LAMMPS, automatically generated by `dpgen`. + `model_devi.out` records the model deviation of concerned labels, energy and force, in MD. It serves as the criterion for selecting which structures and doing Ab-initio calculations. By `head model_devi.out`, you will see: ``` # step max_devi_e min_devi_e avg_devi_e max_devi_f min_devi_f avg_devi_f 0 9.343707e-01 2.288385e-01 3.738665e-01 8.401729e-03 6.477382e-03 7.289600e-03 10 9.371803e-01 2.274761e-01 3.749860e-01 1.348967e-02 5.478629e-03 1.160542e-02 20 9.424105e-01 2.288360e-01 3.771461e-01 2.169293e-02 8.194449e-03 1.773667e-02 30 9.489608e-01 2.292192e-01 3.798002e-01 1.405901e-02 5.863203e-03 1.192985e-02 40 9.525256e-01 2.271607e-01 3.812842e-01 5.866476e-03 4.941065e-03 5.518061e-03 ``` `max_devi_f` is the error indicator that we use. #### 02.fp Now let's go back to folder `ch4_previous_example` and enter `iter.000000/02.fp`. You will see: ``` candidate.shuffled.000.out rest.shuffled.000.out data.000 task.000.0000?? ``` + `candidate.shuffle.000.out`: records which structures will be seleted from last step `01.model_devi`. There are always far more candidates than the maximum `fp_task_max` you expect to calculate at one time. In this condition, `dpgen` will randomly choose up to `fp_task_max` strucures and form the folder `task.*.0000*`. + `rest.shuffle.000.out`: record the other structures where our model's is either too accurate (`max_devi_f` is less than `model_devi_f_trust_lo`, no need to calculate any more), or too inacurate (lager than `model_devi_f_trust_lo`, there may be some error). + `data.000` After Ab-initio calculations, `dpgen` will collect these data and convert them into the format `deepmd-kit` needs. In the next iteration'. ## Results Return to the folder `previous_example` By ` wc -l iter.000000/02.fp/*out`, we will see ``` 557 iter.000000/02.fp/candidate.shuffled.000.out 453 iter.000000/02.fp/rest_accurate.shuffled.000.out 0 iter.000000/02.fp/rest_failed.shuffled.000.out 1010 total ``` This means there are 557 out of 1010 structures in the fisrt iteration which could be candidates for DFT calculations. In `param.json`, we set `fp_task_max` to 30. So 30 structures are actually selected for DFT calculations. However, in the second iteration `iter.000001`, when doing `wc -l iter.000001/02.fp/*out`, you may find ``` 29 iter.000001/02.fp/candidate.shuffled.001.out 2981 iter.000001/02.fp/rest_accurate.shuffled.001.out 0 iter.000001/02.fp/rest_failed.shuffled.001.out 3010 total ``` It shows that there remain only 29 candidates. Notice that the number of MD steps we set in `iter.000001` is 3 times longer than in `iter.000000`. The last iteration `iter.000002` will only contain `00.train` and will return 4 DP models of good qualities for CH4 at 50K. # Preparation for `CP` on Traverse ```bash cd some_workspace cp -r /home/hsinyu/share/csi_qe_gpu_2019 . source /home/hsinyu/q-e-nesap/get_module_traverse.sh ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.