---
title: 'ANN TFlow Documents'
disqus: hackmd
---

ANN TFlow
===

## Table of Contents

[TOC]

Introduction
---

ANN-TFlow is a TensorFlow-based Neural Network model open-source package. This project is being developed at Academia Sinica, Taiwan, in the Research Center for Applied Sciences, primarily by Chun-Wei Pao and I-Ta Hsieh, and is released under the GNU General Public License.

ANN-TFlow is designed to easily bring machine learning to atomistic calculations. This allows one to predict (or really, interpolate) calculations on the potential energy surface. The ANN-TFlow works by first learning from quantum mechanical calculations (usually VASP) that can provide energy and forces as a function of atomic coordinates. The predictions of both energy and force from ANN-TFlow can take place with arbitrary accuracy, approaching that of the original calculator. As an improvement of the Atomistic Machine-learning Package (AMP) open-source package, ANN-TFlow allows multiple GPU training that significantly accelerates the training of Neural Networks and increases the total samples in the training set as well.

ANN-TFlow is designed to integrate closely with the TensorFlow, Atomistic Machine-learning Package (AMP), and Atomic Simulation Environment (ASE). The interface is in pure python. Several compute-heavy parts of the codes, such as calculations of fingerprint and fingerprint-prime, will have Numba (A High-Performance Python Compiler) version to accelerate the calculations in the near future.

Using ANN_TFlow
---
### User Flow

graph TD
  DFT --> ASE_DB --> FP/dFP --> Train_NN --> MD_Simulation --> Get_New_Stuct --> DFT

### Multi-GPUs Training Procedure

### Results
Example: High Entropy Alloy (Co, Ti, Ni, Zr, Hf)

| | Energy RMSE | Force RMSE |
| --------------------------------- | ------------------- | ------------------- |
|(1) L-BFGS + Max-min | $1.32\times10^{-3}$ | $8.43\times10^{-2}$ |
|(2) L-BFGS + Mean-Stdev | $1.16\times10^{-3}$ | $7.94\times10^{-2}$ |
|(3) Adam + Max-min + Batch training| $8.85\times10^{-3}$ | $1.70\times10^{-1}$ |

Parity
- Energy

- Force

Example
---
### Training ML Potential

import os
# set -1 to use cpu
#os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"

from ann_modules import NeuralNetwork
from amp.descriptor.gaussian import make_symmetry_functions

e_list = ['Co', 'Hf', 'Ni', 'Ti', 'Zr']
G = make_symmetry_functions(elements=e_list, type='G2', etas=[0.05,2.,6.,20.,40.,80.])
G += make_symmetry_functions(elements=e_list, type='G4', etas=[0.005], zetas=[1.,2.,4.,16.], gammas=[+1.,-1.])
G = {'Co': G,'Hf': G,'Ni': G,'Ti': G,'Zr': G}

model = NeuralNetwork(images='md.db',sampling="0:1000:1",
                      validation_images='md.db',validation_sampling='1:1000:2',
                      parameters=None, parameters_epoch=None,
                      action='training', hiddenlayers=(10, 10, 10), cores=16,
                      stdscaler='normalize', Gs=G, cutoff=5.5,
                      multi_gpus=True, gpu_max_usage=None,
                      energy_coefficient=1.0, force_coefficient=0.05,
                      convergenceCriteria={'energy_rmse': 0.001, 'energy_maxresid': None, \
                                           'force_rmse': 0.1, 'force_maxresid': None})

model.train(optimizer='l-bfgs', batch=None, checkpoint=2000, maxEpochs=50000,
            lbfgs_correction_pairs=50, # for l-bfgs
            learning_rate=0.001, amsgrad=False, # for adam
            elastic_net_lambda=None, elastic_net_alpha=0., gradient_noise=False)

### Predict Energy and Forces

import os
# set -1 to use cpu
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
#os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"

from ann_modules import NeuralNetwork

model = NeuralNetwork(images='md.db', sampling=':', 
                      parameters='ANN-tflow.checkpoint', parameters_epoch=None,
                      action="calculation")

energy = model.calculate_energy()
force = model.calculate_force()

print(energy)
print(force)

### Plot Parity

import os
# set -1 to use cpu
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
#os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"

from ann_modules import ParityPlot

ParityPlot(images='md.db', sampling=None, 
           parameters='ANN-tflow.checkpoint', parameters_epoch=None,
           force_predict=True)

### Transform ANN-TFlow Results into LAMMPS PairANN Input Format

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

import ann2amplmp

ann2amplmp.write_lmp(params_in='ANN-tflow.checkpoint', epoch=None)

Module Documentation
---
### NeuralNetwork
A TensorFlow-based Neural Network model which constructs a neural network by using TensorFlow 2.0. This funciton also allows for GPU acceleration.

- #### Args:
**images:** (*str*) List of ASE atoms objects that are being trained.

**sampling:** (*str* or *int*) A str or int about sampling method of data point in images. E.g. 100, '50:500', '0:500:2'. Default is None.

**validation_images:** (*str*) List of ASE atoms objects that are being validated while training. Default is None.

**validation_sampling:** (*str* or *int*) A str or int about sampling method of data point in validation_images. Default is None.

**parameters:** (*str*) File of AMP or ANN-tflow that contains parameters. Default is None.

**parameters_epoch:** (*int*) Epoch of weights in the checkpoint file that you want to use as initial weights. Default is None.

**action:** (*str*) The action you want to do with the class NeuralNetwork. Set "training" to fit an ANN model, "calculation" to predict energy and forces with given parameters. Default is "training".

**hiddenlayers:** (*tuple* or *dict*) Structure of the neural network. Can either be in the format (int,int,int), where each element represnts the size of a layer and there and the length of the list is the number of layers, or dictionary format of the network structure for each element type. E.g. {'Cu': (10, 5, 5), 'Pt': (10, 5)}. Default is (5, 5).

**cores:** (*int*) The number of iterations allowed tensorflow probability l-bfgs optimizer and AMP calculator to run in parallel. Default is 8.

**stdscaler:** (*str*) Normalize method of input fingerprints and fingerprint-primes. 2 methods are allowed, 'normalize' and 'zero_mean'. Default is 'normalize'.

**Gs:** (*dict* or *list* or *None*) Dictionary of symbols and lists of dictionaries for making symmetry functions. Either auto-genetrated by AMP, or given in the following form, E.g.

Gs = {"Te": [{"type":"G2", "element":"O", "eta":10.},
{"type":"G4", "elements":["O", "Au"],
"eta":5., "gamma":1., "zeta":1.0}],
"Bi": [{"type":"G2", "element":"O", "eta":2.},
{"type":"G4", "elements":["O", "Au"],
"eta":2., "gamma":1., "zeta":5.0}]}

You can use amp.model.gaussian.make_symmetry_functions to help create these lists of dictionaries. Default is None.

**cutoff:** (*float*) Cutoff represent the radius above which neighbor interactions are ignored. Default is 6.5 Angstroms.

**multi_gpus:** (*bool*) Set this to True allows using multiple GPUs in training procedure. Default is False.

**gpu_max_usage:** (*int*) Maximum memory (in MB) to allocate on the GPUs device. None to use all GPUs device memories. Default is None.

**energy_coefficient:** (*float*) Used to adjust the loss function; this is the initial weight applied to the energy component. Default is 1.0.

**force_coefficient:** (*float* or *None*) Used to adjust the loss function; this is the initial weight applied to the force component. Default is 0.04.

**convergenceCriteria:** (*dict*) Dictionary of convergence criteria, analogous to the main AMP convergence criteria dictionary. Default is

{'energy_rmse': 0.001, 'energy_maxresid': None,
'force_rmse' : None, 'force_maxresid' : None}

Note: you can turn off force training by setting both 'force_rmse' and 'force_maxresid' to None.

### ANNDataSet
Get data sets of ANN models from db files for ANN models by using AMP and ASE.

- #### Args:
**force_training:** (*bool*) Fit the model w or w/o force training. If 'True', the functions below will calculate fingerprint-primes. Default is False.

**stdscaler:** (*str*) Normalize method of input fingerprints and fingerprint-primes. 2 methods are allowed, 'normalize' and 'zero_mean'. Default is 'normalize'.

**elements:** (*List*) A list contains all types of element in training sets.

**fprange:** (*dict*) A Dictionary contains range of each fingerprint for all elements. Which is needed by normalize method.

**mean_stdev:** (*dict*) A Dictionary contains mean and standard deviatiob of each fingerprint for all elements. Which is needed by zero_mean method.

**Gs:** (*dict* or *list* or *None*) Dictionary of symbols and lists of dictionaries for making symmetry functions. Either auto-genetrated by AMP, or given in the following form, E.g.

Gs = {"Te": [{"type":"G2", "element":"O", "eta":10.},
{"type":"G4", "elements":["O", "Au"],
"eta":5., "gamma":1., "zeta":1.0}],
"Bi": [{"type":"G2", "element":"O", "eta":2.},
{"type":"G4", "elements":["O", "Au"],
"eta":2., "gamma":1., "zeta":5.0}]}

You can use amp.model.gaussian.make_symmetry_functions to help create these lists of dictionaries. Default is None.

**cutoff:** (*float*) Cutoff represent the radius above which neighbor interactions are ignored. Default is 6.5 Angstroms.

**cores:** (*int*) The number of iterations allowed tensorflow probability l-bfgs optimizer and AMP calculator to run in parallel. Default is 8.

### Training
Main functions of training loops.

- #### Args:
**model:** (*tensorflow/keras model object*) An instance of a tensorflow/keras ANN model.

**HL:** (*dict*) A dictionary of all hidden layers.

**elements:** (*list*) A list of all types of elements.

**force_training:** (*bool*) Fit the model w or w/o force training. Default is False.

**batch_training:** (*bool*) Fit the model w or w/o batch training. Default is False.

**coefficient:** (*list*) A list of int used to adjust the loss function; this is the weights applied to the energy and force components. E.g. [1.0, 0.04]

**convergenceCriteria:** (*dict*) Dictionary of convergence criteria, analogous to the main AMP convergence criteria dictionary.

**totalImages:** (*int*) A int of total images of training set.

**v_totalImages:** (*int*) A int of total images of validation set.

**descriptor_parameters:** (*dict*) A dictionary of Gaussian parameters from AMP calculator.

**stdscaler:** (*str*) Normalize method of input fingerprints and fingerprint-primes. 2 methods are allowed, 'normalize' and 'zero_mean'. Default is 'normalize'.

**fprange:** (*dict*) A Dictionary contains range of each fingerprint for all elements. Which is needed by normalize method.

**mean_stdev:** (*dict*) A Dictionary contains mean and standard deviatiob of each fingerprint for all elements. Which is needed by zero_mean method.

**hidden_layers_struct:** (*dict* of *tuples*) A dictionary of tuples contains the structure of the neural network for each type of elements. E.g. {'Cu': (10, 5, 5), 'Pt': (10, 5)}.

**multi_gpus:** (*bool*) Set this to True allows using multiple GPUs in training procedure. Default is False.

**gpus:** (*list*) A list of GPU names that can be used by tensorflow.

**validation:** (*bool*) A bool to control if validation set should be calculated.

### Calculation
Calculators of energy and force

- #### Args:
**elements:** (*list*) A list of all types of elements.

### ParityPlot
Calculate and Plot Parities.

- #### Args:
**images:** (*str*) List of ASE atoms objects that are being trained.

**sampling:** (*str* or *int*) A str or int about sampling method of data point in images. E.g. 100, '50:500', '0:500:2'. Default is None.

**parameters:** (*str*) File of AMP or ANN-tflow that contains parameters. Default is None.

**parameters_epoch:** (*int*) Epoch of weights in the checkpoint file that you want to use as initial weights. Default is None.

**force_predict:** (*bool*) Force prediction. Default is False.

Theory
---
### Atomic representation of potential energy

### Gaussian descriptor
The Gaussian descriptor creates feature vectors for the input layer of the atomic neural network based on the Behler scheme, and defaults to a small set of reasonable values. In ANN-TFlow, the Gaussian descriptor functions were divided into 2 categories, $G^{II}$ (the radial descriptor) and $G^{IV}$ (the
angular descriptor), and can be expressed as

- $G^{II}_{i}=\displaystyle\sum_{i\neq j}exp(-\eta\dfrac{R^{2}_{ij}}{R^{2}_c})f_c(R_{ij})$

- $G^{IV}_{i}=2^{1-\zeta}\displaystyle\sum_{i,j\neq k}(1+\lambda cos\theta_{ijk})^\zeta exp(-\eta\dfrac{R^{2}_{ij}+R^{2}_{ik}+R^{2}_{jk}}{R^{2}_c})f_c(R_{ij}) f_c(R_{ik})f_c(R_{jk})$

where Rc is the cutoff distance for the descriptor functions and
$\eta$ and $\zeta$ are redefined parameters for the descriptors. The cutoff function fc can be expressed as

- $\begin{equation}
f_c(R)=\left\{
\begin{aligned}
0.5[1+cos(\dfrac{\pi R}{R_c})] & , & R<R_c \\
0 & , & R>R_c
\end{aligned}
\right.
\end{equation}$

### Neural network model

### Normalization
In machine learning, The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information. The ANN-TFlow provides 2 normaliztion methods for users.

- Max-min

$-1 + 2 \times\dfrac{Fp-min(Fp)}{Max(Fp)-min(Fp)}$

- Mean-Stdev

$\dfrac{Fp-mean(Fp)}{stdev(Fp)}$

### Regularization
In order to avoid overfitting, the elastic net, a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods, is used in the ANN-TFlow training process.

$𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒\{𝑆𝑆𝐸+𝑃\}$

- Elastic Net

$P = \lambda[(1-\alpha) \times 0.5 \displaystyle\sum_{k=1}^{N}W_i^2 + \alpha \displaystyle\sum_{k=1}^{N}|W_i|]$

- Lasso Regression (L1)

$P = \lambda \displaystyle\sum_{k=1}^{N}|W_i|$

- Ridge Regression (L2)

$P = 0.5\lambda \displaystyle\sum_{k=1}^{N}W_i^2$

Release Note
---

### 0.1.0-Beta
- Release date: March 19, 2020

### v 1.0
- Release date: July 30, 2020

### v 2.0
- In progress

gantt
    dateFormat  YYYY-MM
    title Project Timeline 
    axisFormat  %Y-%m
    
    section V2
    Modularization   :done, OO, 2021-06, 2w
    Numba            :after OO, 4w

References
---
1. [Atom-centered symmetry functions for constructing high-dimensional neural network potentials, J. Behler, J. Chem. Phys. 134(7), 074106 (2011)](https://doi.org/10.1063/1.3553717)
2. [Amp: A modular approach to machine learning in atomistic simulations, A. Khorshidi, and A.A. Peterson, Comput. Phys. Commun. 207, 310–324 (2016)](https://doi.org/10.1016/j.cpc.2016.05.010)
3. [Fast and Accurate Artificial Neural Network Potential Model for MAPbI3 Perovskite Materials, Hsin-An Chen and Chun-Wei Pao, ACS Omega 2019 4 (6), 10950-10959 (2019)](https://doi.org/10.1021/acsomega.9b00378)
## Appendix and FAQ

:::info
**Find this document incomplete?** Leave a comment!
:::

###### tags: `AMP` `ML Potential` `TensorFlow`