Protein sequence understanding
===
###### tags: `Paper`、`Database`、`Tools`

# Motivation
NLP Understanding --> Protein sequence understanding
* 宋代李清照《一剪梅·红藕香残玉簟秋》
紅藕香殘玉簟秋。輕解羅裳，獨上蘭舟。雲中誰寄錦書來，雁字回時，月滿西樓。
花自飄零水自流。一種相思，兩處閒愁。此情無計可消除，才下眉頭，卻上心頭。
* [Amino Acids and Protein Sequences](https://www.news-medical.net/life-sciences/Amino-Acids-and-Protein-Sequences.aspx#)
* [Venn diagram for amino acid properties](https://www.google.com/url?sa=i&source=images&cd=&ved=2ahUKEwjz7sLy4MrkAhUhxIsBHTgoDC4QjRx6BAgBEAQ&url=https%3A%2F%2Fwww.researchgate.net%2Ffigure%2FVenn-diagram-for-amino-acid-properties-according-to-Russell_fig1_235324128&psig=AOvVaw0nfTf1Qi9VIKWFn_lOem4Z&ust=1568359561916051)
* [BERT_word vs character]

# NLP History
{%pdf https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf %}

# Journal Paper
* Mean Spectral Normalization of Deep Neural Networks for Embedded Automation
Paper: https://arxiv.org/pdf/19 07.04003v1.pdf
Codes: https://github.com/AntixK/mean-spectral-norm

* "Learning Role-based Graph Embeddings" (IJCAI 2018). Role2vec is a node embedding method which creates typed motif-based node representations for downstream tasks such as node classification, vertex level regression and link prediction.
https://github.com/benedekrozemberczki/role2vec

* Towards algorithmic analytics for large-scale datasets
(Nature Machine Intelligencevolume 1, pages296–306 (2019))

* [Computational and Structural Biotechnology Journal (CSBJ)](https://www.journals.elsevier.com/computational-and-structural-biotechnology-journal)
* [Plosone](https://journals.plos.org/plosone/)
* [Bioinformatics and Biology Insights](https://journals.sagepub.com/home/bbi)
* [BMC Bioinformatics](https://bmcbioinformatics.biomedcentral.com/)

# Database
* [amigo](http://amigo.geneontology.org/amigo)
* Signal Peptide
* Phosphorylation sites

# Methods
* Transformer
* BERT (Bidirectional Encoder Representations from Transformers)
[BERT 科普文](https://leemeng.tw/attack_on_bert_transfer_learning_in_nlp.html?fbclid=IwAR2OI9-B00YAQdw-KaNrGNOTjhQli2JS20nmwWo10LX9s0O2d3MV4rc3Dq0)
BERT used in the pre-training to fine-tune pre-learned word embeddings. This results in a network which can produce different embeddings of same word based on its context. Later, this pre-trained network can be used for task specific training with some additional layers.
BERT is more about sentence vectors than word vectors. In the paper they initialise with random vectors for the word embeddings.
* BERT-DST (Dialogue State Tracking, 2019/07/05)
Here is a really cool variation of BERT, namely BERT-DST
Paper is here: https://arxiv.org/pdf/1907.03040.pdf
Github code: https://github.com/guanlinchao/bert-dst
* ELMO (Embedding from Language MOdel)
* GPT (Generative Pre-Training)
* XLNet [科普文](https://zhuanlan.zhihu.com/p/70257427?fbclid=IwAR3yCMCcXXjgSIcQkua3TBVqGO1JJ92aX6W1JOvxaHZDr0-_H6ZKnbG6NUE)
1. 与Bert采取De-noising Autoencoder方式不同的新的预训练目标：Permutation Language Model(简称PLM)；这个可以理解为在自回归LM模式下，如何采取具体手段，来融入双向语言模型。这个是XLNet在模型角度比较大的贡献，确实也打开了NLP中两阶段模式潮流的一个新思路。
2. 引入了Transformer-XL的主要思路：相对位置编码以及分段RNN机制。实践已经证明这两点对于长文档任务是很有帮助的；
3. 加大增加了预训练阶段使用的数据规模；Bert使用的预训练数据是BooksCorpus和英文Wiki数据，大小13G。XLNet除了使用这些数据外，另外引入了Giga5，ClueWeb以及Common Crawl数据，并排掉了其中的一些低质量数据，大小分别是16G,19G和78G。可以看出，在预训练阶段极大扩充了数据规模，并对质量进行了筛选过滤。这个明显走的是GPT2.0的路线。

# Tools
* [Seaborn](https://seaborn.pydata.org/)

# phosphorylation (磷酸化位點)

# 20 amios

- [iPPI-PseAAC(CGR) note](https://hackmd.io/oFJHh1ofT_mBdqOeSyv50Q?both)
- [Protein sequence understanding note from TONY](https://hackmd.io/CXGYQvwzSOOvH5d2qSTd8g?view)
- [Paper](https://hackmd.io/B6FE-O8CSNK05_cngzupNw)