Intro to Natural Language Processing

--- tags: NLP, natural language processing, machine learning, deep learning, neural networks --- # Intro to Natural Language Processing (under construction ... see also [Controlled Natural Language](https://hackmd.io/@alexhkurz/ryJmdyP6_), [Attempto](https://hackmd.io/@alexhkurz/S1aIoprpO), [Semantic Parsing](https://hackmd.io/@alexhkurz/rk8spB9hu), [Transformers](https://hackmd.io/@alexhkurz/SJs_B0tlF)) ## References - Lectures and Courses on NLP - Stanford [Natural Language Processing with Deep Learning](http://web.stanford.edu/class/cs224n/). Archived [2019 Version](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/). - Books on NLP - Bird, Klein, and Loper: [Natural Language Processing with Python](https://www.nltk.org/book), 2019. - Jurafsky, Martin: [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/) - Eisenstein: [Natural Language Processing](https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf) - Ch.6: [Video by Noah Smith](https://drive.google.com/file/d/1cK43rSzH491oI9NIrLlDAeP8P2F7LXTJ/view) - Ch.18: Machine Translation. - Deep Learning - Goodfellow, Bengio, Courville: [Deep Learning](https://www.deeplearningbook.org/). - Michael Nielsen: [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) - [Geometric Deep Learning](https://geometricdeeplearning.com) - [Dive into Deep Learning](https://d2l.ai/), (Chapter 14,15) - Google, Machine Learning Foundations (with Python notebooks). - [Ep #8 - Tokenization for Natural Language Processing](https://www.youtube.com/watch?v=f5YJA5mQD5c) - [Ep #9 - Using the Sequencing APIs](https://www.youtube.com/watch?v=L3suP4g8p7U) - [Ep #10 - Using NLP to build a sarcasm classifier](https://www.youtube.com/watch?v=-8XmD2zsFBI) - [https://copilot.github.com/](https://copilot.github.com/) ## Further Articles [Neural Machine Translation: A Review and Survey](https://arxiv.org/pdf/1912.02047.pdf), 2020. [Universal Dependencies: A cross-linguistic typology](https://nlp.stanford.edu/~manning/papers/USD_LREC14_UD_revision.pdf) [Globally Normalized Transition-Based Neural Networks](https://arxiv.org/pdf/1603.06042.pdf) [Anaphora and Coreference Resolution: A Review](https://arxiv.org/pdf/1805.11824.pdf) [Stanford Log-linear Part-Of-Speech Tagger](https://nlp.stanford.edu/software/tagger.shtml) [Watson and Question Answering](https://researcher.watson.ibm.com/researcher/files/us-mike.barborak/WatsonInAIMagReprint.pdf) ... [DeepQA](https://researcher.watson.ibm.com/researcher/view_group_pubs.php?grp=2099&t=1) ... [AllenNLP]() ... [Semantic Parsing](https://guide.allennlp.org/semantic-parsing-seq2seq#1) ... Learning Context-Free Grammars: [Wikipedia](https://en.wikipedia.org/wiki/Grammar_induction) [Yes you should understand backprop](https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b) [`npm install compromise`](https://github.com/spencermountain/compromise/) ## Digressions What is a language? There is a language in the real world. There is language as a probability distribution. And there is langauge as given by data. So we need to think about at least three different notions. [^ontology] [^ontology]: There is a metaphysical problem here. Are the languages in the real world real? Does English exist? Or is "English" just a name without a precise meaning that we use to communicate a variety of vage phenomena? Certainly we are confident that we understand sentences such as "I translated this text from English to German." But that does not mean that I can give a definition of "English", that I can answer the question: What is English?