---
# System prepended metadata

title: Deep learning summarization

---

# Deep learning summarization

![](https://www.reddit.com/r/MachineLearning/comments/mbhewa/d_advanced_takeaways_from_fastai_book/)

I’ve also put the insights into a deck on save all to help you remember them over the long-term. I would massively recommend using a spaced repetition app (video explanation) like anki or save all for the things you learn otherwise you’ll just forget so much of what is important. Here’s the takeaways:
Neural Network Training Fundamentals

1. Always start an ML project by producing simple baselines

+ If is binary classification then could even be as simple as predicting the most common class in the training dataset

+ Other baselines: linear regression, random forest, boosting etc…

    Then you can use your baseline to clean your data by looking at the datapoints it gets most incorrect and checking to see if they are actually classified correctly in the data

2.  In general you can also leverage your baselines to help debug your models

+ e.g. if you make your neural network 1 layer then it should be able to match the performance of a linear regression baseline, if it doesn’t then you have a bug!

+ e.g. if adding a feature improves the performance of linear regression then it should probably also improve the performance of your neural net unless you have a bug!

    Hyperparameter optimisation can help a bit (especially for the learning rate) but in general there are default hyperparameters that can do quite well and so closely optimising the hyperparameters should be one of the last things you try rather than the first

3. If you know something about the problem then try to inject it as an inductive bias into the training process

+ e.g. if some of your features are related in a sequential way then incorporate them into training separately using an RNN

+ e.g. if you know the output should only be between -3 and 3 then use sigmoid to design the final layer so that it forces the output of the network to be in this range

Transfer Learning

1. Always use transfer learning if you can by finding a model pre-trained for a similar task and then fine-tune that model for your particular task

    e.g. see huggingface for help with this in NLP

2. Gradual unfreezing and discriminative learning rates work well when fine-tuning a transfer learned model

3. Gradual unfreezing = freeze earlier layers and train the later layers only, then gradually unfreeze the earlier layers one by one

4. Discriminative learning rates = having different learning rates per layer of your network (usually earlier layers have smaller learning rates than later layers)

Tricks to Deal with Overfitting

1. Best way to deal with overfitting is by getting more data. Exhaust this first before you start regularising with other methods

2. Data augmentation is really powerful and now possible with text as well as images:

- Image data augmentation - crop, pad, squish and resize images

- Text data augmentation - negate words, replace words with similes, perturb word embeddings (nice github repo for this)

3. Mixup regularisation = create new data by averaging together training datapoints

4. Backwards training (NLP only): train an additional separate model that is fed text backwards and then average the outputs of your two models to get your final prediction

Other Tricks to Improve Performance

1. Test time augmentation = at test time, use the average prediction from many augmented versions of the input as your prediction rather than just the prediction from the true input

2. 1 cycle training = when you increase and reduce the learning rate throughout training in a circular fashion (usually makes a huge difference)

3. Learning rate finder algorithm = algorithm that Fast AI provide to help you automatically discover roughly the best learning rate

4. Never use one-hot encodings, use embeddings instead, even in tabular data!

5. Using AdamW instead of Adam can help a little bit

6. Lower precision training can help and on pytorch lightning is just a simple flag you can set

7. For regression problems if you know the output should be within a range then its good to use sigmoid to force the neural net output to be within this range

    I.e. make the network output: min_value + sigmoid(output) * (max_value - min_value)

8. Clustering your features can help you identify which ones are the most redundant and then removing the can help performance

9. Label smoothing = use 0.1 and 0.9 instead of 0 and 1 for label targets (can smoothen training)

10. Don’t dichotomise your data, if your output is continuous then its better to train the network to predict continuous values rather than turning it into a classification problem

11. Progressive resizing = train model on smaller resolution images first, then increase resolution gradually (can speed up training a lot)

12. Strategically using bottleneck layers to force the network to form more compact representations of the data at different points can be helpful

13. Try using skip connections as they can help smooth out the loss surface