Try   HackMD

Introduction to PyTorch:

An Intro Guide for High-School Students Preparing for AI Olympiads

PyTorch is a powerful and versatile library for numerical computing and machine learning. It’s built on Python, a language known for its simplicity and vast ecosystem. While AI can be implemented in other programming languages, the overwhelming majority of open-source tools and frameworks—like TensorFlow, Scikit-learn, and PyTorch—are written in Python. This makes Python the de facto standard for artificial intelligence and machine learning.

This note introduces three key aspects of PyTorch: its capabilities for numerical programming, its use of automatic differentiation, and its features for building neural networks and differentiable programs.

1. Numerical Programming with PyTorch

PyTorch as a Numerical Programming Tool

PyTorch is, first and foremost, a numerical programming package, akin to a "calculator app" for Python. It provides tools to perform mathematical operations efficiently and is especially optimized for large-scale computations like those in machine learning. Another example of a numerical programming library is NumPy, but PyTorch goes beyond it by offering features like GPU support and automatic differentiation, which make it more suitable for deep learning.

Tensors and Their Operations

The primary data structure in PyTorch is the tensor, a generalization of scalars, vectors, and matrices. Tensors can have arbitrary dimensions and shapes:

  • Scalars: A single number, shape [].
  • Vectors: A 1D array, shape [n].
  • Matrices: A 2D array, shape [m, n].
  • Higher dimensions: For example, a 3D tensor might have shape [batch_size, height, width].

PyTorch supports a wide range of operations, from elementwise functions (addition, multiplication, exponentiation) to matrix multiplication using the @ operator.

Example:

import torch
x = torch.tensor([[1, 2], [3, 4]])
y = torch.tensor([[5, 6], [7, 8]])
z = x @ y  # Matrix multiplication

Vectorized Code for Efficiency

One of the key principles in PyTorch is vectorization: performing operations directly on entire tensors instead of iterating through their elements. This approach is much faster because PyTorch uses highly optimized C libraries for tensor operations. For example:

# Inefficient Python loop
for i in range(len(a)):
    c[i] = a[i] + b[i]

# Efficient PyTorch operation
c = a + b

CPU, GPU, and TPU

PyTorch supports computations on multiple platforms:

  • CPU: A general-purpose processor with a small number of cores (4–32) optimized for diverse tasks.
  • GPU (Graphics Processing Unit): A highly parallel processor with thousands of cores (e.g., NVIDIA GPUs can have 2,000–10,000 cores). GPUs are optimized for repeated, parallel computations, such as matrix multiplication or elementwise operations. They are designed with memory layouts that ensure data is physically close to the computing cores, minimizing delays.
  • TPU (Tensor Processing Unit): A specialized processor for machine learning, developed by Google, optimized for matrix and tensor computations.

GPUs excel at operations like:

  1. Matrix multiplication: Repeated multiplications and additions in parallel.
  2. Elementwise matrix operations: Applying a function (e.g., exponentiation) to each element of a matrix.

To use a GPU in PyTorch, you move tensors to the GPU with .cuda():

x = x.cuda()  # Move tensor to GPU
y = y.cuda()
z = x @ y     # Perform computation on GPU

To bring tensors back to the CPU, use .cpu():

z = z.cpu()

Experimenting in Google Colab

In Google Colab, you can request a GPU backend by selecting Runtime > Change runtime type > GPU. This allows you to compare the speed of computations on the CPU and GPU. For example, try multiplying two large matrices and observe the difference.

Common Mistakes with GPU

To fully benefit from the GPU, ensure:

  1. All tensors involved in a computation are moved to the GPU.
  2. Avoid frequent transfers between CPU and GPU memory, as this is a common bottleneck.
  3. Optimize GPU throughput by structuring computations to minimize idle cores.

2. Automatic Differentiation in PyTorch

Computational Graphs and Gradients

PyTorch builds a computational graph in the background whenever you perform tensor operations. This graph tracks the sequence of operations and is essential for automatic differentiation, a method for computing gradients (derivatives). Gradients are critical for training machine learning models, as they indicate how to adjust model parameters to reduce error.

PyTorch uses reverse-mode automatic differentiation, commonly referred to as backpropagation, to calculate gradients efficiently. Backpropagation uses the chain rule of calculus and is akin to dynamic programming, where intermediate results are reused.

For more details, see this note on automatic differentiation.

Using Gradients in PyTorch

  1. Enabling Gradients: To enable gradient tracking, set requires_grad=True when creating a tensor: 
    ​​​x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
  2. Backpropagation: Call .backward() to compute gradients: 
    ​​​y = x ** 2
​​​y.sum().backward()  # Computes dy/dx for each x
​​​print(x.grad)       # Prints gradients
  3. Disabling Gradients: Use torch.no_grad for inference to save memory and computation: 
    ​​​with torch.no_grad():
​​​    y = model(x)
  4. Graph Management: After .backward(), the computational graph is destroyed by default. Use .detach() to keep a tensor without tracking its history: 
    ​​​detached_x = x.detach()

3. Building Neural Networks with PyTorch

Neural Networks and Differentiable Programming

PyTorch simplifies the creation of neural networks, which are composed of layers performing differentiable computations. These layers are combined to process inputs, transform data, and generate outputs.

Here’s a simple example of a multi-layer perceptron (MLP):

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 50),  # Input size 10, output size 50
    nn.ReLU(),          # Activation function
    nn.Linear(50, 1)    # Output size 1
)

Loss Functions and Optimization

  1. Loss Functions: Quantify the difference between predictions and true values (e.g., mean squared error or cross-entropy loss).
  2. Optimization Algorithms: Use gradients to adjust model parameters, e.g., stochastic gradient descent (SGD) or Adam.

Example:

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Training loop (simplified)
for data, target in dataloader:
    optimizer.zero_grad()
    predictions = model(data)
    loss = loss_fn(predictions, target)
    loss.backward()
    optimizer.step()

Additional Utilities

PyTorch provides tools to streamline workflows:

  • Dataset and Dataloader Classes: Simplify data handling and batching.
  • Transformations: Facilitate data preprocessing (e.g., normalization, augmentation).

For more advanced details, explore the Elements of Differentiable Programming.

PyTorch is a flexible and efficient framework for numerical programming, automatic differentiation, and building neural networks. With these tools, you can create powerful machine learning models and explore the exciting world of AI.

Last changed by 
 
fhuszar
0
73

Read more

DeepNN Notes on Inductive Biases of Neural Architectures

Lecture notes on general algorithms for calculating Jacobians and gradients, to motivate the use of backpropagation in deep learning.

Feb 6, 2025
DeepNN Notes on The Recent History of Deep Learning

Timeline 1: Breakthroughs in Applications2012 - AlexNet/ImageNet Challenge: AlexNet’s victory in the ImageNet competition marked the resurgence of deep learning, demonstrating the power of CNNs for computer vision.2015 - DQN-Atari: Deep Q-Networks (DQN) achieved human-level performance on Atari games, showcasing the potential of deep reinforcement learning.2015 - Neural Machine Translation (NMT): Google Translate adopted neural machine translation with sequence-to-sequence models and attention, significantly improving translation quality.2015 - Image-to-text captioning: Models like Show, Attend and Tell combined CNNs and RNNs to generate accurate image captions, signaling the rise of multimodal AI.2016 - WaveNet: DeepMind introduced WaveNet, revolutionizing text-to-speech synthesis with natural-sounding, human-like audio.2016 - AlphaGo: DeepMind’s AlphaGo defeated the world champion Go player, demonstrating the potential of reinforcement learning for strategic games.2017 - OpenAI Dota 2 (OpenAI Five): Reinforcement learning models defeated professional players in Dota 2, a complex multiplayer online battle arena game.2019 - AlphaStar (StarCraft II): DeepMind’s AlphaStar reached Grandmaster level in StarCraft II, showcasing AI’s ability to handle real-time strategy games with long time horizons.2019 - GPT-2: OpenAI’s GPT-2 demonstrated the potential of large-scale language models, generating coherent and contextually relevant text.2020 - AlphaFold: DeepMind’s AlphaFold solved the protein folding problem, a decades-old challenge in molecular biology.2021 - AlphaCode/Codex/Copilot (LLMs for text and code): Models like Codex powered applications like GitHub Copilot, revolutionizing programming by enabling natural language coding.2022 - Minerva/LLEMMA/AlphaProof/AlphaGeometry: Advanced models tackled complex mathematical reasoning, formal proofs, and problem-solving in geometry.Timeline 2: MethodsFoundational Architectures (Vision and General Methods):1998 - Basic CNNs: Yann LeCun’s work on convolutional neural networks (CNNs) for handwritten digit recognition (LeNet) established the foundation for modern vision models.2012 - AlexNet: Revived deep learning by scaling up CNNs and using GPUs to win the ImageNet challenge.2014 - VGG: Introduced a very deep architecture with smaller convolutional filters, setting a benchmark for simple, deep CNN designs.2015 - ResNet: Introduced residual connections, enabling the training of very deep networks and surpassing human-level performance in image recognition.2020 - Vision Transformers (ViT): Applied Transformer architectures to vision tasks, achieving state-of-the-art performance by leveraging self-attention mechanisms.Generative Models:2013 - Variational Autoencoders (VAE): Provided a probabilistic framework for generating data while learning latent representations.2014 - Generative Adversarial Networks (GANs): Introduced by Ian Goodfellow, GANs became the foundation for realistic image synthesis and adversarial training.2021 - Diffusion Models: Models like DDPM (Denoising Diffusion Probabilistic Models) enabled state-of-the-art image and audio generation, powering tools like Stable Diffusion and DALL-E 2.Text Models:1997 - LSTM: Long Short-Term Memory networks addressed the vanishing gradient problem, enabling RNNs to capture long-term dependencies.2017 - Attention: The Attention mechanism (introduced in the "Attention is All You Need" paper) allowed models to focus on specific parts of input sequences, revolutionizing sequence modeling.2017 - Transformer: Introduced as a self-attention-based architecture, it became the backbone of nearly all modern NLP and multimodal AI systems.2022 - RLHF (Reinforcement Learning from Human Feedback): Enabled fine-tuning of language models like GPT-3.5 and ChatGPT to align with human intent.2023 - RLAIF (Reinforcement Learning from AI Feedback): Improved alignment efficiency by leveraging model-generated feedback for fine-tuning large language models.Timeline 3: Responsible AI and AI Safety2014 - Adversarial Examples: Szegedy et al. discovered adversarial examples, showing that small, imperceptible changes to input data could drastically alter model predictions, raising concerns about robustness.2016 - Fairness - Equality of Opportunity: The "Equality of Opportunity in Supervised Learning" paper by Hardt et al. introduced algorithmic fairness by ensuring equalized error rates across groups.2016 - Differentially Private SGD: Abadi et al. introduced techniques for training deep learning models with differential privacy, balancing privacy guarantees with utility for sensitive datasets.2016 - GradCAM: Gradient-weighted Class Activation Mapping provided a visual explanation of deep learning models by highlighting important regions in input data.2018 - Gender Shades: Joy Buolamwini and Timnit Gebru’s study revealed biases in commercial facial recognition systems, catalyzing fairness research.2018–2023 - Mechanistic Interpretability: Pioneered by Anthropic’s interpretability research and OpenAI’s circuits work, aiming to reverse-engineer neural network reasoning by analyzing weights and activations.2022 - Constitutional AI: Anthropic introduced Constitutional AI to align language models with human values through rule-based principles, avoiding harmful behaviors without direct human feedback.2023 - Safety Benchmarks (e.g., HELM): The Holistic Evaluation of Language Models formalized evaluation protocols for fairness, bias, and alignment.Timeline 4: Theory of Deep Learning2017 - Understanding Deep Learning Requires Rethinking Generalization: Zhang et al. showed that deep networks can memorize random labels yet still generalize well, challenging traditional theories.2018 - Neural Tangent Kernel (NTK): Jacot et al. showed that deep networks in the infinite-width limit behave like kernel methods, providing insights into gradient descent dynamics.2018 - Deep Linear Models: Saxe et al. studied deep linear networks, simplifying analysis while retaining key properties of deep models.2018–2020 - SGD Implicit Bias: Research showed that stochastic gradient descent (SGD) biases models toward solutions with favorable generalization properties.2019 - Double Descent: Revealed that test error decreases as model capacity grows beyond a critical point, challenging the bias-variance trade-off.2019 - Lottery Ticket Hypothesis: Frankle and Carbin found that large networks contain sparse sub-networks ("winning tickets") that match the performance of the full model.2020 - Scaling Laws: OpenAI and DeepMind demonstrated how model performance scales predictably with size, data, and compute.2021 - Recursive Training: Models can generate high-quality data for self-training, creating a virtuous cycle of performance improvement.2021 - Grokking: Power et al. observed that models sometimes generalize long after fitting the training data, highlighting the interplay of optimization and regularization.2022 - Transcendence (Beyond Memorization): A recent paper explored neural networks’ ability to extrapolate rules beyond training distributions.2022 - Theory of In-context Learning in LLMs: Explored how large language models encode and utilize training distribution statistics to perform implicit meta-learning.2023 - Rule Extrapolation in LLMs: Research formalized how LLMs generalize beyond seen examples, connecting this to abstraction and scaling laws.

Jan 30, 2025
abc conjecture with rationals

My colleague Challenger is working on the abc conjecture. This is one of those problems that are relatively easy to state, but which turn out to be very complicated mathematically.

Dec 12, 2024
AI Overview Session Plan

I usually start with asking the audience to volunteer definitions, thoughts on AI. I ask them to tell me What AI is. I wait a few minutes then usually a few participants will start saying ideas. It's important to remind them that there is no good answer, and they can say something moderately silly. If there's no volunteers or very few, I sometimes prompt the audience (anyone thought of robots?)

Dec 10, 2024
Read more from fhuszar
Published on HackMD