# **Top 140 PyTorch Interview Questions and Answers**
This comprehensive guide covers essential PyTorch interview questions across multiple categories, with detailed explanations for each.these 140 carefully curated questions represent the most important concepts you'll encounter in PyTorch interviews.
---
## **Table of Contents**
1. [PyTorch Fundamentals](#pytorch-fundamentals)
2. [Tensor Operations](#tensor-operations)
3. [Autograd System](#autograd-system)
4. [Neural Network Building](#neural-network-building)
5. [Model Training and Optimization](#model-training-and-optimization)
6. [Computer Vision with PyTorch](#computer-vision-with-pytorch)
7. [NLP with PyTorch](#nlp-with-pytorch)
8. [Advanced PyTorch Features](#advanced-pytorch-features)
9. [Performance Optimization](#performance-optimization)
10. [PyTorch Ecosystem](#pytorch-ecosystem)
11. [Debugging and Troubleshooting](#debugging-and-troubleshooting)
12. [PyTorch vs Other Frameworks](#pytorch-vs-other-frameworks)
13. [Deployment and Productionization](#deployment-and-productionization)
14. [Recent Developments](#recent-developments)
---
## **PyTorch Fundamentals**
### **1. What is PyTorch and how does it differ from other deep learning frameworks?**
PyTorch is an open-source deep learning framework developed by Facebook's AI Research lab. Key differences from other frameworks:
- **Dynamic computation graphs** (define-by-run) vs TensorFlow's static graphs (define-and-run)
- **Pythonic interface** that feels more natural to Python developers
- **Strong GPU acceleration** support through CUDA
- **Tighter integration** with the Python ecosystem
- **Easier debugging** due to immediate execution model
- **Research-focused** with rapid adoption in academic papers
### **2. Explain the difference between eager execution and graph execution in PyTorch**
**Eager execution** (default in PyTorch):
- Operations execute immediately
- Code executes in the order written
- Easy to debug with standard Python tools
- More intuitive for Python developers
- Example: `x = torch.tensor([1, 2, 3]); y = x + 2`
**Graph execution** (via `torch.compile` or TorchScript):
- Builds a computational graph first
- Optimizes the graph before execution
- Better performance for production deployment
- Similar to TensorFlow's graph mode
- Example: `model = torch.compile(model)`
### **3. What are the main components of the PyTorch ecosystem?**
1. **PyTorch Core**: Tensor computation with strong GPU acceleration
2. **TorchScript**: For serializing and optimizing models
3. **TorchVision**: Computer vision models, datasets, and transforms
4. **TorchText**: Natural language processing tools and datasets
5. **TorchAudio**: Audio processing tools and datasets
6. **TorchServe**: Model serving library
7. **TorchElastic**: Distributed training with fault tolerance
8. **PyTorch Lightning**: High-level interface for cleaner code
9. **Captum**: Model interpretability and understanding
10. **TorchMetrics**: Collection of machine learning metrics
### **4. How does PyTorch handle memory management compared to NumPy?**
PyTorch:
- Uses a caching memory allocator for GPU tensors
- Reuses memory blocks to reduce fragmentation
- Has better GPU memory management
- Provides tools like `torch.cuda.empty_cache()`
- Allows pinned memory for faster CPU-GPU transfers
- Has gradient memory that needs to be managed
NumPy:
- Uses standard Python memory allocation
- No special memory management for accelerators
- Simpler memory model but less optimized for deep learning
### **5. What is the difference between `torch.Tensor` and `torch.autograd.Variable`?**
In older versions of PyTorch (pre-0.4), `Variable` was a wrapper around `Tensor` that supported autograd. Since PyTorch 0.4, `Tensor` and `Variable` have been merged:
- **Current approach**: All tensors can track computation history if `requires_grad=True`
- **Simplified API**: No need to wrap tensors in Variables
- **Backward compatibility**: `Variable` still exists but is deprecated
```python
# Current approach (PyTorch 1.0+)
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
y.backward(torch.tensor([1.0, 1.0, 1.0]))
```
### **6. Explain the difference between `torch.tensor()` and `torch.Tensor()`**
- **`torch.tensor(data)`**: Creates a tensor from data with inference of data type
- Example: `torch.tensor([1, 2, 3])` creates a float tensor
- **`torch.Tensor(data)`**: An alias for `torch.FloatTensor(data)` (always creates float)
- Example: `torch.Tensor([1, 2, 3])` creates a float tensor even if data is integer
Best practice is to use `torch.tensor()` as it's more explicit about data types.
### **7. What is the purpose of the `requires_grad` parameter in PyTorch tensors?**
`requires_grad` determines whether PyTorch should track operations on a tensor for automatic differentiation:
- When `True`: PyTorch builds a computational graph to compute gradients
- When `False`: No gradient tracking, saving memory and computation
- Default: `False` for tensors created with `torch.tensor()`, `True` for model parameters
```python
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
y.backward(torch.tensor([1.0, 1.0, 1.0]))
print(x.grad) # tensor([2., 2., 2.])
```
### **8. How do you move tensors between CPU and GPU in PyTorch?**
```python
# Create tensor on CPU
x = torch.tensor([1, 2, 3])
# Move to GPU (if available)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = x.to(device)
# Or using specific methods
x = x.cuda() # Move to GPU
x = x.cpu() # Move back to CPU
# Create directly on device
x = torch.tensor([1, 2, 3], device=device)
```
### **9. What is the difference between `detach()` and `detach().clone()`?**
- **`tensor.detach()`**: Creates a tensor that shares storage with the original but doesn't require gradients
- Changes to the detached tensor affect the original
- Example: `y = x.detach(); y[0] = 5` also changes `x[0]`
- **`tensor.detach().clone()`**: Creates a complete copy that doesn't share storage
- Changes to the cloned tensor don't affect the original
- Example: `y = x.detach().clone(); y[0] = 5` doesn't change `x[0]`
### **10. Explain the difference between `torch.no_grad()`, `torch.inference_mode()`, and `torch.set_grad_enabled()`**
- **`torch.no_grad()`**: Context manager that disables gradient calculation
- Reduces memory consumption and speeds up computations
- Used during inference/validation
```python
with torch.no_grad():
output = model(input)
```
- **`torch.inference_mode()`**: More efficient version of `no_grad()` introduced in PyTorch 1.9
- Even better performance for inference
- Disables view tracking and version counter updates
```python
with torch.inference_mode():
output = model(input)
```
- **`torch.set_grad_enabled(mode)`**: Conditionally enables/disables gradient calculation
```python
# Equivalent to no_grad() when mode=False
with torch.set_grad_enabled(train_mode):
output = model(input)
```
---
## **Tensor Operations**
### **11. How do you create a tensor of zeros with shape (3, 4) in PyTorch?**
```python
# Method 1
zeros = torch.zeros(3, 4)
# Method 2
zeros = torch.zeros((3, 4))
# Method 3
zeros = torch.zeros(3, 4, dtype=torch.float32, device='cpu')
```
### **12. What is the difference between `view()`, `reshape()`, and `resize_()`?**
- **`view()`**: Creates a new tensor with the same data but different shape
- Only works if the tensor is contiguous in memory
- Doesn't copy data (returns a view)
- Example: `x.view(2, 3)`
- **`reshape()`**: Similar to view but will copy data if needed to make it contiguous
- More flexible than view
- Example: `x.reshape(2, 3)`
- **`resize_()`**: Changes the shape of the tensor in-place
- Can change the total number of elements
- May lose data if new size is smaller
- Example: `x.resize_(2, 3)`
### **13. How do you perform matrix multiplication in PyTorch?**
Multiple ways to perform matrix multiplication:
```python
a = torch.randn(3, 4)
b = torch.randn(4, 5)
# Method 1: torch.mm (for 2D matrices only)
c = torch.mm(a, b)
# Method 2: torch.matmul (more general)
c = torch.matmul(a, b)
# Method 3: @ operator (Python 3.5+)
c = a @ b
# Method 4: tensor method
c = a.matmul(b)
```
### **14. What is the difference between `torch.sum()` and `tensor.sum()`?**
No functional difference - they are equivalent:
```python
x = torch.tensor([1, 2, 3])
# Both are identical
result1 = torch.sum(x)
result2 = x.sum()
# Can also specify dimension
result3 = torch.sum(x, dim=0)
result4 = x.sum(dim=0)
```
### **15. How do you concatenate tensors along a specific dimension?**
```python
x = torch.tensor([[1, 2], [3, 4]])
y = torch.tensor([[5, 6], [7, 8]])
# Concatenate along rows (dimension 0)
z0 = torch.cat((x, y), dim=0)
# tensor([[1, 2],
# [3, 4],
# [5, 6],
# [7, 8]])
# Concatenate along columns (dimension 1)
z1 = torch.cat((x, y), dim=1)
# tensor([[1, 2, 5, 6],
# [3, 4, 7, 8]])
```
### **16. Explain how broadcasting works in PyTorch with an example**
Broadcasting allows operations between tensors of different shapes by "stretching" smaller tensors.
Rules:
1. Dimensions are aligned from right to left
2. Dimensions are compatible if equal or one of them is 1
3. Missing dimensions are treated as 1
Example:
```python
x = torch.randn(4, 1, 5) # Shape: (4, 1, 5)
y = torch.randn(3, 1) # Shape: (3, 1)
# Broadcasting rules:
# x: (4, 1, 5)
# y: (1, 3, 1) [after prepending 1 and aligning right]
# Result: (4, 3, 5)
z = x + y # Works due to broadcasting
```
### **17. How do you create a diagonal matrix from a vector in PyTorch?**
```python
# Method 1: torch.diag
vector = torch.tensor([1, 2, 3])
diagonal_matrix = torch.diag(vector)
# tensor([[1, 0, 0],
# [0, 2, 0],
# [0, 0, 3]])
# Method 2: Using eye and multiplication
diagonal_matrix = torch.eye(3) * vector
# Method 3: For creating a batch of diagonal matrices
vectors = torch.tensor([[1, 2, 3], [4, 5, 6]])
diagonal_matrices = torch.diag_embed(vectors)
```
### **18. What is the difference between `squeeze()` and `unsqueeze()`?**
- **`squeeze()`**: Removes dimensions of size 1
```python
x = torch.zeros(2, 1, 3)
y = x.squeeze() # Shape: (2, 3)
y = x.squeeze(1) # Shape: (2, 3) - only removes dim 1
```
- **`unsqueeze()`**: Adds a dimension of size 1
```python
x = torch.zeros(2, 3)
y = x.unsqueeze(0) # Shape: (1, 2, 3)
y = x.unsqueeze(1) # Shape: (2, 1, 3)
```
### **19. How do you index and slice tensors in PyTorch?**
Similar to NumPy:
```python
x = torch.arange(9).view(3, 3)
# tensor([[0, 1, 2],
# [3, 4, 5],
# [6, 7, 8]])
# Basic indexing
print(x[1, 2]) # tensor(5)
# Slicing
print(x[0:2, 1:3]) # tensor([[1, 2], [4, 5]])
# Boolean masking
mask = x > 4
print(x[mask]) # tensor([5, 6, 7, 8])
# Fancy indexing
indices = torch.tensor([0, 2])
print(x[indices, indices]) # tensor([0, 8])
```
### **20. How do you compute the softmax function across a specific dimension?**
```python
x = torch.randn(2, 3)
# Using torch.softmax
y = torch.softmax(x, dim=1)
# Manual implementation
exp_x = torch.exp(x)
y_manual = exp_x / torch.sum(exp_x, dim=1, keepdim=True)
# Using F.softmax from nn.functional
import torch.nn.functional as F
y_func = F.softmax(x, dim=1)
```
---
## **Autograd System**
### **21. Explain how PyTorch's autograd system works**
PyTorch's autograd:
- Builds a dynamic computation graph on-the-fly
- Tracks all operations on tensors with `requires_grad=True`
- Computes gradients using reverse-mode automatic differentiation
- Stores gradients in the `.grad` attribute of tensors
- Uses the chain rule to compute gradients efficiently
When you call `.backward()`:
1. Computes gradients of the output with respect to all inputs
2. Accumulates gradients in the `.grad` attribute
3. Frees the computation graph (unless `retain_graph=True`)
### **22. What is the purpose of the `backward()` function in PyTorch?**
The `backward()` function:
- Computes gradients of the current tensor with respect to graph leaves
- Is typically called on a scalar loss value
- Populates the `.grad` attribute of all tensors with `requires_grad=True`
- Implements backpropagation through the computation graph
```python
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
z = y.sum()
# Compute gradients
z.backward()
# Access gradients
print(x.grad) # tensor([2., 2., 2.])
```
### **23. Why do we need to zero the gradients before calling `backward()`?**
Gradients are accumulated by default (not overwritten), so if you don't zero them:
- Gradients from previous iterations accumulate
- Leads to incorrect gradient values
- Causes the optimizer to take incorrect steps
```python
# Typical training loop
optimizer.zero_grad() # Zero the gradients
loss.backward() # Compute gradients
optimizer.step() # Update parameters
```
### **24. What is the difference between `detach()` and `stop_gradient`?**
- **`detach()`**: Creates a tensor that shares storage but doesn't require gradients
- Returns a new tensor that doesn't track history
- Example: `y = x.detach()`
- **`torch.no_grad()` context manager**: Temporarily disables gradient calculation
- More efficient for multiple operations
- Example:
```python
with torch.no_grad():
y = x * 2
```
PyTorch doesn't have a direct `stop_gradient` like TensorFlow, but `detach()` serves a similar purpose.
### **25. How do you compute gradients for non-scalar outputs?**
For non-scalar outputs, you need to provide a gradient tensor of the same shape:
```python
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
# For non-scalar outputs, provide gradient weights
grad_tensor = torch.tensor([0.1, 1.0, 0.01])
y.backward(gradient=grad_tensor)
print(x.grad) # tensor([0.2, 2.0, 0.02])
```
### **26. What is the purpose of `retain_graph` in the `backward()` function?**
`retain_graph`:
- When `True`, keeps the computation graph after backward pass
- Needed when you want to call `backward()` multiple times on the same graph
- Uses more memory as the graph isn't freed
- Default is `False` (graph is freed after backward)
```python
# Example where we need to compute multiple backward passes
loss1 = compute_loss1(model)
loss1.backward(retain_graph=True)
loss2 = compute_loss2(model)
loss2.backward()
```
### **27. How do you create a custom autograd function in PyTorch?**
By subclassing `torch.autograd.Function`:
```python
class CustomFunction(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
# Save tensors for backward pass
ctx.save_for_backward(input)
# Compute forward pass
return input * 2
@staticmethod
def backward(ctx, grad_output):
# Retrieve saved tensors
input, = ctx.saved_tensors
# Compute gradients
grad_input = grad_output * 2
return grad_input
# Usage
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = CustomFunction.apply(x)
```
### **28. What is the difference between `torch.autograd.grad()` and `tensor.backward()`?**
- **`tensor.backward()`**:
- Called on a scalar tensor
- Populates `.grad` attribute of input tensors
- Implicitly computes gradients of the tensor w.r.t. all inputs
- Example: `loss.backward()`
- **`torch.autograd.grad()`**:
- Computes and returns gradients explicitly
- Can compute gradients for non-scalar outputs
- Doesn't modify tensors, just returns gradients
- Example: `grads = torch.autograd.grad(loss, parameters)`
### **29. How do you compute second-order derivatives (Hessian) in PyTorch?**
```python
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x[0]**2 + x[1]**3
# First derivatives
grads = torch.autograd.grad(y, x, create_graph=True)[0]
# Second derivatives (Hessian)
hessian = []
for i in range(len(x)):
hessian.append(torch.autograd.grad(grads[i], x, retain_graph=True)[0])
hessian = torch.stack(hessian)
```
### **30. What is the purpose of `create_graph` in `torch.autograd.grad()`?**
`create_graph`:
- When `True`, creates a graph of the derivative operations
- Needed when you want to compute higher-order derivatives
- Allows taking gradients of gradients
- Uses more memory as it keeps the computation graph
- Default is `False`
```python
# For computing second derivatives
first_derivative = torch.autograd.grad(loss, parameters, create_graph=True)
second_derivative = torch.autograd.grad(first_derivative, parameters)
```
---
## **Neural Network Building**
### **31. How do you define a custom neural network in PyTorch?**
By subclassing `nn.Module`:
```python
import torch.nn as nn
import torch.nn.functional as F
class MyNetwork(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(MyNetwork, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.fc2 = nn.Linear(hidden_size, output_size)
self.relu = nn.ReLU()
def forward(self, x):
x = self.relu(self.fc1(x))
x = self.fc2(x)
return x
# Usage
model = MyNetwork(784, 256, 10)
```
### **32. What is the difference between `nn.Module` and `nn.Sequential`?**
- **`nn.Module`**:
- Base class for all neural network modules
- Requires defining `__init__` and `forward` methods
- More flexible for complex architectures
- Allows custom forward passes and control flow
- **`nn.Sequential`**:
- Container for sequential models
- Automatically defines `forward` as sequence of modules
- Simpler syntax for straightforward architectures
- Less flexible for complex models
```python
# Sequential approach
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10)
)
# Module approach (more flexible)
class FlexibleModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.fc2 = nn.Linear(256, 10)
def forward(self, x):
# Can include custom logic
if x.mean() > 0:
x = F.relu(self.fc1(x))
else:
x = torch.tanh(self.fc1(x))
return self.fc2(x)
```
### **33. How do you access and modify model parameters in PyTorch?**
```python
model = MyNetwork(784, 256, 10)
# Access parameters
for param in model.parameters():
print(param.shape)
# Access named parameters
for name, param in model.named_parameters():
print(f"{name}: {param.shape}")
# Modify specific parameter
with torch.no_grad():
model.fc1.weight[0, 0] = 1.0
# Freeze specific layer
for param in model.fc1.parameters():
param.requires_grad = False
```
### **34. What is the purpose of `nn.Parameter` in PyTorch?**
`nn.Parameter`:
- Is a subclass of `torch.Tensor`
- Automatically registered as a parameter when assigned to a module
- Included in `model.parameters()` and `model.named_parameters()`
- Used for trainable weights in neural networks
```python
class CustomLayer(nn.Module):
def __init__(self, input_dim, output_dim):
super().__init__()
# This is automatically registered as a parameter
self.weight = nn.Parameter(torch.randn(input_dim, output_dim))
self.bias = nn.Parameter(torch.zeros(output_dim))
```
### **35. How do you implement a custom layer in PyTorch?**
By subclassing `nn.Module`:
```python
class CustomLayer(nn.Module):
def __init__(self, input_dim, output_dim):
super(CustomLayer, self).__init__()
self.weight = nn.Parameter(torch.randn(input_dim, output_dim))
self.bias = nn.Parameter(torch.zeros(output_dim))
def forward(self, x):
return torch.mm(x, self.weight) + self.bias
# Usage in a model
class MyModel(nn.Module):
def __init__(self):
super().__init__()
self.custom_layer = CustomLayer(10, 5)
def forward(self, x):
return self.custom_layer(x)
```
### **36. What is the difference between `F.relu()` and `nn.ReLU()`?**
- **`nn.ReLU()`**:
- A module that can be added to `nn.Sequential`
- Has state (can be registered in a model)
- Example: `self.relu = nn.ReLU()`
- **`F.relu()`**:
- A functional version (no state)
- Used directly in forward pass
- More memory efficient for one-time use
- Example: `x = F.relu(x)`
Best practice:
- Use `nn.ReLU` when defining layers in `__init__`
- Use `F.relu` for one-time operations in forward
### **37. How do you implement a residual connection in PyTorch?**
```python
class ResidualBlock(nn.Module):
def __init__(self, channels):
super(ResidualBlock, self).__init__()
self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.bn1 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
residual = x
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out += residual # Residual connection
return self.relu(out)
```
### **38. How do you handle variable-length sequences in RNNs?**
Using `pack_padded_sequence` and `pad_packed_sequence`:
```python
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
# Sort sequences by length (descending)
lengths = [5, 4, 3, 2, 1]
seq_tensor = torch.randn(5, 10, 20) # (batch, seq, features)
# Pack sequences
packed = pack_padded_sequence(seq_tensor, lengths, batch_first=True, enforce_sorted=False)
# Process through RNN
rnn = nn.LSTM(20, 30, batch_first=True)
output, (hn, cn) = rnn(packed)
# Unpack sequences
output, lengths = pad_packed_sequence(output, batch_first=True)
```
### **39. How do you implement a custom loss function in PyTorch?**
By subclassing `nn.Module` or using a function:
```python
# As a function
def custom_loss(output, target):
loss = torch.mean((output - target) ** 2)
return loss
# As a module (better for complex losses)
class CustomLoss(nn.Module):
def __init__(self, alpha=0.5):
super(CustomLoss, self).__init__()
self.alpha = alpha
def forward(self, output, target):
mse = F.mse_loss(output, target)
l1 = F.l1_loss(output, target)
return self.alpha * mse + (1 - self.alpha) * l1
# Usage
criterion = CustomLoss(alpha=0.7)
loss = criterion(predictions, targets)
```
### **40. What is the difference between `nn.CrossEntropyLoss` and `nn.NLLLoss`?**
- **`nn.CrossEntropyLoss`**:
- Combines `LogSoftmax` and `NLLLoss` in one
- Expects raw scores (logits) as input
- Example: `loss = nn.CrossEntropyLoss()(logits, targets)`
- **`nn.NLLLoss`**:
- Expects log probabilities as input
- Typically used after `LogSoftmax`
- Example:
```python
log_probs = F.log_softmax(logits, dim=1)
loss = nn.NLLLoss()(log_probs, targets)
```
---
## **Model Training and Optimization**
### **41. What is the typical training loop structure in PyTorch?**
```python
model = MyModel()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(num_epochs):
for inputs, labels in train_loader:
# Zero gradients
optimizer.zero_grad()
# Forward pass
outputs = model(inputs)
loss = criterion(outputs, labels)
# Backward pass
loss.backward()
# Update parameters
optimizer.step()
```
### **42. How do you implement learning rate scheduling in PyTorch?**
Using `torch.optim.lr_scheduler`:
```python
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100):
train(...)
scheduler.step() # Update learning rate
```
Common schedulers:
- `StepLR`: Decay LR by gamma every step_size epochs
- `MultiStepLR`: Decay LR at specific milestones
- `ReduceLROnPlateau`: Reduce LR when metric plateaus
- `CosineAnnealingLR`: Cosine annealing schedule
### **43. How do you save and load a trained model in PyTorch?**
```python
# Save entire model
torch.save(model, 'model.pth')
# Save model state dict (recommended)
torch.save(model.state_dict(), 'model_weights.pth')
# Load model
model = MyModel()
model.load_state_dict(torch.load('model_weights.pth'))
model.eval() # Set to evaluation mode
# Save checkpoint with optimizer state
torch.save({
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': loss,
}, 'checkpoint.pth')
# Load checkpoint
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
```
### **44. What is the purpose of `model.train()` and `model.eval()`?**
- **`model.train()`**:
- Sets the model to training mode
- Enables layers like Dropout and BatchNorm to behave as during training
- Example: Dropout randomly zeros activations
- **`model.eval()`**:
- Sets the model to evaluation mode
- Disables layers like Dropout
- Sets BatchNorm to use population statistics
- Should be used during validation and testing
```python
# Training
model.train()
train_loss = train_one_epoch(model, train_loader)
# Validation
model.eval()
with torch.no_grad():
val_loss = validate(model, val_loader)
```
### **45. How do you handle imbalanced datasets in classification tasks?**
Several approaches:
1. **Class weights**:
```python
class_weights = torch.tensor([0.1, 0.3, 0.6])
criterion = nn.CrossEntropyLoss(weight=class_weights)
```
2. **Oversampling**:
```python
from torch.utils.data import WeightedRandomSampler
sampler = WeightedRandomSampler(weights, len(dataset))
loader = DataLoader(dataset, sampler=sampler)
```
3. **Focal loss** (reduces weight of well-classified examples):
```python
class FocalLoss(nn.Module):
def __init__(self, gamma=2, alpha=None):
super().__init__()
self.gamma = gamma
self.alpha = alpha
def forward(self, inputs, targets):
CE_loss = F.cross_entropy(inputs, targets, reduction='none')
pt = torch.exp(-CE_loss)
focal_loss = (1 - pt) ** self.gamma * CE_loss
return focal_loss.mean()
```
### **46. How do you implement early stopping in PyTorch?**
```python
best_val_loss = float('inf')
patience = 5
counter = 0
for epoch in range(num_epochs):
# Training and validation
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), 'best_model.pth')
counter = 0
else:
counter += 1
if counter >= patience:
print(f"Early stopping after {epoch} epochs")
break
```
### **47. What is gradient clipping and how do you implement it in PyTorch?**
Gradient clipping prevents exploding gradients by limiting the norm of gradients:
```python
# During training loop
optimizer.zero_grad()
loss.backward()
# Clip gradients before optimizer step
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
```
Common methods:
- `clip_grad_norm_`: Clips by norm
- `clip_grad_value_`: Clips by value
### **48. How do you implement transfer learning in PyTorch?**
```python
# Load pretrained model
model = torchvision.models.resnet18(pretrained=True)
# Freeze all layers
for param in model.parameters():
param.requires_grad = False
# Replace the classifier
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, num_classes)
# Train only the classifier
optimizer = torch.optim.SGD(model.fc.parameters(), lr=0.001)
# Or unfreeze some layers
for param in model.layer4.parameters():
param.requires_grad = True
```
### **49. How do you implement k-fold cross-validation in PyTorch?**
```python
from sklearn.model_selection import KFold
k_folds = 5
kfold = KFold(n_splits=k_folds, shuffle=True)
for fold, (train_ids, test_ids) in enumerate(kfold.split(dataset)):
# Create data subsets
train_subsampler = torch.utils.data.SubsetRandomSampler(train_ids)
test_subsampler = torch.utils.data.SubsetRandomSampler(test_ids)
# Create data loaders
train_loader = DataLoader(dataset, batch_size=32, sampler=train_subsampler)
test_loader = DataLoader(dataset, batch_size=32, sampler=test_subsampler)
# Train model
model = MyModel()
train(model, train_loader)
# Evaluate
accuracy = evaluate(model, test_loader)
print(f'Fold {fold} Accuracy: {accuracy}')
```
### **50. How do you implement mixed precision training in PyTorch?**
Using `torch.cuda.amp`:
```python
from torch.cuda.amp import autocast, GradScaler
model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters())
scaler = GradScaler()
for inputs, targets in train_loader:
optimizer.zero_grad()
# Runs the forward pass with autocasting
with autocast():
outputs = model(inputs)
loss = criterion(outputs, targets)
# Scales loss and calls backward()
scaler.scale(loss).backward()
# Unscales gradients and calls optimizer step
scaler.step(optimizer)
# Updates the scale for next iteration
scaler.update()
```
---
## **Computer Vision with PyTorch**
### **51. How do you load and transform images using TorchVision?**
```python
from torchvision import datasets, transforms
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
train_dataset = datasets.ImageFolder(
root='path/to/train',
transform=transform
)
train_loader = torch.utils.data.DataLoader(
train_dataset,
batch_size=32,
shuffle=True,
num_workers=4
)
```
### **52. How do you implement data augmentation in PyTorch?**
Using TorchVision transforms:
```python
train_transform = transforms.Compose([
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),
transforms.RandomRotation(15),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
# For more advanced augmentations
import torchvision.transforms.functional as F
class CustomAugmentation:
def __call__(self, img):
if random.random() > 0.5:
img = F.hflip(img)
if random.random() > 0.5:
img = F.rotate(img, 10)
return img
```
### **53. How do you use a pretrained CNN model from TorchVision?**
```python
import torchvision.models as models
# Load pretrained model
model = models.resnet50(pretrained=True)
# Set to evaluation mode
model.eval()
# Prepare input
input_image = load_and_preprocess_image('image.jpg')
input_batch = input_image.unsqueeze(0) # Add batch dimension
# Get predictions
with torch.no_grad():
output = model(input_batch)
# Get predicted class
_, predicted_idx = torch.max(output, 1)
```
### **54. How do you implement a custom CNN architecture in PyTorch?**
```python
class CustomCNN(nn.Module):
def __init__(self, num_classes=10):
super(CustomCNN, self).__init__()
self.features = nn.Sequential(
nn.Conv2d(3, 64, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
nn.Conv2d(64, 128, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
)
self.classifier = nn.Sequential(
nn.Linear(128 * 7 * 7, 512),
nn.ReLU(inplace=True),
nn.Dropout(0.5),
nn.Linear(512, num_classes)
)
def forward(self, x):
x = self.features(x)
x = torch.flatten(x, 1)
x = self.classifier(x)
return x
```
### **55. How do you implement transfer learning for image classification?**
```python
# Load pretrained model
model = models.resnet18(pretrained=True)
# Freeze all parameters
for param in model.parameters():
param.requires_grad = False
# Replace the final fully connected layer
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, num_classes)
# Train only the classifier
optimizer = torch.optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)
# Or unfreeze some layers
for param in model.layer4.parameters():
param.requires_grad = True
# Create optimizer for all trainable parameters
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()),
lr=0.001, momentum=0.9)
```
### **56. How do you visualize feature maps in a CNN?**
```python
def visualize_feature_maps(model, image, layer_idx=0):
# Get intermediate layer output
activations = []
def hook_fn(module, input, output):
activations.append(output)
# Register hook
hook = model.features[layer_idx].register_forward_hook(hook_fn)
# Forward pass
with torch.no_grad():
_ = model(image.unsqueeze(0))
# Remove hook
hook.remove()
# Get activations
activation = activations[0][0].detach()
# Plot
grid = torchvision.utils.make_grid(activation.unsqueeze(1), nrow=8, padding=1)
plt.imshow(grid.permute(1, 2, 0))
plt.show()
```
### **57. How do you implement object detection with PyTorch?**
Using TorchVision's detection models:
```python
import torchvision
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision.transforms import functional as F
# Load pretrained model
model = fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
# Prepare image
image = load_image('image.jpg')
image_tensor = F.to_tensor(image).unsqueeze(0)
# Get predictions
with torch.no_grad():
predictions = model(image_tensor)
# Process predictions
boxes = predictions[0]['boxes']
labels = predictions[0]['labels']
scores = predictions[0]['scores']
# Filter by score
threshold = 0.5
mask = scores > threshold
boxes = boxes[mask]
labels = labels[mask]
```
### **58. How do you implement semantic segmentation in PyTorch?**
```python
import torchvision
from torchvision.models.segmentation import deeplabv3_resnet101
# Load pretrained model
model = deeplabv3_resnet101(pretrained=True)
model.eval()
# Prepare image
image = load_image('image.jpg')
image_tensor = F.to_tensor(image).unsqueeze(0)
# Get prediction
with torch.no_grad():
output = model(image_tensor)['out'][0]
# Get segmentation map
segmentation = output.argmax(0)
```
### **59. How do you implement image captioning in PyTorch?**
```python
class ImageCaptioner(nn.Module):
def __init__(self, vocab_size, embed_size, hidden_size):
super().__init__()
# CNN for image features
self.cnn = torchvision.models.resnet18(pretrained=True)
self.cnn = nn.Sequential(*list(self.cnn.children())[:-1])
# LSTM for language modeling
self.embed = nn.Embedding(vocab_size, embed_size)
self.lstm = nn.LSTM(embed_size + 512, hidden_size, batch_first=True)
self.linear = nn.Linear(hidden_size, vocab_size)
def forward(self, images, captions, lengths):
# Extract image features
with torch.no_grad():
features = self.cnn(images)
features = features.view(features.size(0), -1)
# Embed captions
embeddings = self.embed(captions)
# Concatenate image features with embeddings
features = features.unsqueeze(1).expand(-1, embeddings.size(1), -1)
embeddings = torch.cat((features, embeddings), dim=2)
# Pack sequences
packed = pack_padded_sequence(embeddings, lengths, batch_first=True, enforce_sorted=False)
# LSTM
hiddens, _ = self.lstm(packed)
# Unpack
outputs, _ = pad_packed_sequence(hiddens, batch_first=True)
# Linear layer
outputs = self.linear(outputs)
return outputs
```
### **60. How do you implement style transfer in PyTorch?**
```python
class StyleTransfer(nn.Module):
def __init__(self, content_img, style_img, model='vgg19'):
super().__init__()
# Load pretrained model
self.vgg = torchvision.models.vgg19(pretrained=True).features
# Freeze parameters
for param in self.vgg.parameters():
param.requires_grad_(False)
# Set content and style images
self.content_img = content_img
self.style_img = style_img
# Define layers for content and style
self.content_layers = ['conv_4']
self.style_layers = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']
def forward(self, input_img):
content_losses = []
style_losses = []
x = input_img
for i, layer in enumerate(self.vgg):
x = layer(x)
name = f'conv_{i+1}'
if name in self.content_layers:
content_losses.append(content_loss(x, self.content_features[name]))
if name in self.style_layers:
style_losses.append(style_loss(x, self.style_features[name]))
return content_losses, style_losses
def style_loss(self, target_features, style_features):
target_gram = gram_matrix(target_features)
style_gram = gram_matrix(style_features)
return F.mse_loss(target_gram, style_gram)
def content_loss(self, target_features, content_features):
return F.mse_loss(target_features, content_features)
def gram_matrix(self, input):
batch_size, channels, h, w = input.size()
features = input.view(batch_size * channels, h * w)
gram = features @ features.t()
return gram / (batch_size * channels * h * w)
```
---
## **NLP with PyTorch**
### **61. How do you tokenize text for NLP tasks in PyTorch?**
Using TorchText or Hugging Face Tokenizers:
```python
# Using TorchText (older approach)
from torchtext.data import Field
TEXT = Field(tokenize='spacy', tokenizer_language='en_core_web_sm')
train_data, test_data = datasets.IMDB.splits(TEXT)
# Using Hugging Face Tokenizers (recommended)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer.tokenize("Hello, world!")
ids = tokenizer.convert_tokens_to_ids(tokens)
encoded = tokenizer("Hello, world!", return_tensors='pt')
```
### **62. How do you handle variable-length sequences in NLP?**
Using padding and packing:
```python
from torch.nn.utils.rnn import pad_sequence
# List of variable-length sequences
sequences = [torch.ones(3), torch.ones(4), torch.ones(5)]
# Pad sequences
padded = pad_sequence(sequences, batch_first=True, padding_value=0)
# Create mask for padding
mask = (padded != 0)
# For RNNs, use pack_padded_sequence
lengths = [len(seq) for seq in sequences]
packed = pack_padded_sequence(padded, lengths, batch_first=True, enforce_sorted=False)
```
### **63. How do you implement an RNN language model in PyTorch?**
```python
class RNNLanguageModel(nn.Module):
def __init__(self, vocab_size, embed_size, hidden_size, num_layers=1):
super().__init__()
self.embed = nn.Embedding(vocab_size, embed_size)
self.rnn = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)
self.linear = nn.Linear(hidden_size, vocab_size)
def forward(self, x, hidden=None):
# x shape: (batch_size, seq_length)
x = self.embed(x) # (batch_size, seq_length, embed_size)
x, hidden = self.rnn(x, hidden)
x = self.linear(x) # (batch_size, seq_length, vocab_size)
return x, hidden
```
### **64. How do you implement a Transformer model in PyTorch?**
Using `nn.Transformer`:
```python
class TransformerModel(nn.Module):
def __init__(self, vocab_size, d_model, nhead, num_encoder_layers, num_decoder_layers):
super().__init__()
self.embedding = nn.Embedding(vocab_size, d_model)
self.pos_encoder = PositionalEncoding(d_model)
self.transformer = nn.Transformer(
d_model=d_model,
nhead=nhead,
num_encoder_layers=num_encoder_layers,
num_decoder_layers=num_decoder_layers
)
self.fc_out = nn.Linear(d_model, vocab_size)
def forward(self, src, tgt, src_mask=None, tgt_mask=None):
src = self.embedding(src) * math.sqrt(self.d_model)
src = self.pos_encoder(src)
tgt = self.embedding(tgt) * math.sqrt(self.d_model)
tgt = self.pos_encoder(tgt)
output = self.transformer(src, tgt, src_mask, tgt_mask)
return self.fc_out(output)
```
### **65. How do you implement attention mechanism in PyTorch?**
Basic attention implementation:
```python
class Attention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.attn = nn.Linear(hidden_size * 2, hidden_size)
self.v = nn.Linear(hidden_size, 1, bias=False)
def forward(self, hidden, encoder_outputs):
# hidden: (batch_size, hidden_size)
# encoder_outputs: (batch_size, seq_len, hidden_size)
batch_size = encoder_outputs.size(0)
seq_len = encoder_outputs.size(1)
# Repeat hidden state seq_len times
hidden = hidden.unsqueeze(1).repeat(1, seq_len, 1)
# Compute energy
energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim=2)))
# Compute attention scores
attention = self.v(energy).squeeze(2)
# Apply softmax to get weights
attention_weights = F.softmax(attention, dim=1)
# Apply attention weights to encoder outputs
context_vector = torch.bmm(attention_weights.unsqueeze(1), encoder_outputs)
context_vector = context_vector.squeeze(1)
return context_vector, attention_weights
```
### **66. How do you fine-tune a BERT model for text classification?**
Using Hugging Face Transformers:
```python
from transformers import BertTokenizer, BertForSequenceClassification
# Load tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
# Tokenize input
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt", padding=True, truncation=True, max_length=128)
# Forward pass
outputs = model(**inputs, labels=torch.tensor([1]))
# Compute loss
loss = outputs.loss
# Backward pass
loss.backward()
```
### **67. How do you handle long documents in Transformer models?**
Several approaches:
1. **Truncation**: Limit to max sequence length
```python
inputs = tokenizer(text, max_length=512, truncation=True, return_tensors="pt")
```
2. **Sliding window**: Process document in chunks
```python
def process_long_document(text, chunk_size=512, stride=256):
tokens = tokenizer(text, return_tensors="pt", add_special_tokens=False)
chunks = []
for i in range(0, len(tokens['input_ids'][0]), stride):
chunk = tokens['input_ids'][0][i:i+chunk_size]
chunks.append(chunk)
return chunks
```
3. **Longformer/BigBird**: Use models designed for long sequences
```python
from transformers import LongformerModel
model = LongformerModel.from_pretrained("allenai/longformer-base-4096")
```
### **68. How do you implement beam search for text generation?**
```python
def beam_search(model, start_token, max_length, beam_width=3):
# Initialize beams
beams = [(start_token, 0.0)] # (sequence, score)
for _ in range(max_length):
all_candidates = []
for seq, score in beams:
# Get next token probabilities
with torch.no_grad():
logits = model(seq)
probs = F.log_softmax(logits[:, -1, :], dim=-1)
# Get top beam_width candidates
top_probs, top_indices = torch.topk(probs, beam_width)
# Create new candidates
for i in range(beam_width):
next_token = top_indices[0, i].item()
next_score = score + top_probs[0, i].item()
all_candidates.append((seq + [next_token], next_score))
# Select top beam_width candidates
ordered = sorted(all_candidates, key=lambda x: x[1], reverse=True)
beams = ordered[:beam_width]
return beams[0][0] # Return best sequence
```
### **69. How do you implement a named entity recognition (NER) model?**
```python
class NERModel(nn.Module):
def __init__(self, vocab_size, tagset_size, embed_dim=100, hidden_dim=200):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim // 2, bidirectional=True, batch_first=True)
self.fc = nn.Linear(hidden_dim, tagset_size)
def forward(self, sentence):
embeds = self.embedding(sentence)
lstm_out, _ = self.lstm(embeds)
tag_space = self.fc(lstm_out)
return F.log_softmax(tag_space, dim=2)
```
### **70. How do you implement a transformer-based machine translation model?**
```python
class TransformerMT(nn.Module):
def __init__(self, src_vocab_size, tgt_vocab_size, d_model, nhead, num_layers):
super().__init__()
self.encoder_embedding = nn.Embedding(src_vocab_size, d_model)
self.decoder_embedding = nn.Embedding(tgt_vocab_size, d_model)
self.pos_encoder = PositionalEncoding(d_model)
self.transformer = nn.Transformer(
d_model=d_model,
nhead=nhead,
num_encoder_layers=num_layers,
num_decoder_layers=num_layers
)
self.fc_out = nn.Linear(d_model, tgt_vocab_size)
self.d_model = d_model
def generate_square_subsequent_mask(self, sz):
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
return mask
def create_padding_mask(self, seq, pad_idx):
return (seq == pad_idx).transpose(0, 1)
def forward(self, src, tgt, src_mask=None, tgt_mask=None, src_padding_mask=None, tgt_padding_mask=None):
src = self.encoder_embedding(src) * math.sqrt(self.d_model)
src = self.pos_encoder(src)
tgt = self.decoder_embedding(tgt) * math.sqrt(self.d_model)
tgt = self.pos_encoder(tgt)
output = self.transformer(
src, tgt,
src_mask=src_mask,
tgt_mask=tgt_mask,
src_key_padding_mask=src_padding_mask,
tgt_key_padding_mask=tgt_padding_mask
)
return self.fc_out(output)
```
---
## **Advanced PyTorch Features**
### **71. What is TorchScript and when would you use it?**
TorchScript is PyTorch's intermediate representation that allows:
- Serializing models from Python
- Running models in non-Python environments
- Optimizing models for production deployment
- Better performance through graph optimization
Use cases:
- Deploying models to production (C++ environments)
- Serving models with TorchServe
- Mobile deployment with PyTorch Mobile
- Model optimization
```python
# Tracing
traced_model = torch.jit.trace(model, example_input)
# Scripting
scripted_model = torch.jit.script(model)
# Save
traced_model.save("model.pt")
# Load in C++
// #include <torch/script.h>
// auto module = torch::jit::load("model.pt");
```
### **72. How do you use `torch.compile()` for model optimization?**
```python
# PyTorch 2.0+ feature
model = MyModel()
optimized_model = torch.compile(model)
# Training loop with optimized model
for inputs, targets in train_loader:
optimizer.zero_grad()
outputs = optimized_model(inputs)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()
```
`torch.compile()`:
- Converts model to optimized TorchDynamo graph
- Applies various optimizations
- Can significantly improve training speed
- Works with both eager and graph execution
### **73. How do you implement custom CUDA kernels in PyTorch?**
Using `torch.cuda.CUDAStream` and custom kernels:
```python
import torch
from torch.utils.cpp_extension import load
# Load custom CUDA kernel
cuda_ext = load(
name='cuda_ext',
sources=['custom_kernel.cu'],
verbose=True
)
# Use custom kernel
output = cuda_ext.custom_function(input)
```
Alternatively, using `torch.cuda.Function`:
```python
class CustomFunction(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
# Custom CUDA implementation
output = custom_cuda_forward(input)
ctx.save_for_backward(input)
return output
@staticmethod
def backward(ctx, grad_output):
input, = ctx.saved_tensors
grad_input = custom_cuda_backward(input, grad_output)
return grad_input
```
### **74. How do you implement distributed training in PyTorch?**
Using `torch.distributed`:
```python
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
def setup(rank, world_size):
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
dist.init_process_group("nccl", rank=rank, world_size=world_size)
def cleanup():
dist.destroy_process_group()
def train(rank, world_size):
setup(rank, world_size)
# Create model and move to GPU
model = MyModel().to(rank)
ddp_model = DDP(model, device_ids=[rank])
# Create dataloader with DistributedSampler
train_sampler = torch.utils.data.distributed.DistributedSampler(
dataset,
num_replicas=world_size,
rank=rank
)
train_loader = torch.utils.data.DataLoader(
dataset,
batch_size=batch_size,
sampler=train_sampler
)
# Training loop
for inputs, labels in train_loader:
inputs, labels = inputs.to(rank), labels.to(rank)
outputs = ddp_model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
cleanup()
```
### **75. How do you use PyTorch with multi-GPU training?**
Using DataParallel or DistributedDataParallel:
**DataParallel (simpler, but less efficient):**
```python
model = MyModel()
model = torch.nn.DataParallel(model)
model = model.to('cuda')
```
**DistributedDataParallel (recommended for multi-GPU):**
```python
# Setup distributed environment
torch.distributed.init_process_group(backend='nccl')
# Move model to current device
local_rank = int(os.environ['LOCAL_RANK'])
torch.cuda.set_device(local_rank)
model = model.to(local_rank)
# Wrap model with DDP
model = torch.nn.parallel.DistributedDataParallel(
model,
device_ids=[local_rank],
output_device=local_rank
)
```
### **76. How do you implement custom autograd functions with CUDA support?**
```python
class CustomFunction(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
# CUDA implementation
output = custom_cuda_forward(input)
ctx.save_for_backward(input)
return output
@staticmethod
def backward(ctx, grad_output):
input, = ctx.saved_tensors
# CUDA implementation
grad_input = custom_cuda_backward(input, grad_output)
return grad_input
# Usage
x = torch.randn(10, requires_grad=True, device='cuda')
y = CustomFunction.apply(x)
```
### **77. How do you use PyTorch with ONNX for model export?**
```python
# Export model to ONNX
dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
model,
dummy_input,
"model.onnx",
export_params=True,
opset_version=11,
do_constant_folding=True,
input_names=['input'],
output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}}
)
# Load and run ONNX model
import onnxruntime
ort_session = onnxruntime.InferenceSession("model.onnx")
outputs = ort_session.run(None, {'input': dummy_input.numpy()})
```
### **78. How do you implement model parallelism in PyTorch?**
Splitting a model across multiple devices:
```python
class ModelParallelModel(nn.Module):
def __init__(self, device0, device1):
super().__init__()
self.device0 = device0
self.device1 = device1
# First part on device0
self.part1 = nn.Sequential(
nn.Linear(1000, 500),
nn.ReLU()
).to(device0)
# Second part on device1
self.part2 = nn.Sequential(
nn.Linear(500, 250),
nn.ReLU(),
nn.Linear(250, 10)
).to(device1)
def forward(self, x):
x = x.to(self.device0)
x = self.part1(x)
x = x.to(self.device1)
x = self.part2(x)
return x
```
### **79. How do you use PyTorch with TensorBoard for visualization?**
```python
from torch.utils.tensorboard import SummaryWriter
# Create writer
writer = SummaryWriter('runs/experiment_1')
# Log scalars
for epoch in range(100):
writer.add_scalar('Loss/train', train_loss, epoch)
writer.add_scalar('Accuracy/val', val_acc, epoch)
# Log histograms
writer.add_histogram('Weights/layer1', model.layer1.weight, epoch)
# Log images
writer.add_image('Images/sample', image, epoch)
# Log graph
writer.add_graph(model, dummy_input)
# Close writer
writer.close()
```
### **80. How do you implement gradient checkpointing in PyTorch?**
```python
# Using torch.utils.checkpoint
from torch.utils.checkpoint import checkpoint
class CheckpointedModel(nn.Module):
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(1000, 1000)
self.layer2 = nn.Linear(1000, 1000)
self.layer3 = nn.Linear(1000, 10)
def forward(self, x):
# Only save inputs for backward pass
x = checkpoint(self.layer1, x)
x = checkpoint(self.layer2, x)
x = self.layer3(x)
return x
```
Gradient checkpointing trades compute for memory by recomputing activations during backward pass.
---
## **Performance Optimization**
### **81. How do you profile PyTorch model performance?**
Using `torch.profiler`:
```python
from torch.profiler import profile, record_function, ProfilerActivity
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
schedule=torch.profiler.schedule(wait=1, warmup=1, active=3),
on_trace_ready=torch.profiler.tensorboard_trace_handler('./log'),
record_shapes=True) as prof:
for step, (inputs, labels) in enumerate(train_loader):
if step >= 1 + 1 + 3:
break
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
prof.step() # Need to call this to record each step
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
```
### **82. How do you optimize data loading in PyTorch?**
Best practices:
- Use `num_workers > 0` in DataLoader
- Use `pin_memory=True` for GPU training
- Use appropriate batch size
- Prefetch data with `prefetch_factor`
- Use memory-mapped files for large datasets
- Consider custom collate functions
```python
train_loader = DataLoader(
dataset,
batch_size=64,
shuffle=True,
num_workers=4,
pin_memory=True,
prefetch_factor=2
)
```
### **83. How do you reduce memory usage during training?**
Strategies:
- Use gradient checkpointing
- Reduce batch size
- Use mixed precision training
- Clear unused variables with `del`
- Use `torch.cuda.empty_cache()`
- Avoid unnecessary intermediate variables
- Use `inplace` operations where safe
- Use memory-efficient architectures
### **84. How do you implement custom CUDA kernels for performance?**
Using CUDA C++ with PyTorch:
1. Write CUDA kernel in `.cu` file
2. Compile with `torch.utils.cpp_extension.load`
3. Call from Python
Example kernel (`custom_kernel.cu`):
```cuda
#include <torch/extension.h>
__global__ void custom_kernel_forward(
const float* input,
float* output,
int size
) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < size) {
output[idx] = input[idx] * 2.0f;
}
}
torch::Tensor custom_forward(torch::Tensor input) {
auto output = torch::empty_like(input);
int size = input.numel();
int threads = 256;
int blocks = (size + threads - 1) / threads;
custom_kernel_forward<<<blocks, threads>>>(
input.data_ptr<float>(),
output.data_ptr<float>(),
size
);
return output;
}
```
### **85. How do you use memory profiling tools with PyTorch?**
Using `torch.cuda.memory_summary()`:
```python
# Print memory summary
print(torch.cuda.memory_summary())
# Track memory allocation
before = torch.cuda.memory_allocated()
# Some operations
after = torch.cuda.memory_allocated()
print(f"Memory used: {(after - before) / 1024**2:.2f} MB")
# Reset memory statistics
torch.cuda.reset_peak_memory_stats()
```
Third-party tools:
- `memory_profiler` package
- `nvprof` for CUDA profiling
- PyTorch Profiler with TensorBoard
### **86. How do you optimize inference performance in PyTorch?**
Strategies:
- Use `torch.inference_mode()` instead of `torch.no_grad()`
- Use TorchScript or ONNX for deployment
- Apply quantization
- Use tensor parallelism
- Optimize model architecture
- Use CUDA Graphs for fixed computation patterns
- Use TensorRT for NVIDIA GPUs
```python
# Quantization example
model.eval()
quantized_model = torch.quantization.quantize_dynamic(
model, {nn.Linear}, dtype=torch.qint8
)
# Inference
with torch.inference_mode():
output = quantized_model(input)
```
### **87. How do you implement quantization in PyTorch?**
Types of quantization:
- **Dynamic quantization**: Weights quantized, activations quantized dynamically
- **Static quantization**: Both weights and activations quantized
- **QAT (Quantization Aware Training)**: Simulate quantization during training
Example (dynamic quantization):
```python
model.eval()
quantized_model = torch.quantization.quantize_dynamic(
model, {nn.LSTM, nn.Linear}, dtype=torch.qint8
)
```
Example (QAT):
```python
# Prepare model for QAT
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
model = torch.quantization.prepare_qat(model)
# Train with QAT
for epoch in range(10):
train_one_epoch(model, train_loader)
# Quantization parameters get updated during training
# Convert to quantized model
quantized_model = torch.quantization.convert(model.eval())
```
### **88. How do you use CUDA Graphs for performance optimization?**
```python
# Record CUDA graph
g = torch.cuda.CUDAGraph()
input = torch.randn(64, 3, 224, 224, device="cuda")
with torch.cuda.graph(g):
output = model(input)
# Replay graph with new inputs
for i in range(100):
input.copy_(next(data_iter))
g.replay()
loss = criterion(output, target)
loss.backward()
```
CUDA Graphs capture a sequence of CUDA operations and replay them with minimal CPU overhead.
### **89. How do you optimize model architecture for better performance?**
Architecture optimization strategies:
- Use depthwise separable convolutions
- Reduce channel dimensions
- Use efficient activation functions (SiLU, HardSwish)
- Apply model pruning
- Use knowledge distillation
- Implement efficient attention mechanisms
- Use neural architecture search
Example (depthwise separable convolution):
```python
class DepthwiseSeparableConv(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size):
super().__init__()
self.depthwise = nn.Conv2d(
in_channels, in_channels, kernel_size,
groups=in_channels, padding=kernel_size//2
)
self.pointwise = nn.Conv2d(
in_channels, out_channels, 1
)
def forward(self, x):
x = self.depthwise(x)
x = self.pointwise(x)
return x
```
### **90. How do you identify and fix bottlenecks in PyTorch training?**
Steps to identify and fix bottlenecks:
1. **Profile the entire pipeline**:
- Use PyTorch Profiler to identify CPU/GPU bottlenecks
- Check data loading, forward pass, backward pass
2. **Check data loading**:
- Increase `num_workers` in DataLoader
- Use `pin_memory=True` for GPU training
- Consider memory-mapped files
3. **Optimize model**:
- Use mixed precision training
- Apply gradient checkpointing
- Optimize model architecture
4. **Check hardware utilization**:
- Monitor GPU utilization with `nvidia-smi`
- Ensure GPU isn't waiting for CPU
---
## **PyTorch Ecosystem**
### **91. What is PyTorch Lightning and how does it differ from vanilla PyTorch?**
PyTorch Lightning is a lightweight wrapper for PyTorch that:
- Organizes training code into logical components
- Handles boilerplate code (training loops, GPU handling)
- Provides built-in support for distributed training
- Simplifies experiment tracking
- Maintains full control over the model
Key differences:
- **Vanilla PyTorch**: More flexible, more boilerplate
- **PyTorch Lightning**: Less boilerplate, more structured
Example Lightning module:
```python
import pytorch_lightning as pl
class LitModel(pl.LightningModule):
def __init__(self):
super().__init__()
self.l1 = nn.Linear(28 * 28, 10)
def forward(self, x):
return torch.relu(self.l1(x.view(x.size(0), -1)))
def training_step(self, batch, batch_idx):
x, y = batch
y_hat = self(x)
loss = F.cross_entropy(y_hat, y)
return loss
def configure_optimizers(self):
return torch.optim.Adam(self.parameters(), lr=0.02)
```
### **92. How do you use TorchVision for image transformations?**
```python
from torchvision import transforms
# Define transformation pipeline
transform = transforms.Compose([
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(brightness=0.2, contrast=0.2),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
# Apply to image
image = Image.open('image.jpg')
transformed_image = transform(image)
# Custom transformations
class CustomTransform:
def __call__(self, img):
# Custom transformation logic
return img.rotate(10)
```
### **93. What is TorchText and how is it used for NLP tasks?**
TorchText (older version) provided:
- Text datasets (IMDB, SST, etc.)
- Tokenization and vocabulary building
- Data iterators for variable-length sequences
However, Hugging Face Transformers has largely superseded TorchText for modern NLP.
Current best practice:
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased')
```
### **94. How do you use Captum for model interpretability?**
```python
from captum.attr import IntegratedGradients, visualization
# Initialize attribution method
ig = IntegratedGradients(model)
# Compute attributions
input = torch.randn(1, 3, 224, 224, requires_grad=True)
target = 0
attributions = ig.attribute(input, target=target)
# Visualize
_ = visualization.visualize_image_attr(
attribution=attributions.squeeze(0).permute(1, 2, 0).detach().numpy(),
original_image=input.squeeze(0).permute(1, 2, 0).detach().numpy(),
method="blended_heat_map",
sign="all",
show_colorbar=True
)
```
### **95. How do you use TorchMetrics for evaluation metrics?**
```python
from torchmetrics import Accuracy, Precision, Recall
# Initialize metrics
accuracy = Accuracy(task="multiclass", num_classes=10)
precision = Precision(task="multiclass", average='macro', num_classes=10)
recall = Recall(task="multiclass", average='macro', num_classes=10)
# Update metrics
for batch in dataloader:
preds, target = model(batch), batch["labels"]
accuracy.update(preds, target)
precision.update(preds, target)
recall.update(preds, target)
# Compute final metrics
acc = accuracy.compute()
prec = precision.compute()
rec = recall.compute()
```
### **96. What is TorchServe and how is it used for model deployment?**
TorchServe is a framework for serving PyTorch models in production.
Basic workflow:
1. Create a model archive
2. Start TorchServe
3. Make predictions via API
Example:
```bash
# Create model archive
torch-model-archiver --model-name my_model \
--version 1.0 \
--model-file model.py \
--serialized-file model.pth \
--handler image_classifier
# Start TorchServe
torchserve --start --model-store model_store --models my_model=my_model.mar
# Make prediction
curl -X POST http://localhost:8080/predictions/my_model -T input.jpg
```
### **97. How do you use PyTorch with Weights & Biases for experiment tracking?**
```python
import wandb
wandb.init(project="my-project", entity="my-entity")
# Log hyperparameters
wandb.config = {
"learning_rate": 0.001,
"batch_size": 32,
"architecture": "ResNet50"
}
# Training loop
for epoch in range(epochs):
# Training code...
# Log metrics
wandb.log({
"loss": train_loss,
"accuracy": train_acc,
"val_loss": val_loss,
"val_accuracy": val_acc
})
# Log model checkpoint
if epoch % 10 == 0:
torch.save(model.state_dict(), "model.pth")
wandb.save("model.pth")
```
### **98. How do you use PyTorch with MLflow for experiment tracking?**
```python
import mlflow
import mlflow.pytorch
mlflow.set_experiment("my-experiment")
with mlflow.start_run():
# Log parameters
mlflow.log_param("learning_rate", 0.001)
mlflow.log_param("batch_size", 32)
# Training loop
for epoch in range(epochs):
# Training code...
# Log metrics
mlflow.log_metric("loss", train_loss, step=epoch)
mlflow.log_metric("accuracy", train_acc, step=epoch)
# Log model
mlflow.pytorch.log_model(model, "model")
```
### **99. How do you use PyTorch with Ray for distributed training?**
```python
import ray
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler
ray.init()
def train_cnn(config, checkpoint_dir=None):
# Training function
model = MyModel()
optimizer = torch.optim.SGD(model.parameters(), lr=config["lr"])
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
model.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
for i in range(100):
# Training code...
# Save checkpoint
with tune.checkpoint_dir(step=i) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((model.state_dict(), optimizer.state_dict()), path)
# Report metrics
tune.report(loss=loss.item(), accuracy=accuracy)
# Run hyperparameter tuning
analysis = tune.run(
train_cnn,
config={
"lr": tune.loguniform(1e-4, 1e-1),
},
num_samples=10,
scheduler=ASHAScheduler(metric="loss", mode="min"),
progress_reporter=CLIReporter(metric_columns=["loss", "accuracy", "training_iteration"])
)
print("Best config: ", analysis.get_best_config(metric="loss", mode="min"))
```
### **100. How do you use PyTorch with ONNX Runtime for inference?**
```python
# Export model to ONNX
torch.onnx.export(model, dummy_input, "model.onnx")
# Run with ONNX Runtime
import onnxruntime as ort
ort_session = ort.InferenceSession("model.onnx")
# Get input names
input_name = ort_session.get_inputs()[0].name
# Run inference
outputs = ort_session.run(None, {input_name: input_numpy})
```
---
## **Debugging and Troubleshooting**
### **101. How do you debug NaN values in PyTorch models?**
Strategies:
- Use `torch.autograd.set_detect_anomaly(True)`
- Check for division by zero
- Check for log of zero/negative values
- Use gradient clipping
- Reduce learning rate
- Add small epsilon to denominators
```python
# Detect anomalies
with torch.autograd.detect_anomaly():
loss = model(inputs)
loss.backward()
```
### **102. How do you fix "CUDA out of memory" errors?**
Solutions:
- Reduce batch size
- Use gradient accumulation
- Clear cache with `torch.cuda.empty_cache()`
- Use mixed precision training
- Apply gradient checkpointing
- Remove unnecessary variables with `del`
- Use smaller model
### **103. How do you debug shape mismatches in PyTorch?**
Strategies:
- Print tensor shapes at each step
- Use assertions in forward pass
- Use shape checking tools
- Use `torch.jit.script` for static checking
```python
def forward(self, x):
print(f"Input shape: {x.shape}")
x = self.conv1(x)
print(f"After conv1: {x.shape}")
x = self.fc(x.view(x.size(0), -1))
print(f"Before fc: {x.shape}")
return x
```
### **104. How do you fix "one of the variables needed for gradient computation has been modified by the inplace operation" error?**
This error occurs when:
- Using in-place operations on tensors that require gradients
- Modifying tensors that are part of the computation graph
Solutions:
- Avoid in-place operations (e.g., use `x = x + 1` instead of `x += 1`)
- Use `detach().clone()` when modifying tensors
- Set `retain_graph=True` in backward if needed multiple times
### **105. How do you debug slow training in PyTorch?**
Steps:
- Profile with PyTorch Profiler
- Check if GPU is being used (`torch.cuda.is_available()`)
- Monitor GPU utilization (`nvidia-smi`)
- Check data loading speed
- Profile individual components (data loading, forward, backward)
- Check if mixed precision is properly configured
### **106. How do you fix "expected scalar type X but found type Y" error?**
This occurs when tensors have different data types.
Solutions:
- Ensure consistent data types
- Use `.to()` to convert tensors
- Check model and input dtypes
```python
# Ensure model and inputs are same dtype
model = model.float() # or .half() for mixed precision
inputs = inputs.float()
```
### **107. How do you debug vanishing/exploding gradients?**
Strategies:
- Print gradient norms
- Use gradient clipping
- Check weight initialization
- Use appropriate activation functions
- Add batch normalization
- Monitor gradient histograms with TensorBoard
```python
# Monitor gradient norms
for name, param in model.named_parameters():
if param.grad is not None:
grad_norm = param.grad.data.norm(2).item()
print(f"{name} gradient norm: {grad_norm}")
```
### **108. How do you fix "dimension out of range" error?**
This occurs when indexing with invalid dimensions.
Solutions:
- Check tensor dimensions with `.shape`
- Verify dimension indices
- Use negative indices carefully
- Check if dimensions were squeezed/unsqueezed
```python
x = torch.randn(3, 4)
print(x.shape) # torch.Size([3, 4])
# This would cause error
# x.mean(dim=2)
# Correct usage
x.mean(dim=1) # OK
```
### **109. How do you debug model not learning (loss not decreasing)?**
Steps:
- Check if gradients are updating parameters
- Verify data preprocessing
- Check learning rate (too high/low)
- Ensure proper initialization
- Check for implementation errors
- Try overfitting a small batch first
```python
# Overfit a small batch to verify implementation
small_batch = next(iter(train_loader))
for _ in range(100):
optimizer.zero_grad()
output = model(small_batch[0])
loss = criterion(output, small_batch[1])
loss.backward()
optimizer.step()
print(loss.item())
```
### **110. How do you fix "expected device X but got device Y" error?**
This occurs when tensors are on different devices.
Solutions:
- Move all tensors to same device
- Use `.to(device)` consistently
- Ensure model and inputs are on same device
```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
inputs = inputs.to(device)
targets = targets.to(device)
```
---
## **PyTorch vs Other Frameworks**
### **111. How does PyTorch differ from TensorFlow?**
| Feature | PyTorch | TensorFlow |
|---------|---------|------------|
| **Execution Model** | Eager execution by default | Graph execution by default (with eager option) |
| **API Style** | Pythonic, object-oriented | More verbose, sometimes functional |
| **Debugging** | Easier (standard Python tools) | More challenging (graph-based) |
| **Deployment** | TorchServe, ONNX | TensorFlow Serving, TFLite |
| **Research Adoption** | Dominant in research | Strong in production |
| **Keras Integration** | TorchKeras (less common) | Native Keras integration |
| **Mobile Deployment** | PyTorch Mobile | TensorFlow Lite |
| **Community** | Strong academic/research | Strong industry |
### **112. When would you choose PyTorch over TensorFlow?**
Choose PyTorch when:
- Working in research/academic setting
- Need flexibility for novel architectures
- Prefer Pythonic debugging experience
- Building custom training loops
- Working with dynamic computation graphs
- Using cutting-edge research models
- Collaborating with researchers (most papers use PyTorch)
### **113. When would you choose TensorFlow over PyTorch?**
Choose TensorFlow when:
- Building production systems with TF Serving
- Need mobile deployment with TFLite
- Using Keras for quick prototyping
- Working with TFX for ML pipelines
- Need strong enterprise support
- Using TPUs for accelerated training
- Integrating with Google Cloud services
### **114. How does PyTorch compare to JAX?**
| Feature | PyTorch | JAX |
|---------|---------|-----|
| **Core Concept** | OOP with modules | Functional transformations |
| **Autograd** | Automatic | Explicit with `grad` |
| **JIT Compilation** | `torch.compile()` | `jit` transformation |
| **Vectorization** | Explicit loops | `vmap` transformation |
| **Parallelization** | DDP, DataParallel | `pmap` transformation |
| **Debugging** | Standard Python | Requires JAX-specific tools |
| **Community** | Large, research-focused | Growing, research-focused |
| **Best For** | Flexible model building | High-performance numerical computing |
### **115. How does PyTorch Lightning compare to Keras?**
| Feature | PyTorch Lightning | Keras |
|---------|-------------------|-------|
| **Underlying Framework** | PyTorch | TensorFlow |
| **Flexibility** | High (full PyTorch access) | Moderate (constrained by TF) |
| **Boilerplate** | Reduces PyTorch boilerplate | Minimal boilerplate |
| **Research Features** | Better for novel research | More opinionated |
| **Distributed Training** | Built-in support | Requires TF distribution strategies |
| **Model Subclassing** | Encouraged | Discouraged (functional API preferred) |
| **Customization** | Easier to customize training loop | More constrained |
### **116. How does PyTorch's autograd compare to TensorFlow's GradientTape?**
Both provide automatic differentiation, but with different approaches:
**PyTorch Autograd**:
- Implicitly tracks operations on tensors with `requires_grad=True`
- Computation graph built dynamically during execution
- Gradients accumulated in `.grad` attribute
- More "invisible" to the user
**TensorFlow GradientTape**:
- Explicit context manager (`with tf.GradientTape() as tape:`)
- Must explicitly watch tensors if not variables
- Returns gradients directly rather than storing them
- More explicit about what's being differentiated
### **117. How does PyTorch's distributed training compare to TensorFlow's?**
**PyTorch Distributed**:
- `torch.distributed` package with multiple backends (NCCL, Gloo, MPI)
- `DistributedDataParallel` for multi-GPU training
- More flexible but requires more setup
- Better for research with custom training loops
**TensorFlow Distribution Strategies**:
- `tf.distribute` API with various strategies
- More integrated with high-level APIs
- Easier setup for standard use cases
- Better for production deployment
### **118. How does PyTorch's deployment story compare to TensorFlow's?**
**PyTorch Deployment**:
- TorchScript for serialization
- TorchServe for serving
- ONNX for cross-framework deployment
- PyTorch Mobile for mobile
- Less mature ecosystem than TensorFlow
**TensorFlow Deployment**:
- TensorFlow Serving for production
- TFX for ML pipelines
- TFLite for mobile
- TF.js for web
- More mature deployment ecosystem
### **119. How does PyTorch's visualization tools compare to TensorFlow's?**
**PyTorch**:
- TensorBoard integration via `torch.utils.tensorboard`
- Less built-in visualization
- More reliant on third-party tools
- Growing ecosystem
**TensorFlow**:
- Native TensorBoard integration
- More comprehensive visualization tools
- Better model introspection
- More mature visualization ecosystem
### **120. How does PyTorch's model zoo compare to TensorFlow Hub?**
**PyTorch Model Zoo**:
- `torchvision.models` for vision models
- Hugging Face Model Hub for NLP
- Growing but less centralized
- More research-focused models
**TensorFlow Hub**:
- Centralized model repository
- More production-ready models
- Better integration with TensorFlow ecosystem
- Wider variety of pre-trained models
---
## **Deployment and Productionization**
### **121. How do you deploy a PyTorch model to production?**
Common approaches:
1. **TorchServe**:
```bash
torch-model-archiver --model-name my_model \
--version 1.0 \
--model-file model.py \
--serialized-file model.pth \
--handler image_classifier
torchserve --start --model-store model_store --models my_model=my_model.mar
```
2. **ONNX Runtime**:
```python
torch.onnx.export(model, dummy_input, "model.onnx")
# Then use ONNX Runtime for inference
```
3. **TorchScript**:
```python
scripted_model = torch.jit.script(model)
scripted_model.save("model.pt")
# Load in C++ or Python
```
4. **REST API with Flask/FastAPI**:
```python
from fastapi import FastAPI, UploadFile
app = FastAPI()
@app.post("/predict/")
async def predict(file: UploadFile):
image = preprocess(file)
with torch.no_grad():
prediction = model(image)
return {"prediction": prediction.tolist()}
```
### **122. How do you convert a PyTorch model to ONNX format?**
```python
# Export model to ONNX
dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
model,
dummy_input,
"model.onnx",
export_params=True,
opset_version=11,
do_constant_folding=True,
input_names=['input'],
output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}}
)
# Verify ONNX model
import onnx
onnx_model = onnx.load("model.onnx")
onnx.checker.check_model(onnx_model)
```
### **123. How do you serve a PyTorch model with TorchServe?**
1. Create a model archive:
```bash
torch-model-archiver --model-name my_model \
--version 1.0 \
--model-file model.py \
--serialized-file model.pth \
--handler image_classifier
```
2. Start TorchServe:
```bash
torchserve --start --model-store model_store --models my_model=my_model.mar
```
3. Make predictions:
```bash
curl -X POST http://localhost:8080/predictions/my_model -T input.jpg
```
4. Custom handler example (`handler.py`):
```python
def handle(data, context):
input_tensor = preprocess(data)
with torch.no_grad():
output = model(input_tensor)
return postprocess(output)
```
### **124. How do you optimize a PyTorch model for mobile deployment?**
Using PyTorch Mobile:
```python
# Trace the model
traced_model = torch.jit.trace(model, example_input)
# Optimize for mobile
optimized_model = optimize_for_mobile(traced_model)
# Save for mobile
optimized_model._save_for_lite_interpreter("model.ptl")
# In Android (Java)
// Module module = Module.load("model.ptl");
// Tensor output = module.forward(IValue.from(input)).toTensor();
```
### **125. How do you implement a REST API for a PyTorch model using FastAPI?**
```python
from fastapi import FastAPI, UploadFile, File
import torch
from PIL import Image
from torchvision import transforms
app = FastAPI()
# Load model
model = torch.jit.load('model.pt')
model.eval()
# Preprocessing
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
@app.post("/predict/")
async def predict(file: UploadFile = File(...)):
# Read and preprocess image
image = Image.open(file.file).convert('RGB')
input_tensor = transform(image).unsqueeze(0)
# Make prediction
with torch.no_grad():
output = model(input_tensor)
# Process output
_, predicted = torch.max(output, 1)
return {"prediction": predicted.item()}
```
### **126. How do you monitor a deployed PyTorch model?**
Monitoring strategies:
- Log prediction metrics and latency
- Track input data distributions
- Monitor for drift in model performance
- Implement health checks
- Set up alerts for anomalies
Example with Prometheus:
```python
from prometheus_client import start_http_server, Counter, Histogram
# Initialize metrics
REQUEST_COUNT = Counter('model_requests_total', 'Total model requests')
REQUEST_LATENCY = Histogram('model_request_latency_seconds', 'Model request latency')
ERROR_COUNT = Counter('model_errors_total', 'Total model errors')
@app.middleware("http")
async def monitor_requests(request, call_next):
REQUEST_COUNT.inc()
start_time = time.time()
try:
response = await call_next(request)
return response
except Exception as e:
ERROR_COUNT.inc()
raise e
finally:
latency = time.time() - start_time
REQUEST_LATENCY.observe(latency)
```
### **127. How do you implement model versioning and rollback in production?**
Strategies:
- Store model artifacts with version numbers
- Use canary deployments for new versions
- Implement A/B testing between versions
- Keep previous versions available for rollback
- Track model performance metrics by version
Example workflow:
1. Train and validate new model version
2. Deploy to staging environment
3. Run A/B test in production (e.g., 95% traffic to v1, 5% to v2)
4. Monitor metrics for new version
5. Gradually shift traffic if metrics are good
6. Keep previous version available for rollback
### **128. How do you handle model drift in production?**
Detection and mitigation:
- Monitor input data distributions
- Track model performance metrics
- Implement statistical tests for drift
- Set up alerts for significant changes
- Retrain models on new data
Example drift detection:
```python
from scipy import stats
def detect_drift(new_data, reference_data, threshold=0.05):
# KS test for continuous features
drift_detected = False
for i in range(new_data.shape[1]):
stat, pvalue = stats.ks_2samp(reference_data[:, i], new_data[:, i])
if pvalue < threshold:
print(f"Drift detected in feature {i}")
drift_detected = True
return drift_detected
```
### **129. How do you implement continuous training for PyTorch models?**
Continuous training pipeline:
1. Monitor data pipeline for new data
2. Trigger retraining when sufficient data accumulates
3. Validate new model against current production model
4. Deploy new model if it meets quality criteria
5. Update reference data for drift detection
Example with Airflow:
```python
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
def train_model(**kwargs):
# Load latest data
data = load_data()
# Train model
model = train(data)
# Validate model
if validate(model):
# Save model
save_model(model)
return "Model trained and validated"
else:
raise Exception("Model validation failed")
dag = DAG('continuous_training', schedule_interval='@daily')
train_task = PythonOperator(
task_id='train_model',
python_callable=train_model,
dag=dag
)
```
### **130. How do you secure a deployed PyTorch model API?**
Security measures:
- Use HTTPS/TLS for encryption
- Implement authentication (API keys, OAuth)
- Validate and sanitize inputs
- Rate limiting to prevent abuse
- Monitor for anomalous requests
- Keep dependencies up to date
Example with FastAPI:
```python
from fastapi import Depends, FastAPI, HTTPException
from fastapi.security import APIKeyHeader
app = FastAPI()
API_KEY = "secret-key"
API_KEY_NAME = "X-API-Key"
api_key_header = APIKeyHeader(name=API_KEY_NAME, auto_error=False)
async def get_api_key(api_key: str = Depends(api_key_header)):
if api_key != API_KEY:
raise HTTPException(
status_code=403, detail="Could not validate API key"
)
return api_key
@app.post("/predict/", dependencies=[Depends(get_api_key)])
async def predict(file: UploadFile = File(...)):
# Prediction code
pass
```
---
## **Recent Developments**
### **131. What are the key features of PyTorch 2.0?**
PyTorch 2.0 introduced:
1. **`torch.compile()`**: Just-in-time compiler for performance optimization
```python
model = torch.compile(model)
```
2. **Improved ONNX export**: Better support for dynamic shapes
3. **Better GPU performance**: Optimizations for NVIDIA GPUs
4. **Enhanced distributed training**: Improved FSDP (Fully Sharded Data Parallel)
5. **Better Windows support**: More complete Windows experience
6. **New memory format**: Channels-last memory format for vision models
### **132. What is FSDP (Fully Sharded Data Parallel) and how does it differ from DDP?**
FSDP:
- Shards model parameters, gradients, and optimizer states across devices
- Reduces memory usage per device
- Enables training very large models
- Better memory efficiency than DDP
DDP:
- Replicates entire model on each device
- Only shards data, not model parameters
- Simpler but less memory efficient
Example FSDP usage:
```python
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = MyModel()
model = FSDP(model)
# Training loop
for inputs, labels in train_loader:
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
```
### **133. What is the state of PyTorch on Apple Silicon (M1/M2 chips)?**
As of 2023:
- Official PyTorch support for Apple Silicon (via MPS backend)
- Good performance for many workloads
- Not all operations supported yet
- Installation:
```bash
conda install pytorch torchvision torchaudio -c pytorch
```
Example usage:
```python
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
model = MyModel().to(device)
```
### **134. What are the latest developments in PyTorch for distributed training?**
Recent improvements:
- **FSDP (Fully Sharded Data Parallel)**: Memory-efficient distributed training
- **Zero Redundancy Optimizer (ZeRO)**: Memory optimization techniques
- **Pipeline parallelism**: Splitting models across devices
- **Better TPU support**: Through integration with XLA
- **Improved fault tolerance**: For long-running distributed jobs
### **135. How is PyTorch evolving for large language model training?**
PyTorch developments for LLMs:
- **FSDP**: For memory-efficient large model training
- **Activation checkpointing**: To reduce memory usage
- **Better mixed precision**: For training stability
- **Improved distributed communication**: For faster training
- **Integration with Hugging Face**: For model sharing and training
### **136. What is TorchDynamo and how does it work?**
TorchDynamo:
- A Python-level compiler for PyTorch
- Part of PyTorch 2.0's `torch.compile()`
- Analyzes Python bytecode to extract computation graphs
- Works with most Python features
- Can target multiple backends (NVFuser, Triton, etc.)
How it works:
1. Intercepts Python bytecode execution
2. Extracts computation graphs from Python control flow
3. Optimizes and compiles graphs
4. Caches compiled graphs for reuse
### **137. What is the role of Triton in PyTorch's ecosystem?**
Triton:
- Open-source programming language for GPU kernel development
- Designed to be accessible to ML researchers
- Used by PyTorch for custom kernels
- Enables writing high-performance GPU code without CUDA
Benefits:
- Easier than writing CUDA
- Automatic optimization
- Good performance
- Integration with PyTorch
### **138. How is PyTorch supporting sparse models and operations?**
PyTorch sparse support:
- `torch.sparse` module for sparse tensors
- Support for COO and CSR formats
- Sparse-dense matrix multiplication
- Sparse optimizers
- Growing support for sparse training
Example:
```python
# Create sparse tensor
i = torch.tensor([[0, 1, 1], [2, 0, 2]])
v = torch.tensor([3, 4, 5], dtype=torch.float32)
sparse_tensor = torch.sparse_coo_tensor(i, v, [2, 4])
# Sparse-dense multiplication
dense_tensor = torch.randn(4, 3)
result = torch.sparse.mm(sparse_tensor, dense_tensor)
```
### **139. What are the latest developments in PyTorch for reinforcement learning?**
Recent RL developments:
- Better support for parallel environments
- Integration with RL libraries like RLlib
- Improved support for custom environments
- Better GPU acceleration for RL algorithms
- Growing ecosystem of RL algorithms
### **140. How is PyTorch evolving for graph neural networks?**
PyTorch GNN developments:
- Better integration with PyTorch Geometric
- Improved support for dynamic graphs
- Better performance for large graphs
- Growing collection of GNN architectures
- Improved distributed training for GNNs
---
## **Conclusion**
This comprehensive list of 140 PyTorch interview questions covers the essential concepts, practical implementations, and advanced features you'll need to know for PyTorch interviews. While there are many more possible questions, these represent the most important and frequently asked topics.
Remember that understanding the underlying concepts is more important than memorizing specific answers. During interviews, interviewers often value your problem-solving approach and understanding of fundamentals more than rote memorization.
For continued learning, consider:
- Reading the PyTorch documentation thoroughly
- Implementing models from research papers
- Contributing to open-source PyTorch projects
- Following PyTorch's GitHub repository for the latest developments
Good luck with your interviews!
Sign in with Wallet
Connect another wallet