Vectors, matrices, and transformations - the language of neural networks
Linear algebra is the mathematical foundation of all machine learning. Every neural network, every transformation, every data representation relies on vectors and matrices.
| # | Topic | Skill |
|---|---|---|
| 1 | Vectors | Create, add, and scale vectors |
| 2 | Dot Product | Compute similarity and projections |
| 3 | Matrices | Multiply matrices and understand shapes |
| 4 | Transpose | Flip rows and columns |
| 5 | Determinant | Check if matrix is invertible |
| 6 | Inverse | Solve linear systems Ax = b |
| 7 | Eigenvalues | Understand PCA dimensionality reduction |
| 8 | NumPy | Implement all operations in code |
Before we dive in, let's decode the symbols you'll see:
Vectors and Scalars:
Operations:
Greek Letters (commonly used):
Special Symbols:
In AI, everything is a vector or matrix:
What is a Vector? A vector is an ordered list of numbers representing a point or direction in space. In AI, a 300-dimensional word embedding is a vector with 300 numbers.
Example: v = [0.2, -0.5, 0.8] is a 3D vector
Vector Operations:
Addition: [1, 2] + [3, 4] = [4, 6] - combine vectors
Scalar Multiplication: 2 × [1, 2] = [2, 4] - scale a vector
Dot Product: [1, 2] · [3, 4] = 1×3 + 2×4 = 11 - measures alignment
Vector Magnitude (Length): ||v|| = √(v₁² + v₂² + ... + vₙ²)
For [3, 4]: √(3² + 4²) = √25 = 5
Unit Vectors (Normalization): Divide by magnitude to get length 1: v̂ = v / ||v||
Why normalize? When comparing semantic similarity, we care about direction (meaning), not magnitude.
Cosine Similarity: cos(θ) = (A · B) / (||A|| × ||B||)
Range: -1 (opposite) to +1 (same direction)
Interactive Example: Compare word embeddings:
This is why RAG systems use cosine similarity to find relevant documents!
Vector Spaces and Dimensions:
Key insight: More dimensions = more information capacity
What is a Matrix? A rectangular array of numbers. In AI, matrices represent:
Example: Matrix A is a 2x2 matrix with values [[1, 2], [3, 4]]
This is a 2×2 matrix (2 rows, 2 columns)
Matrix Multiplication: The core operation in deep learning. To multiply A×B:
Example: (2×3) × (3×2) → (2×2) result
Step-by-Step Visual Example:
A = [[1, 2, 3], B = [[7, 8], Result = [[?, ?], [4, 5, 6]] [9, 10], [?, ?]] [11, 12]]
To get result[0,0] (first row, first column):
To get result[0,1]:
Final: [[58, 64], [139, 154]]
The Pattern: Each element = dot product of corresponding row and column!
Why Matrix Multiplication Matters: Every neural network layer does: output = input × weights + bias
For a layer transforming 512D → 256D:
Matrix Transpose: Flip rows and columns: Aᵀ
If A = [[1, 2], [3, 4]] then Aᵀ = [[1, 3], [2, 4]]
Used in backpropagation to reverse gradient flow.
Identity Matrix (I): Diagonal 1s, rest 0s. Acts like multiplying by 1.
I = [[1, 0], [0, 1]] A × I = I × A = A
Matrix Inverse: Undo a transformation: A × A⁻¹ = I
Used in solving linear systems and some optimization algorithms.
Eigenvalues and Eigenvectors: Special vectors that only get scaled (not rotated) by a matrix: A × v = λ × v
Where:
Visual Intuition: Imagine stretching a rubber sheet:
Real Example: Covariance matrix of data:
PCA uses this to reduce 1000 dimensions to 2 for visualization!
Try It: Matrix [[3, 1], [0, 2]] has eigenvector [1, 0] with eigenvalue 3
Matrix Rank: Number of linearly independent rows/columns.
Full rank = maximum information, no redundancy. Low rank = compressed representation (useful in matrix factorization).
Tensor Operations: Tensors are multi-dimensional arrays:
PyTorch and TensorFlow operate on tensors.
Word Embeddings (Vector Arithmetic):
king - man + woman ≈ queen
Because embeddings capture semantic relationships as geometric relationships in vector space.
Neural Network Layers:
Each layer is matrix multiplication: output = input @ weights + bias (@ is matrix multiplication in Python)
Attention Mechanisms (Transformers):
Attention(Q, K, V) = softmax(Q × Kᵀ / √d) × V
Where Q, K, V are matrices derived from input embeddings.
Batch Processing: Process multiple inputs simultaneously using matrix operations:
This is why GPUs are fast - optimized for matrix multiplication.
Dimensionality Reduction (PCA):
Reduces 10,000D features to 100D while keeping most information.
Recommendation Systems: Matrix factorization: R ≈ U × Vᵀ
Finds latent features explaining preferences.
Let's see how these concepts translate to Python code you'll write daily:
Scalars - Simple Numbers:
import numpy as np
# Scalars in AI
learning_rate = 0.001
model_accuracy = 0.95
loss_value = 2.34
# In Python, just regular float or int
print(type(learning_rate)) # <class 'float'>
Vectors - 1D Arrays:
# Customer features: [age, income, years_as_customer]
customer = np.array([35, 75000, 3])
# Word embedding (simplified)
word_embedding = np.array([0.2, -0.5, 0.8, 0.1, -0.3])
print(f"Customer shape: {customer.shape}") # (3,)
print(f"Embedding shape: {word_embedding.shape}") # (5,)
Matrices - 2D Arrays:
# Dataset: 3 customers, 4 features each
data = np.array([
[25, 50000, 1, 0], # customer 1
[35, 75000, 3, 1], # customer 2
[45, 90000, 5, 1] # customer 3
])
# Neural network weights
weights = np.random.randn(4, 8) # 4 inputs to 8 neurons
print(f"Data shape: {data.shape}") # (3, 4)
print(f"Weights shape: {weights.shape}") # (4, 8)
Dot Product - Core Calculation:
# Features and weights
features = np.array([1.0, 2.0, 3.0])
weights = np.array([0.5, 0.3, 0.2])
# Method 1: Using np.dot
output1 = np.dot(features, weights)
# Method 2: Using @ operator (preferred)
output2 = features @ weights
# Method 3: Manual calculation
output3 = np.sum(features * weights)
print(f"All equal: {output1} = {output2} = {output3}") # 1.7
Matrix Multiplication - Neural Network Forward Pass:
# Batch of 32 samples, each with 512 features
input_data = np.random.randn(32, 512)
# Layer weights: transform 512 -> 256 dimensions
layer_weights = np.random.randn(512, 256)
bias = np.random.randn(256)
# Forward pass through layer
layer_output = input_data @ layer_weights + bias
print(f"Input shape: {input_data.shape}") # (32, 512)
print(f"Weights shape: {layer_weights.shape}") # (512, 256)
print(f"Output shape: {layer_output.shape}") # (32, 256)
Transpose - Backpropagation:
A = np.array([[1, 2], [3, 4]])
# Transpose using .T
A_T = A.T
print(f"Original:
{A}")
print(f"Transposed:
{A_T}")
# Common in backprop: gradient flows backward
gradient = np.random.randn(32, 256)
weights_gradient = input_data.T @ gradient # (512, 32) @ (32, 256) = (512, 256)
Vector Operations - All at Once:
v = np.array([3, 4])
w = np.array([1, 2])
# Addition
v_plus_w = v + w
print(f"v + w = {v_plus_w}") # [4, 6]
# Scalar multiplication
scaled = 2 * v
print(f"2 * v = {scaled}") # [6, 8]
# Dot product
dot = v @ w
print(f"v · w = {dot}") # 11
# Magnitude (L2 norm)
magnitude = np.linalg.norm(v)
print(f"||v|| = {magnitude}") # 5.0
# Normalization
unit_v = v / np.linalg.norm(v)
print(f"v̂ = {unit_v}") # [0.6, 0.8]
print(f"||v̂|| = {np.linalg.norm(unit_v)}") # 1.0
Cosine Similarity - Text/Embedding Comparison:
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# Word embeddings (simplified)
king = np.array([0.5, 0.8, 0.2])
queen = np.array([0.6, 0.7, 0.3])
dog = np.array([-0.3, 0.1, 0.9])
print(f"king-queen similarity: {cosine_similarity(king, queen):.3f}") # ~0.97
print(f"king-dog similarity: {cosine_similarity(king, dog):.3f}") # ~0.12
1. Dimension Mismatches:
A = np.random.randn(3, 4)
B = np.random.randn(5, 3)
# This will ERROR: shapes (3,4) and (5,3) not aligned
try:
result = A @ B
except ValueError as e:
print(f"Error: {e}")
# ALWAYS check shapes first
print(f"A shape: {A.shape}, B shape: {B.shape}")
# Fix: transpose B so (5,3) -> (3,5), then (3,4) @ (4,3) works
B_correct = np.random.randn(4, 5)
result = A @ B_correct # (3,4) @ (4,5) = (3,5)
2. Element-wise vs Matrix Multiplication:
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
# Element-wise multiplication (Hadamard product)
element_wise = A * B # [[1*5, 2*6], [3*7, 4*8]] = [[5, 12], [21, 32]]
# Matrix multiplication
matrix_mult = A @ B # [[1*5+2*7, 1*6+2*8], [3*5+4*7, 3*6+4*8]] = [[19, 22], [43, 50]]
print(f"Element-wise (A * B):
{element_wise}")
print(f"Matrix mult (A @ B):
{matrix_mult}")
# Use the RIGHT one for your math!
# Neural networks need @ (matrix mult), not * (element-wise)
3. Forgetting Transposes:
# Computing covariance requires transpose
X = np.random.randn(100, 5) # 100 samples, 5 features
# WRONG: X @ X gives (100,5) @ (100,5) - dimension error!
# RIGHT: X.T @ X gives (5,100) @ (100,5) = (5,5) covariance matrix
cov_matrix = X.T @ X / 100
print(f"Covariance shape: {cov_matrix.shape}") # (5, 5)
4. Broadcasting Surprises:
# Adding bias to network layer
output = np.random.randn(32, 10) # 32 samples, 10 neurons
bias = np.random.randn(10) # 10 bias values
# This works! NumPy broadcasts bias across all 32 samples
result = output + bias # (32,10) + (10,) = (32,10)
# But this might not do what you expect
wrong_bias = np.random.randn(32, 1)
result2 = output + wrong_bias # (32,10) + (32,1) broadcasts to (32,10)
# Each sample gets DIFFERENT bias - usually not intended!
Pro Tips:
One-liner for each concept:
| Concept | Key Takeaway |
|---|---|
| Scalar | Single number (5, 0.001, -3.14) |
| Vector | Ordered list of numbers representing direction + magnitude |
| Matrix | 2D array - used for data batches and neural network weights |
| Dot Product | Measures alignment: positive=same direction, zero=perpendicular |
| Magnitude | Vector length: ||v|| = √(v₁² + v₂² + ...) |
| Unit Vector | Normalized to length 1: v̂ = v / ||v|| |
| Cosine Similarity | Range -1 to +1, measures semantic similarity in embeddings |
| Matrix Multiply | (m,n) @ (n,p) → (m,p) - core neural network operation |
| Transpose | Flip rows/columns: A.T swaps (m,n) → (n,m) |
| Eigenvalue/vector | Av = λv - used in PCA for dimensionality reduction |
Essential Commands:
# Vector operations
v = np.array([3, 4])
np.linalg.norm(v) # Magnitude: 5.0
v / np.linalg.norm(v) # Unit vector: [0.6, 0.8]
a @ b # Dot product (scalar result)
# Matrix operations
A @ B # Matrix multiplication
A.T # Transpose
A.shape # Check dimensions (rows, cols)
# Cosine similarity
np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
The Golden Rules:
Video Courses:
Books:
Interactive Visualizations:
GitHub Repositories:
Papers: