Multiplication in NumPy: A Practical Guide
0) Mental model (fast)
Use '@' / [Link] for linear algebra (matrix × vector/matrix), including batched
matmuls.
Use '*' / [Link] for element-wise (Hadamard) operations like gating/masks.
Use [Link] mainly for vector inner products (or legacy matrix multiply).
Use [Link] to build a rank-1 matrix from two vectors.
Use broadcasting to apply per-feature/per-time scaling or bias across batches.
For higher-rank tensors, use [Link] or [Link].
1) Element-wise (Hadamard) multiplication
What: Multiply entries one-by-one, without mixing features or examples.
NumPy: A * B or [Link](A, B)
Shapes: Same shape, or broadcastable to the same shape.
import numpy as np
a = [Link]([1, 2, 3])
b = [Link]([10, 20, 30])
a * b # → array([10, 40, 90])
A = [Link]([[1, 2, 3],
[4, 5, 6]]) # (2,3)
S = [Link]([[ 1, 10, 100],
[ 1, 10, 100]]) # (2,3)
A * S
# → [[1, 20, 300],
# [4, 50, 600]]
Use for: LSTM/GRU gates (ft * c_prev, it * c~t, ot * tanh(c_next)), masks, feature-wise
scaling.
Pitfall: Don’t replace with [Link] — that mixes units.
2) Dot product (inner product of vectors)
What: Multiply corresponding entries and sum to a scalar.
NumPy: [Link](a, b) or a @ b for 1-D vectors.
Shapes: (n,) · (n,) → () (scalar)
a = [Link]([1, 2, 3])
b = [Link]([10, 20, 30])
[Link](a, b) # or a @ b
# → 140
Use for: Similarity, projections, components of matrix multiply.
Note: For 2-D arrays, [Link] does matrix multiply; prefer '@' for clarity.
3) Matrix–vector / Matrix–matrix multiplication
What: Linear algebra product (row × column, sum over shared dimension).
NumPy: A @ B or [Link](A, B)
Shapes: (m×k) @ (k×n) → (m×n)
A = [Link]([[1, 2, 3],
[4, 5, 6]]) # (2,3)
b = [Link]([10, 20, 30]) # (3,)
A @ b
# → array([140, 320])
B = [Link]([[1, 0],
[0, 1],
[1, 1]]) # (3,2)
A @ B
# → [[4, 5],
# [10,11]]
Use for: Linear layers (W @ x + b), RNN/LSTM/GRU gate pre-activations.
Dot vs matmul: matmul/@ supports batched matmuls with broadcasting; dot has different
N-D rules and is less explicit.
4) Outer product
What: All pairwise products of two vectors → rank-1 matrix.
NumPy: [Link](a, b)
Shapes: (m,) ⊗ (n,) → (m, n)
a = [Link]([1, 2, 3])
b = [Link]([10, 20])
[Link](a, b)
# → [[10, 20],
# [20, 40],
# [30, 60]]
Use for: Constructing rank-1 matrices, factorizations. Not a substitute for matrix multiply.
5) Scalar multiplication
What: Multiply every element by a scalar.
3 * [Link]([[1,2,3],[4,5,6]])
# → [[ 3, 6, 9],
# [12, 15, 18]]
Use for: Rescaling activations, losses, gradients.
6) Broadcasting (element-wise with automatic expansion)
What: NumPy auto-expands dimensions of size 1 to match the other operand.
Rules: Compare shapes from right to left; two dimensions are compatible if equal or one is
1; missing leading dims are treated as 1.
A = [Link]([[1, 2, 3],
[4, 5, 6]]) # (2,3)
v = [Link]([1, 10, 100]) # (3,)
A * v
# → [[ 1, 20, 300],
# [ 4, 50, 600]]
M = [Link]((5,1)) # (5,1)
w = [Link]([1,2,3]) # (3,)
M * w
# → shape (5,3), each row is [1,2,3]
Use for: Feature-wise scale/bias applied across batch/time, e.g., Z + b with [Link] == (n,1).
7) Cross product (3-D vectors)
What: Vector orthogonal to both inputs (right-hand rule).
NumPy: [Link](u, v)
Shapes: (3,) × (3,) → (3,)
[Link]([1,0,0], [0,1,0])
# → [0,0,1]
Use for: Geometry/physics; rarely used in standard DL layers.
8) Kronecker product (bonus)
What: Block-wise multiplication of matrices.
NumPy: [Link](A, B)
Use for: Certain signal processing / structured linear algebra.
9) Tensor contractions (higher-rank)
What: Generalized multiplications that sum over specified axes.
[Link] example: X:(batch,T,d), W:(d,h) → (batch,T,h) by contracting over d
[Link](X, W, axes=([2],[0])) # → (batch, T, h)
[Link] example (explicit, readable subscripts):
# Same as tensordot above:
[Link]('btd,dh->bth', X, W)
# Attention scores: Q:(b,h,t,d), K:(b,h,s,d) → scores:(b,h,t,s)
[Link]('bhtd,bhsd->bhts', Q, K)
Use for: Batched matmuls, attention, multi-axis reductions. Tip: start with tensordot, switch
to einsum for clarity/perf.
10) LSTM/GRU tie-in (why Hadamard matters)
LSTM per time step: use '@' for gate pre-activations and '*' (Hadamard) to apply gates per
unit.
concat = [Link]((a_prev, xt), axis=0) # (n_a+n_x, m)
ft = sigmoid(Wf @ concat + bf) # (n_a, m)
it = sigmoid(Wi @ concat + bi) # (n_a, m)
cct = [Link] (Wc @ concat + bc) # (n_a, m)
c_next = ft * c_prev + it * cct # ELEMENT-WISE (Hadamard)
ot = sigmoid(Wo @ concat + bo) # (n_a, m)
a_next = ot * [Link](c_next) # ELEMENT-WISE (Hadamard)
GRU is similar: element-wise gates like z_t * a_prev + (1 - z_t) * a~_t.
11) Common pitfalls & quick fixes
Confusing '*' and '@': linear layer/gate pre-act → '@'; gating/masking → '*'
Silent broadcasting bugs: check shapes; consider np.broadcast_shapes([Link],
[Link])
Row vs column vectors: (n,) vs (n,1) vs (1,n); keep biases as (n,1) to broadcast across
batch m
Wrong concat axis in RNNs: (n_a,m) with (n_x,m) → concatenate along axis=0
[Link] on higher-rank arrays: prefer '@'/matmul for matrices/batches; clearer
semantics
Summation losing a dimension: use keepdims=True to preserve axes (e.g., [Link](axis=1,
keepdims=True))
12) Quick “which one to use?” guide
Per-unit gating/mask → element-wise '*'
Single similarity score of two vectors → a @ b or [Link](a, b)
Affine/gate pre-activations → W @ x + b
Feature-wise bias/scale across batch → broadcasting with + / *
Make a rank-1 matrix from vectors → [Link](a, b)
Batched multi-axis ops → [Link] / [Link]