Physics-Informed Neural Networks

A Comprehensive Study Guide with PyTorch

“The idea is to encode the governing physical laws directly into the
loss function of a neural network, so that the network is forced to
respect the physics of the problem.”
— Raissi, Perdikaris & Karniadakis, 2019

Topics Covered
ˆ Mathematical foundations of PINNs

ˆ Automatic differentiation with PyTorch

ˆ Loss function formulation

ˆ Network architecture design

ˆ Training strategies and optimizers

ˆ Case studies: Heat, Burgers, Schrödinger equations

ˆ Failure modes and debugging

ˆ Advanced topics: domain decomposition, adaptive sam-

pling

PyTorch ≥ 2.0 — Python ≥ 3.9 — 2024

Deep Learning for PDEs Physics-Informed Neural Networks

Contents

1 Introduction to Physics-Informed Neural Networks 2

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Core Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Comparison with Classical Methods . . . . . . . . . . . . . . . . . . . . . . 2

2 Mathematical Foundations 2
2.1 Universal Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Collocation-Based Residual . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 The PINN Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.4 Automatic Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 PyTorch Implementation: Foundations 3

3.1 Environment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2 Automatic Differentiation for PDEs . . . . . . . . . . . . . . . . . . . . . . 4
3.3 The Neural Network Architecture . . . . . . . . . . . . . . . . . . . . . . . 5
3.3.1 Standard MLP for PINNs . . . . . . . . . . . . . . . . . . . . . . . 5
3.3.2 Modified MLP (mPINN) . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3.3 Fourier Feature Network . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Case Study 1: The Heat Equation 8

4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2 Full PyTorch Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 8

5 Case Study 2: Burgers’ Equation 10

5.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Two-Phase Training: Adam + L-BFGS . . . . . . . . . . . . . . . . . . . . 11

6 Training Strategies and Best Practices 12

6.1 Loss Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
6.1.1 Manual Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.1.2 NTK-Based Adaptive Weighting . . . . . . . . . . . . . . . . . . . . 13
6.2 Collocation Point Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.2.1 Latin Hypercube Sampling . . . . . . . . . . . . . . . . . . . . . . . 13
6.2.2 Residual-Based Adaptive Sampling (RAD) . . . . . . . . . . . . . . 14
6.3 Learning Rate Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7 Inverse Problems 15
7.1 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

8 Failure Modes and Debugging 16

8.1 Common Failure Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.2 Input/Output Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.3 Gradient Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
8.4 Monitoring with TensorBoard . . . . . . . . . . . . . . . . . . . . . . . . . 18

1
Deep Learning for PDEs Physics-Informed Neural Networks

9 Advanced Topics 18
9.1 Causal Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.2 Domain Decomposition (XPINNs) . . . . . . . . . . . . . . . . . . . . . . . 19
9.3 SIREN: Sinusoidal Representation Networks . . . . . . . . . . . . . . . . . 19

10 Evaluation and Visualization 20

10.1 Relative L2 Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.2 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

11 Quick Reference Checklist 22

2
Deep Learning for PDEs Physics-Informed Neural Networks

1 Introduction to Physics-Informed Neural Networks

1.1 Motivation
Classical numerical methods (FEM, FDM, FVM) solve partial differential equations by
discretizing the domain into a mesh. While powerful, they face challenges:

ˆ High computational cost in 3D or time-dependent problems

ˆ Difficulty with irregular geometries

ˆ Poor scalability to inverse problems

ˆ Inability to naturally incorporate observational data

Physics-Informed Neural Networks (PINNs), introduced by Raissi et al. (2019), offer

a mesh-free alternative that uses neural networks as universal function approximators
while enforcing physical laws through the loss function.

1.2 Core Idea

Given a general PDE:

N [u](x, t) = f (x, t), (x, t) ∈ Ω × [0, T ] (1)

with boundary conditions B[u] = g on ∂Ω and initial condition u(x, 0) = u0 (x),

we approximate u(x, t) ≈ uθ (x, t) where uθ is a neural network with parameters θ.
The key insight: automatic differentiation allows us to compute N [uθ ] exactly (up
to floating-point precision), so we can penalize PDE residuals directly in the loss.

1.3 Comparison with Classical Methods

Property Classical (FEM/FDM) PINN

Domain discretization Required (mesh) Not required
Handles irregular geometry Difficult Natural
Inverse problems Requires re-solve Unified framework
Data assimilation Not native Natural
Convergence guarantees Strong Limited
High dimensions Exponential cost Feasible
Accuracy Very high Moderate

2 Mathematical Foundations
2.1 Universal Approximation

Universal Approximation Theorem (informal)

For any continuous function f : Rn → R on a compact set and ε > 0, there exists

3
Deep Learning for PDEs Physics-Informed Neural Networks

a neural network uθ with a single hidden layer such that:

sup |f (x) − uθ (x)| < ε

x∈K

This guarantees that the solution u can in principle be approximated by uθ .

2.2 Collocation-Based Residual

The PDE residual at a collocation point (xr , tr ) is:

r(xr , tr ; θ) = N [uθ ](xr , tr ) − f (xr , tr ) (2)

The PINN enforces this to be zero at a finite set of collocation points {(xir , tir )}N r
i=1
sampled from Ω × [0, T ].

2.3 The PINN Loss Function

The total loss is a weighted sum of three terms:

L(θ) = λr Lr (θ) + λbc Lbc (θ) + λic Lic (θ) (3)

where:
Nr
1 X 2
Lr (θ) = r(xir , tir ; θ) (PDE residual) (4)
Nr i=1
Nbc
1 X 2
Lbc (θ) = uθ (xjbc , tjbc ) − g(xjbc , tjbc ) (boundary condition) (5)
Nbc j=1

Nic
1 X 2
Lic (θ) = uθ (xkic , 0) − u0 (xkic ) (initial condition) (6)
Nic k=1

The weights λr , λbc , λic > 0 balance the competing objectives.

2.4 Automatic Differentiation

Unlike finite differences, AD computes exact derivatives of uθ through the computational
graph. In PyTorch, this is done via [Link].
For a scalar output u and input x:
∂uθ
= [Link](u, x, create graph=True) (7)
∂x
The flag create graph=True is critical — it allows computing higher-order deriva-
tives by keeping the gradient computation in the graph.

3 PyTorch Implementation: Foundations

3.1 Environment Setup

4
Deep Learning for PDEs Physics-Informed Neural Networks

Install dependencies

1 # Install required packages

2 # pip install torch torchvision numpy matplotlib scipy
3

4 import torch
5 import torch . nn as nn
6 import numpy as np
7 import matplotlib . pyplot as plt
8 from torch . optim import Adam , LBFGS
9 from torch . optim . lr_scheduler import StepLR ,
CosineAnne alingL R
10

11 # Set device
12 device = torch . device ( ’ cuda ’ if torch . cuda . is_available ()
else ’ cpu ’)
13 print ( f " Using device : { device } " )
14

15 # Set random seed for reproducibility

16 torch . manual_seed (42)
17 np . random . seed (42)

3.2 Automatic Differentiation for PDEs

Computing PDE derivatives with autograd

1 def gradient ( outputs , inputs , create_graph = True ) :

2 """ Compute first - order gradient via autograd . """
3 return torch . autograd . grad (
4 outputs , inputs ,
5 grad_outputs = torch . ones_like ( outputs ) ,
6 create_graph = create_graph ,
7 retain_graph = True
8 ) [0]
9

10 def laplacian (u , x ) :
11 """ Compute Laplacian d ^2 u / dx ^2. """
12 u_x = gradient (u , x ) # first derivative
13 u_xx = gradient ( u_x , x ) # second derivative
14 return u_xx
15

16 # Example : compute u_t and u_xx from a network output

17 x = torch . linspace ( -1 , 1 ,
100) . unsqueeze (1) . requires_grad_ ( True )
18 t = torch . linspace (0 , 1 ,
100) . unsqueeze (1) . requires_grad_ ( True )
19

20 # Suppose u = network (x , t )
21 # u_t = gradient (u , t )

5
Deep Learning for PDEs Physics-Informed Neural Networks

22 # u_xx = laplacian (u , x )

Important: requires grad

Inputs to the network must have requires grad=True for autograd to track oper-
ations. Forgetting this is the most common source of errors in PINN implementa-
tions.

3.3 The Neural Network Architecture

3.3.1 Standard MLP for PINNs

Standard PINN Network

1 class PINN ( nn . Module ) :
2 """
3 Standard Multi - Layer Perceptron for PINNs .
4 Input : (x , t ) -> Output : u (x , t )
5 """
6 def __init__ ( self , layers , activation = nn . Tanh () ) :
7 super ( PINN , self ) . __init__ ()
8 self . activation = activation
9 self . layers = nn . ModuleList ()
10

11 # Build layers
12 for i in range ( len ( layers ) - 1) :
13 self . layers . append ( nn . Linear ( layers [ i ] ,
layers [ i +1]) )
14

15 # Weight initialization ( Xavier for tanh )

16 self . _init_weights ()
17

18 def _init_weights ( self ) :

19 for layer in self . layers :
20 if isinstance ( layer , nn . Linear ) :
21 nn . init . xavier_normal_ ( layer . weight )
22 nn . init . zeros_ ( layer . bias )
23

24 def forward ( self , x , t ) :

25 # Concatenate inputs
26 inp = torch . cat ([ x , t ] , dim =1)
27

28 # Hidden layers with activation

29 for layer in self . layers [: -1]:
30 inp = self . activation ( layer ( inp ) )
31

32 # Output layer ( no activation )

33 out = self . layers [ -1]( inp )
34 return out

6
Deep Learning for PDEs Physics-Informed Neural Networks

35

36 # Example : 2 inputs , 4 hidden layers of 64 neurons , 1 output

37 layers = [2 , 64 , 64 , 64 , 64 , 1]
38 model = PINN ( layers ) . to ( device )
39 print ( model )
40 print ( f " Parameters : { sum ( p . numel () for p in
model . parameters () ) : ,} " )

3.3.2 Modified MLP (mPINN)

The modified MLP introduced by Wang et al. (2022) improves training by adding two
auxiliary networks U and V :
Modified MLP Architecture
1 class ModifiedMLP ( nn . Module ) :
2 """
3 Modified MLP with encoder branches U and V .
4 Helps mitigate spectral bias ( slow learning of high - freq
features ) .
5 Reference : Wang et al . , 2022
6 """
7 def __init__ ( self , layers , activation = nn . Tanh () ) :
8 super ( ModifiedMLP , self ) . __init__ ()
9 self . act = activation
10 in_dim = layers [0]
11 hidden = layers [1]
12

13 # Encoder branches
14 self . U = nn . Linear ( in_dim , hidden )
15 self . V = nn . Linear ( in_dim , hidden )
16

17 # Hidden layers
18 self . hidden = nn . ModuleList (
19 [ nn . Linear ( hidden , hidden ) for _ in
range ( len ( layers ) - 2) ]
20 )
21 self . out = nn . Linear ( hidden , layers [ -1])
22

23 self . _init_weights ()
24

25 def _init_weights ( self ) :

26 for m in self . modules () :
27 if isinstance (m , nn . Linear ) :
28 nn . init . xavier_normal_ ( m . weight )
29 nn . init . zeros_ ( m . bias )
30

31 def forward ( self , x , t ) :

32 inp = torch . cat ([ x , t ] , dim =1)

7
Deep Learning for PDEs Physics-Informed Neural Networks

33 U = self . act ( self . U ( inp ) )

34 V = self . act ( self . V ( inp ) )
35

36 H = inp
37 for layer in self . hidden :
38 # Multiplicative gate with encoder branches
39 H = self . act ( layer ( H ) ) * U + (1 -
self . act ( layer ( H ) ) ) * V
40

41 return self . out ( H )

3.3.3 Fourier Feature Network

Fourier features help overcome spectral bias — the tendency of MLPs to learn low-
frequency functions first:
Fourier Feature Embedding

1 class Fo ur ie rF ea tu re PI NN ( nn . Module ) :
2 """
3 PINN with random Fourier feature embedding .
4 Helps capture high - frequency solution features .
5 """
6 def __init__ ( self , layers , sigma =1.0) :
7 super ( FourierFeaturePINN , self ) . __init__ ()
8 in_dim = layers [0]
9 embed_dim = layers [1] // 2 # Half for sin , half for
cos
10

11 # Fixed random Fourier features ( not trained )

12 B = torch . randn ( in_dim , embed_dim ) * sigma
13 self . register_buffer ( ’B ’ , B ) # not a parameter
14

15 # MLP after embedding

16 mlp_layers = [ layers [1]] + layers [2:]
17 self . mlp = nn . Sequential (
18 *[
19 nn . Sequential ( nn . Linear ( mlp_layers [ i ] ,
mlp_layers [ i +1]) ,
20 nn . Tanh () )
21 for i in range ( len ( mlp_layers ) - 2)
22 ],
23 nn . Linear ( mlp_layers [ -2] , mlp_layers [ -1])
24 )
25

26 def forward ( self , x , t ) :

27 inp = torch . cat ([ x , t ] , dim =1)
28 # Fourier embedding
29 proj = inp @ self . B

8
Deep Learning for PDEs Physics-Informed Neural Networks

30 embed = torch . cat ([ torch . sin ( proj ) ,

torch . cos ( proj ) ] , dim =1)
31 return self . mlp ( embed )

4 Case Study 1: The Heat Equation

4.1 Problem Formulation
The 1D heat equation:
∂u ∂ 2u
= α 2, x ∈ [−1, 1], t ∈ [0, 1] (8)
∂t ∂x
with:

u(x, 0) = sin(πx) (IC) (9)

u(−1, t) = u(1, t) = 0 (BCs) (10)
2
The analytical solution is u(x, t) = e−απ t sin(πx).

4.2 Full PyTorch Implementation

Heat Equation PINN

1 import torch
2 import torch . nn as nn
3 import numpy as np
4

5 class HeatPINN ( nn . Module ) :

6 def __init__ ( self , alpha =0.01) :
7 super () . __init__ ()
8 self . alpha = alpha
9 self . net = nn . Sequential (
10 nn . Linear (2 , 64) , nn . Tanh () ,
11 nn . Linear (64 , 64) , nn . Tanh () ,
12 nn . Linear (64 , 64) , nn . Tanh () ,
13 nn . Linear (64 , 64) , nn . Tanh () ,
14 nn . Linear (64 , 1)
15 )
16 # Xavier initialization
17 for m in self . net :
18 if isinstance (m , nn . Linear ) :
19 nn . init . xavier_normal_ ( m . weight )
20 nn . init . zeros_ ( m . bias )
21

22 def forward ( self , x , t ) :

23 inp = torch . cat ([ x , t ] , dim =1)
24 return self . net ( inp )
25

9
Deep Learning for PDEs Physics-Informed Neural Networks

26 def pde_residual ( self , x , t ) :

27 u = self . forward (x , t )
28 u_t = torch . autograd . grad (
29 u , t , grad_outputs = torch . ones_like ( u ) ,
30 create_graph = True ) [0]
31 u_x = torch . autograd . grad (
32 u , x , grad_outputs = torch . ones_like ( u ) ,
33 create_graph = True ) [0]
34 u_xx = torch . autograd . grad (
35 u_x , x , grad_outputs = torch . ones_like ( u_x ) ,
36 create_graph = True ) [0]
37 return u_t - self . alpha * u_xx # residual
38

39 def sample_points ( N_r , N_bc , N_ic , device ) :

40 """ Sample collocation , BC , and IC points . """
41 # PDE collocation points ( interior )
42 x_r = ( torch . rand ( N_r , 1) * 2 -
1) . to ( device ) . requires_grad_ ( True )
43 t_r = torch . rand ( N_r , 1) . to ( device ) . requires_grad_ ( True )
44

45 # Boundary : x = -1 and x = 1
46 t_bc = torch . rand ( N_bc , 1) . to ( device )
47 x_bc_l = - torch . ones ( N_bc // 2 , 1) . to ( device )
48 x_bc_r = torch . ones ( N_bc // 2 , 1) . to ( device )
49 x_bc = torch . cat ([ x_bc_l , x_bc_r ] , dim =0)
50 t_bc = torch . cat ([ t_bc , t_bc ] , dim =0)
51 u_bc = torch . zeros ( N_bc , 1) . to ( device )
52

53 # Initial condition : t = 0
54 x_ic = ( torch . rand ( N_ic , 1) * 2 - 1) . to ( device )
55 t_ic = torch . zeros ( N_ic , 1) . to ( device )
56 u_ic = torch . sin ( np . pi * x_ic )
57

58 return ( x_r , t_r ) , ( x_bc , t_bc , u_bc ) , ( x_ic , t_ic , u_ic )

59

60 def train_heat_pinn ( epochs =5000 , lr =1 e -3) :

61 device = torch . device ( ’ cuda ’ if
torch . cuda . is_available () else ’ cpu ’)
62 model = HeatPINN ( alpha =0.01) . to ( device )
63 optimizer = torch . optim . Adam ( model . parameters () , lr = lr )
64 scheduler = torch . optim . lr_scheduler . StepLR (
65 optimizer , step_size =1000 , gamma =0.5)
66

67 # Loss weights
68 lam_r , lam_bc , lam_ic = 1.0 , 10.0 , 10.0
69

70 ( x_r , t_r ) , ( x_bc , t_bc , u_bc ) , ( x_ic , t_ic , u_ic ) = \

71 sample_points (2000 , 200 , 200 , device )
72

73 for epoch in range ( epochs ) :

10
Deep Learning for PDEs Physics-Informed Neural Networks

74 optimizer . zero_grad ()
75

76 # PDE loss
77 res = model . pde_residual ( x_r , t_r )
78 loss_r = torch . mean ( res **2)
79

80 # Boundary loss
81 u_pred_bc = model ( x_bc , t_bc )
82 loss_bc = torch . mean (( u_pred_bc - u_bc ) **2)
83

84 # Initial condition loss

85 u_pred_ic = model ( x_ic , t_ic )
86 loss_ic = torch . mean (( u_pred_ic - u_ic ) **2)
87

88 # Total loss
89 loss = lam_r * loss_r + lam_bc * loss_bc + lam_ic *
loss_ic
90 loss . backward ()
91 optimizer . step ()
92 scheduler . step ()
93

94 if epoch % 500 == 0:
95 print ( f " Epoch { epoch :5 d } | Loss :
{ loss . item () :.4 e } "
96 f " | r : { loss_r . item () :.4 e } "
97 f " | bc : { loss_bc . item () :.4 e } "
98 f " | ic : { loss_ic . item () :.4 e } " )
99

100 return model

101

102 model = train_heat_pinn ( epochs =5000)

5 Case Study 2: Burgers’ Equation

5.1 Problem Formulation
The 1D viscous Burgers’ equation:

∂u ∂u ∂ 2u
+u = ν 2, x ∈ [−1, 1], t ∈ [0, 1] (11)
∂t ∂x ∂x
with:

u(x, 0) = − sin(πx) (12)

u(−1, t) = u(1, t) = 0 (13)

This is a nonlinear PDE that develops a sharp shock at t ≈ 0.5 for small ν, making it
a common PINN benchmark.

11
Deep Learning for PDEs Physics-Informed Neural Networks

5.2 Implementation

Burgers’ Equation Residual

1 class BurgersPINN ( nn . Module ) :

2 def __init__ ( self , nu =0.01/ np . pi ) :
3 super () . __init__ ()
4 self . nu = nu
5 self . net = nn . Sequential (
6 nn . Linear (2 , 100) , nn . Tanh () ,
7 nn . Linear (100 , 100) , nn . Tanh () ,
8 nn . Linear (100 , 100) , nn . Tanh () ,
9 nn . Linear (100 , 100) , nn . Tanh () ,
10 nn . Linear (100 , 100) , nn . Tanh () ,
11 nn . Linear (100 , 1)
12 )
13

14 def forward ( self , x , t ) :

15 return self . net ( torch . cat ([ x , t ] , dim =1) )
16

17 def residual ( self , x , t ) :

18 u = self . forward (x , t )
19

20 u_t = torch . autograd . grad (

21 u , t , torch . ones_like ( u ) , create_graph = True ) [0]
22 u_x = torch . autograd . grad (
23 u , x , torch . ones_like ( u ) , create_graph = True ) [0]
24 u_xx = torch . autograd . grad (
25 u_x , x , torch . ones_like ( u_x ) ,
create_graph = True ) [0]
26

27 # Burgers : u_t + u * u_x - nu * u_xx = 0

28 return u_t + u * u_x - self . nu * u_xx
29

30 def loss_fn ( model , x_r , t_r , x_bc , t_bc , u_bc , x_ic , t_ic ,
u_ic ,
31 lam_r =1.0 , lam_bc =10.0 , lam_ic =10.0) :
32 res = model . residual ( x_r , t_r )
33 loss_r = ( res **2) . mean ()
34 loss_bc = (( model ( x_bc , t_bc ) - u_bc ) **2) . mean ()
35 loss_ic = (( model ( x_ic , t_ic ) - u_ic ) **2) . mean ()
36 return lam_r * loss_r + lam_bc * loss_bc + lam_ic * loss_ic

5.3 Two-Phase Training: Adam + L-BFGS

A common and effective strategy is to use Adam first (fast global exploration) followed
by L-BFGS (accurate local refinement):

12
Deep Learning for PDEs Physics-Informed Neural Networks

Two-Phase Training

1 def two_ ph as e_t ra in in g ( model , data , adam_epochs =5000 ,

lbfgs_epochs =500) :
2 x_r , t_r , x_bc , t_bc , u_bc , x_ic , t_ic , u_ic = data
3

4 # --- Phase 1: Adam ---

5 optimizer = Adam ( model . parameters () , lr =1 e -3)
6 for epoch in range ( adam_epochs ) :
7 optimizer . zero_grad ()
8 loss = loss_fn ( model , x_r , t_r , x_bc , t_bc , u_bc ,
x_ic , t_ic , u_ic )
9 loss . backward ()
10 optimizer . step ()
11 if epoch % 1000 == 0:
12 print ( f " [ Adam ] Epoch { epoch }: Loss =
{ loss . item () :.4 e } " )
13

14 # --- Phase 2: L - BFGS ---

15 optimizer_lbfgs = LBFGS (
16 model . parameters () ,
17 lr =1.0 ,
18 max_iter =50 ,
19 history_size =50 ,
20 tolerance_grad =1 e -7 ,
21 tolerance_change =1 e -9 ,
22 line_search_fn = ’ strong_wolfe ’
23 )
24

25 def closure () :
26 optimizer_lbfgs . zero_grad ()
27 loss = loss_fn ( model , x_r , t_r , x_bc , t_bc , u_bc ,
x_ic , t_ic , u_ic )
28 loss . backward ()
29 return loss
30

31 for epoch in range ( lbfgs_epochs ) :

32 optimizer_lbfgs . step ( closure )
33 if epoch % 100 == 0:
34 loss = closure ()
35 print ( f " [L - BFGS ] Epoch { epoch }: Loss =
{ loss . item () :.4 e } " )
36

37 return model

6 Training Strategies and Best Practices

6.1 Loss Weighting
Poorly balanced loss terms are the #1 cause of PINN failure. Several strategies exist:

13
Deep Learning for PDEs Physics-Informed Neural Networks

6.1.1 Manual Weighting

Set λr , λbc , λic manually. Rule of thumb: upweight BC and IC relative to PDE residual
since they have fewer points but must be satisfied exactly.

6.1.2 NTK-Based Adaptive Weighting

Adaptive Loss Weighting (simplified)

1 def com p u te _ n t k_ w e ig h t s ( model , losses_dict ) :

2 """
3 Estimate gradient magnitudes for each loss term
4 and compute balancing weights .
5 Wang et al . (2021) - NTK - based adaptive weighting .
6 """
7 grad_norms = {}
8 for name , loss in losses_dict . items () :
9 grads = torch . autograd . grad (
10 loss , model . parameters () ,
11 retain_graph = True , allow_unused = True
12 )
13 norm = sum (
14 g . norm () **2 for g in grads if g is not None
15 ) . sqrt ()
16 grad_norms [ name ] = norm . item ()
17

18 total_norm = sum ( grad_norms . values () )

19 weights = { k : total_norm / ( len ( grad_norms ) * v )
20 for k , v in grad_norms . items () }
21 return weights
22

23 # Usage in training loop :

24 # losses = { ’ r ’: loss_r , ’ bc ’: loss_bc , ’ ic ’: loss_ic }
25 # weights = c o m pu t e _n t k _w e i gh t s ( model , losses )
26 # loss = sum ( weights [ k ] * v for k , v in losses . items () )

6.2 Collocation Point Sampling

6.2.1 Latin Hypercube Sampling
Better than pure random for low-discrepancy coverage:
Latin Hypercube Sampling

1 from scipy . stats import qmc

2

3 def la t i n _ h y p e r c u b e _ s a m p l e (N , domain , device ) :

4 """
5 Sample N points from domain using Latin Hypercube
Sampling .

14
Deep Learning for PDEs Physics-Informed Neural Networks

6 domain : list of ( min , max ) tuples per dimension

7 """
8 d = len ( domain )
9 sampler = qmc . LatinHypercube ( d = d )
10 sample = sampler . random ( n = N ) # shape (N , d ) , values in
[0 ,1]
11

12 # Scale to domain
13 l_bounds = [ b [0] for b in domain ]
14 u_bounds = [ b [1] for b in domain ]
15 sample = qmc . scale ( sample , l_bounds , u_bounds )
16

17 return torch . tensor ( sample ,

dtype = torch . float32 ) . to ( device )
18

19 # Example : sample (x , t ) from [ -1 ,1] x [0 ,1]

20 pts = l a t i n _ h y p e r c u b e _ s a m p l e (2000 , [( -1 , 1) , (0 , 1) ] , device )
21 x_r = pts [: , 0:1]. requires_grad_ ( True )
22 t_r = pts [: , 1:2]. requires_grad_ ( True )

6.2.2 Residual-Based Adaptive Sampling (RAD)

Resample points where the PDE residual is largest:
Residual-Adaptive Collocation

1 def res a m p l e _ c o l l o c a t i o n ( model , N_cand , N_keep , domain ,

device ) :
2 """
3 Generate N_cand candidates , keep N_keep with highest
residual .
4 Called every K epochs to adaptively focus on hard
regions .
5 """
6 # Generate candidates
7 pts = l a t i n _ h y p e r c u b e _ s a m p l e ( N_cand , domain , device )
8 x_c = pts [: , 0:1]. requires_grad_ ( True )
9 t_c = pts [: , 1:2]. requires_grad_ ( True )
10

11 # Evaluate residual
12 with torch . no_grad () :
13 res = model . residual ( x_c , t_c ) . abs () . squeeze ()
14

15 # Keep top N_keep by residual magnitude

16 _ , idx = torch . topk ( res , N_keep )
17 x_new = pts [ idx , 0:1]. detach () . requires_grad_ ( True )
18 t_new = pts [ idx , 1:2]. detach () . requires_grad_ ( True )
19 return x_new , t_new

15
Deep Learning for PDEs Physics-Informed Neural Networks

6.3 Learning Rate Scheduling

LR Scheduling Strategies

1 optimizer = Adam ( model . parameters () , lr =1 e -3)

2

3 # Option 1: Step decay

4 scheduler1 = torch . optim . lr_scheduler . StepLR (
5 optimizer , step_size =2000 , gamma =0.5)
6

7 # Option 2: Cosine annealing ( smooth decay )

8 scheduler2 = torch . optim . lr_scheduler . Cosin eAnnea lingLR (
9 optimizer , T_max =10000 , eta_min =1 e -5)
10

11 # Option 3: Reduc eLROnP lateau ( adaptive )

12 scheduler3 = torch . optim . lr_scheduler . Reduc eLROnP lateau (
13 optimizer , mode = ’ min ’ , factor =0.5 ,
14 patience =500 , verbose = True )
15

16 # In training loop :
17 # scheduler1 . step () # for Step and Cosine
18 # scheduler3 . step ( loss . item () ) # for Reduc eLROnP lateau

7 Inverse Problems
7.1 Concept
One of PINN’s most powerful applications: identify unknown PDE parameters from
sparse observations.
For example, given noisy measurements {(xid , tid , uid )} of a Burgers solution with un-
known viscosity ν, we can learn ν simultaneously with the network.

7.2 Implementation

Inverse Problem: Identifying Viscosity

1 class In ve rs eB ur ge rs PI NN ( nn . Module ) :
2 def __init__ ( self ) :
3 super () . __init__ ()
4 self . net = nn . Sequential (
5 nn . Linear (2 , 100) , nn . Tanh () ,
6 nn . Linear (100 , 100) , nn . Tanh () ,
7 nn . Linear (100 , 100) , nn . Tanh () ,
8 nn . Linear (100 , 1)
9 )
10 # nu is a LEARNABLE parameter ( initialized near 0)
11 self . log_nu = nn . Parameter ( torch . tensor ( -5.0) )
12

13 @property

16
Deep Learning for PDEs Physics-Informed Neural Networks

14 def nu ( self ) :
15 return torch . exp ( self . log_nu ) # ensure positivity
16

17 def forward ( self , x , t ) :

18 return self . net ( torch . cat ([ x , t ] , dim =1) )
19

20 def residual ( self , x , t ) :

21 u = self (x , t )
22 u_t = torch . autograd . grad (
23 u , t , torch . ones_like ( u ) , create_graph = True ) [0]
24 u_x = torch . autograd . grad (
25 u , x , torch . ones_like ( u ) , create_graph = True ) [0]
26 u_xx = torch . autograd . grad (
27 u_x , x , torch . ones_like ( u_x ) ,
create_graph = True ) [0]
28 return u_t + u * u_x - self . nu * u_xx
29

30 def train_inverse ( model , x_data , t_data , u_data ,

31 x_r , t_r , epochs =10000) :
32 opt = Adam ( model . parameters () , lr =1 e -3)
33

34 for epoch in range ( epochs ) :

35 opt . zero_grad ()
36

37 # Data fidelity loss ( from observations )

38 u_pred = model ( x_data , t_data )
39 loss_data = (( u_pred - u_data ) **2) . mean ()
40

41 # Physics loss
42 res = model . residual ( x_r , t_r )
43 loss_pde = ( res **2) . mean ()
44

45 loss = loss_data + loss_pde

46 loss . backward ()
47 opt . step ()
48

49 if epoch % 1000 == 0:
50 print ( f " Epoch { epoch } | Loss : { loss . item () :.4 e } "
51 f " | nu : { model . nu . item () :.6 f } " )

8 Failure Modes and Debugging

17
Deep Learning for PDEs Physics-Informed Neural Networks

8.1 Common Failure Modes

Symptom Cause Fix

Loss NaN immedi- Learning rate too high, or Reduce LR, add gradient
ately gradient explosion clipping
Loss stagnates at ∼ Poor initialization, bad Xavier init, rebalance λ
10−1 weighting
BC/IC satisfied but Insufficient collocation Increase Nr , use adaptive
interior wrong points sampling
Correct trend, Missing physical normaliza- Normalize inputs/outputs
wrong amplitude tion to [−1, 1]
Works at t = 0, Temporal causal error Time marching, causal
fails at large t training
Oscillations in solu- Spectral bias Fourier features, SIREN ac-
tion tivations

8.2 Input/Output Normalization

Normalizing Inputs

1 class NormalizedPINN ( nn . Module ) :

2 """ PINN with input normalization for better
conditioning . """
3 def __init__ ( self , layers , x_range , t_range ) :
4 super () . __init__ ()
5 self . x_min , self . x_max = x_range
6 self . t_min , self . t_max = t_range
7 self . net = PINN ( layers )
8

9 def normalize ( self , x , t ) :

10 """ Map inputs to [ -1 , 1]. """
11 x_n = 2.0*( x - self . x_min ) /( self . x_max - self . x_min )
- 1.0
12 t_n = 2.0*( t - self . t_min ) /( self . t_max - self . t_min )
- 1.0
13 return x_n , t_n
14

15 def forward ( self , x , t ) :

16 x_n , t_n = self . normalize (x , t )
17 return self . net ( x_n , t_n )

18
Deep Learning for PDEs Physics-Informed Neural Networks

8.3 Gradient Clipping

Gradient Clipping

1 # After loss . backward () , before optimizer . step () :

2 torch . nn . utils . clip_grad_norm_ ( model . parameters () ,
max_norm =1.0)
3 optimizer . step ()

8.4 Monitoring with TensorBoard

TensorBoard Logging

1 from torch . utils . tensorboard import SummaryWriter

2

3 writer = SummaryWriter ( ’ runs / pinn_experiment ’)

4

5 for epoch in range ( epochs ) :

6 # ... training ...
7 writer . add_scalar ( ’ Loss / total ’ , loss . item () , epoch )
8 writer . add_scalar ( ’ Loss / pde ’ , loss_r . item () , epoch )
9 writer . add_scalar ( ’ Loss / bc ’ , loss_bc . item () , epoch )
10 writer . add_scalar ( ’ Loss / ic ’ , loss_ic . item () , epoch )
11 writer . add_scalar ( ’ LR ’ , optimizer . param_groups [0][ ’ lr ’] ,
epoch )
12

13 writer . close ()
14 # Run : tensorboard -- logdir = runs

9 Advanced Topics
9.1 Causal Training
For long time horizons, enforce that the network learns the solution causally (early time
must be accurate before later time is penalized):
Causal Loss Weighting

1 def causal_loss ( model , x_r , t_r , eps =1.0) :

2 """
3 Weight PDE residuals so that early - time errors
4 suppress late - time gradients .
5 Reference : Wang et al . , 2022 ( Respecting Causality )
6 """
7 res = model . residual ( x_r , t_r ) **2
8

9 # Sort by time
10 t_vals , sort_idx = t_r . squeeze () . sort ()

19
Deep Learning for PDEs Physics-Informed Neural Networks

11 res_sorted = res [ sort_idx ]

12

13 # Causal weight : w_i = exp ( - eps * sum_ {j < i } r_j )

14 cumsum = torch . cumsum ( res_sorted . detach () , dim =0)
15 weights = torch . exp ( - eps * cumsum )
16

17 return ( weights * res_sorted ) . mean ()

9.2 Domain Decomposition (XPINNs)

Split the domain into subdomains and train separate networks, matching solutions at
interfaces:

uθ1 (x, t) ≈ uθ2 (x, t), x ∈ Γ12 (14)

Interface Matching Loss

1 def interface_loss ( model1 , model2 , x_int , t_int ) :

2 """
3 Enforce continuity of solution and flux across interface .
4 """
5 u1 = model1 ( x_int , t_int )
6 u2 = model2 ( x_int , t_int )
7

8 # Solution continuity
9 loss_u = (( u1 - u2 ) **2) . mean ()
10

11 # Flux continuity : du / dx must match

12 u1_x = torch . autograd . grad (
13 u1 , x_int , torch . ones_like ( u1 ) , create_graph = True ) [0]
14 u2_x = torch . autograd . grad (
15 u2 , x_int , torch . ones_like ( u2 ) , create_graph = True ) [0]
16 loss_flux = (( u1_x - u2_x ) **2) . mean ()
17

18 return loss_u + loss_flux

9.3 SIREN: Sinusoidal Representation Networks

SIREN uses sin activations with careful initialization, making them ideal for PINNs solv-
ing oscillatory PDEs:
SIREN Network
1 class SirenLayer ( nn . Module ) :
2 def __init__ ( self , in_dim , out_dim , omega_0 =30.0 ,
is_first = False ) :
3 super () . __init__ ()
4 self . omega_0 = omega_0

20
Deep Learning for PDEs Physics-Informed Neural Networks

5 self . linear = nn . Linear ( in_dim , out_dim )

6 # SIREN - specific initialization
7 with torch . no_grad () :
8 if is_first :
9 self . linear . weight . uniform_ ( -1/ in_dim ,
1/ in_dim )
10 else :
11 bound = np . sqrt (6/ in_dim ) / omega_0
12 self . linear . weight . uniform_ ( - bound , bound )
13

14 def forward ( self , x ) :

15 return torch . sin ( self . omega_0 * self . linear ( x ) )
16

17 class SIREN ( nn . Module ) :

18 def __init__ ( self , layers , omega_0 =30.0) :
19 super () . __init__ ()
20 net = [ SirenLayer ( layers [0] , layers [1] ,
21 omega_0 = omega_0 , is_first = True ) ]
22 for i in range (1 , len ( layers ) -2) :
23 net . append ( SirenLayer ( layers [ i ] , layers [ i +1] ,
omega_0 ) )
24 net . append ( nn . Linear ( layers [ -2] , layers [ -1]) )
25 self . net = nn . Sequential (* net )
26

27 def forward ( self , x , t ) :

28 return self . net ( torch . cat ([ x , t ] , dim =1) )

10 Evaluation and Visualization

10.1 Relative L2 Error

Evaluation Metrics
1 def relativ e_l2_e rror ( u_pred , u_exact ) :
2 """ Standard PINN benchmark metric . """
3 return ( torch . norm ( u_pred - u_exact ) /
4 torch . norm ( u_exact ) ) . item ()
5

6 @torch . no_grad ()
7 def evaluate_model ( model , x_test , t_test , u_exact ) :
8 model . eval ()
9 u_pred = model ( x_test , t_test )
10 err = r elativ e_l2_e rror ( u_pred , u_exact )
11 print ( f " Relative L2 Error : { err :.4 e } " )
12 return u_pred , err

21
Deep Learning for PDEs Physics-Informed Neural Networks

10.2 Visualization

Plotting PINN solution

1 import matplotlib . pyplot as plt

2 import numpy as np
3

4 @torch . no_grad ()
5 def plot_solution ( model , nx =200 , nt =200 , device = ’ cpu ’) :
6 x = np . linspace ( -1 , 1 , nx )
7 t = np . linspace (0 , 1 , nt )
8 X , T = np . meshgrid (x , t )
9

10 x_flat = torch . tensor ( X . ravel () , dtype = torch . float32 ,

11 device = device ) . unsqueeze (1)
12 t_flat = torch . tensor ( T . ravel () , dtype = torch . float32 ,
13 device = device ) . unsqueeze (1)
14

15 u_pred = model ( x_flat , t_flat ) . cpu () . numpy () . reshape ( nt ,

nx )
16

17 fig , axes = plt . subplots (1 , 2 , figsize =(12 , 4) )

18

19 # Heatmap
20 im = axes [0]. pcolormesh (X , T , u_pred , cmap = ’ RdBu_r ’ ,
21 shading = ’ auto ’)
22 plt . colorbar ( im , ax = axes [0])
23 axes [0]. set_xlabel ( ’x ’) ; axes [0]. set_ylabel ( ’t ’)
24 axes [0]. set_title ( ’ PINN Prediction $ u (x , t ) $ ’)
25

26 # Slices
27 for ti , label in zip ([0.25 , 0.5 , 0.75] ,
28 [ ’t =0.25 ’ , ’t =0.5 ’ , ’t =0.75 ’ ]) :
29 idx = int ( ti * nt )
30 axes [1]. plot (x , u_pred [ idx ] , label = label )
31 axes [1]. legend () ; axes [1]. set_xlabel ( ’x ’)
32 axes [1]. set_title ( ’ Solution slices ’)
33

34 plt . tight_layout ()
35 plt . savefig ( ’ pinn_solution . png ’ , dpi =150)
36 plt . show ()

22
Deep Learning for PDEs Physics-Informed Neural Networks

11 Quick Reference Checklist

PINN Implementation Checklist

1. Inputs: Set requires grad=True on all spatial and temporal inputs

2. Derivatives: Use create graph=True for all autograd calls (enables higher-
order)

3. Normalization: Normalize inputs to [−1, 1] and outputs to similar scale

4. Architecture: Start with tanh, 4–6 layers, 64–128 neurons per layer

5. Initialization: Xavier for tanh; SIREN init for sin activations

6. Loss weights: Upweight BC/IC (typically 10× the PDE residual weight)

7. Sampling: Use LHS or quasi-random; increase Nr near complex regions

8. Optimizer: Adam (5k–20k steps) → L-BFGS for refinement

9. Monitor: Track each loss component separately; use TensorBoard

10. Validate: Compare against known analytical solutions; report relative L2 error

References
1. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural
networks: A deep learning framework for solving forward and inverse problems
involving nonlinear partial differential equations. Journal of Computational Physics,
378, 686–707.

2. Wang, S., Teng, Y., & Perdikaris, P. (2021). Understanding and mitigating gradient
flow pathologies in physics-informed neural networks. SIAM Journal on Scientific
Computing, 43(5), A3055–A3081.

3. Wang, S., Yu, X., & Perdikaris, P. (2022). When and why PINNs fail to train: A
neural tangent kernel perspective. Journal of Computational Physics, 449, 110768.

4. Jagtap, A. D., & Karniadakis, G. E. (2020). Extended physics-informed neural

networks (XPINNs): A generalized space-time domain decomposition based deep
learning framework. Communications in Computational Physics, 28(5).

5. Sitzmann, V., et al. (2020). Implicit neural representations with periodic activation
functions. NeurIPS, 33.

6. Tancik, M., et al. (2020). Fourier features let networks learn high frequency func-
tions in low dimensional domains. NeurIPS, 33.

23