0% found this document useful (0 votes)
30 views4 pages

Efficient LLM Model Allocation Algorithm

The document outlines algorithms for efficient partitioning and allocation of large language models (LLMs) across devices, considering factors like model size, device capabilities, and network latency. It includes methods for optimal device selection, resource computation, and handling partition constraints. Additionally, it describes a task-to-expert allocation process to assign incoming tasks to the best expert-node pairs based on execution time and communication latency.

Uploaded by

anikhasan64445
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
30 views4 pages

Efficient LLM Model Allocation Algorithm

The document outlines algorithms for efficient partitioning and allocation of large language models (LLMs) across devices, considering factors like model size, device capabilities, and network latency. It includes methods for optimal device selection, resource computation, and handling partition constraints. Additionally, it describes a task-to-expert allocation process to assign incoming tasks to the best expert-node pairs based on execution time and communication latency.

Uploaded by

anikhasan64445
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Algorithm 1 Efficient LLM Partitioning and Allocation

Require: Set of models M, Set of devices D, Network latency matrix L


Ensure: Optimized allocation of models to devices
1: Sort models M by size or priority
2: for each model m ∈ M do
3: Identify devices Dfit ⊆ D that can fit m entirely
4: if Dfit ̸= ∅ then
5: Assign m to device d∗ = arg max FLOPS(d)
d∈Dfit
6: continue
7: end if
8: Partition m into layers {l1 , l2 , ..., ln }
9: Initialize empty current group G = ∅, previous device dprev = ∅
10: for each layer li in m do
11: Append li to G
12: Compute required resources for G
13: Identify feasible devices Dgroup ⊆ D
14: if Dgroup = ∅ then
15: if dprev = ∅ then
16: Abort: No allocation possible for li
17: end if
18: Allocate G \ {li } to dprev
19: Reset G = {li }
20: end if
21: Select optimal device:
 
−FLOPS(d)
d∗ = arg min + L(dprev , d)
d∈Dgroup 1GPU (d)

22: Assign G to d∗
23: Update dprev = d∗
24: end for
25: end for

1 Proposed Algorithm

1
Algorithm 2 Efficient LLM Partitioning and Allocation
Require: Set of models M, Set of devices D, Network latency matrix L
Ensure: Optimized allocation of models to devices
1: Sort models M by size or priority
2: for each model m ∈ M do
3: Identify devices Dfit ⊆ D that can fit m entirely
4: if Dfit ̸= ∅ then
5: Assign m to device d∗ = arg max FLOPS(d)
d∈Dfit
6: continue
7: end if
8: Partition m into layers {l1 , l2 , ..., ln }
9: Initialize empty current group G = ∅, previous device dprev = ∅
10: for each layer li in m do
11: Append li to G
12: Compute required resources for G
13: Identify feasible devices Dgroup ⊆ D
14: if Dgroup = ∅ then
15: if dprev = ∅ then
16: Abort: No allocation possible for li
17: end if
18: Allocate G \ {li } to dprev
19: Reset G = {li }
20: end if
21: Select optimal device:
 
−FLOPS(d)
d∗ = arg min + L(dprev , d)
d∈Dgroup 1GPU (d)

22: Assign G to d∗
23: Update dprev = d∗
24: end for
25: end for

2
Algorithm 3 Efficient LLM Partitioning and Allocation with Partition Con-
straint
Require: Set of models M, Set of devices D, Network latency matrix L, Max-
imum partitions Pmax
Ensure: Optimized allocation of models to devices
1: Sort models M by size or priority
2: for each model m ∈ M do
3: Identify devices Dfit ⊆ D that can fit m entirely
4: if Dfit ̸= ∅ then
5: Assign m to device d∗ = arg max FLOPS(d)
d∈Dfit
6: continue
7: end if
8: Partition m into layers {l1 , l2 , ..., ln }
9: Initialize empty current group G = ∅, previous device dprev = ∅
10: Set partition count P = 0
11: for each layer li ∈ m do
12: if P ≥ Pmax then
13: Abort: Partitioning exceeds allowed limit Pmax
14: end if
15: Append li to G
16: Compute required resources for G
17: Identify feasible devices Dgroup ⊆ D
18: if Dgroup = ∅ then
19: if dprev = ∅ then
20: Abort: No allocation possible for li
21: end if
22: Allocate G \ {li } to dprev
23: Reset G = {li }
24: Increment partition count P = P + 1
25: end if
26: Select optimal device:
 
∗ −FLOPS(d)
d = arg min + L(dprev , d)
d∈Dgroup 1GPU (d)

27: Assign G to d∗
28: Update dprev = d∗
29: end for
30: end for

3
Algorithm 4 Task-to-Expert Allocation
Require: Incoming task T , Set of experts E, Set of nodes N , Load matrix L,
Latency matrix Λ
Ensure: Assign task T to an optimal expert-node pair
1: Extract task type τ (T ), complexity c(T ), and required model type
2: Identify relevant experts ET ⊆ E that match τ (T )
3: if ET = ∅ then
4: Abort: No suitable expert found
5: end if
6: Initialize best assignment (e∗ , n∗ ) ← (∅, ∅), minimum cost Cmin ← ∞
7: for each expert e ∈ ET do
8: for each node n hosting e do
9: Compute expected execution time:

c(T )
Texec =
FLOPS(n)

10: Compute communication latency:

Tcomm = Λ(nclient , n)

11: Compute total cost:

C = Texec + Tcomm + L(n)

12: if C < Cmin then


13: Update best assignment (e∗ , n∗ ) ← (e, n)
14: Update Cmin ← C
15: end if
16: end for
17: end for
18: if (e∗ , n∗ ) = (∅, ∅) then
19: Abort: No feasible allocation found
20: else
21: Assign task T to expert e∗ on node n∗
22: Update node load: L(n∗ ) ← L(n∗ ) + c(T )
23: end if

You might also like