Positional Encodings Anchor Spatial Structure in

Vision Transformers: A Geometric Perspective on

Robustness

Anonymous Author(s)
Affiliation
Address
email

Abstract

1 Positional embeddings (PEs) in Vision Transformers (ViTs) are known to impact

2 performance and robustness, but their role in shaping internal spatial representations
3 is not well understood. In this work, we study how different forms of PEs influence
4 the representational geometry of ViTs and how these changes relate to robustness
5 under content-disrupting distribution shifts. We introduce a metric, the Spatial
6 Similarity Distance Correlation (SSDC), to quantify spatial structure in token
7 representations. Using this metric, we show that ViTs trained without PEs still
8 develop non-trivial spatial structure, but this structure is driven by visual content
9 and collapses under token permutation. In contrast, we find that all PEs considered
10 (learned absolute, sinusoidal, and rotary) are associated with a consistent shift
11 toward an index-anchored spatial organization. Representations in these models
12 remain stable under perturbations that disrupt content, and exhibit substantially
13 improved robustness to such distributional shifts. We further show that while
14 different PEs produce distinct depth-wise trajectories of spatial structure, their
15 robustness properties are largely similar (with secondary variation across encoding
16 schemes), suggesting that robustness appears to depend on the presence of a
17 stable positional reference frame more than it depends on the specific encoding
18 mechanism. These results offer a geometric account of how positional encodings
19 shape internal representations, with implications for the principled design of future
20 encoding schemes.

21 1 Introduction

22 Vision Transformers (ViTs) model images as sequences of patch tokens processed by self-attention
23 [Dosovitskiy et al., 2021]. Unlike convolutional architectures, they lack built-in inductive biases
24 toward locality and translation equivariance, and instead rely on positional embeddings (PEs) to inject
25 spatial information, enabling the model to distinguish tokens originating from different locations.
26 While PEs are designed to provide positional information, this design does not determine how that
27 signal is integrated into internal representations. In particular, it remains unclear whether positional
28 information organizes token representations into similarity structures anchored to absolute indices, or
29 whether spatial structure continues to arise primarily from visual content.
30 Prior work shows that ViTs retain substantial performance even when positional information is
31 removed or degraded [Dosovitskiy et al., 2021, Chu et al., 2023], suggesting that spatial relationships
32 can partially emerge from patch content alone. This raises a central question: if spatial structure can
33 arise without explicit positional guidance, what functional role do positional embeddings play?

Submitted to 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Do not distribute.
34 Existing studies have largely addressed this question through downstream performance comparisons
35 or architectural variations. While informative, these approaches provide limited insight into how
36 positional information shapes internal representations. In particular, it remains unclear whether
37 different positional encoding schemes (learned absolute, sinusoidal, or rotary) induce distinct spatial
38 reasoning strategies, or whether their effects on robustness arise from a shared mechanism.
39 In this work, we adopt a geometric perspective. We analyze the evolution of token representations
40 across the transformer stack using tools from representational geometry [Raghu et al., 2021], introduc-
41 ing the Spatial Similarity Distance Correlation (SSDC) as a probe of spatial structure. Critically, we
42 use SSDC in conjunction with a random permutation intervention at inference to distinguish whether
43 spatial organization is anchored to token indices or driven by patch content. We compare models
44 trained with learned absolute positional embeddings (APE), sinusoidal encodings, rotary embeddings
45 (RoPE), and no positional embeddings, and evaluate their robustness to distributional shifts.
46 Our central finding is that the specific encoding mechanism matters less than the presence of a
47 consistent positional signal. We show that:
48 • Positional encodings are associated with index-based spatial organization: All PE types
49 shift ViTs away from purely content-driven spatial structure toward representations that
50 remain partially anchored to token indices under permutation.
51 • This shift, not the encoding form, is associated with robustness: Despite differing in
52 how spatial structure develops across depth, APE, sinusoidal, and RoPE models exhibit
53 broadly comparable robustness to content-disrupting distributional shifts (despite consistent
54 but smaller differences between encoding schemes), while models lacking index-based
55 organization are substantially more fragile.
56 • A stable positional reference frame is strongly implicated in robustness: Using Random
57 Permutation Training (RPT), which preserves PEs but destroys index-to-location consistency,
58 we find that robustness is greatly reduced when a consistent positional frame cannot be
59 learned.
60 Together, these results provide a unified, geometric account of how positional encodings shape internal
61 representations and why they remain critical for robust visual recognition, though we emphasize that
62 the evidence is intervention-based rather than strictly causal.

63 2 Related Work
64 Positional Information in Vision Transformers
65 The standard Vision Transformer (ViT) breaks the permutation invariance of self-attention by adding
66 learnable absolute positional embeddings (PEs) to patch tokens [Dosovitskiy et al., 2021], establishing
67 the dominant paradigm for spatial encoding. However, ViTs retain substantial performance when
68 positional information is degraded or removed [Dosovitskiy et al., 2021, Chu et al., 2023], suggesting
69 that spatial structure can partially emerge from patch content alone.
70 Similar observations have been reported beyond vision. Recent work on decoder-only transformers
71 shows that models trained without PEs can recover positional information implicitly and tend to
72 rely on relative positions in practice [Kazemnejad et al., 2023]. Earlier findings in convolutional
73 networks further demonstrate that substantial positional information can be learned implicitly from
74 architectural biases such as zero-padding [Islam* et al., 2020]. Together, these results suggest that
75 explicit positional signals are not strictly required for structured spatial information to emerge.
76 This creates a central puzzle: if spatial structure can arise without explicit positional guidance, what
77 functional role do PEs play? Prior work has primarily addressed this question through architectural
78 variants [d’Ascoli et al., 2022, Liu et al., 2021, Heo et al., 2024] or performance comparisons [Doso-
79 vitskiy et al., 2021, Chu et al., 2023], leaving their mechanistic impact on internal representations
80 largely unexplored.
81 Representational Analysis of Transformers
82 A separate line of work studies the geometry and dynamics of transformer representations. Early
83 analyses compare ViT and CNN representations [Raghu et al., 2021], revealing differences in spatial
84 organization. Subsequent work examines how attention transforms representations [Kobayashi

2
85 et al., 2021], how representational rank evolves with depth [Dong et al., 2021], and how token
86 representations tend to homogenize in deeper layers [Bhojanapalli et al., 2021]. The residual stream
87 framework provides a useful lens for analyzing these dynamics [Elhage et al., 2021]. However,
88 these approaches do not isolate the causal role of positional embeddings, nor do they connect
89 representational structure to robustness.
90 Robustness of Visual Models
91 Vision Transformers exhibit distinct robustness profiles compared to convolutional networks. Prior
92 work shows that transformers are generally more robust to certain spatial perturbations but can
93 be more sensitive to texture-based changes [Bhojanapalli et al., 2021]. Additional studies report
94 favorable out-of-distribution generalization properties for ViTs [Paul and Chen, 2022], connecting to
95 broader findings on shape versus texture bias in visual recognition [Geirhos et al., 2019]. While the
96 impact of positional embeddings on robustness has been observed (particularly that models trained
97 with PEs exhibit better robustness profiles than models trained without them) [Mao et al., 2021], the
98 relationship between a model’s spatial organization strategy (whether anchored to absolute position
99 or inferred from content) and its robustness to distributional shifts remains poorly understood.
100 Our Contribution
101 We connect these lines of work by showing that positional embeddings are associated with a shift
102 toward index-based spatial organization, and that this shift (rather than the specific encoding mech-
103 anism) appears to be a dominant correlate of robustness. Using SSDC and controlled permutation
104 interventions (RPT and RPI), we provide a geometric account of how positional information shapes
105 internal representations and why it improves robustness beyond merely breaking permutation invari-
106 ance.

107 3 Preliminaries

108 3.1 Vision Transformer Architecture and Positional Encodings

109 All models are Vision Transformers trained from scratch on ImageNet-100, with approximately 22M
110 parameters (details in Appendix A). Images are partitioned into fixed-size patches, projected into
111 token embeddings, and processed by a stack of self-attention and feedforward layers.
112 Since self-attention is permutation invariant, positional encodings are required to inject spatial
113 information. We consider three commonly used PE schemes, all adapted to 2D grids:
114 Learned Absolute Positional Embeddings (APE): learnable vectors added to token embeddings
115 before the first transformer block, establishing a fixed index-to-location mapping.
116 Sinusoidal Positional Embeddings: fixed, deterministic encodings constructed from sinusoidal
117 functions applied independently along spatial axes and added to token embeddings.
118 Rotary Positional Embeddings (RoPE): position-dependent rotations applied to query and key
119 vectors within each attention layer, introducing positional information multiplicatively.
120 These approaches differ in parameterization (learned vs. fixed) and integration (additive vs. multi-
121 plicative), enabling comparison of how different positional signals shape internal representations.

122 3.2 Index-Based and Content-Based Spatial Organization

123 We distinguish between two qualitatvely distinct modes of spatial organization in token representa-
124 tions.
125 Index-based spatial organization refers to representations whose similarity structure depends
126 systematically on token position. Tokens that are spatially proximate tend to have more similar
127 representations by virtue of their indices, and this structure persists under disruptions to patch content.
128 This definition is behavioral and does not assume explicit coordinate representations.
129 Content-based spatial organization refers to representations in which similarity is driven primarily
130 by patch content. Spatial structure arises indirectly from natural image statistics and degrades under
131 transformations that disrupt content or token ordering.

3
132 In practice, models may exhibit both behaviors; the key distinction is which signal dominates and
133 whether spatial structure persists when content-based cues are disrupted.

134 4 Methods

135 4.1 Residual Stream Geometry

136 At selected layers, we extract the residual stream as a matrix R ∈ RT ×C , where T is the number
137 of tokens and C the embedding dimension. We compute pairwise cosine similarities between unit-
138 normalized token representations to form a symmetric similarity matrix, averaged across the batch
139 dimension.

140 4.2 Spatial Similarity Distance Correlation

141 Let S ∈ RT ×T denote the token similarity matrix, and let pi ∈ Z2 denote the spatial coordinates of
142 token i. Define the spatial distance matrix D by Dij = ∥pi − pj ∥1 . We define SSDC as the Spearman
143 rank correlation between similarity and negative spatial distance over all token pairs:

SSDC = ρSpearman ({Sij }i<j , {−Dij }i<j ) .

144 Higher SSDC values indicate that spatially proximate tokens tend to have more similar representations.
145 We use Spearman rank correlation to remain agnostic to the precise functional form relating spatial
146 distance and representational similarity.
147 Importantly, SSDC should be interpreted as a coarse proxy for spatial organization rather than a
148 direct measurement of a specific mechanism. Absolute values may reflect multiple factors (e.g., data
149 statistics, architectural biases), and therefore SSDC is primarily used comparatively (to track changes
150 across depth and to measure sensitivity to controlled interventions).

151 4.3 Random Permutation at Inference (RPI)

152 To distinguish index-based from content-based organization, we randomly permute token order at
153 inference while keeping positional indices fixed. This breaks the correspondence between token order
154 and spatial location. Under this setup, spatial structure driven purely by patch content is expected
155 to be disrupted, as spatially adjacent tokens no longer correspond to neighboring image patches.
156 In contrast, if a model has learned representations that depend systematically on token indices via
157 positional signals, some spatial structure may persist or be partially recoverable.
158 As a result, SSDC under RPI should be interpreted as an indicator of the extent to which spatial
159 organization depends on token indices, rather than as a definitive separation between index-based and
160 content-based mechanisms

161 4.4 Random Permutation during Training (RPT)

162 Random Permutation Training (RPT) applies a fresh random permutation to the token sequence
163 at every forward pass during training. At each batch, patch tokens are shuffled while positional
164 embeddings remain fixed to their original indices, breaking the consistent mapping between token
165 index and spatial location. This prevents the model from learning a stable index-based spatial
166 organization despite the presence of positional signals.

167 4.5 Positional Embedding Magnitude Scaling

168 We scale positional embeddings at inference by a factor α, replacing ei with αei . This provides a
169 continuous intervention on positional signal strength without retraining. We apply this to APE and
170 sinusoidal models; an equivalent scaling for RoPE is not directly defined due to its multiplicative
171 formulation.

4
 0.80 0.70
Untrained Ablated

0.75 0.65

SSDC 0.70 0.60

SSDC
0.65 0.55

0.60 0.50

Untrained Ablated
0.55 0.45 Ablated
Intact
0.50 0.40
0 2 4 6 8 10 0 2 4 6 8 10
Layer Layer

(a) Evolution of SSDC across depth on untrained (b) Evolution of SSDC across depth for on un-
ablated models trained ablated models, trained ablated models,
and intact (trained with APE) models
Figure 1: (a) SSDC grows weakly and remains at a relatively high value across layers, indicating
static spatial correlations induced by architectural and data priors rather than learning.
(b) While untrained ablated models exhibit relatively high but slowly varying SSDC consistent
with static data and architectural priors, trained ablated models display a sharp increase in early
layers, indicating the emergence of learned spatial structure despite the absence of explicit positional
encoding.

172 4.6 Fragility Score

173 We quantify robustness using the Fragility Score (FS):

Ashift
FS = 1 − ,
Anormal
174 where Anormal and Ashift denote accuracy on clean and shifted data. Higher values indicate greater
175 sensitivity to distributional shift.

176 5 Results
177 5.1 Architectural Priors Induce Static Spatial Correlations at Initialization

178 Experimental Setup: We evaluate SSDC across all layers of untrained ablated models on the
179 Imagenet-100 dataset. Unless stated otherwise, all reported results are averaged over 4 random seeds.
180 Shaded regions in figures indicate variability across runs (±1 standard deviation).
181 Results: The untrained ablated model exhibits a substantial non-zero SSDC (approximately
182 0.57–0.64) with only a weak, gradual increase across depth (Figure 1a). This behavior is highly
183 consistent across runs and reflects static spatial correlations induced by architectural priors and the
184 inherent structure of natural images, rather than learned spatial reasoning.
185 Crucially, this baseline highlights that SSDC should not be interpreted as a standalone metric whose
186 absolute magnitude reflects the presence or strength of learned spatial organization. Even in the
187 absence of training, relatively high SSDC values emerge. Instead, the layer-wise dynamics of SSDC
188 (in particular, the rate and pattern of change across depth) are the informative signal. In contrast to the
189 shallow, nearly static progression observed here, trained models exhibit rapid and structured changes
190 in SSDC (e.g., sharp increases in early layers), indicating the emergence of learned spatial structure.
191 This establishes a static baseline, allowing us to distinguish genuinely learned spatial organization
192 from correlations that arise purely from architectural and data-driven effects.

193 5.2 Emergence of Spatial Structure Without Positional Encoding

194 Experimental Setup: To investigate whether spatial structure can emerge in the absence of explicit
195 positional information, we evaluate SSDC across all layers of untrained ablated models, trained
196 ablated models, and trained intact (APE) models on the Imagenet-100 dataset.

5
 0.6 0.100
Ablated
0.5 0.075 RPT

0.050
0.4

SSDC under RPI

SSDC under RPI
0.025
0.3
0.000
0.2
−0.025
0.1
−0.050
APE
0.0 RoPE −0.075
Sinusoidal PEs
−0.1 −0.100
0 2 4 6 8 10 0 2 4 6 8 10
Layer Layer

(a) Models with positional encodings. APE, SinuPE, (b) Models without a stable positional reference
and RoPE models exhibit substantial SSDC recovery frame. Ablated and RPT models collapse to near-zero
under RPI, indicating spatial organization anchored to SSDC across all layers under RPI, indicating that their
token indices. RoPE shows a gradual accumulation of spatial structure is entirely content-driven and does
structure across depth, while APE and SinuPE exhibit not survive token permutation.
earlier, injection-like peaks.
Figure 2: SSDC under random permutation at inference (RPI). RPI disrupts the correspondence
between token content and spatial position. Only models that anchor spatial structure to token indices
exhibit SSDC recovery after permutation. In contrast, models lacking a consistent positional mapping
collapse to near-zero SSDC, revealing a purely content-based spatial organization.

197 Results: Figure 1b compares the layer-wise evolution of SSDC for an untrained ablated model,
198 a trained ablated model, and a trained model with positional embeddings. The untrained ablated
199 model exhibits relatively high SSDC (approximately 0.57–0.64) with only weak growth across depth,
200 reflecting static spatial correlations induced by architectural and data priors rather than learning.
201 In contrast, the trained ablated model shows a qualitatively different trajectory: starting from lower
202 SSDC, it exhibits a sharp increase in early layers followed by continued growth. This dynamic pattern
203 closely resembles that of the trained model with positional embeddings. The key distinction is not
204 absolute SSDC magnitude, but its evolution—static in the untrained model, structured and rapidly
205 changing after training.
206 These results indicate that non-trivial spatial structure emerges during training even without positional
207 embeddings. This is consistent with the non-trivial performance of ablated models and prior evidence
208 that transformers can implicitly recover positional information.
209 We emphasize that this emergent structure is not equivalent to that induced by positional embed-
210 dings. Rather, this establishes that spatial organization can arise without explicit positional signals,
211 motivating a more precise characterization of its underlying mechanism in the next section.

212 5.3 Disentangling Index-Based and Content-Based Spatial Organization

213 Experimental Setup: To distinguish between index-based and content-based spatial organization,
214 we evaluate SSDC across all layers under a Random Permutation at Inference (RPI) intervention.
215 Concretely, patch tokens are randomly permuted before being processed by the transformer, while
216 positional embedding indices (when present) remain fixed to their original spatial locations. This
217 operation disrupts the correspondence between token content and spatial position, while preserving
218 any mapping between token indices and positional embeddings.
219 Under this setup, any spatial structure that arises purely from patch content is destroyed, as spatially
220 adjacent tokens no longer correspond to neighboring image patches. In contrast, if a model has
221 learned to anchor its representations to absolute token indices via positional embeddings, spatial
222 structure can be re-established through the fixed positional signal. As a result, SSDC recovery under
223 RPI serves as a probe for index-based spatial organization: models that rely on absolute positional
224 information exhibit non-trivial SSDC despite permutation, whereas models that rely on content-based
225 cues collapse to near-zero SSDC.

6
 Gaussian Blur Fragility Score
0.30

JPEG Fragility Score

0.6

0.25
0.5

0.20
0.4

0.15
0.3

APE RoPE SPE Ablated RPT APE RoPE SPE Ablated RPT
Model Condition Model Condition

(a) Gaussian Blur (σ = 2.5). Fragility scores under a (b) JPEG Compression (quality = 5). Fragility
mild perturbation that removes high-frequency detail scores under a strong content-disrupting transforma-
while largely preserving global spatial structure. Dif- tion. Models with positional encodings exhibit sub-
ferences between models are present but compressed, stantially lower fragility than ablated and RPT models,
reflecting the weaker disruption of content-based cues. with RoPE achieving the lowest fragility overall.
Figure 3: Robustness to distributional shifts. Fragility scores across model variants under two
perturbation regimes. The gap between models with and without a stable positional reference frame is
most pronounced under strong content disruption (JPEG), while remaining consistent but attenuated
under milder perturbations (Gaussian blur).

226 We evaluate this behavior across models trained with learned absolute positional embeddings (APE),
227 sinusoidal encodings, rotary embeddings (RoPE), no positional embeddings (ablated), and under
228 Random Permutation Training (RPT). RPT preserves the presence of positional embeddings while
229 disrupting the consistency between token indices and spatial locations during training; we include it
230 here for completeness, and analyze its implications for robustness in the following subsection.
231 Results: Models trained without positional embeddings exhibit a complete collapse of SSDC under
232 RPI across all layers, suggesting that their spatial structure is predominantly content-driven under this
233 probe. Despite exhibiting non-trivial SSDC in the unpermuted setting (Section 5.2), this structure
234 does not survive disruption of patch content, indicating that it is not anchored to token indices.
235 In contrast, all models trained with positional embeddings show substantial SSDC recovery under
236 RPI, indicating representations that are more consistent with index-anchored spatial organization.
237 However, the nature of this recovery differs across encoding schemes. For APE and sinusoidal
238 embeddings, SSDC exhibits a rapid increase in early layers following permutation, reaching a peak
239 within the first few layers before stabilizing or slightly decreasing. This behavior suggests that spatial
240 structure is injected early in the network via additive positional signals.
241 RoPE models display a qualitatively different trajectory: SSDC increases more gradually and
242 continues to grow with depth, without a pronounced early-layer peak. This indicates that positional
243 information is integrated progressively throughout the network, consistent with its multiplicative
244 incorporation into attention mechanisms.
245 RPT models, despite having positional embeddings present, fail to exhibit meaningful SSDC recovery
246 under RPI, behaving similarly to fully ablated models. This suggests that the mere presence of
247 positional embeddings is insufficient; a consistent mapping between token indices and spatial locations
248 during training appears necessary for index-based spatial organization to emerge.
249 Taken together, these results establish that positional embeddings are associated with a shift from
250 content-based to index-based spatial organization, and that this shift depends critically on the stability
251 of the positional reference frame rather than on the architectural presence of positional signals alone.

252 5.4 Robustness to Content-Preserving and Content-Disrupting Perturbations

253 Experimental Setup: To evaluate how spatial organization strategy influences robustness, we
254 measure performance under distribution shifts that perturb image content while preserving global
255 structure. We consider two transformations:

7
256 JPEG Compression: We apply aggressive compression (quality = 5), introducing blocking artifacts
257 that strongly disrupt local texture statistics while preserving coarse spatial layout. This provides a
258 targeted probe of reliance on content-based cues.
259 Gaussian Blur: We apply Gaussian blur with standard deviation σ = 2.5, attenuating high-frequency
260 detail while preserving low-frequency structure. This constitutes a milder perturbation than JPEG.
261 For each model, we compute the Fragility Score (FS), defined as the relative drop in accuracy under
262 each transformation.
263 We include Random Permutation Training (RPT) as a critical control. RPT preserves the presence
264 of positional embeddings but prevents the formation of a stable index-based spatial organization,
265 allowing us to distinguish between the mere presence of positional signals and the emergence of a
266 consistent positional reference frame.
267 We emphasize that these robustness results are limited to content-disrupting perturbations (e.g.,
268 compression artifacts and blur) and do not necessarily generalize to other forms of distribution shift
269 such as geometric transformations or semantic shifts. Results: Under JPEG compression, models
270 with positional encodings exhibit substantially lower fragility (APE and sinusoidal: ∼0.43, RoPE:
271 ∼0.30) than ablated and RPT models (∼0.66). This large gap indicates that robustness to severe
272 content degradation is strongly influenced by the presence of a stable positional reference frame.
273 Within PE-based models, RoPE consistently achieves lower fragility, suggesting a secondary effect
274 of the encoding mechanism.
275 Under Gaussian blur, the same ordering is preserved but differences are attenuated (RoPE: ∼0.15,
276 APE: ∼0.17–0.20, sinusoidal: ∼0.22, RPT: ∼0.25, ablated: ∼0.30). Because blur preserves global
277 structure, it provides a weaker test of reliance on content-based cues, reducing the separation between
278 models.
279 Taken together, these results support a two-level interpretation: (1) the emergence of a stable
280 positional reference frame appears to be a dominant factor associated with robustness, and (2) the
281 specific encoding mechanism introduces secondary variation, with RoPE exhibiting consistently
282 lower fragility.
283 Crucially, the poor robustness of RPT models shows that the mere presence of positional embeddings
284 is insufficient: robustness appears to rely on learning a consistent mapping between token indices and
285 spatial locations. This provides evidence for a relationship between the spatial organization patterns
286 identified earlier and downstream robustness.

287 5.5 Linking Index-Based Spatial Organization to Robustness via Positional Scaling

288 Experimental Setup: To probe the relationship between spatial organization and robustness, we
289 require a controlled intervention that selectively disrupts index-based spatial structure while preserving
290 the rest of the model. We achieve this by scaling the magnitude of learned absolute positional
291 embeddings (APE) at inference time by a factor α ∈ [0, 1], without retraining.
292 While this intervention operates on positional embeddings, our goal is not to study positional signal
293 strength per se, but to use it as a mechanism to continuously degrade the model’s index-based spatial
294 organization. To measure the integrity of this organization, we evaluate Spatial Similarity Distance
295 Correlation (SSDC) under Random Permutation at Inference (RPI), as introduced in Section 5.3.
296 Under RPI, any recovered spatial structure must be anchored to token indices rather than content. We
297 therefore interpret SSDC recovery as a proxy for the presence of index-based spatial organization.
298 Importantly, SSDC recovery approaching zero does not imply that the model is actively constructing
299 a content-based spatial structure; rather, it indicates that index-based spatial organization is no longer
300 preserved under permutation.
301 To summarize this behavior compactly, we define:

∆SSDC = SSDClayer1 − SSDClayer0 ,

302 which captures the immediate recovery of spatial structure after the first encoder block under RPI.
303 Since SSDC at layer 0 is identically zero under permutation, ∆SSDC serves as a direct measure of
304 early-layer index-based spatial organization.

8
305 We jointly analyze ∆SSDC and the Fragility Score (FS) across varying α. For clarity, we report
306 representative magnitudes illustrating distinct regimes, with the full results provided in the Appendix.

α ∆SSDC (RPI) Fragility Score

Mean Std Mean Std
1.0 0.4725 0.0228 0.4338 0.0127
0.8 0.3125 0.0259 0.4780 0.0146
0.7 0.1845 0.0342 0.5145 0.0145
0.5 0.0475 0.0083 0.5975 0.0202
0.4 0.0000 0.0000 0.6272 0.0189
Table 1: Effect of positional embedding magnitude α on index-based spatial organization and
robustness. ∆SSDC (measured under RPI) captures the recovery of index-based spatial structure after
the first encoder block. As α decreases, ∆SSDC collapses, indicating the breakdown of index-based
spatial organization, while fragility increases sharply in the same regime before plateauing once
spatial structure is lost.

307 Results: We observe a clear correspondence between the degradation of index-based spatial organi-
308 zation and the loss of robustness.
309 At high magnitudes (α ≥ 0.9), models exhibit strong SSDC recovery (∆SSDC ≈ 0.37–0.47),
310 indicating intact index-based spatial organization. In this regime, fragility remains relatively low and
311 stable (FS ≈ 0.44–0.46), suggesting that robustness is preserved when spatial structure is intact.
312 As α decreases into an intermediate regime (0.8 ≥ α ≥ 0.5), SSDC recovery drops sharply
313 (∆SSDC ≈ 0.30 → 0.02), reflecting the progressive breakdown of index-based spatial organization.
314 This degradation is accompanied by a pronounced increase in fragility (FS ≈ 0.48 → 0.63). Notably,
315 the most significant increases in fragility occur precisely where SSDC recovery is actively decreasing,
316 indicating that robustness degradation is strongly correlated with the loss of spatial structure.
317 Below a critical threshold (α ≤ 0.4), SSDC recovery collapses to zero (∆SSDC ≈ 0), indicating that
318 index-based spatial organization is no longer recoverable under permutation. In this regime, fragility
319 continues to increase, but only marginally (FS ≈ 0.65 → 0.685). This suggests that once spatial
320 organization is fully disrupted, further degradation in robustness is no longer driven by changes in
321 spatial structure, but instead reflects secondary effects such as reduced representational quality or
322 distribution mismatch induced by scaling.
323 A complementary effect is observed at high magnitudes: when index-based spatial organization
324 is already fully intact, small reductions in α have limited impact on fragility. Together, these
325 observations reveal three regimes: (1) a stable regime with intact spatial organization and low
326 fragility, (2) a transition regime where spatial structure degrades and fragility increases sharply, and
327 (3) a collapsed regime where spatial organization is absent and fragility plateaus.
328 Overall, these results provide evidence that robustness may be driven in part by the presence of
329 index-based spatial organization. Positional scaling serves only as a means of intervention; the
330 observed changes in robustness track the degradation of spatial structure rather than the magnitude of
331 the positional signal itself.

332 6 Limitations
333 The findings reported here are based on ViT-S models trained from scratch on ImageNet-100, and it
334 remains an open question whether the observed relationships between positional encoding, index-
335 based spatial organization, and robustness generalize to larger architectures, pre-trained models, or
336 models fine-tuned from large-scale checkpoints. The robustness evaluation is specifically scoped
337 to content-disrupting perturbations (JPEG compression and Gaussian blur); we make no claims
338 about spatial perturbations, adversarial shifts, or semantic distribution changes, and these may
339 involve different mechanisms. SSDC is used as a coarse proxy for spatial organization rather than
340 a direct measurement of a specific representational mechanism, and its interpretation depends on
341 the comparative and intervention-based framing established in Section 5.1. Finally, the positional
342 scaling experiment (Section 5.5) conflates spatial organization degradation with changes in raw

9
343 positional signal magnitude, and while the three-regime structure is consistent with a mediating role
344 for index-based organization, alternative pathways cannot be fully excluded.

345 7 Conclusion
346 We studied how positional encodings shape spatial organization in Vision Transformers and how
347 this relates to robustness under content-disrupting perturbations. Using SSDC and permutation-
348 based interventions, we found that spatial structure emerges even without positional encodings, but
349 is largely content-driven and does not persist under token permutation. In contrast, models with
350 positional encodings exhibit representations that remain more consistent with index-anchored spatial
351 organization under the same probe.
352 Across experiments, robustness under content-disrupting shifts is closely associated with the presence
353 of a stable positional reference frame rather than the mere presence of positional embeddings. This
354 is supported by the behavior of RPT models and by positional scaling, where robustness degrades
355 alongside reductions in measured index-anchored spatial structure. Differences between encoding
356 schemes persist but appear secondary to this effect.
357 Overall, our results suggest that positional encodings may contribute to robustness by supporting a
358 stable positional reference frame in the learned representations. We emphasize that this conclusion is
359 based on intervention-based evidence and should be interpreted as identifying a strong relationship
360 rather than a fully isolated causal mechanism.

361 References
362 Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and
363 Andreas Veit. Understanding robustness of transformers for image classification. pages 10211–
364 10221, 10 2021. doi: 10.1109/ICCV48922.2021.01007.
365 Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional
366 encodings for vision transformers. In The Eleventh International Conference on Learning Repre-
367 sentations, 2023. URL [Link]
368 Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure
369 attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, edi-
370 tors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of
371 Proceedings of Machine Learning Research, pages 2793–2803. PMLR, 18–24 Jul 2021. URL
372 [Link]
373 Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
374 Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit,
375 and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale.
376 In International Conference on Learning Representations, 2021. URL [Link]
377 net/forum?id=YicbFdNTTy.
378 Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent
379 Sagun. Convit: improving vision transformers with soft convolutional inductive biases*. Journal
380 of Statistical Mechanics: Theory and Experiment, 2022(11):114005, nov 2022. doi: 10.1088/
381 1742-5468/ac9830. URL [Link]
382 Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda
383 Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli,
384 Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal
385 Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris
386 Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
387 [Link]
388 Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and
389 Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias
390 improves accuracy and robustness. In International Conference on Learning Representations,
391 2019. URL [Link]

10
392 Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for
393 vision transformer. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy,
394 September 29–October 4, 2024, Proceedings, Part X, page 289–305, Berlin, Heidelberg, 2024.
395 Springer-Verlag. ISBN 978-3-031-72683-5. doi: 10.1007/978-3-031-72684-2_17. URL https:
396 //[Link]/10.1007/978-3-031-72684-2_17.
397 Md Amirul Islam*, Sen Jia*, and Neil D. B. Bruce. How much position information do convolutional
398 neural networks encode? In International Conference on Learning Representations, 2020. URL
399 [Link]
400 Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan, Payel Das, and Siva Reddy. The impact
401 of positional encoding on length generalization in transformers. In Thirty-seventh Conference on
402 Neural Information Processing Systems, 2023. URL [Link]
403 Drrl2gcjzl.
404 Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Incorporating Residual and
405 Normalization Layers into Analysis of Masked Language Models. In Marie-Francine Moens,
406 Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Con-
407 ference on Empirical Methods in Natural Language Processing, pages 4547–4568. Association
408 for Computational Linguistics, November 2021. doi: 10.18653/v1/[Link]-main.373. URL
409 [Link]
410 Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining
411 Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows . In 2021
412 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002. IEEE
413 Computer Society, October 2021. doi: 10.1109/ICCV48922.2021.00986. URL [Link]
414 [Link]/10.1109/ICCV48922.2021.00986.
415 Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue.
416 Towards robust vision transformer. 2022 IEEE/CVF Conference on Computer Vision and Pattern
417 Recognition (CVPR), pages 12032–12041, 2021. URL [Link]
418 CorpusID:235211752.
419 Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. Proceedings of the AAAI
420 Conference on Artificial Intelligence, 36(2):2071–2081, Jun. 2022. doi: 10.1609/aaai.v36i2.20103.
421 URL [Link]
422 Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.
423 Do vision transformers see like convolutional neural networks? In A. Beygelzimer, Y. Dauphin,
424 P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems,
425 2021. URL [Link]

426 A Experimental Setup and Hyperparameters

427 B Technical appendices and supplementary material

428 Technical appendices with additional results, figures, graphs, and proofs may be submitted with the
429 paper submission before the full submission deadline (see above). You can upload a ZIP file for
430 videos or code, but do not upload a separate PDF file for the appendix. There is no page limit for the
431 technical appendices.
432 Note: Think of the appendix as “optional reading” for reviewers. The paper must be able to stand
433 alone without the appendix; for example, adding critical experiments that support the main claims to
434 an appendix is inappropriate.

11
 Table 2: Model architecture and training hyperparameters used in all experiments.
Parameter Value
Input & Tokenization
Input resolution 224 × 224
Patch size 16 × 16
Number of patches 196
Input channels (C) 3
ViT Architecture
Embedding dimension (D) 384
Number of encoder layers 12
Number of attention heads 8
Key/query dimension (dk ) 48
Dropout (embedding) 0.15
Dropout (attention) 0.15
Dropout (MLP) 0.15
Stochastic depth rate 0.2
Training Hyperparameters
Batch size 128
Optimizer Adam
Learning rate 1 × 10−3
Weight decay 5 × 10−2
Adam β1 0.9
Adam β2 0.999
Training epochs 60

435 NeurIPS Paper Checklist

436 1. Claims
437 Question: Do the main claims made in the abstract and introduction accurately reflect the
438 paper’s contributions and scope?
439 Answer: [Yes]
440 Justification: The abstract and introduction accurately describe the paper’s contributions
441 and scope. Claims are presented with appropriate hedging reflecting the intervention-based
442 nature of the evidence, and the scope is explicitly limited to content-disrupting distributional
443 shifts. The paper does not claim results beyond the experimental conditions studied.
444 Guidelines:
445 • The answer [N/A] means that the abstract and introduction do not include the claims
446 made in the paper.
447 • The abstract and/or introduction should clearly state the claims made, including the
448 contributions made in the paper and important assumptions and limitations. A [No] or
449 [N/A] answer to this question will not be perceived well by the reviewers.
450 • The claims made should match theoretical and experimental results, and reflect how
451 much the results can be expected to generalize to other settings.
452 • It is fine to include aspirational goals as motivation as long as it is clear that these goals
453 are not attained by the paper.
454 2. Limitations
455 Question: Does the paper discuss the limitations of the work performed by the authors?
456 Answer: [Yes]
457 Justification: A separate Limitations section is included in the paper.
458 Guidelines:

12
459 • The answer [N/A] means that the paper has no limitation while the answer [No] means
460 that the paper has limitations, but those are not discussed in the paper.
461 • The authors are encouraged to create a separate “Limitations” section in their paper.
462 • The paper should point out any strong assumptions and how robust the results are to
463 violations of these assumptions (e.g., independence assumptions, noiseless settings,
464 model well-specification, asymptotic approximations only holding locally). The authors
465 should reflect on how these assumptions might be violated in practice and what the
466 implications would be.
467 • The authors should reflect on the scope of the claims made, e.g., if the approach was
468 only tested on a few datasets or with a few runs. In general, empirical results often
469 depend on implicit assumptions, which should be articulated.
470 • The authors should reflect on the factors that influence the performance of the approach.
471 For example, a facial recognition algorithm may perform poorly when image resolution
472 is low or images are taken in low lighting. Or a speech-to-text system might not be
473 used reliably to provide closed captions for online lectures because it fails to handle
474 technical jargon.
475 • The authors should discuss the computational efficiency of the proposed algorithms
476 and how they scale with dataset size.
477 • If applicable, the authors should discuss possible limitations of their approach to
478 address problems of privacy and fairness.
479 • While the authors might fear that complete honesty about limitations might be used by
480 reviewers as grounds for rejection, a worse outcome might be that reviewers discover
481 limitations that aren’t acknowledged in the paper. The authors should use their best
482 judgment and recognize that individual actions in favor of transparency play an impor-
483 tant role in developing norms that preserve the integrity of the community. Reviewers
484 will be specifically instructed to not penalize honesty concerning limitations.
485 3. Theory assumptions and proofs
486 Question: For each theoretical result, does the paper provide the full set of assumptions and
487 a complete (and correct) proof?
488 Answer: [N/A]
489 Justification: [TODO]
490 Guidelines:
491 • The answer [N/A] means that the paper does not include theoretical results.
492 • All the theorems, formulas, and proofs in the paper should be numbered and cross-
493 referenced.
494 • All assumptions should be clearly stated or referenced in the statement of any theorems.
495 • The proofs can either appear in the main paper or the supplemental material, but if
496 they appear in the supplemental material, the authors are encouraged to provide a short
497 proof sketch to provide intuition.
498 • Inversely, any informal proof provided in the core of the paper should be complemented
499 by formal proofs provided in appendix or supplemental material.
500 • Theorems and Lemmas that the proof relies upon should be properly referenced.
501 4. Experimental result reproducibility
502 Question: Does the paper fully disclose all the information needed to reproduce the main ex-
503 perimental results of the paper to the extent that it affects the main claims and/or conclusions
504 of the paper (regardless of whether the code and data are provided or not)?
505 Answer: [Yes]
506 Justification: Full training details, hyperparameters, and model architecture specifications
507 are provided in Appendix A. All metrics (SSDC, Fragility Score) are formally defined in
508 Section 4 (and more details are provided in Appendix B), and all interventions (RPI, RPT,
509 magnitude scaling) are described in sufficient detail to reproduce the experimental setup.
510 Guidelines:
511 • The answer [N/A] means that the paper does not include experiments.

13
512 • If the paper includes experiments, a [No] answer to this question will not be perceived
513 well by the reviewers: Making the paper reproducible is important, regardless of
514 whether the code and data are provided or not.
515 • If the contribution is a dataset and/or model, the authors should describe the steps taken
516 to make their results reproducible or verifiable.
517 • Depending on the contribution, reproducibility can be accomplished in various ways.
518 For example, if the contribution is a novel architecture, describing the architecture fully
519 might suffice, or if the contribution is a specific model and empirical evaluation, it may
520 be necessary to either make it possible for others to replicate the model with the same
521 dataset, or provide access to the model. In general. releasing code and data is often
522 one good way to accomplish this, but reproducibility can also be provided via detailed
523 instructions for how to replicate the results, access to a hosted model (e.g., in the case
524 of a large language model), releasing of a model checkpoint, or other means that are
525 appropriate to the research performed.
526 • While NeurIPS does not require releasing code, the conference does require all submis-
527 sions to provide some reasonable avenue for reproducibility, which may depend on the
528 nature of the contribution. For example
529 (a) If the contribution is primarily a new algorithm, the paper should make it clear how
530 to reproduce that algorithm.
531 (b) If the contribution is primarily a new model architecture, the paper should describe
532 the architecture clearly and fully.
533 (c) If the contribution is a new model (e.g., a large language model), then there should
534 either be a way to access this model for reproducing the results or a way to reproduce
535 the model (e.g., with an open-source dataset or instructions for how to construct
536 the dataset).
537 (d) We recognize that reproducibility may be tricky in some cases, in which case
538 authors are welcome to describe the particular way they provide for reproducibility.
539 In the case of closed-source models, it may be that access to the model is limited in
540 some way (e.g., to registered users), but it should be possible for other researchers
541 to have some path to reproducing or verifying the results.
542 5. Open access to data and code
543 Question: Does the paper provide open access to the data and code, with sufficient instruc-
544 tions to faithfully reproduce the main experimental results, as described in supplemental
545 material?
546 Answer: [No]
547 Justification: Code and data are not released with this submission. However, all experimen-
548 tal details necessary for reproduction are provided in the paper and appendix, including
549 full architecture specifications, hyperparameters, formal metric definitions, and detailed
550 descriptions of all interventions. ImageNet-100 is a publicly available dataset. We intend to
551 release code upon publication.
552 Guidelines:
553 • The answer [N/A] means that paper does not include experiments requiring code.
554 • Please see the NeurIPS code and data submission guidelines ([Link]
555 public/guides/CodeSubmissionPolicy) for more details.
556 • While we encourage the release of code and data, we understand that this might not
557 be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not
558 including code, unless this is central to the contribution (e.g., for a new open-source
559 benchmark).
560 • The instructions should contain the exact command and environment needed to run to
561 reproduce the results. See the NeurIPS code and data submission guidelines (https:
562 //[Link]/public/guides/CodeSubmissionPolicy) for more details.
563 • The authors should provide instructions on data access and preparation, including how
564 to access the raw data, preprocessed data, intermediate data, and generated data, etc.
565 • The authors should provide scripts to reproduce all experimental results for the new
566 proposed method and baselines. If only a subset of experiments are reproducible, they
567 should state which ones are omitted from the script and why.

14
568 • At submission time, to preserve anonymity, the authors should release anonymized
569 versions (if applicable).
570 • Providing as much information as possible in supplemental material (appended to the
571 paper) is recommended, but including URLs to data and code is permitted.
572 6. Experimental setting/details
573 Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa-
574 rameters, how they were chosen, type of optimizer) necessary to understand the results?
575 Answer: [Yes]
576 Justification: Hyperparameters and training details are provided in Appendix A.
577 Guidelines:
578 • The answer [N/A] means that the paper does not include experiments.
579 • The experimental setting should be presented in the core of the paper to a level of detail
580 that is necessary to appreciate the results and make sense of them.
581 • The full details can be provided either with the code, in appendix, or as supplemental
582 material.
583 7. Experiment statistical significance
584 Question: Does the paper report error bars suitably and correctly defined or other appropriate
585 information about the statistical significance of the experiments?
586 Answer: [Yes]
587 Justification: For every result in the paper, we report the mean across 4 random seeds and
588 include error bars in every figure and table.
589 Guidelines:
590 • The answer [N/A] means that the paper does not include experiments.
591 • The authors should answer [Yes] if the results are accompanied by error bars, confidence
592 intervals, or statistical significance tests, at least for the experiments that support the
593 main claims of the paper.
594 • The factors of variability that the error bars are capturing should be clearly stated (for
595 example, train/test split, initialization, random drawing of some parameter, or overall
596 run with given experimental conditions).
597 • The method for calculating the error bars should be explained (closed form formula,
598 call to a library function, bootstrap, etc.)
599 • The assumptions made should be given (e.g., Normally distributed errors).
600 • It should be clear whether the error bar is the standard deviation or the standard error
601 of the mean.
602 • It is OK to report 1-sigma error bars, but one should state it. The authors should
603 preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis
604 of Normality of errors is not verified.
605 • For asymmetric distributions, the authors should be careful not to show in tables or
606 figures symmetric error bars that would yield results that are out of range (e.g., negative
607 error rates).
608 • If error bars are reported in tables or plots, the authors should explain in the text how
609 they were calculated and reference the corresponding figures or tables in the text.
610 8. Experiments compute resources
611 Question: For each experiment, does the paper provide sufficient information on the com-
612 puter resources (type of compute workers, memory, time of execution) needed to reproduce
613 the experiments?
614 Answer: [No]
615 Justification: Detailed compute resource information is not reported in this submission.
616 Experiments were conducted on TPU hardware. We note that the experimental scale (ViT-S,
617 ImageNet-100) is modest and reproducible on commonly available academic compute; full
618 training and hyperparameter details are provided in Appendix A.

15
619 Guidelines:
620 • The answer [N/A] means that the paper does not include experiments.
621 • The paper should indicate the type of compute workers CPU or GPU, internal cluster,
622 or cloud provider, including relevant memory and storage.
623 • The paper should provide the amount of compute required for each of the individual
624 experimental runs as well as estimate the total compute.
625 • The paper should disclose whether the full research project required more compute
626 than the experiments reported in the paper (e.g., preliminary or failed experiments that
627 didn’t make it into the paper).
628 9. Code of ethics
629 Question: Does the research conducted in the paper conform, in every respect, with the
630 NeurIPS Code of Ethics [Link]
631 Answer: [Yes]
632 Justification: The research involves standard machine learning experimentation on publicly
633 available image datasets. No human subjects, sensitive data, or high-risk applications are
634 involved. Anonymity is preserved in this submission.
635 Guidelines:
636 • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of
637 Ethics.
638 • If the authors answer [No], they should explain the special circumstances that require a
639 deviation from the Code of Ethics.
640 • The authors should make sure to preserve anonymity (e.g., if there is a special consid-
641 eration due to laws or regulations in their jurisdiction).
642 10. Broader impacts
643 Question: Does the paper discuss both potential positive societal impacts and negative
644 societal impacts of the work performed?
645 Answer: [N/A]
646 Justification: This work advances scientific understanding of representational geometry in
647 Vision Transformers. It does not involve human subjects, sensitive data, or applications with
648 direct societal impact, and we do not identify meaningful pathways to misuse.
649 Guidelines:
650 • The answer [N/A] means that there is no societal impact of the work performed.
651 • If the authors answer [N/A] or [No], they should explain why their work has no societal
652 impact or why the paper does not address societal impact.
653 • Examples of negative societal impacts include potential malicious or unintended uses
654 (e.g., disinformation, generating fake profiles, surveillance), fairness considerations
655 (e.g., deployment of technologies that could make decisions that unfairly impact specific
656 groups), privacy considerations, and security considerations.
657 • The conference expects that many papers will be foundational research and not tied
658 to particular applications, let alone deployments. However, if there is a direct path to
659 any negative applications, the authors should point it out. For example, it is legitimate
660 to point out that an improvement in the quality of generative models could be used to
661 generate Deepfakes for disinformation. On the other hand, it is not needed to point out
662 that a generic algorithm for optimizing neural networks could enable people to train
663 models that generate Deepfakes faster.
664 • The authors should consider possible harms that could arise when the technology is
665 being used as intended and functioning correctly, harms that could arise when the
666 technology is being used as intended but gives incorrect results, and harms following
667 from (intentional or unintentional) misuse of the technology.
668 • If there are negative societal impacts, the authors could also discuss possible mitigation
669 strategies (e.g., gated release of models, providing defenses in addition to attacks,
670 mechanisms for monitoring misuse, mechanisms to monitor how a system learns from
671 feedback over time, improving the efficiency and accessibility of ML).

16
672 11. Safeguards
673 Question: Does the paper describe safeguards that have been put in place for responsible
674 release of data or models that have a high risk for misuse (e.g., pre-trained language models,
675 image generators, or scraped datasets)?
676 Answer: [N/A]
677 Justification: This paper poses no such risks.
678 Guidelines:
679 • The answer [N/A] means that the paper poses no such risks.
680 • Released models that have a high risk for misuse or dual-use should be released with
681 necessary safeguards to allow for controlled use of the model, for example by requiring
682 that users adhere to usage guidelines or restrictions to access the model or implementing
683 safety filters.
684 • Datasets that have been scraped from the Internet could pose safety risks. The authors
685 should describe how they avoided releasing unsafe images.
686 • We recognize that providing effective safeguards is challenging, and many papers do
687 not require this, but we encourage authors to take this into account and make a best
688 faith effort.
689 12. Licenses for existing assets
690 Question: Are the creators or original owners of assets (e.g., code, data, models), used in
691 the paper, properly credited and are the license and terms of use explicitly mentioned and
692 properly respected?
693 Answer: [Yes]
694 Justification: The ImageNet-100 dataset is a subset of ImageNet, which is properly cited.
695 No proprietary code or models are used in this work.
696 Guidelines:
697 • The answer [N/A] means that the paper does not use existing assets.
698 • The authors should cite the original paper that produced the code package or dataset.
699 • The authors should state which version of the asset is used and, if possible, include a
700 URL.
701 • The name of the license (e.g., CC-BY 4.0) should be included for each asset.
702 • For scraped data from a particular source (e.g., website), the copyright and terms of
703 service of that source should be provided.
704 • If assets are released, the license, copyright information, and terms of use in the
705 package should be provided. For popular datasets, [Link]/datasets
706 has curated licenses for some datasets. Their licensing guide can help determine the
707 license of a dataset.
708 • For existing datasets that are re-packaged, both the original license and the license of
709 the derived asset (if it has changed) should be provided.
710 • If this information is not available online, the authors are encouraged to reach out to
711 the asset’s creators.
712 13. New assets
713 Question: Are new assets introduced in the paper well documented and is the documentation
714 provided alongside the assets?
715 Answer: [N/A]
716 Justification: This paper does not release new assets.
717 Guidelines:
718 • The answer [N/A] means that the paper does not release new assets.
719 • Researchers should communicate the details of the dataset/code/model as part of their
720 submissions via structured templates. This includes details about training, license,
721 limitations, etc.

17
722 • The paper should discuss whether and how consent was obtained from people whose
723 asset is used.
724 • At submission time, remember to anonymize your assets (if applicable). You can either
725 create an anonymized URL or include an anonymized zip file.
726 14. Crowdsourcing and research with human subjects
727 Question: For crowdsourcing experiments and research with human subjects, does the paper
728 include the full text of instructions given to participants and screenshots, if applicable, as
729 well as details about compensation (if any)?
730 Answer: [N/A]
731 Justification: This paper does not involve crowdsourcing nor research with human subjects.
732 Guidelines:
733 • The answer [N/A] means that the paper does not involve crowdsourcing nor research
734 with human subjects.
735 • Including this information in the supplemental material is fine, but if the main contribu-
736 tion of the paper involves human subjects, then as much detail as possible should be
737 included in the main paper.
738 • According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
739 or other labor should be paid at least the minimum wage in the country of the data
740 collector.
741 15. Institutional review board (IRB) approvals or equivalent for research with human
742 subjects
743 Question: Does the paper describe potential risks incurred by study participants, whether
744 such risks were disclosed to the subjects, and whether Institutional Review Board (IRB)
745 approvals (or an equivalent approval/review based on the requirements of your country or
746 institution) were obtained?
747 Answer: [N/A]
748 Justification: The paper does not involve crowdsourcing nor research with human subjects.
749 Guidelines:
750 • The answer [N/A] means that the paper does not involve crowdsourcing nor research
751 with human subjects.
752 • Depending on the country in which research is conducted, IRB approval (or equivalent)
753 may be required for any human subjects research. If you obtained IRB approval, you
754 should clearly state this in the paper.
755 • We recognize that the procedures for this may vary significantly between institutions
756 and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the
757 guidelines for their institution.
758 • For initial submissions, do not include any information that would break anonymity (if
759 applicable), such as the institution conducting the review.
760 16. Declaration of LLM usage
761 Question: Does the paper describe the usage of LLMs if it is an important, original, or
762 non-standard component of the core methods in this research? Note that if the LLM is used
763 only for writing, editing, or formatting purposes and does not impact the core methodology,
764 scientific rigor, or originality of the research, declaration is not required.
765 Answer: [N/A]
766 Justification: The core method development in this research does not involve LLMS as any
767 important, original, or non-standard components.
768 Guidelines:
769 • The answer [N/A] means that the core method development in this research does not
770 involve LLMs as any important, original, or non-standard components.
771 • Please refer to our LLM policy in the NeurIPS handbook for what should or should not
772 be described.

18