Code Prompting Elicits Conditional Reasoning Abilities

in Text+Code LLMs

Haritz Puerto1 , Martin Tutek2 * , Somak Aditya3 , Xiaodan Zhu1,4 , Iryna Gurevych1
1
Ubiquitous Knowledge Processing Lab (UKP Lab),
TU Darmstadt and Hessian Center for AI ([Link])
2
Technion – IIT, 3 IIT Kharagpur, 4 Queen’s University
[Link]

Abstract
Reasoning is a fundamental component of lan-
guage understanding. Recent prompting tech-
arXiv:2401.10065v3 [[Link]] 28 Sep 2024

niques, such as chain of thought, have consis-

tently improved LLMs’ performance on vari-
ous reasoning tasks. Nevertheless, there is still
little understanding of what triggers reasoning
abilities in LLMs in the inference stage. In
this paper, we investigate the effect of the in-
put representation on the reasoning abilities Figure 1: Code prompting converts a natural language
of LLMs. We hypothesize that representing problem into a code prompt and prompts a large lan-
natural language tasks as code can enhance spe- guage model with such code to generate an answer.
cific reasoning abilities such as entity tracking
or logical reasoning. To study this, we pro-
pose code prompting, a methodology we opera- (Liu et al., 2020, 2023a; Sinha et al., 2019), and
tionalize as a chain of prompts that transforms commonsense-focused reasoning (Madaan et al.,
a natural language problem into code and di- 2022; Liu et al., 2022a,b; Wang et al., 2023). Condi-
rectly prompts the LLM using the generated
tional reasoning, a primary yet complex reasoning
code without resorting to external code execu-
tion. We find that code prompting exhibits a ability that draws alternative conclusions depend-
high-performance boost for multiple LLMs (up ing on the fulfillment of certain conditions, remains
to 22.52 percentage points on GPT 3.5, 7.75 on understudied. These conditions are stated in the
Mixtral, and 16.78 on Mistral) across multiple text, making the problem self-contained, which
conditional reasoning datasets. We then con- allows us to study the semantic inferencing capa-
duct comprehensive experiments to understand bilities of the underlying model, i.e., identifying
how the code representation triggers reasoning relevant premises and ascertaining the presence
abilities and which capabilities are elicited in
of total evidence (Nolt et al., 1988; Cabria and
the underlying models. Our analysis on GPT
3.5 reveals that the code formatting of the input Magnini, 2014) without the requirement for, and
problem is essential for performance improve- confounding effects of external knowledge. Condi-
ment. Furthermore, the code representation im- tional reasoning is also a fundamental form of rea-
proves sample efficiency of in-context learning soning useful in many practical scenarios, such as
and facilitates state tracking of entities.1 answering real-world questions about the eligibility
for a visa or a loan. Despite the recent introduction
1 Introduction
of some benchmarks (Saeidi et al., 2018; Sun et al.,
Reasoning is a fundamental component of both 2022; Kazemi et al., 2023), conditional reasoning
human and artificial intelligence (AI) and the back- abilities of LLMs remain understudied.
bone of many NLP tasks. Recently, intensive stud- Recently, researchers have analyzed the syner-
ies have been performed on different aspects or gies between LLMs and symbolic interpreters to
types of reasoning such as mathematical reason- improve performance on reasoning tasks (Gao
ing (Patel et al., 2021; Chen et al., 2021b; Cobbe et al., 2023; Chen et al., 2023; Lyu et al., 2023).
et al., 2021), various kinds of logical reasoning These works transform structured reasoning prob-
* Work done while author was at TU Darmstadt. lems, such as mathematic or symbolic reasoning,
1
Our code and prompts are available at this URL. into code and run it on an external interpreter. In
such a setup, LLMs are mainly focused on natu- reasoning abilities, showing that prompting
ral language representation aspects and planning with code yields largely improved variable
how to solve the problem, while the actual logi- state tracking.
cal reasoning is offloaded to an external execution
module, confounding our understanding of the rea-
soning In particular, the fundamental questions of 2 Background and Related Work
what contributes to the reasoning abilities and how LLM Types. We categorize LLMs into three
reasoning abilities are triggered in LLMs remain types according to their intended use: i) LLMs for
open. Nevertheless, pretraining on code is con- natural language text (text-only LLMs), ii) LLMs
sidered an important component that contributes for coding tasks (code-only LLMs), and iii) LLMs
to and explains the improved reasoning ability of for natural language and coding tasks (text+code
LLMs. State-of-the-art LLMs such as GPT 3.5 (Ko- LLMs). The intended use of text-only LLMs (Zhang
jima et al., 2022), GPT 4 (OpenAI, 2023), Mixtral et al., 2022; Touvron et al., 2023a) is to process
(Jiang et al., 2024), and Mistral 7B (Jiang et al., and generate natural language text such as answers
2023) have been pretrained on both text and code to questions. The intended use of code-only LLMs
and have demonstrated considerable boosts in many (Li et al., 2023b; Roziere et al., 2023) is to pro-
reasoning benchmarks. cess and generate code. Lastly, text+code LLMs
In this work, we analyze whether one can elicit are equally capable of solving natural language
improved conditional reasoning abilities in LLMs and coding tasks. Examples of this are GPT 3.5
by merely changing the input format, i.e., from text (Kojima et al., 2022), Mixtral (Jiang et al., 2024),
to code. We constrain our experiments to text+code and Mistral (Jiang et al., 2023). In this work, we
LLMs to run text and code inputs on the same un- focus on text+code LLMs because of their ability
derlying model. In this way, we can avoid the to process two types of input representations inter-
confounding factor of different pretraining data of changeably: natural language text and code. Using
specialized text and code LLMs. To understand such models eliminates the confounding effect of
the benefit of code as an intermediate representa- fine-tuning between model variants specialized for
tion, we devise a chain of prompts, code prompting, only text or code.
that transforms a natural language (NL) task into
code and directly prompts the LLM with the gener- Augmenting text with code. Most works that
ated code. The code contains the logical structure generate code to solve natural language tasks use
needed to solve the problem, along with the orig- an external symbolic interpreter to run the result-
inal natural language text as code comments. An ing code. Chen et al. (2023) and Gao et al. (2023)
illustration is provided in Figure 1. Our contribu- showed consistent gains on mathematical problems,
tions are summarized as follows: symbolic reasoning, and algorithmic problems by
• We propose a methodology to investigate how using LLMs aided by external code interpreters.
the input representation impacts the reasoning Lyu et al. (2023) further report improvements in
abilities of text+code LLMs. boolean multi-hop QA, planning, and relational
inference. In contrast, Ye et al. (2023) used an ex-
• We operationalize such methodology by intro- ternal automated theorem prover with declarative
ducing a chain of prompts that transforms a code and showed consistent gains w.r.t. imperative
NL task into code, which is then sent back to code-interpreter-aided LLMs on arithmetic reason-
the LLM to generate NL answers. ing, logical reasoning, symbolic reasoning, and
regex synthesis tasks. Pan et al. (2023) did not
• We conduct a comprehensive study to com-
use any interpreter and instead created programs
pare code prompts with text prompts, show-
composed of multiple subroutines and used smaller
ing (i) large performance gains on the three
specialized models to run them. In this way, they
LLMs (up to 22.52 points for GPT3.5, up to
outperform text prompts on text LLMs for fact-
7.75 for Mixtral, and up to 16.78 for Mistral),
checking tasks. Lastly, Li et al. (2023a) runs pieces
while (ii) being more efficient with regard to
of code in an LLM to update the program state
the number of demonstrations.
when the Python interpreter fails due to a code
• We conduct extensive analysis to understand exception and shows performance gains on BIG-
why code prompts efficiently elicit conditional Bench Hard (Suzgun et al., 2022). All these works
Figure 2: Code prompting converts natural language descriptions into code to be solved with a large language
model. The figure shows a transformed instance from the ConditionalQA dataset.

investigate how to best use an external symbolic in- To the best of our knowledge, only the work of
terpreter to aid an LLM in solving reasoning tasks, Hussain et al. (2023) investigates the conditional
i.e., they run code and therefore have a program reasoning abilities of LLMs. However, they only
state with variables and its values. However, we analyze the abilities of text LLMs after training
do not employ any external symbolic reasoner, and them on ConditionalQA (Sun et al., 2022).
we do not run code. We investigate the reasoning
abilities of LLMs under different input representa- 3 Code Prompting
tions (i.e., text and code). Our code prompts are We posit that each LLM encodes a set of capabili-
not executed; they are simply read by the LLM and ties, such as mathematical, logical, or conditional
used to generate a natural language answer. reasoning. However, not all of them are used for
every input instance, even if they would be use-
Some works suggest that code LLMs may pos-
ful. We hypothesize that the input representation
sess superior reasoning abilities than text LLMs.
plays a pivotal role in eliciting such capabilities.
Madaan et al. (2022) investigate whether code
Prior works show that LLMs trained on a combi-
LLMs are superior at structured reasoning than
nation of text and code exhibit superior reasoning
text LLMs. They observe that code LLMs can gen-
abilities (Kojima et al., 2022; OpenAI, 2023; Jiang
erate graphs that link commonsense concepts better
et al., 2024, 2023). Therefore, we conjecture that
than text LLMs. Liu et al. (2023b) investigate code
a code representation of a natural language (NL)
prompts in abductive and counterfactual reasoning
problem may trigger some of these reasoning abil-
tasks and report superior results than text prompts
ities encoded in text+code LLMs. More formally,
on code-davinci (Ouyang et al., 2022), a code
we wonder whether exists some space S 2 with an
LLM. However, code prompts exhibit mixed re-
associated function f that transforms a natural lan-
sults on text-davinci-002 (Ouyang et al., 2022),
guage problem p ∈ N into that space, such that,
a text LLM. We attribute this to the fact that while
when prompting an LLM with the representation
this model includes some code in its pretraining cor-
of p in such space yields better results according to
pus, it is not explicitly trained for code generation
some evaluation function σ.
and, in general, performs poorly on code generation
tasks (Chen et al., 2021a). Therefore, the effect of ∃S, f : N → S, σ(LLM (f (p)) ≥ σ(LLM (p))
the input representation on the reasoning abilities of
text+code LLMs remains unclear. Furthermore, the We fix S to the programming language space and
reasons behind the superior performance of code define code prompts f (p) as prompts that model
prompts in code LLMs also remain unclear. In our a natural language problem with code. We also
work, we aim to answer whether code prompts can 2
Since the input of LLMs must be strings, S must be a set
elicit conditional reasoning abilities in text+code of all possible sentences constructed using some alphabet and
LLMs and the reasons behind this. grammar.
define f as a prompt that transforms the NL text we focus on the main partition, which includes
into code. f (p) code follows the original NL text three subsets BGQA-1, BGQA-2, and BGQA-3, where
as much as possible. We use a simple structured the number indicates the reasoning hops needed to
code that contains the logical structure needed to answer. Lastly, while ShARC encompasses dialogue
solve the problem, along with the original NL text generation, we aim to evaluate specific capabilities
as code comments. In particular, it creates vari- unrelated to conversational flow. Therefore, we
ables for key entities in the question and documents isolated the QA pairs from the provided dialogues,
and if blocks for each conditional statement in the resulting in a dataset where the model has to answer
documents. Figure 2 exemplifies this transforma- yes, no, or not enough information.3 We include
tion and Appendix D provides more details of the more details about the datasets in Appendix B, a
code features. Lastly, we define code prompting formal definition of the prompts in Appendix C,
as LLM (f (p)), a chain of prompts that i) trans- and examples in Appendix Q.
form the NL text into code, and ii) use this code to
prompt the LLM to generate the answer in natural 4.3 Models
language. Figure 1 illustrates this pipeline.
It is important to note that the code is not ex- We perform our study using text+code LLMs be-
ecuted per se and therefore, there is no program cause of their ability to process text and code inter-
state. We simply prompt the LLM with the code changeably. We do not employ code-only LLMs
and ask the LLM to generate a natural language because their intended use does not include solving
answer based on the content of such code. This natural language tasks (Roziere et al., 2023), as
setup allows us to investigate the effect of the input required in our case. Similarly, we do not employ
representation on text+code LLMs. text-only LLMs because they cannot generate code.
Furthermore, using text+code LLMs also allow us
4 Experimental Setup to eliminate the confouding effect of fine-tuning
between model variants specialized for only text or
4.1 Task Setup code. We corroborate the shortcomings of text-only
We evaluate the conditional reasoning abilities of and code-only LLMs with additional experiments
the LLMs under different prompting methods us- on CodeLLaMA (Roziere et al., 2023) and LLaMA
ing the traditional question-answering task. The 2 (Touvron et al., 2023b) in Appendix G and H.
input is a question and a document, and the model We employ OpenAI’s gpt-35-turbo, Mixtral
needs to produce the answer, which can be a span 8x7B (Jiang et al., 2024), and Mistral 7B (Jiang
of the input document, yes or no. Given that chain- et al., 2023). The use of these models allows us to
of-thought prompting (CoT, Wei et al. 2022) has investigate whether our hypothesis holds across all
been shown to improve the reasoning abilities of available sizes of text+code LLMs. We execute our
LLMs, we instruct the model to generate a CoT prompts with in-context learning and provide one
before the final answer in all prompting methods. demonstration per class. More details on the LLM
Since we force code prompting to generate a natu- setup are provided in Appendix E.
ral language answer, we also force it to generate a
natural language CoT. 4.4 Evaluation

4.2 Datasets We follow the evaluation metrics used in the orig-

inal datasets. For CondQA, we report the F1 token
Throughout our experiments, we use three question-
overlap between the predicted answer and the la-
answering (QA) datasets for conditional reason-
bel, while for BGQA and ShARC, we report the macro
ing: ConditionalQA (CondQA; Sun et al., 2022), a
F1 score. We run the main experiments two times
scenario-based question answering (QA) dataset,
with different random seeds (0 and 1). We report
BoardgameQA (BGQA; Kazemi et al., 2023), a
the average and standard deviation performance
boardgame-base QA dataset with conflicting rules,
across these runs. For the subsequent analyses of
and ShARC (Saeidi et al., 2018), a conversational
code prompts, we run each experiment once only
QA dataset with natural language rules. Solving
on GPT 3.5 due to the inference costs.
these datasets requires advanced conditional and
compositional reasoning capabilities. 3
In the full task, not enough information would trigger
We focus on the QA task of CondQA. For BGQA, another step in a pipeline to generate a follow-up question.
 Model Prompt CondQA ShARC BGQA-1 BGQA-2 BGQA-3 ∆CP
Test Set
Text 58.70 62.95 51.15 37.42 27.77
GPT 3.5 Code 60.60 54.98 58.67 55.56 50.29 8.42
Text 48.17 53.77 56.38 39.64 30.15
Mixtral Code 44.73 59.06 53.33 47.39 44.72 4.22
Text 35.74 43.60 47.40 48.78 47.86
Mistral Code 33.28 49.92 53.80 51.27 48.79 2.74
Dev Set
Text 56.54 ± 0.08 64.10 ± 0.10 53.16 ± 1.67 33.71 ± 10.37 31.5 ± 13.39
GPT 3.5 Code 57.64 ± 1.42 58.54 ± 1.22 68.60 ± 1.09 55.85 ± 4.06 47.57 ± 2.68 9.84
Text 46.60 ± 0.99 55.71 ± 2.51 58.31 ± 1.77 36.77 ± 0.09 32.06 ± 1.79
Mixtral Code 40.88 ± 1.84 58.96 ± 3.44 57.94 ± 5.52 45.32 ± 0.54 38.90 ± 7.33 2.51
Text 28.84 ± 0.02 37.56 ± 0.78 47.61 ± 0.92 47.29 ± 1.97 46.56 ± 2.92
Mistral Code 28.26 ± 10.03 53.42 ± 0.93 52.21 ± 0.95 54.27 ± 1.42 45.22 ± 10.75 5.10

Table 1: Comparison (F1 score) of text prompt and code prompts. All results use one demonstration per class.
∆CP = Code Prompt - Text Prompt, i.e., the average performance gain from code prompts across all datasets.

5 Experiments through code prompts for most datasets (i.e., GPT-

3.5 in 4/5, Mixtral in 3/5, Mistral in 4/5). Notably,
We devise a set of experiments to analyze and quan- code prompts consistently surpass text prompts on
tify whether the code representation of a natural BGQA-2 and BGQA-3, the most reasoning-intensive
language prompt (i.e., code prompts) elicits condi- datasets (see Appendix B), for all models. This
tional reasoning abilities and why. We first com- is particularly evident for GPT-3.5, where gains
pare the performance of the two prompting meth- exceed 18 points. Conversely, the advantage is
ods — text prompts and code prompts on three narrower on CondQA, where the linguistic dimen-
LLMs across three datasets (§5.1). We then con- sion plays the biggest role (see Appendix B). This
duct extensive ablation experiments on the dev set suggests that code prompts elicit conditional rea-
of the datasets with GPT 3.5, the best-performing soning abilities and are most suited for reasoning-
and largest model, to understand the reason behind intensive tasks. Furthermore, in the cases where
the performance gain from code prompting. In text prompts are superior, their average gains are
particular, we study whether code syntax or the im- only 4.23. In contrast, code prompts lead to signifi-
plicit text simplification from the code translation is cantly larger mean gains of 8.53 in the cases where
what improves performance (Section 5.2). We also they are superior. Additionally, an experiment with
check if the improvement is caused by the mod- Phi-2, a small language model, reveals a substan-
els merely being exposed to code within prompts tial 15-point performance improvement using code
and not necessarily the instances translated to code prompts (see Appendix I).
(Section 5.3). Furthermore, we show that code
prompting is more sample efficient (Section 5.4) To shed light on the performance gains driven by
when compared to text prompting and that models code prompts, we delve into the confusion matri-
prompted with code exhibit superior state tracking ces (attached in Appendix N) and discover that text
capabilities (Section 5.5). Lastly, we conduct a prompts in Mistral predict “not enough information”
human evaluation that confirms the faithfulness of much less than code prompts for BGQA. This is par-
the generated code in Section 6. ticularly noticeable in BGQA-1, where text prompts
do not predict a single “not enough information,”
5.1 Code Prompting Improves over Text while code prompts do. On the other hand, text
Prompting prompts in GPT 3.5 and Mixtral overpredict “not
Table 1 shows the model performance on the de- enough information” on BGQA and ShARC, leading
velopment and test sets. Code prompts outperform to a low number of true positives for the conclusive
text prompts in the majority of cases on the test answers. We hypothesize that this model hesita-
set (11 out of 15). This trend holds true across tion could stem from the alignment tax (Ouyang
models, with each achieving peak performance et al., 2022) of reinforcement learning from human
feedback models. This potential barrier may be Figure 2 would be back-transformed as Key en-
alleviated by code prompts because they indicate tity: husband pass away. To transform the if state-
to the model the variable that answers the question ments, we create a translation prompt by providing
and instruct the model to track the entailment status four demonstrations. These demonstrations sim-
of variables within the given code. ply translate the conditional statements within the
These consistent and substantial gains from code code-formatted instance back into natural language.
prompts are obtained despite a straightforward We also translate the variables in the same manner.
transformation of text prompts, which does not This makes the back-translated text as close as pos-
incorporate new information, as shown in Figure 2. sible to the code text. We provide examples of this
This finding strongly suggests that code possesses in Table 16 from Appendix P.
specific characteristics that effectively elicit condi- Results. The results5 in Table 2 show that (1)
tional reasoning abilities within text+code LLMs. prompting with atomic statements does not reach
the performance of code prompts, and (2) mapping
5.2 Code Syntax Elicits Reasoning Abilities back from code to NL results in a performance
We now want to delve into the source of the perfor- drop compared to code prompts. These findings
mance gains observed when using code prompting. suggest that code prompts enhance LLM perfor-
We investigate whether these improvements stem mance beyond mere text simplification. This con-
from the simplification of text into premises fa- clusion is supported by the observation that these
cilitated by code, effectively reducing the task to alternative text simplification approaches, despite
a form of semantic inference within the linguis- offering similar semantics to code prompts, fail
tic dimension, or if there are inherent properties to replicate the performance gains observed with
of code syntax that contribute to enhanced perfor- code prompts. Therefore, these results imply that
mance. To investigate this, we devise experiments specific syntactic features embedded within code
with prompts that represent the intermediate states directly contribute to performance improvement.
between natural language and code. Lastly, our evaluation on BGQA-3 reveals a sig-
I. Atomic Statements. Inspired by Min et al. nificantly larger performance decline when using
(2023), we transform each NL sentence4 into a atomic statements compared to back-translated
sequence of atomic statements, which we then ap- code. This disparity likely stems from the dataset’s
pend to the original sentence. In this way, the inherent structure. The method we employ for gen-
atomic statements can be seen as defining variables erating atomic statements (Min et al., 2023) was
for each key entity in the text. Hence, this new specifically designed for general text formats like
prompt would resemble code but without control Wikipedia pages. However, BGQA is a logic-based
flow and in natural language form. The prompt dataset where input "facts" are already presented as
retains access to the original instance text (i.e., no minimally informative statements, deviating from
loss of information) but is also augmented by sim- the typical structure of general documents. As a
plified sentences in the form of atomic statements. result, generating atomic statements from these
This setup allows us to investigate whether the sim- sentences can unintentionally disrupt the sentence
plicity of the input triggers improves reasoning abil- structure, making it difficult to track the attributes
ities, regardless of the text and code syntax. of subjects and objects within the text. This ob-
servation is further supported by our results on
II. Back-Translated Code. In our second experi-
CondQA, a dataset with longer documents, where
ment, we investigate whether the semantics of the
atomic statements achieve higher performance than
code statements and not the code syntax are the rea-
back-translated code.
son behind the performance boost. For this purpose,
we back-transform the code prompts into NL such
5.3 Code Semantics are Important
that the reasoning statements (i.e., the if conditions)
are clearly and concisely stated in natural language. Previously, we have shown that code syntax is nec-
Specifically, we map every variable into the for- essary to elicit the reasoning abilities of text+code
mat Key entity: variable without snake case. For 5
We do not conduct ablation tests on ShARC because,
instance, the variable husband_pass_away from as explained in Section 5, these ablations aim to understand
why code prompts outperform text prompts using the highest
4
We only transform the facts in BGQA since transforming performing model. Results for Mistral and Mixtral are shown
the rules into atomic statements as well yields worse results. in Appendix J.
 Dataset ∆ Atomic St. ∆ Code → NL Prompt CQA CQA-YN BG1 BG2 BG3
CondQA −2.66 −4.72 Anonym. −1.62 −2.90 −6.60 −4.80 −4.00
Random −3.40 −2.67 −7.40 −9.20 −9.80
BGQA-1 −4.37 −1.43 - Comments N.A. −14.02 −16.70 −16.20 −5.20
BGQA-2 −8.72 −5.39
BGQA-3 −19.26 −3.68
Table 3: Performance gap to code prompts for each code
Table 2: Performance gap of atomic statements and perturbation. cQA stands for CondQA, CQA-YN for
back-translated code when compared to code prompts the partition of CondQA with yes-no answers, BG for
using GPT 3.5. Results from the dev set of each dataset. BGQA. Results reported on the dev set of each dataset.

LLMs. Now, we aim to investigate which aspects on CondQA and BGQA-1. This can be interpreted as
of code are pivotal. In particular, we evaluate the the model being able to identify the irrelevance of
impact of retaining the natural language text of the the code to the text. Hence, the model disregards
original instance within the code comments and the the code to solely focus on the code comments (i.e.,
importance of the code semantics. To analyze the the natural language text). This could be possible
former, we have (1) removed the code comments thanks to the provided demonstrations, which show
that include the original natural language text from answers that only refer to the natural language text.
the input and evaluated the performance of the new
prompts. To analyze the latter, we (2) perturbed These results confirm that code alone does not
the code to anonymize the variables and functions, trigger reasoning abilities, and instead, the combi-
as well as (3) added random code whose seman- nation of code that represents the original natural
tics are completely irrelevant to the original natural language instance and the NL text is able to unlock
language text. In the latter two cases, the code com- the potential of LLMs. We observe similar patterns
ments remain unmodified (examples illustrating on Mistral and Mixtral in Appendix J.
them are provided in Table 17 from Appendix P).
Since CondQA includes span answers and removing
5.4 Code Prompts are More Sample-Efficient
the NL text would make it impossible for the model
at Eliciting Reasoning Abilities
to generate the span, we only report performance
on the yes-no answers partition (CondQA-YN).
Given our observations that code prompts trig-
Table 3 shows that removing the NL text in the
ger conditional reasoning abilities better than text
code comments yields a performance drop of 14.02
prompts, it is natural to ask the follow-up question:
points on CondQA and a performance drop between
are code prompts also more sample-efficient than
16.7 and 5.2 on BGQA. This significant and consis-
text prompts? To answer this, we evaluate how the
tent decrease in all datasets confirms that retaining
overall performance of GPT 3.5 changes with re-
NL text in comments is vital for the LLM to under-
spect to the number of demonstrations for the two
stand the input problem.
prompting methods.
Effect of Code Perturbations. Code perturba- Figure 3 shows that when we only provide
tions (anonymous code and random code) confirm one demonstration per class (i.e., answer type in
the importance of code semantics in eliciting rea- our datasets), the performance gap is the largest
soning abilities. When we use anonymized code, across all datasets. As expected, this gap de-
we observe a performance reduction of almost 2 creases when we provide more demonstrations.
points on CondQA and a decrease between 6.6 and 4 Moreover, we also observe that code prompts with
in BGQA. The decrease is even larger when the code only one demonstration per class even outperform
is randomized, with drops of more than 3 points text prompts with three demonstrations per class,
on CondQA and between 7.4 and 9.8 on BGQA. This which further shows the sample efficiency of code
more pronounced drop is expected since the seman- prompts. These results indicate that code prompts
tics and logic of the code mismatch the NL text, trigger conditional reasoning more efficiently than
whereas anonymous code maintains the same logic text prompts on GPT 3.5, and this is one of the
on both NL and code. Furthermore, we also ob- reasons for its superior performance. We conduct
serve that the performance drop of random code additional analysis on Mistral and Mixtral in Ap-
prompts is similar to that of text prompts (Table 1) pendix K.
 Correct Ans. Incorrect Ans.
Dataset Text Code Text Code
CondQA 71.08 4.39 60.79 11.39
BGQA-1 39.33 8.84 51.65 22.12
BGQA-2 44.79 15.04 52.54 24.75
BGQA-3 54.01 14.21 52.13 16.98

Table 4: Comparison of the percentage of memory er-

Figure 3: Performance comparison of GPT 3.5 between rors made by GPT 3.5. For each dataset, we separately
text (green) and code prompts (blue) using 1, 2, and 3 compute memory errors for the instances where the
demonstrations per class. Results reported on dev sets. model gives the correct and incorrect answers. Lower is
better. Results from the dev set of each dataset.

5.5 Code Prompts Improve Variable Tracking

in LLMs
Does Generated Text reflect Model Beliefs? As
We hypothesize that one of the reasons for the supe-
the generated text may not be faithful to the internal
rior performance of code prompting is an improved
beliefs of the model (Lyu et al., 2023), we first test
ability to identify and track the states of key vari-
the validity of this experiment as a proxy metric
ables or concepts. This hypothesis is based on the
of the internal belief of the model. To do this,
intuition that, for natural language in general, lo-
we compare the memory error percentage of the
cal context is the most important part to generate
prompting methods in instances where the model
the next token (Khandelwal et al., 2018; Sun et al.,
solves (i.e., correct instances) and does not solve
2021). However, generating code is often more
(i.e., incorrect instances) the question. If incorrect
challenging because code frequently refers to pre-
instances yield a higher memory error, this would
viously defined functions and variables, which can
indicate that the model struggles more to remember
be dozens or even hundreds of lines apart. This
the variable states on those instances, which in turn
resembles multi-hop reasoning, where the model
would make it more likely to fail when conducting
may need to reference a key entity dozens of lines
the reasoning process. Therefore, our probes would
before. Therefore, an improved ability to look for
be a proxy metric of the internal belief of the model.
distant co-references caused by training on code
can be beneficial for multi-hop reasoning, which is Table 4 shows the results of this comparison. We
also needed to solve our datasets. observe that all prompting methods in all datasets
To test our hypothesis, we devise the following consistently make more memory mistakes on in-
experiment. Firstly, we define reasoning step as correct instances than on correct instances, with
each output sentence split by “\n.” After generating the exception of text prompts on CondQA. However,
each reasoning step, we stop the model generation the memory error in this case is significantly high,
and query about all key entities defined in the input which may suggest that the model is not able to
prompt. In the case of text prompts, we query the track entities correctly in either case. Therefore,
model whether the given facts are true or not, and we can use this experiment as a proxy measure of
for code prompts, we query for the value of the the memory of the model.
(boolean) variables. In all cases, the model only
has to generate True, False, a string, or unknown. Code Prompting improves State Track-
Then, we compare the percentage of errors in text ing. From Table 4, we further observe that Text
and code prompts. This number represents the Prompts make significantly more memory errors
memory errors committed by the model. The more than code prompts on all datasets on GPT 3.5
memory errors there are, the more difficult it is for (Results for Mistral and Mixtral are provided in
the model to track and remember entities/variables. Appendix M). Specifically, the gap is consistently
We provide further details on how we extracted the more than 30% with peaks on CondQA (66.69%)
key entities to ask for, how we identified the reason- and BGQA-3 (39.8%). Therefore, this experiment
ing steps in the chain of thought used to stop the empirically confirms our hypothesis that code
model for conducting the probes, and examples of prompts improve state tracking of the key entities
the prompt probes in Appendix L and its Table 18. and variables when compared to text prompts.
6 Human Analysis of the Generated Code in the case of Mistral 7B, we observed a bit worse
results. Only 4 out of the 10 cases are perfect trans-
We conduct a small human evaluation to confirm
lations. In two cases, there is no code, and instead,
the faithfulness of the generated code to the source
the model only generated the original natural lan-
natural language text. We evaluate the code gen-
guage text in code comments. We also observe
erations of all our models on ten random samples
one case where the first half of the text is correctly
from the dev set of CondQA, ShARC, and BGQA (in
translated into code but the second half only con-
particular, we use BGQA-1 partition). We check for
tains the code comments representing the natural
perfect translations, and for the failing cases, we
language text. We also observe one case where the
analyze the errors.
code is correct, but the indentation is wrong; all
BGQA. We observe perfect translations in all mod- code blocks are under the first if statement, which
els for all the analyzed samples. We attribute this should not be like that. Lastly, we find two cases
effectiveness to the close alignment between the where the if statements do not contain an execution
natural language documents and first-order logic. body. Notably, even in the cases where the code
is not perfect, the original semantics from the nat-
ShARC. We observe that GPT 3.5 generates per-
ural language remain untouched because they are
fect translations in all cases except one. However,
preserved through code comments.
this case is a corner case where the document is
This analysis confirms that, in general, the trans-
irrelevant to the question, and therefore, there is
lated code faithfully represents the semantics of the
no answer. Furthermore, the document is only one
source natural language text.
line. Consequently, the model does not generate
code and simply keeps the text as a code comment.
In the case of Mixtral 8x7B, we observed perfect 7 Conclusions and Future Work
code translations for 70% of the samples. One of
This work demonstrates that the code representa-
the failing cases assings as the question variable
tion of a natural language task (i.e., code prompts)
a variable that is actually from the conversation
can elicit reasoning abilities in large language mod-
history, no the quesiton. Another error case ex-
els (LLMs) of text and code. These code prompts
hibits wrong value assingments to some variables.
contain the original natural language (NL) formu-
They should be none, but they are assinged true
lation as a code comment and code that formulates
and false. The last case is the corner case explain
the logic of the text. To create these code prompts,
above. As for Mistral 7B, we find that 60% of
we use in-context learning to teach an LLM how
the analyzed samples have a perfect translation. In
to conduct such a transformation automatically.
the remaining 40%, we observe three cases with
Through multiple experiments on three LLMs and
no question variable and the same corner case as
three datasets, we show that code prompts trigger
before. However, the semantics of the natural lan-
conditional reasoning abilities, with large perfor-
guage text remain, thanks to the code comments.
mance gains w.r.t. text prompts (up to 22.52 per-
CondQA. We observe that GPT 3.5 generates a centage points on GPT 3.5, 7.75 on Mixtral, and
perfect translation in 8 out of the 10 cases. In these 16.78 Mistral). Our experiments reveal that even
two cases with errors, we observe that most of the simple code can be beneficial as long as it closely
code is correct, but in both cases, one conditional follows the semantics of the NL text and is accom-
statement is missed. The model directly generates panied by the original NL text. We also show that
the body of the if statement without the correspond- code prompts are more sample-efficient than text
ing if. It is also worth noting that the code is of prompts and that their performance boost stems
high quality, including data structures as ditionar- from their superior ability to identify and track the
ies, generating code that explains tables, and also state of key variables or entities, a central aspect of
generates lists of strings. In the case of Mixtral the logical dimension of semantic inference.
8x7B, we obtain perfect translations in 6 out of 10 In our future work, we plan to extend to a wider
cases. All the failing cases exhibit the same type of range of reasoning abilities, such as multi-hop rea-
error: there is a variable statement without a prior soning, to understand the capacity and generaliz-
if condition. It is worth noting that the code, in ability of code prompting. We also plan to investi-
general, is of high quality, contains data structures gate how pretraining on text, code, and text+code
such as dictionaries and even process tables. Lastly, affects the triggering of LLMs’ reasoning abilities.
Limitations Ethics and Broader Impact Statement
Our work aims to improve the reasoning abilities of
Transforming a natural language problem into code
LLMs. The use of code prompts may also simplify
requires an intermediate step that raises the cost of
the explainability of the model responses since we
the whole pipeline. However, this mapping is not a
can inspect the entailment status of the variables.
complicated task, as even the smallest models we
We hope these results contribute to enhancing the
considered were able to perform it successfully in
trustworthiness and safety of LLMs. Nevertheless,
an in-context learning setup. Therefore, we believe
every development may pose some risks. In our
it would be possible to train a small generative
case, the improvement of the reasoning abilities
model to do it instead of using a large language
in LLMs may be utilized by malicious actors to
model. In this way, we could minimize the cost
propagate more persuasive disinformation.
of using code prompts without affecting its perfor-
mance. Acknowledgements
We only ran the experiments on the dev set with
two different random seeds due to the costs of This work has been funded by the German Re-
running large language models and because we search Foundation (DFG) as part of the UKP-
prioritized experimenting on multiple models and SQuARE project (grant GU 798/29-1), by the Ger-
datasets. Nevertheless, the results of all models man Federal Ministry of Education and Research
exhibit similar patterns, which confirms the repre- and the Hessian Ministry of Higher Education,
sentativeness of our results. Also, we conduct the Research, Science and the Arts within their joint
ablations only on GPT 3.5, the best-performing and support of the National Research Center for Ap-
largest model. However, confirming that the find- plied Cybersecurity ATHENE, and by the Euro-
ings from these ablations also hold on the smaller pean Union (ERC, InterText, 101054961). Views
models would be interesting. and opinions expressed are, however, those of the
author(s) only and do not necessarily reflect those
This work focuses only on analyzing the effects
of the European Union or the European Research
of code representations for natural language tasks.
Council. Neither the European Union nor the grant-
However, it could be possible that other input rep-
ing authority can be held responsible for them. So-
resentation spaces also elicit reasoning abilities.
mak Aditya acknowledges travel support from the
We limit the scope of this work to only the space
European Union’s Horizon 2020 research and inno-
of simple structured languages (more details on
vation program under grant agreement No 951847.
Appendix D) because prior research suggests that
We gratefully acknowledge the support of Mi-
pretraining on code improves the reasoning abili-
crosoft with a grant for access to OpenAI GPT
ties of LLMs, but it might be possible that certain
models via the Azure cloud (Accelerate Founda-
natural languages such as German or Chinese, or
tion Model Academic Research).
certain types of programming languages, such as
Lastly, we thank Max Glockner, Jonathan Ton-
declarative or logical, also elicit certain abilities.
glet, Sheng Lu, and the anonymous reviewers for
Similarly, we do not conduct experiments on mul-
their insightful comments and suggestions on a
tiple code generation methods because our goal is
prior draft of this paper.
to analyze whether the mere change of the repre-
sentation can elicit reasoning abilities and not an
analysis of the best coding style. References
Our tasks require instruction following abilities,
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten
so we do not conduct comparisons of base vs. chat Bosma, Henryk Michalewski, David Dohan, Ellen
models. Future work could investigate whether Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021.
instruction tuning has an impact on the LLMs’ abil- Program synthesis with large language models. arXiv
ities to understand code. preprint arXiv:2108.07732.

Lastly, we conduct our experiments on data in Elana Cabria and Bernardo Magnini. 2014. Decom-
English. Analyzing whether our findings hold true posing semantic inference. Linguistic Issues in Lan-
guage Technology, 9.
in other languages would be interesting. However,
the lack of conditional reasoning datasets in other Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan,
languages would make this study difficult. Henrique Ponde de Oliveira Pinto, Jared Kaplan,
 Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky.
Brockman, et al. 2021a. Evaluating large lan- 2018. Sharp nearby, fuzzy far away: How neural lan-
guage models trained on code. arXiv preprint guage models use context. In Proceedings of the 56th
arXiv:2107.03374. Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 284–294,
Wenhu Chen, Xueguang Ma, Xinyi Wang, and Melbourne, Australia. Association for Computational
William W. Cohen. 2023. Program of thoughts Linguistics.
prompting: Disentangling computation from reason-
ing for numerical reasoning tasks. Transactions on Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu-
Machine Learning Research. taka Matsuo, and Yusuke Iwasawa. 2022. Large lan-
guage models are zero-shot reasoners. In Advances in
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena
Neural Information Processing Systems, volume 35,
Shah, Iana Borova, Dylan Langdon, Reema Moussa,
pages 22199–22213. Curran Associates, Inc.
Matt Beane, Ting-Hao Huang, Bryan Routledge, and
William Yang Wang. 2021b. FinQA: A dataset of nu- Tamera Lanham, Anna Chen, Ansh Radhakrishnan,
merical reasoning over financial data. In Proceedings Benoit Steiner, Carson Denison, Danny Hernan-
of the 2021 Conference on Empirical Methods in Nat- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jack-
ural Language Processing, pages 3697–3711, Online son Kernion, et al. 2023. Measuring faithful-
and Punta Cana, Dominican Republic. Association ness in chain-of-thought reasoning. arXiv preprint
for Computational Linguistics. arXiv:2307.13702.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian,
Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen,
Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-
Nakano, et al. 2021. Training verifiers to solve math Fei, Fei Xia, and Brian Ichter. 2023a. Chain of code:
word problems. arXiv preprint arXiv:2110.14168. Reasoning with a language model-augmented code
emulator. arXiv preprint arXiv:2312.04474.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon,
Pengfei Liu, Yiming Yang, Jamie Callan, and Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas
Graham Neubig. 2023. Pal: program-aided lan- Muennighoff, Denis Kocetkov, Chenghao Mou, Marc
guage models. In Proceedings of the 40th Interna- Marone, Christopher Akiki, Jia LI, Jenny Chim,
tional Conference on Machine Learning, ICML’23. Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo,
[Link]. Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier,
Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Lo-
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio gesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muh-
César Teodoro Mendes, Allie Del Giorno, Sivakanth tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Ja-
Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo son T Stillerman, Siva Sankalp Patel, Dmitry Ab-
de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all ulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang,
you need. arXiv preprint arXiv:2306.11644. Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni,
Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav
Syed-Amad Hussain, Parag Pravin Dakle, SaiKrishna Timor, Jennifer Ding, Claire S Schlesinger, Hailey
Rallabandi, and Preethi Raghavan. 2023. Towards Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra,
leveraging llms for conditional qa. arXiv preprint Alex Gu, Carolyn Jane Anderson, Brendan Dolan-
arXiv:2312.01143. Gavitt, Danish Contractor, Siva Reddy, Daniel Fried,
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz
sch, Chris Bamford, Devendra Singh Chaplot, Diego Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha,
de las Casas, Florian Bressand, Gianna Lengyel, Guil- Leandro Von Werra, and Harm de Vries. 2023b. Star-
laume Lample, Lucile Saulnier, Lélio Renard Lavaud, coder: may the source be with you! Transactions on
Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Machine Learning Research. Reproducibility Certifi-
Thibaut Lavril, Thomas Wang, Timothée Lacroix, cation.
and William El Sayed. 2023. Mistral 7b. arXiv
preprint arXiv:2310.06825. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue,
and Wenhu Chen. 2024. Long-context llms strug-
Albert Q Jiang, Alexandre Sablayrolles, Antoine gle with long in-context learning. arXiv preprint
Roux, Arthur Mensch, Blanche Savary, Chris Bam- arXiv:2404.02060.
ford, Devendra Singh Chaplot, Diego de las Casas,
Emma Bou Hanna, Florian Bressand, et al. 2024. Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan
Mixtral of experts. arXiv preprint arXiv:2401.04088. Duan, Ming Zhou, and Yue Zhang. 2023a. Logiqa
2.0—an improved dataset for logical reasoning in
Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung natural language understanding. IEEE/ACM Trans-
Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ra- actions on Audio, Speech, and Language Processing,
machandran. 2023. BoardgameQA: A dataset for 31:2947–2962.
natural language reasoning with contradictory infor-
mation. In Thirty-seventh Conference on Neural In- Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He,
formation Processing Systems Datasets and Bench- Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi.
marks Track, pages 1–23. 2022a. Rainier: Reinforced knowledge introspector
 for commonsense question answering. In Proceed- Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan
ings of the 2022 Conference on Empirical Methods Luu, William Yang Wang, Min-Yen Kan, and Preslav
in Natural Language Processing, pages 8938–8958, Nakov. 2023. Fact-checking complex claims with
Abu Dhabi, United Arab Emirates. Association for program-guided reasoning. In Proceedings of the
Computational Linguistics. 61st Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- 6981–7004, Toronto, Canada. Association for Com-
ter West, Ronan Le Bras, Yejin Choi, and Hannaneh putational Linguistics.
Hajishirzi. 2022b. Generated knowledge prompting
for commonsense reasoning. In Proceedings of the Arkil Patel, Satwik Bhattamishra, and Navin Goyal.
60th Annual Meeting of the Association for Compu- 2021. Are NLP models really able to solve simple
tational Linguistics (Volume 1: Long Papers), pages math word problems? In Proceedings of the 2021
3154–3169, Dublin, Ireland. Association for Compu- Conference of the North American Chapter of the
tational Linguistics. Association for Computational Linguistics: Human
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Language Technologies, pages 2080–2094, Online.
Yile Wang, and Yue Zhang. 2020. Logiqa: A chal- Association for Computational Linguistics.
lenge dataset for machine reading comprehension
with logical reasoning. In Proceedings of the Twenty- Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten
Ninth International Joint Conference on Artificial Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,
Intelligence, IJCAI-20, pages 3622–3628. Interna- Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.
tional Joint Conferences on Artificial Intelligence Code llama: Open foundation models for code. arXiv
Organization. Main track. preprint arXiv:2308.12950.

Xiao Liu, Da Yin, Chen Zhang, Yansong Feng, and Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer
Dongyan Zhao. 2023b. The magic of IF: Investi- Singh, Tim Rocktäschel, Mike Sheldon, Guillaume
gating causal reasoning abilities in large language Bouchard, and Sebastian Riedel. 2018. Interpretation
models of code. In Findings of the Association for of natural language rules in conversational machine
Computational Linguistics: ACL 2023, pages 9009– reading. In Proceedings of the 2018 Conference on
9022, Toronto, Canada. Association for Computa- Empirical Methods in Natural Language Processing,
tional Linguistics. pages 2087–2097, Brussels, Belgium. Association
for Computational Linguistics.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang,
Delip Rao, Eric Wong, Marianna Apidianaki, and Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle
Chris Callison-Burch. 2023. Faithful chain-of- Pineau, and William L. Hamilton. 2019. CLUTRR:
thought reasoning. arXiv preprint arXiv:2301.13379. A diagnostic benchmark for inductive reasoning from
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, text. In Proceedings of the 2019 Conference on
and Graham Neubig. 2022. Language models of code Empirical Methods in Natural Language Processing
are few-shot commonsense learners. In Proceedings and the 9th International Joint Conference on Natu-
of the 2022 Conference on Empirical Methods in Nat- ral Language Processing (EMNLP-IJCNLP), pages
ural Language Processing, pages 1384–1403, Abu 4506–4515, Hong Kong, China. Association for Com-
Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
putational Linguistics.
Haitian Sun, William Cohen, and Ruslan Salakhutdinov.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, 2022. ConditionalQA: A complex reading compre-
Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- hension dataset with conditional answers. In Pro-
moyer, and Hannaneh Hajishirzi. 2023. FActScore: ceedings of the 60th Annual Meeting of the Associa-
Fine-grained atomic evaluation of factual precision tion for Computational Linguistics (Volume 1: Long
in long form text generation. In Proceedings of the Papers), pages 3627–3637, Dublin, Ireland. Associa-
2023 Conference on Empirical Methods in Natural tion for Computational Linguistics.
Language Processing, pages 12076–12100, Singa-
pore. Association for Computational Linguistics. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-
Micke, and Mohit Iyyer. 2021. Do long-range lan-
John Eric Nolt, Dennis Rohatyn, and Achille Varzi. guage models actually use long-range context? In
1988. Schaum’s outline of logic. McGraw Hill Pro- Proceedings of the 2021 Conference on Empirical
fessional. Methods in Natural Language Processing, pages 807–
OpenAI. 2023. Gpt-4 technical report. arXiv preprint 822, Online and Punta Cana, Dominican Republic.
arXiv:2303.08774. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se-
Carroll Wainwright, Pamela Mishkin, Chong Zhang, bastian Gehrmann, Yi Tay, Hyung Won Chung,
Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi,
2022. Training language models to follow instruc- Denny Zhou, and Jason Wei. 2022. Challenging
tions with human feedback. Advances in Neural big-bench tasks and whether chain-of-thought can
Information Processing Systems, 35:27730–27744. solve them. arXiv preprint arXiv:2210.09261.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier (Appendix D, E, and F). Subsequently, we perform
Martinet, Marie-Anne Lachaux, Timothée Lacroix, experiments on code-only and text-only LLMs to
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
show that code prompting should only be used on
Azhar, et al. 2023a. Llama: Open and effi-
cient foundation language models. arXiv preprint text+code LLMs (Appendix G and H). Then, we
arXiv:2302.13971. show experiments on small LLMs (Appendix I),
ablations on local LLMs (Appendix J, K, L, and M.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay We conclude with the confusion matrices for the
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti main experiments Appendix N, prompt examples
Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton O, P, and Q.
Ferrer, Moya Chen, Guillem Cucurull, David Esiobu,
Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, B Datasets
Cynthia Gao, Vedanuj Goswami, Naman Goyal, An-
thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan ConditionalQA is a QA dataset where the an-
Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
swers are applicable under specific scenarios (i.e.,
Isabel Kloumann, Artem Korenev, Punit Singh Koura,
Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- conditional answers). Therefore, along with each
ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- question, the dataset provides a scenario that de-
tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- scribes the background of the person posing such
bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- a question. Questions require multi-hop, composi-
stein, Rashi Rungta, Kalyan Saladi, Alan Schelten,
Ruan Silva, Eric Michael Smith, Ranjan Subrama- tional, and conditional logic over documents about
nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- public policies (e.g., the eligibility for a subsidy).
lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Answers can be a span of the document, yes, and
Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, no. We use an oracle retriever to select the relevant
Melanie Kambadur, Sharan Narang, Aurelien Ro-
driguez, Robert Stojnic, Sergey Edunov, and Thomas passages to the question so that we can isolate the
Scialom. 2023b. Llama 2: Open foundation and fine- analysis of conditional reasoning abilities in LLMs
tuned chat models. arXiv preprint arXiv:2307.09288. from the retrieval component. The expected out-
put is a chain of thought (CoT; Wei et al. 2022)
Wenya Wang, Vivek Srikumar, Hannaneh Hajishirzi,
and Noah A. Smith. 2023. Elaboration-generating followed by the final answer. To create the CoT,
commonsense question answering at scale. In Pro- we use the annotated evidence sentences. We use
ceedings of the 61st Annual Meeting of the Associa- an oracle retriever to retrieve the relevant passages
tion for Computational Linguistics (Volume 1: Long to the question. This retriever is based on the sen-
Papers), pages 1619–1635, Toronto, Canada. Associ-
ation for Computational Linguistics. tences annotated as evidence for the answer (i.e.,
rationales). We concatenate all sections that in-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten clude one rationale and use the resulting passage
Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le,
and Denny Zhou. 2022. Chain-of-thought prompt-
as input document.
ing elicits reasoning in large language models. In
Advances in Neural Information Processing Systems,
BoardgameQA is a dataset that evaluates the abil-
volume 35, pages 24824–24837. Curran Associates, ity to reason with contradictory information guided
Inc. by preferences. For example, given a question
about traveling abroad, information found online
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023.
Satlm: Satisfiability-aided language models using about regulations can be contradictory because
declarative prompting. In Proceedings of NeurIPS, rules may change over time. Answering questions
pages 1–33. in this dataset requires complex multi-hop reason-
ing with conditional, deductive, and compositional
Susan Zhang, Stephen Roller, Naman Goyal, Mikel
Artetxe, Moya Chen, Shuohui Chen, Christopher De- abilities. The domain of the problems is board
wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. games, which allows us to analyze the conditional
Opt: Open pre-trained transformer language models. reasoning abilities in a completely different domain
arXiv preprint arXiv:2205.01068. from CondQA. BGQA is divided into multiple parti-
tions focusing on different characteristics, such as
A List of Appendices
the depth of the reasoning tree, the need for exter-
We start the appendices with details on the datasets nal information, etc. We focus on the main par-
and prompts (Appendix B and C). Then, we pro- tition and its subpartitions (i.e., BGQA-1, BGQA-2,
vide additional details on the features of the gener- BGQA-3), where the number refers to the number
ated code, the setup of the LLMs, and their costs of reasoning hops required to answer the ques-
tion. This dataset also includes annotated chain-of- As far as we know, these datasets do not contain
thoughts (CoT); therefore, we use their annotated any personal information or offensive content. Al-
input (“example”) as the input prompt and their though we did not explicitly analyze this, the au-
annotated CoT (“proof ”) as the expected output. thors of these datasets did not mention including
such content, and we did not observe such content
ShARC is a conversational QA dataset with nat- during our use of the datasets. All these datasets
ural language rules where most questions are un- are in English.
derspecified. Therefore, the model may need to
ask a follow-up question to know more about the Dataset Training Dev Test License
background of the interlocutor to return an answer.
CondQA 2338 285 804 BSD 2
The documents are of legal domain retrieved from BGQA-1 1000 500 1000 CC BY
the web pages of different governments and state BGQA-2 1000 500 1000 CC BY
agencies. Since this is a conversational QA and we BGQA-3 1000 500 1000 CC BY
are not interested in evaluating the conversational ShARC 21890 2270 8276 CC-BY-SA-3.0
abilities of LLMs, we transform the task into regu-
lar QA, instead of conversational QA. To do this, Table 5: Sizes of the datasets.
the model must answer yes, no, or not enough infor-
mation for each question. In the original task, not C Prompt Formulation
enough information, would lead to the generation
of a follow-up question. CONDQA. Firstly, we define the different compo-
nents of a data point: scenario (S), question (Q),
Complexity of the datasets. We analyze the document (D), rationales (R), and answer (A).
complexity of the datasets by counting the percent- Then, the text prompt tp is defined as follows:
age of reasoning operations (i.e., if statements) in
the code prompt generated by GPT 3.5. This analy- tp = "Question:" + S + Q + "Document:" + D
sis shows that the most difficult dataset is BGQA-3 +"Let’s think step by step"
with 21.58% of reasoning operations, followed (1)
by BGQA-2 (16.99%), CondQA (14.66%), BGQA-1
(10.55%), and lastly, ShARC (8.32%). where + represents the string concatenation op-
We also analyze the length of the documents of erator. Then, the output format, to is:
each dataset and find that BGQA-3 has the longest to = R + "Answer:" + A (2)
documents with an average of 39 lines of code, fol-
lowed by CondQA (38), BGQA-2 (25), ShARC (22), For code prompts, we first define a function
and lastly BGQA-1 (15). It is worth noting that C : NL → C that maps a natural language text into
the documents from CondQA are the short docu- code as shown in Figure 2. Then, we define code
ments extracted with the oracle retriever described prompt cp as follows:
above, instead of the full documents, which are
cp = "#Question:" + C(S) + C(Q)+
much longer (up to 9k tokens).
These two analyses suggest that BGQA-3 and "#Document:" + C(D) (3)
BGQA-2 are the most reasoning-intensive datasets +"#Let’s think step by step"
due to the high proportion of reasoning operations.
Similarly, we define the output format, co, as:
In contrast, CondQA is the dataset where the lin-
guistic dimension plays the biggest role because
co = R + "#Answer:" + A (4)
their documents are among the longest ones while
it contains much less proportion of reasoning opera- BGQA. Firstly, we define the components of a data
tions than the other datasets with similar document point in this dataset: facts (F ), rules (R), and ques-
lengths. tions (Q). Therefore, our text prompt is defined as
follows6 :
Dataset sizes, licenses, and safety. The sizes
and licenses of all the datasets used in this work tp = F + R + Q (5)
are provided in Table 5. Our use of these datasets 6
BGQA provides a field example with all the variables of
is consistent with their intended use, i.e., academic the dataset concatenated with descriptions. We use this field
research to evaluate question-answering systems. as text prompt.
This dataset also provides the CoT that leads to the E LLM Setup
answer. Therefore, we use that CoT as the expected
output. The exact models we used are the following: gpt-
For code prompts, we follow the same approach 3.5-16k-0613 for CondQA and BGQA. For ShARC,
as with the previous dataset. We define code since the documents are shorter, we used GPT-3.5-
prompts, cp, as follows: 0301 due to the lower costs. In both cases, we
run the models through the Azure AI service. We
also use Mixtral 8x7B with 4-bit quantization for
tp = C(F ) + C(R) + C(Q) (6)
all the datasets using one Nvidia A100 in our own
with the output format (co) being: server. Lastly, we use Mistral 7B v0.1 for CondQA
and BGQA. However, this model yields very poor
co = C(cot) (7) results on ShARC, so we use the instruct-v0.2 vari-
ant to be able to make a fair comparison between
ShARC. Firstly, we define the components of a text and code prompts on this dataset using Mistral
data point in this dataset: question (Q), scenario 7B. We use fp16 quantization for the Mistral 7B
(S), document (D), and conversation history (H). experiments and run them on our own server with
Then, the text prompt tp is defined as follows: one Nvidia A100.
All of our prompting methods are implemented
tp = "Question:" + S + Q + "Document:" + D using the Langchain library.7 We set the decoding
temperature to zero and use greedy sampling to
+"Conversation history:" + H
make the outputs deterministic. For each experi-
+"What is the answer to the question:" + Q ment, we use a random sample from the training
(8) set as demonstrations. The LLM generating the
code for code prompts is the same one as the one
the output format is the answer label directly, which running the code to generate the final answer. We
can be yes, no, or not enough information. evaluate each model and prompt in the dev set of
Similarly to the other datasets, we defined code each dataset with two random seeds. Since the
prompts cp as follows: demonstrations are selected randomly, the seed de-
termines them. The seed that yields the best per-
tp = "#Question:" + C(S) + C(Q)+ formance on the dev set is then used for the final
+"#Document:" + C(D) evaluation on the test set.
+"#Conversation history:" + C(H) The number of demonstrations used to translate
+"#What is the answer to the question:" + C(Q) the documents into code is specified in Table 6.
(9) Note that this number differs from the number of
demonstrations used to generate the answer, which
Lastly, the output format is the answer label di- is always three.
rectly, which in this case are True, False, or None. We use chain of thoughts (CoT) based on the pro-
vided annotations of the datasets. We do not use
D Coding Features advanced CoT methods for text prompts because
our aim is to quantify how much improvement we
To generate code as close as possible to the NL text, can get by transforming the natural language CoT
we use a programming language based on a simpli- into code syntax, and therefore, the natural lan-
fication of Python. We only use boolean variables guage text and code must be as close as possible.
or variables that contain lists of strings. Variables The use of advanced CoT methods would also be
follow the snake case naming convention. We also reflected in the code syntax, making the experi-
employ if statements to model conditional reason- mental setup more complicated without providing
ing, but we do not use loops, functions, or classes. better insights.
We create a code comment with the original NL The best random seeds found (and consequently
text for each input sentence, and right after the code used for the test set evaluation) are described in
comment, we generate the code that represents the Table 7 and Table 8.
semantics of that sentence. However, we do not
7
enforce the generated code to be a runnable script. [Link]
 Dataset GPT Mixtral Mistral Dataset GPT Mixtral Mistral
CondQA 4 4 4 CondQA 0 1 0
ShARC 5 4 4 ShARC 0 0 0
BGQA-1 4 3 3 BGQA-1 1 0 1
BGQA-2 4 3 4 BGQA-2 0 1 1
BGQA-3 4 3 4 BGQA-3 0 1 0

Table 6: Number of demonstrations for code transla- Table 8: Best seeds for text prompts
tions. Note this is not the number of demonstrations to
generate the answer.
Furthermore, CondQA requires generating a natu-
Dataset GPT Mixtral Mistral ral language answer that is a span of the document.
The use of code to generate a natural language span
CondQA 0 0 0
of a document is also far from the fine-tuning tasks
ShARC 0 1 1
of this model. This would explain why the code
BGQA-1 1 0 1
representation is worse than the text representation.
BGQA-2 1 0 0
It is particularly interesting to see the results on
BGQA-3 0 1 0
ShARC. After manually inspecting the outputs, we
observe that Code Llama can successfully generate
Table 7: Best seeds for code prompts
the code corresponding to the natural language in-
put. However, when it is prompted with such code
F Costs and the question variable, the model does not gen-
erate the value of the variable (i.e., true, false,
Running a data instance from ConditionalQA with
or none). Instead, it generates \n. The reasons
gpt-3.5-16k-0613 using code prompts costs $0.04
behind this remain unclear and would require fur-
while with text prompts $0.01. On BoardgameQA-
ther investigation, which is out of the scope of this
depth 3 (i.e., the partition with the most expensive
paper.
prompts), with the same model, the costs per ques-
However, we observe a different behavior on
tion are $0.02 and $0.03 for text and code prompts,
BGQA. In this dataset, code prompts outperform
respectively. Lastly, on ShaRC, using gpt-3.5-0301,
text prompts. We attribute this to the high align-
the costs per question are $0.0006 and $0.005 for
ment with the first-order logic of this dataset, which
text and code prompts, respectively.
makes it closer to the intended use of the model.
G Code-only LLMs Nevertheless, it is important to note that these re-
sults are not intended to be comprehensive enough
Although our work focuses on text+code LLMs to conclude that code LLMs or Code Llama can
because they are the only type of LLMs whose or cannot solve natural language tasks, which is
intended use includes natural language and cod- out of the scope of this work. Instead, they sim-
ing tasks, we conduct a small experiment on Code ply seem to confirm the warnings of the authors
Llama (Roziere et al., 2023), a code-only LLM. It of Code Llama, i.e., this model is not intended for
is important to note that their authors advise against natural language tasks.
using this model on natural language tasks because
their intended use is in code generation tasks only.
Table 9 shows the results of Code Llama on our Model Text Prompt Code Prompt
datasets. Firstly, we can observe that code prompts CondQA 31.58 26.34
perform significantly worse than text prompts on ShARC 58.33 18.62
CondQA and ShARC despite being a code LLM. We BGQA1 44.38 44.78
can attribute this to the nature of these datasets and BGQA2 44.59 49.41
the intended use of the model. These datasets re- BGQA3 40.88 46.44
quire a strong comprehension of natural language
documents and dialogues and answering natural Table 9: Text and code prompts results in Code Llama
language questions about them. This is far from the 7B - Instruct with one demonstration.
intended use of the model (i.e., generating code).
H Text-only LLMs Prompt BGQA-1
As briefly mentioned in Section 4.3, text-only Text 33.20 ± 1.42
LLMs are not expected to perform well with code Code 48.32 ± 1.65
prompting and should not be used for this as they
are not explicitly trained on code. For example, Table 11: Comparison of text prompt and code prompts
on the MBPP coding benchmark (Austin et al., with Phi-2 on the validation set. Metric: F1 score. One
2021), LLaMA 2 scores 26.1% , while Mistral 7B demonstration per class is provided.
(a text+code LLM) achieves 47.5% and code-llama
2 7B, achieves 52.5% (Jiang et al., 2023). Table 10 Prompt BG1 BG2 BG3
further proves our claim. LLaMA 2 7B-Chat (Tou-
Atomic St. -4.29 -8.09 -2.19
vron et al., 2023b) with code prompting never out-
NLCode -2.7 -4.36 -5.97
performs text prompting due to its lack of code
Anonym. Code -0.45 -7.17 -4.52
understanding.
Rnd Code -4.27 -11.16 -13.93
- Comments -0.79 -9.88 -29.21
Prompt CQA ShARC BG1 BG2
Text 29.79 46.16 51.85 39.23 Table 12: Performance drop w.r.t. Code Prompting on
Code 21.32 24.74 45.16 37.66 Mistral.

Table 10: LLaMA 2 7B Chat Results. Code prompts

cannot perfom well on text-only LLMs.
atomic statements and natural language code abla-
tions improve performance.

I Results on Small LMs with Short Prompt BG1 BG2 BG3

Context Window Atomic St. 1.47 -4.01 -7.61
We have shown the effectiveness of code prompting NLCode 0.9 -2.82 -3.18
in the most popular sizes of LLMs in table 1 from Anonym. Code 0.21 -16.59 1.25
section 5.1. However, it is becoming increasingly Rnd Code -15.88 -9.56 -11.31
popular the development of small language mod- - Comments -12.81 -7.14 5.38
els (sLMs) due to their cheaper inferece cost and
Table 13: Performance drop w.r.t. Code Prompting on
higher token thoughput (Gunasekar et al., 2023).
Mixtral.
Therefore, we have conducted a preliminary ex-
periment with Phi-28 , a text+code model of 2.7B
parameters on BGQA-1 to show that our prompting K Number of Demonstrations on Local
methodology also holds in sLMs. As we can show LLMs
on table 11, code prompting yields a remarkable
performance boost of 15 points. However, due We conduct experiments with Mistral and Mixtral
to the limited context window of Phi-2, it is not on BGQA because it is the dataset where we clearly
straightforward to conduct in-context learning on see a difference between text and code prompts
our other datasets. with different numbers of demonstrations. Table 14
shows that Mistral achieves the best results with
J Ablations on Local LLMs just one demonstration for both prompting methods.
We attribute this to the long length and complexity
Table 12 shows that all ablations to code prompt- of the demonstrations9 , which can confuse LLMs,
ing in Mistral yield significant performance drops especially those that are small, which is the case
similar to those observed in GPT 3.5 for Mistral. Li et al. (2024) shows that Mistral
Table 13 shows that most ablations to code dramatically decreases its performance as the num-
prompting in Mixtral also yield significant perfor- ber of input tokens increases. However, very large
mance drops similar to those observed in GPT 3.5, LMs are more resilient to the input length. Text
except on BGQA-1, where text prompts outperform prompts on Mixtral achieve the best results with at
code prompts. Therefore, it is expected that the
9
We cannot even provide three demonstrations for BGQA-2
8
[Link] and 3 due to the length of each demonstration.
Model Prompt # Dem. BG1 BG2 BG3 makes this experiment very costly. Thus, we aim
to maximize the number of instances probed while
1 48.26 45.90 48.63
keeping the costs down. To do so, we use a sam-
Text 2 45.86 48.82 47.92
ple of 50 instances for each dataset partition of
3 44.25 N.A. N.A.
Mistral BoardgameQA, except for Board3, where we used
1 51.54 55.28 52.83 20 instances (≈ 700 probes) because of the cost of
Code 2 51.85 50.51 48.66 the experiment. Due to the length of the demonstra-
3 39.58 N.A. N.A. tions of ConditionalQA and its impact on the costs,
1 57.06 36.71 30.8 we sample five facts and three partial CoTs for each
Text 2 65.49 39.83 33.23 instance, yielding an upper-bound of 15 probes per
3 63.73 N.A. N.A. instance, and run the probes for 30 instances for
Mixtral each dataset partition (i.e., correct and incorrect
1 61.85 44.93 35.22 instances).
Code 2 58.44 45.70 39.43
3 61.3 N.A. N.A. Prompt Probes. In all cases, we follow the fol-
lowing format: Sys. Prompt; ICL Demonstrations;
Table 14: Text and code prompting performance on Input Instance; Partial CoT; Probe.
Mistral and Mixtral under a different number of demon- The probe for text and code prompts in
strations. Significantly best results in bold. BoardgameQA is: “Now, I want to ask you about
the value of some key entities you used. Your an-
least two demonstrations, while code prompts only swers must be ‘yes‘, ‘no‘, or ‘unknown‘. It is very
achieve significantly better results with more than important that you only write one word. Is it true
one demonstration in BGQA-3. that {fact}?”
The probe for text prompts in ConditionalQA
L Variable Tracking Setup is: “Now, I want to ask you about the value of
Extracting key entities in BoardgameQA. This some key entities you used. Your answers must be
dataset provides a list of “facts,” which are short “True”, “False”, “unknown”, or a string. It is very
and concise sentences describing the state of a key important that you only write the exact value. From
entity. Therefore, we use them without alterations the speaker perspective, is it true that {fact}?”
as the key entities to ask for. The probe for code prompts in ConditionalQA
is: “Now, I want to ask you about the value of
Extracting key entities in ConditionalQA. This some key entities you used. Your answers must
dataset provides a scenario describing the back- be “True”, “False”, “unknown”, or a string. It is
ground information of the person posing the answer. very important that you only write the exact value.
Since this scenario is a free-form text, we follow What is the value of the variable {var}?” A real
(Min et al., 2023) to extract atomic statements and example is provided in Table 18.
use them as the key entities to ask for.
M Memory Errors on Local LLMs
Code Prompting variables . To probe the vari-
able tracking abilities of code prompts, we use the Analyzing the memory errors in Mixtral and Mis-
variables defined in the “facts” and “scenario” of tral becomes much more challenging than in GPT
BoardgameQA and ConditionalQA, respectively. 3.5 due to the lower performance of the models.
Table 15 shows that, in general, Mixtral makes
Probing memory at different steps in the Chain-
significantly more memory errors than GPT 3.5.
of-Thought. Inspired by Lanham et al. (2023),
On BGQA-1, we observed more memory errors on
we truncate the Chain-of-Thought (CoT) at differ-
the questions where the model fails than when the
ent completion states and probe the memory of the
model returns the correct answer, as expected. We
model. To break down the CoT, we split it by the
further see that code prompts make fewer memory
character “\n”, which usually represents the end of
errors than text prompts, confirming the results of
a reasoning step. This is possible because our in-
GPT 3.5. However, the margins are much narrower
context learning demonstrations follow this format.
than in GPT 3.5.
Number of probes. For each dataset instance, we On BGQA-2 and 3, the model surprisingly makes
run num_f acts × num_steps_cot probes, which fewer memory errors on the wrong answer parti-
 Correct Ans. Incorrect Ans. Table 17 shows examples of the multiple code
Text Code Text Code ablations we conducted in Section 5.3. Random
code replaces the code with a piece of code from
BGQA-1 58.5 54.0 60.4 55.6
another data point. In this way, the semantics of
BGQA-2 70.8 77.6 66.0 80.0
the text and code mismatch while we keep the code
BGQA-3 65.2 80.4 62.8 94.4
syntactically correct.
Table 15: Percentage of memory errors in Mixtral. Q Prompt Examples

tion. However, in both cases, the percentage is so

high that makes the test unreliable, as we observe in
CondQA in GPT 3.5. In the specific case of Mistral,
we were not able to run this experiment success-
fully due to the model not understanding the task.
We believe our experiment setup for memory error
analysis only works well on very high-performing
models such as GPT. To analyze them on smaller
models such as Mistral and Mixtral, we would need
to devise other types of experiments, which is out
of the scope of this work. We encourage further re-
search on entity tracking methods to analyze LLMs
and leave this as future work.

N Confusion Matrices
Figure 4 shows the confusion matrices of all our
models using text and code prompts for all the
datasets except CondQA. We cannot include this
one because it is a span-extraction task, not a clas-
sification task.

O Atomic Statements
Original sentence: <p>Applying for the legal right
to deal with someone’s property, money and posses-
sions (their estate) when they die is called applying
for probate.</p> Atomic statements: Applying for
the legal right is a process. The process is called
’applying for probate’. The legal right is to deal
with someone’s property, money, and possessions.
The someone is a person who has died. The prop-
erty, money, and possessions are collectively called
the ’estate’.

P Examples of Code Ablations

An example of a back-translated code into natural
language is provided in Table 16. We can observe
in both examples that the resulting natural language
(NL) text is extremely similar to the original code.
In addition, in the second example (BGQA), Rule2
is much simpler after the back-translation than its
original description in NL.
 Type Text
Code # <p>You can apply to become the estate’s administrator if you are 18 or over and
you are the most ‘entitled’ inheritor of the deceased’s estate. This is usually the
deceased’s closest living relative.</p>
if applicant_age >= 18 and entitled_inheritor and closest_relative:
can_apply_estate_administrator = True
Code → NL <p>You can apply to become the estate’s administrator if you are 18 or over and
you are the most ‘entitled’ inheritor of the deceased’s estate. This is usually the
deceased’s closest living relative.</p>
if you are 18 or over and you are the most entitled inheritor of the deceased’s
estate and you are the closest living relative, you can apply to become the estate’s
administrator
Code # Rule2: Be careful when something removes from the board one of the pieces of
the dog and also becomes an enemy of the catfish because in this case it will surely
not burn the warehouse of the mosquito (this may or may not be problematic)
rule2(something) = remove(something, piece_of(dog)) & enemy(something, cat-
fish) => not burn(something, warehouse_of(mosquito))
Code → NL Rule2: If something removes from the board one of the pieces of the dog and
also becomes an enemy of the catfish, then it does not burn the warehouse of the
mosquito

Table 16: Example of a back-translation NL → C in ConditionalQA and BGQA-3. Text in bold represents the main
modification.

Type Text
Original # <p>To be eligible you must have left your country and be unable to go back because
Code you fear persecution.</p>
if left_country_and_fear_persecution:
eligible_for_asylum = True
Anonymous # <p>To be eligible you must have left your country and be unable to go back because
Code you fear persecution.</p>
if var_1
var_2 = True
Random # <p>To be eligible you must have left your country and be unable to go back because
Code you fear persecution.</p>
if value_of_property_gone_down_by_more_than_50:
eligible_to_claim = True
getting_housing_benefit = True

Table 17: Examples code ablations.

 Section Role Message
Problem Human Question: My brother and his wife are in prison for car-
instance rying out a large fraud scheme. Their 7 and 8 year old
children have been living with me for the last 4 years. I want to become
their Special Guardian to look after them permanently. How long will it be
before I hear back from the court?
Document: <h1>What is a special guardian</h1> <p>You can apply to be
a child’s special guardian when they cannot live with their birth parents and
adoption is not right for them.</p> ...
Answers can be "yes" or "no". Let’s think step by step:
Partial AI <p>Within 10 days of receiving your application the court will send you a
CoT case number and a date for a meeting to set out:</p>\n
Probe Human Now, I want to ask you about the value of some key entities you used. Your
answers must be ‘True‘, ‘False‘, ‘unknown‘, or a string. It is very important
that you only write the exact value. From the speaker perspective, is it true
that the children have been living with me for the last 4 years?
Probe AI True

Table 18: Variable Tracking Example. Underlined text represents the variable to probe. Partial CoT is not the
complete answer. The generation was stopped, and only the first step was used in this probe.

System: You are a helpful assistant that answers questions given a document. Answers must be a short
span of the document. You have to extract the span from the document. Do not write anything else. I will
give you some examples first.
ICL Demonstrations...
Human: Question: My brother and his wife are in prison for carrying out a large fraud scheme. Their
7 and 8 year old children have been living with me for the last 4 years. I want to become their Special
Guardian to look after them permanently. How long will it be before I hear back from the court?
Document: <h1>What is a special guardian</h1>
<p>You can apply to be a child’s special guardian when they cannot live with their birth parents and
adoption is not right for them.</p>
<p>You’ll be responsible for looking after the child until they’re 18 (unless the court takes your responsi-
bility away earlier).</p>
<p>You’ll make all day to day decisions about the child, for example schooling and medical treatment.
You do not have to discuss these decisions with the birth parents.</p>
<p>You’ll need to get the consent of everyone who has parental responsibility for the child before you
make some important decisions, for example:</p>
<li>changing the child’s surname</li>
<li>putting the child up for adoption</li>
<li>taking the child abroad for more than 3 months</li>
<li>the child having surgery for reasons other than improving health, such as circumcision, sterilisation or
cosmetic surgery</li>
<p>If you cannot get consent, you can ask the court to decide. Use the form ‘Make an application in
existing court proceedings related to children’ (form C2).</p>
<h1>After you apply</h1>
<p>Within 10 days of receiving your application the court will send you a case number and a date for a
meeting to set out:</p>
<li>a timetable for your case</li>
<li>how it will be dealt with</li>
<p>This meeting is called a ‘first directions hearing’.</p>
<p>You must go to all hearings you’re told to unless the court excuses you. If you’re not able to go,
contact the court office.</p> Answers must be a short span of the document. You have to extract the span
from the document. Do not write anything else. Let’s think step by step:

Table 19: Text prompt Example for ConditionalQA

System: You are a helpful assistant. Your task is to process a pseudo-code that describes a question and a
document. You need to reason using that document and the comments to return the answers. Answers
must be a short span of the document. You have to extract the span from the code comments. Do not write
anything else. I will give you some examples first.
ICL Demonstrations...
Human: # Question: My brother and his wife are in prison for carrying out a large fraud scheme. Their
7 and 8 year old children have been living with me for the last 4 years. I want to become their Special
Guardian to look after them permanently. How long will it be before I hear back from the court?
maximum_redundancy_pay = 16320
housing_standards_and_procedures_in_Northern_Ireland = True
ensure_vehicle_taxed_in_UK = True immigration_advisers_can_help_with_representation_at_tribunal =
True
supply_protective_clothing_and_equipment = True
CBT_required_for_moped_and_motorcycle = True
court_response_time = None # This is the variable that answers the question
# <h1>What is a special guardian</h1>
# <p>You can apply to be a child’s special guardian when they cannot live with their birth parents and
adoption is not right for them.</p>
if attorneys_appointed_jointly:
all_attorneys_must_agree_to_make_decision = True
disability_or_severe_disability_element_of_working_tax_credit = True
mugging_without_physical_harm_emergency = True
# <p>You’ll be responsible for looking after the child until they’re 18 (unless the court takes your
responsibility away earlier).</p>
work_temporarily_for_hirer = True
# <p>You’ll make all day to day decisions about the child, for example schooling and medical treatment.
You do not have to discuss these decisions with the birth parents.</p>
accounts_and_tax_returns_cover_financial_year = "1 June to 31 May"
employer_operating_PAYE = True
# <p>You’ll need to get the consent of everyone who has parental responsibility for the child before you
make some important decisions, for example:</p>
# <li>changing the child’s surname</li>
# <li>putting the child up for adoption</li>
# <li>taking the child abroad for more than 3 months</li>
# <li>the child having surgery for reasons other than improving health, such as circumcision, sterilisation
or cosmetic surgery</li>
managed_by_fit_and_proper_persons = True
check_court_order_for_authorization = True
considering_fostering = True
if not_connected_to_mains_sewer:
septic_tank_used = True
can_claim_tax_relief_if_taxed_twice = True
extra_support_for_disability = True
if operator_of_septic_tank_or_treatment_plant:
follow_general_binding_rules = True
# <p>If you cannot get consent, you can ask the court to decide. Use the form ‘Make an application in
existing court proceedings related to children’ (form C2).</p>
appeals_decision_time = "several months"
if worker and informal_resolution_not_satisfactory:
formal_grievance_complaint_possible = True
time_limit_for_backdating_claims_services = 6
# <h1>After you apply</h1>
# <p>Within 10 days of receiving your application the court will send you a case number and a date for a
meeting to set out:</p>
# <li>a timetable for your case</li>
# <li>how it will be dealt with</li>
# <p>This meeting is called a ‘first directions hearing’.</p>
committee_recommendations_go_to_Prime_Minister = True
check_adviser_registration = True
meet_manning_levels = True
recognised_as_charity_or_CASC = True
apply_for_visa_for_other_reasons = True
debt_paid_off = True
if special_educational_needs_and_disabilities:
affects_behaviour_or_socialisation = True
# <p>You must go to all hearings you’re told to unless the court excuses you. If you’re not able to go,
contact the court office.</p>
payslip_can_include_tax_code = True
VAT_zero_rate = 0
gas_equipment_installed_and_maintained_by_Gas_Safe_registered_engineer = True
# Question: My brother and his wife are in prison for carrying out a large fraud scheme. Their 7 and 8
year old children have been living with me for the last 4 years. I want to become their Special Guardian to
look after them permanently. How long will it be before I hear back from the court?
# Answers must be a short span of the document. You have to extract the span from the code comments.
Do not write anything else.
# Let’s think step by step:

Table 20: Code Prompt Example for ConditionalQA

System: You are a question-answering system that solves the problem of reasoning with contradictory
information guided by preferences over sources of information. You must explain your answers step
by step.
ICL Demonstrations ...
Human: A few players are playing a boardgame
The current state of the game is as follows
The amberjack struggles to find food
And the rules of the game are as follows
Rule1: If the amberjack has difficulty to find food, then the amberjack removes from the board one
of the pieces of the carp
Based on the game state and the rules and preferences, does the amberjack remove from the board
one of the pieces of the carp?
AI:

Table 21: Text prompt Example for BGQA-1

System: You are a large language model of code that can interpret code. You are given a pseudo-code
that resembles to first-order logic that models some scenario. You will be given a question and you
have to answer it step by step. You can use a rule if and only if you know the antecedent of the rule.
ICL Demonstrations
Human: # A few players are playing a boardgame
# The rules of the game are as follows
# Rule1: If the amberjack has difficulty to find food, then the amberjack removes from the board one
of the pieces of the carp.
rule1() = difficulty_finding_food(amberjack) => remove_piece(amberjack, carp)
# The current state of the game is as follows
# The amberjack struggles to find food.
difficulty_finding_food(amberjack) = True
# Based on the game state and the rules and preferences, does the amberjack remove from the board
one of the pieces of the carp?
question = remove_piece(amberjack, carp)
AI:

Table 22: Code prompt Example for BGQA-1

 Figure 4: Confusion matrices of text and code prompts for each model on all datasets.

System: You are a question answering system that answers questions given a document and a conversation
history. The conversation history gives information about the background of the person posing the question.
You must answer ‘yes‘, ‘no‘, or ‘not enough information‘ to the question and nothing else.
ICL Demonstrations...
Human: Question: The item is not equipment for audio books or newspapers, and I’m not selling lifeboats
or anything related to that. It’s for medicine and medicinal ingredients. Can I apply zero VAT to this item?
Document:
## Items that qualify for the zero rate
You may be able to apply zero VAT when you sell the following to an eligible charity:
* equipment for making ‘talking’ books and newspapers
* lifeboats and associated equipment, including fuel
* medicine or ingredients for medicine
* resuscitation training models
Conversation history:
Q: Is it equipment for making ‘talking’ books and newspapers?
A: No
Q: Are you selling lifeboats and associated equipment, including fuel?
A: No
Q: Are you selling medicine or ingredients for medicine?
A: Yes
What is the answer to the question: Can I apply zero VAT to this item? You must answer ‘yes‘, ‘no‘, or
‘not enough information‘ to the question and nothing else.
AI:

Table 23: Text prompt Example for ShARC.

System: You are a question-answering system that answers questions based on a document, and conversa-
tion history. The text is pseudo-code that models the document and conversation history. You must run
the code and update the value of the variable that answers the question. The values can be True, False, or
None.
ICL Demonstrations...
Human:
# Question: # The item is not equipment for audio books or newspapers, and I’m not selling lifeboats or
anything related to that. It’s for medicine and medicinal ingredients. Can I apply zero VAT to this item?
equipment_for_audio_books_or_newspapers = False
selling_lifeboats_or_related_equipment = False
selling_medicine_or_ingredients_for_medicine = True
can_apply_zero_VAT = None # This is the variable that answers the question.
# Other variables needed for the document:
# Document:
## Items that qualify for the zero rate
# You may be able to apply zero VAT when you sell the following to an eligible charity:
# * equipment for making ‘talking’ books and newspapers
if equipment_for_audio_books_or_newspapers:
can_apply_zero_VAT = False
# * lifeboats and associated equipment, including fuel
if selling_lifeboats_or_related_equipment:
can_apply_zero_VAT = False
# * medicine or ingredients for medicine
if selling_medicine_or_ingredients_for_medicine:
can_apply_zero_VAT = True
# * resuscitation training models
resuscitation_training_models = None
can_apply_zero_VAT =
AI:

Table 24: Code prompt Example for ShARC.