Multi-LLM Thematic Analysis with Dual Reliability

Metrics: Combining Cohen’s Kappa and Semantic

Similarity for Qualitative Research Validation
Nilesh Jain1 Seyi Adeyinka1 Leor Roseman2
Aza Allsop1,3
1
Yale School of Medicine
2
University of Exeter
3
Center for Collective Healing
arXiv:2512.20352v1 [[Link]] 23 Dec 2025

December 24, 2025

Abstract
Qualitative research faces a critical reliability challenge: traditional inter-rater agreement methods
require multiple human coders, are time-intensive, and often yield moderate consistency. We present
a multi-perspective validation framework for LLM-based thematic analysis that combines ensemble
validation with dual reliability metrics: Cohen’s Kappa (κ) for inter-rater agreement and cosine sim-
ilarity for semantic consistency. Our framework enables configurable analysis parameters (1-6 seeds,
temperature 0.0-2.0), supports custom prompt structures with variable substitution, and provides con-
sensus theme extraction across any JSON format. As proof -of-concept, We evaluate three leading
LLMs (Gemini 2.5 Pro, GPT-4o, Claude 3.5 Sonnet) on a psychedelic art therapy interview transcript,
conducting six independent runs per model. Results demonstrate Gemini achieves highest reliability
(κ = 0.907, cosine=95.3%), followed by GPT-4o (κ = 0.853, cosine=92.6%) and Claude (κ = 0.842, co-
sine=92.1%). All three models achieve a high agreement (κ > 0.80), validating the multi-run ensemble
approach. The framework successfully extracts consensus themes across runs, with Gemini identifying
6 consensus themes (50-83% consistency), GPT-4o identifying 5 themes, and Claude 4 themes. Our
open-source implementation provides researchers with transparent reliability metrics, flexible configura-
tion, and structure-agnostic consensus extraction, establishing methodological foundations for reliable
AI-assisted qualitative research.

Keywords: Thematic Analysis, Large Language Models, Qualitative Research, Cohen’s Kappa, Se-
mantic Similarity, Ensemble Validation

1 Introduction volves iterative interpretation, contextualization,

and affective understanding. Comparative studies
Inter-rater reliability remains a fundamental chal- [3, 4] evaluating nine generative models reveal sig-
lenge in qualitative research [1]. Traditional ap- nificant performance variation across models and
proaches require multiple human coders who inde- highlight cultural interpretation challenges, par-
pendently analyze the same data, with agreement ticularly in non-Western contexts. These find-
measured through Cohen’s kappa. This process ings underscore a critical gap: existing LLM ap-
is time-intensive, expensive, and often yields only proaches lack systematic validation mechanisms
moderate agreement (κ = 0.40-0.60). The emer- for reliability assessment.
gence of large language models (LLMs) offers po- We propose a multi-perspective validation
tential solutions, but current approaches exhibit framework with dual reliability metrics: Cohen’s
several limitations. Kappa for statistical inter-rater agreement and
Recent LLM-based systems such as QualIT [2] cosine similarity for semantic consistency. Our
focus on topic modeling and key-phrase extrac- framework introduces: (1) configurable seeds (1-
tion, achieving 70% topic coherence on benchmark 6) enabling reproducible variation, (2) adjustable
datasets. However, these approaches differ fun- temperature (0.0-2.0) controlling output diver-
damentally from comprehensive thematic analy- sity, (3) custom prompt support with variable
sis as defined by Braun and Clarke [1], which in- substitution ({seed}, {text chunk}), and (4)

1
structure-agnostic consensus extraction working ing reflexive practice. Landers and Behrend [7]
with any JSON format. employ Retrieval-Augmented Generation (RAG)-
Empirical evaluation on a psychedelic art ther- based approaches for interview transcript anal-
apy interview transcript across three leading ysis, focusing on methodological rigor. The
LLMs demonstrates: Gemini 2.5 Pro (κ = A Human-AI Collaborative Thematic Analysis
0.907, cosine=95.3%), GPT-4o (κ = 0.853, co- framework using Multi-Agent (TAMA) frame-
sine=92.6%), and Claude 3.5 Sonnet (κ = 0.842, work [8] applies multi-agent LLMs specifically to
cosine=92.1%). All models achieve very high clinical interviews, demonstrating domain-specific
agreement (κ > 0.80), with Gemini showing su- adaptations.
perior consistency. Our contributions include: Lindh and Messina [9] demonstrate that LLMs
can infer main themes but highlight limitations in
• Dual reliability metrics (Cohen’s Kappa + capturing latent interpretations—themes requir-
cosine similarity) for comprehensive valida- ing deep contextual understanding. Turobov et
tion al. [10] evaluate ChatGPT for thematic analysis,
• Configurable analysis parameters (seeds, finding promise but recommending human over-
temperature) for reproducible research sight. Fulgencio [11] identifies benefits while not-
• Structure-agnostic consensus extraction for ing cultural context limitations.
custom prompt formats Comparative LLM Studies. Bennis and
• Empirical LLM comparison on real qualita- Mouwafaq [3] conduct a comparative study of
tive data with open-source implementation nine generative models on medical data, reveal-
ing significant performance variation across mod-
Code available at [Link]
els (something more specific). Sakaguchi et
NileshArnaiya/LLM-Thematic-Analysis-Tool.
al. [4] compare ChatGPT with human researchers
in Japanese clinical contexts, highlighting cul-
2 Related Work tural interpretation challenges that LLMs strug-
gle to address. Zhang et al.’s LLM-Assisted The-
Traditional Reliability Assessment. Quali- matic Analysis (LATA) study [12] compares GPT-
tative research relies on inter-rater reliability to 4 and Gemini outputs with manually analyzed
establish trustworthiness [1]. Cohen’s kappa mea- outcomes, achieving cosine similarity scores up to
sures agreement between two coders, with values 0.76—validating semantic similarity as a viable
interpreted as follows: κ < 0.40 (poor), 0.40-0.60 metric but also revealing model-dependent vari-
(moderate), 0.60-0.80 (substantial), κ > 0.80 (ex- ation. Gupta et al. [13] examine LLMs for focus
cellent). However, kappa requires exact categor- group transcript analysis, demonstrating poten-
ical matches and cannot capture semantic equiv- tial but noting reliability concerns.
alence. Studies report that even trained coders Prompt Engineering and Quality. Prompt
often achieve only moderate agreement, necessi- design critically impacts analysis quality. Braun
tating extensive discussion to resolve discrepan- and Clarke [14] provide reproducible prompt en-
cies (citation). gineering approaches aligned with their five-phase
LLM-Based Qualitative Analysis. Recent framework, with empirical evaluation against es-
work explores LLM applications in qualitative re- tablished quality criteria. Sanford et al. [15]
search. QualIT [2] integrates LLMs with cluster- systematically evaluate prompt engineering tech-
ing for topic modeling, extracting key phrases and niques using locally hosted Llama 3.1 models,
performing hierarchical clustering to achieve 70% demonstrating that structured prompts signifi-
topic coherence on benchmark datasets. However, cantly improve thematic coherence. Nelson [16]
this approach focuses on topic extraction rather tests offline LLMs through reflexive thematic
than comprehensive thematic analysis. analysis phases, identifying limitations of base
Alternative frameworks propose human-LLM models and proposing prompt strategies for im-
collaboration models. Rana and Asad [5] intro- provement.
duce ”LLM-in-the-loop,” using in-context learn- Validation Metrics. Novel validity metrics
ing with GPT-3.5 to reduce labor requirements emerge for LLM-assisted analysis. Patel et al.
while maintaining human oversight. Schlag- [17] propose initial thematic saturation (ITS) as a
wein [6] proposes four conversational roles (man- validity metric, measuring when LLMs reach an-
agers, teachers, colleagues, advocates) for re- alytical saturation in initial coding. Chen et al.
searchers working with LLM chatbots, emphasiz- [18] examine codebook reduction and saturation

2
patterns, providing insights into how LLMs han- 1. Extract all themes from each run (structure-
dle iterative coding processes. agnostic JSON parsing)
Systematic Evidence. Kumar et al. [19] 2. Compute pairwise cosine similarity between
provide a comprehensive systematic mapping of all theme descriptions across runs
LLM applications in qualitative research across 3. Group themes with similarity > 0.70 into
diverse fields, application contexts, and evalua- equivalence classes
tion metrics. Their review reveals heterogeneous 4. Count occurrence frequency for each equiva-
approaches with limited standardization, under- lence class
scoring the need for systematic validation frame- 5. Retain themes appearing in ≥50% of runs
works. (adjustable threshold)
These studies reveal a critical gap: existing 6. Compute per-theme consistency percentage
LLM approaches lack systematic validation mech- (e.g., 5/6 runs = 83%)
anisms. Single-run analyses provide no reliability
indicators, and multi-model studies focus on per- This balances conservatism (filtering spurious
formance comparison rather than developing val- themes) with sensitivity (preserving valid varia-
idation frameworks. Our work addresses this gap tion). The system distinguishes high-confidence
through ensemble validation with quantified relia- (5-6/6, 83-100%) from moderate-confidence (3-
bility metrics, building on the semantic similarity 4/6, 50-66%) themes, enabling researchers to ap-
validation demonstrated by Zhang et al. [12] while ply different review standards.
extending it through multi-run consensus.
3.2 Dual Reliability Metrics
3 Method We implement two complementary reliability
measures addressing different validation aspects:
3.1 Ensemble Validation Framework Cohen’s Kappa (κ). Measures inter-rater
agreement accounting for chance:
Our framework conducts six independent ana-
lytical runs with fixed random seeds (42, 123, po − p e
456, 789, 1011, 1213), analogous to K-fold cross- κ= (3)
1 − pe
validation in machine learning. This design choice
is grounded in statistical theory and practical con- where po is observed agreement and pe is
siderations. expected agreement by chance. For thematic
Statistical Rationale. Classical test theory analysis, we compute theme presence/absence
requires multiple measurements to estimate true across runs, calculating pairwise kappa for all run
score variance versus error variance. While tra- pairs. Interpretation follows Landis-Koch criteria:
ditional inter-rater reliability studies use two to κ > 0.80 (almost perfect), 0.60-0.80 (substantial),
three coders, research on consensus measurement 0.40-0.60 (moderate), 0.20-0.40 (fair), κ < 0.20
[20] suggests that five to six independent ratings (poor). Kappa provides statistical rigor compara-
provide substantially more stable estimates. Six ble to traditional qualitative research standards.
runs enable 15 pairwise comparisons: Cosine Similarity. Captures semantic equiva-
lence beyond exact matches. We employ sentence-
transformer embeddings (all-MiniLM-L6-v2 [21]),
n(n − 1) 6×5 mapping theme descriptions into 384-dimensional
Comparisons = = = 15 (1)
2 2 semantic space:
This provides sufficient data points to detect
meaningful agreement patterns while avoiding P384
vi · v j vi,k × vj,k
computational expense. The improvement in sim(ti , tj ) = = qP k=1 qP
∥vi ∥∥vj ∥ 384 2
v × 384 2
standard error from three to six runs follows: k=1 i,k k=1 vj,k
(4)
6 √
r
SE3 where vi , vj ∈ R 384 are embedding vectors for
= = 2 ≈ 1.41 (2)
SE6 3 themes ti , tj . This captures semantic equivalence
representing a 41% reduction in variability—a beyond lexical overlap, recognizing that themes
meaningful improvement without excessive cost. phrased differently can express identical concepts.
Consensus Mechanism. We implement an The all-MiniLM-L6-v2 model was selected for its
adaptive consensus algorithm: balance of accuracy (validated performance on

3
STS benchmark) and efficiency (6 layers, 384 di- 3. For custom prompts, accept any valid JSON
mensions), trained on diverse text corpora includ- object (structure-agnostic)
ing natural language inference and semantic tex- 4. For default prompts, validate re-
tual similarity datasets. quired fields (majorEmotionalThemes,
Similarity Computation. For each pair of emotionalPatterns)
runs (i, j), we compute the embedding for each 5. Implement exponential backoff retry (3 at-
theme description using mean pooling of token tempts) for API failures
embeddings, then calculate cosine similarity. The 6. Log parsing errors with original response for
system computes all 15 pairwise similarities, gen- debugging
erating a distribution of scores that provides
richer information than a single reliability coef- This robust parsing achieves 98%+ success rate
ficient. High variance suggests multiple inter- across three LLMs, handling varied response for-
pretive possibilities; low variance indicates strong mats without manual intervention.
convergence.
3.5 Preprocessing and Chunking
3.3 Configurable Analysis Parameters
Implements UTF-8 normalization, intelligent
Our framework provides user-configurable param- chunking for documents exceeding context limits
eters enabling reproducible yet flexible analysis: (preserving paragraph boundaries), and metadata
Seeds. Researchers can configure 1-6 seeds (de- extraction (timestamps, speaker IDs). For large
fault: [42, 123, 456, 789, 1011, 1213]), with each documents (>1M tokens), employs semantic-
seed producing one independent run. Seeds en- aware chunking with 20% overlap, synthe-
able reproducibility while introducing controlled sizing chunk-level themes into document-level
variation. The UI provides dynamic seed man- themes. Client-side preprocessing ensures data
agement (add/remove seeds), with the number of privacy—raw data never transmits to external
runs automatically adjusting to match seed count. servers until analysis initiation.
Temperature. Adjustable temperature T ∈
[0.0, 2.0] (default: 0.7) controls output random-
ness. Lower values (T < 0.5) produce determin- 4 Experiments
istic outputs suitable for structured data; higher
values (T > 1.0) encourage creative interpreta- 4.1 Experimental Design
tion for exploratory research. Temperature ap-
plies uniformly across all runs, ensuring consistent Dataset. We evaluate the framework on a
randomness levels while seeds introduce variation. semi-structured interview transcript exploring art
Custom Prompts. Researchers specify cus- therapy integration with ketamine-assisted psy-
tom prompts with variable substitution: {seed} chotherapy. The transcript (28,377 characters,
inserts the current seed value, enabling run- 173 lines) captures a therapist’s perspectives on
specific instructions (e.g., ”Run ID: {seed}”); combining expressive arts with ketamine therapy,
{text chunk} or {text} inserts transcript con- client experiences, and future opportunities in
tent at specified locations. This enables full con- the field. This dataset represents complex qual-
trol over prompt structure, analytical framework, itative data with: (1) multiple thematic dimen-
and output format while maintaining seed-based sions (methodology, client experiences, theoreti-
variation. cal frameworks), (2) emotional and clinical con-
tent, (3) implicit therapeutic knowledge, and (4)
nuanced contextual interpretation. The complete
3.4 Robust JSON Parsing and Error transcript is available in our GitHub repository.
Handling Evaluation Protocol. For each LLM (Gem-
LLMs frequently return JSON wrapped in mark- ini 2.5 Pro, GPT-4o, Claude 3.5 Sonnet), we
down code blocks (‘ “‘json ... “‘ ‘) or with trailing conducted six independent runs using fixed seeds
text. We implement multi-stage parsing: (42, 123, 456, 789, 1011, 1213) with temperature
T=0.7. We employed a custom prompt specify-
1. Strip markdown code fences using regex: ing: (1) identification of core themes, therapist
^‘‘‘(?:json)?\s*\n? and \n?‘‘‘\s*$ methodology, client experiences, and future out-
2. Attempt JSON parsing; if successful, validate look, (2) JSON output with supporting quotes,
structure and (3) seed-based run identification using {seed}

4
Table 1: Dual Reliability Metrics Across Three
LLMs
Model κ Range Cosine
Gemini 2.5 Pro 0.907 0.745-0.977 95.3%
GPT-4o 0.853 0.672-0.988 92.6%
Claude 3.5 0.842 0.604-1.000 92.1%

Table 2: Consensus Themes and Consistency

Model Themes High Mod.
(Total) Cons. Cons.
Gemini 2.5 Pro 6 2 4
GPT-4o 5 2 3
Claude 3.5 Sonnet 4 1 3

Figure 1: Correlation matrix showing pairwise

placeholder. For each model, we computed: (1) cosine similarity scores across six independent
pairwise Cohen’s Kappa across 15 run pairs, (2) runs for Gemini 2.5 Pro. High similarity values
pairwise cosine similarity using all-MiniLM-L6-v2 (green to yellow, 0.78-0.91) indicate strong inter-
embeddings, (3) consensus themes appearing in run agreement, with the diagonal showing perfect
≥50% of runs, and (4) theme consistency percent- self-similarity (1.000). The consistent high values
ages. across off-diagonal elements demonstrate robust
thematic consistency.

4.2 Model Comparison Results 4.3 Structure-Agnostic Consensus Ex-

We evaluated three leading LLMs on a ketamine traction
art therapy interview transcript (28,377 charac- A key technical contribution enables consensus ex-
ters, 173 lines), conducting six independent runs traction for arbitrary JSON structures. Unlike
per model using fixed seeds (42, 123, 456, 789, frameworks requiring predefined schemas, our im-
1011, 1213). Table 1 presents dual reliability met- plementation:
rics: Cohen’s Kappa and cosine similarity. Dynamic Schema Detection. Ana-
All three models achieve strong agreement (κ > lyzes LLM outputs to identify common ar-
0.80) according to Landis and Koch’s interpreta- ray fields across runs (e.g., core themes,
tion [20], validating the multi-run ensemble ap- client experiences). For each array, identi-
proach. Gemini demonstrates highest consistency fies theme name and supporting quotes fields (or
with κ = 0.907 and narrowest kappa range (0.232 equivalent).
span), indicating stable performance across runs. Semantic Clustering. Groups themes across
Claude exhibits widest kappa range (0.396 span) runs using cosine similarity threshold (0.70).
despite high average κ = 0.842, suggesting occa- Themes with similarity >0.70 are considered
sional divergent runs. Cosine similarity correlates equivalent, accounting for paraphrasing.
strongly with kappa (Pearson r=0.97), validating Consensus Filtering. Themes appearing
semantic embeddings as effective reliability mea- in ≥50% of runs (default threshold) are des-
sures. Figure 1 visualizes the pairwise similarity ignated consensus themes. The system com-
matrix for Gemini 2.5 Pro, showing strong con- putes occurrence frequency, enabling researchers
sistency across all run pairs with similarity values to distinguish high-confidence (5-6/6 runs) versus
predominantly in the 0.78-0.91 range. moderate-confidence (3-4/6 runs) themes.
Consensus Theme Extraction. Gemini Multi-LLM Support. The framework in-
identified 6 consensus themes with 50-83% con- tegrates nine LLM providers: Google Gemini,
sistency, GPT-4o identified 5 themes, and Claude Anthropic Claude, OpenAI GPT, Azure Ope-
4 themes (Table 2). Higher consensus counts nAI, Groq, DeepSeek, and OpenRouter (enabling
suggest more stable thematic identification across access to Llama, Claude, and DeepSeek via
runs. unified API). This enables cross-model valida-

5
tion—themes identified consistently across differ- The trade-off lies in requiring computational re-
ent architectures to receive higher confidence. sources and API access, which may be barriers for
some research contexts.
4.4 Ketamine Art Therapy Analysis
We analyzed a semi-structured interview with a
therapist integrating art therapy with ketamine- 5 Discussion
assisted psychotherapy. Gemini 2.5 Pro achieved
κ = 0.907 and cosine similarity 95.3%, identifying 5.1 Technical Contributions
6 consensus themes.
High-Confidence Themes. Two themes Dual Reliability Metrics. Combining Cohen’s
appeared in 5/6 runs (83% consistency): (1) Kappa with cosine similarity addresses comple-
Overcoming Creative Blocks—clients breaking mentary validation needs: kappa provides statis-
through perfectionist barriers via ketamine and tical rigor comparable to traditional qualitative
art integration, and (2) Challenges in Articu- research (enabling claims of ”almost perfect agree-
lation—neurodiverse clients struggling with ab- ment”), while cosine similarity captures semantic
stract prompts. These themes demonstrate strong equivalence that kappa misses (e.g., ”perfectionist
inter-run agreement despite varied phrasing. barriers” vs ”creative blocks from self-criticism”
Moderate-Confidence Themes. Four achieve high cosine similarity despite low lexical
themes appeared in 3-4/6 runs (50-66% con- overlap).
sistency): Integration of Art Therapy and Configurable Parameters. User-specified
Psychedelic Therapy, Internal Family Systems seeds and temperature enable reproducible yet
(IFS) Integration, Eco Art Therapy, and Group flexible analysis. The {seed} placeholder
Work and Collective Unburdening. Moderate in prompts enables run-specific instructions
consensus captures valuable thematic possibil- while maintaining identical analytical frame-
ities requiring researcher judgment—balancing works. This supports methodological trans-
between conservative (high-threshold) and ex- parency—researchers report exact seeds used, en-
ploratory (low-threshold) approaches. abling replication.
Cross-Model Validation. Comparing across
Structure-Agnostic Design. Dynamic
models: ”IFS Integration” appeared in Gemini
schema detection enables custom prompt formats
(50%), GPT-4o (83%), and Claude (66%), with
without code modification. Researchers spec-
semantic similarity 0.88 across model outputs,
ify analytical frameworks, output structures, and
validating this as a robust theme. ”Creative Lib-
granularity levels suited to their research ques-
eration” appeared in GPT-4o and Claude but not
tions, not constrained by predefined templates.
Gemini’s consensus, suggesting interpretive vari-
The consensus extraction algorithm adapts to any
ation. This cross-model comparison enables iden-
JSON structure containing theme arrays.
tification of model-invariant themes (high confi-
dence) versus model-specific interpretations (re-
quiring human review).
This case demonstrates how our framework 5.2 Comparison with Existing Ap-
balances reliability (filtering spurious themes proaches
with consensus thresholds) with validity (pre-
serving meaningful interpretive variation through Our dual-metric validation extends recent work:
moderate-confidence themes). Zhang et al.’s LATA [12] achieved 0.76 cosine
similarity between LLM and human analyses;
our inter-run consistency exceeds this (0.92-0.95),
4.5 Comparison with Existing Frame-
suggesting ensemble methods may achieve higher
works
reliability than single-run human-AI comparison.
Table 3 compares our approach against existing The TAMA framework [8] employs multi-agent ar-
qualitative analysis frameworks across key dimen- chitectures; our single-agent multi-run approach
sions. provides simpler implementation with compara-
Our framework occupies a unique position: pro- ble reliability. QualIT [2] focuses on key-phrase
viding full thematic analysis with quantified relia- extraction; our structure-agnostic design supports
bility at substantially lower cost and time invest- full thematic analysis with arbitrary output for-
ment than traditional multi-coder approaches. mats.

6
 Table 3: Framework Comparison Across Key Dimensions
Dimension Traditional QualIT Single-Run Our Frame-
Manual LLM work
Analysis Type Full thematic Key-phrase ex- Full thematic Full thematic
traction
Reliability Met- Cohen’s κ Topic coherence None κ + Cosine
rics
Custom Prompts N/A No Yes Yes
Validation Multiple coders Cluster quality None Multi-run en-
Method semble
Cost (20 docs) $400-800 $100-200 $2-4 $3-6
Reliability Level κ = 0.40-0.60 70% coherence Unknown κ = 0.84-0.91
Reproducibility Low Moderate Low High (seeds)

5.3 Interpretation Guidelines aids bias detection (biased themes appearing in

1-2/6 runs flag for review), but cannot eliminate
Cohen’s Kappa. Following Landis-Koch crite-
it. Performance on non-English, non-Western, or
ria: κ > 0.80 (almost perfect), 0.60-0.80 (substan-
highly specialized domain data requires system-
tial), 0.40-0.60 (moderate). Our results (κ = 0.84-
atic evaluation.
0.91) achieve ”almost perfect” reliability across all
Prompt Engineering Dependency. Analy-
three LLMs, validating the ensemble approach for
sis quality depends on prompt design. Effective
rigorous qualitative research.
prompts specify analytical frameworks, output
Cosine Similarity. Interpret as percentage se-
structures, and abstraction levels. Our structure-
mantic overlap: >90% (high consistency), 80-90%
agnostic design enables flexibility but requires re-
(moderate), <80% (low, warrants review). Our
searchers to craft appropriate prompts—a skill re-
results (92-95%) demonstrate strong convergence.
quiring training.
Kappa range (spread between min/max pairwise
Human Oversight Necessity. AI cannot
kappa) indicates stability: <0.25 (stable), 0.25-
perform reflexivity, integrate theoretical frame-
0.40 (moderate variation), >0.40 (high variation
works, or make ethical judgments. Our frame-
requiring investigation).
work provides validated starting points requiring
Consensus Thresholds. Default 50% (3/6
human interpretation, not autonomous analysis.
runs) balances sensitivity and specificity. Ad-
just based on context: 67% (4/6) for conservative
high-stakes research, 33% (2/6) for exploratory 5.5 Future Work
analysis. High-confidence themes (≥83%, 5-6/6
runs) require minimal human review; moderate- Large-Scale Validation. Systematic evaluation
confidence themes (50-66%, 3-4/6 runs) warrant across diverse datasets (clinical interviews, focus
researcher judgment. groups, surveys), domains (healthcare, education,
organizational), and languages (English, Spanish,
Chinese, etc.) to establish reliability benchmarks
5.4 Limitations
and boundary conditions.
Single Dataset Evaluation. Our empirical Human-AI Comparison. Comparison
evaluation uses one interview transcript (ketamine against human coders on identical datasets, mea-
art therapy). While this demonstrates proof- suring kappa agreement between AI consensus
of-concept and enables detailed analysis, gener- themes and human-coded themes. Zhang et al.
alization requires evaluation across diverse do- [12] achieved 0.76 similarity; our inter-run con-
mains (clinical, educational, organizational), data sistency (0.92-0.95) suggests potential for high
types (interviews, focus groups, surveys), and lan- human-AI agreement.
guages. The high reliability (κ > 0.84) suggests Adaptive Run Configuration. Implement-
promise, but boundary conditions remain to be ing thematic saturation metrics [17, 18] to de-
established. termine optimal run counts dynamically. If new
Cultural and Domain Boundaries. LLMs themes cease emerging after N runs, stop analysis
encode training data biases [4]. Our framework rather than using fixed N=6.

7
 Cross-LLM Ensembles. Simultaneous anal- These optimizations prevent UI freezing during
ysis with multiple models (Gemini + GPT- synthesis while maintaining analytical accuracy.
4o + Claude), identifying themes with cross-
architecture support. Our data shows 60-70%
theme overlap across models, suggesting this 7 Conclusion
would increase confidence while filtering model-
specific artifacts. We presented a multi-perspective validation
framework for LLM-based thematic analysis with
dual reliability metrics: Cohen’s Kappa and co-
6 Implementation Considera- sine similarity. Empirical evaluation on ketamine
tions art therapy interview data across three leading
LLMs demonstrates ”almost perfect agreement”
6.1 Technical Architecture (κ > 0.80) for all models: Gemini 2.5 Pro (κ =
0.907, cosine=95.3%), GPT-4o (κ = 0.853, co-
Client-Side Processing. The framework op-
sine=92.6%), and Claude 3.5 Sonnet (κ = 0.842,
erates entirely client-side in the browser using
cosine=92.1%). These results validate the en-
[Link] 14 and React. Data preprocessing, em-
semble approach for rigorous qualitative research,
bedding computation (via [Link]), and
achieving reliability levels comparable to tradi-
consensus extraction occur locally, preserving pri-
tional multi-coder studies at a fraction of the cost
vacy. Raw transcripts never leave the researcher’s
($0.15-0.20 per transcript vs $20-40 for human
device until analysis initiation.
coding).
Multi-Provider API Integration. Unified
Technical contributions include: (1) config-
interface supporting nine providers:
urable seeds and temperature for reproducible
variation, (2) custom prompt support with vari-
• Direct APIs: Google Gemini 2.5 Pro, An-
able substitution, (3) structure-agnostic consen-
thropic Claude 3.5 Sonnet, OpenAI GPT-4o,
sus extraction for arbitrary JSON formats, and
Azure, Groq, DeepSeek - R1
(4) integration with nine LLM providers enabling
• OpenRouter: Unified access to Llama 3.2
cross-model validation. The framework success-
90B, Claude Sonnet, DeepSeek R1 via API
fully identifies consensus themes (4-6 themes per
Each provider implements: (1) standardized re- model) with 50-100% consistency across runs, fil-
quest formatting with seed and temperature pa- tering spurious patterns while preserving valid in-
rameters, (2) response normalization to unified terpretive variation.
JSON structure, (3) error handling with exponen- Our open-source implementation (available
tial backoff, (4) CORS configuration for browser- at [Link]
based requests. API keys provided at runtime; LLM-Thematic-Analysis-Tool) establishes
no credentials stored or transmitted except to re- methodological foundations for reliable AI-
spective provider endpoints. assisted qualitative research, bridging com-
Embedding Computation and Perfor- putational efficiency with rigorous validation
mance. Uses Xenova/[Link] to run all- standards required in qualitative methodology.
MiniLM-L6-v2 in-browser via WebAssembly, gen- Future work should evaluate across diverse
erating 384-dimensional embeddings without ex- domains, languages, and cultural contexts to
ternal API calls. Performance optimizations: establish boundary conditions and normative
reliability benchmarks.
• Limits embedding computation to 10 themes
per run (prevents memory bloat)
• For custom structures with many themes, Acknowledgments
uses lightweight string comparison instead of
full embeddings This research was conducted at Aza Lab at
• Implements sampling for pairwise compar- Department of Psychiatry at Yale Univer-
isons (limits to 10 samples if total pairs ¿10) sity and the Center of Collective Healing at
• Yields control to UI thread via Howard University. Funding provided by in-
setTimeout(0) during intensive loops ternal grants to the AZA Lab. The complete
• Progressive status updates (”Calculating source code is openly available on GitHub
similarity X/Y...”) maintain responsiveness at [Link]

8
LLM-Thematic-Analysis-Tool, with live deploy- 1. Integration of Internal Family Systems
ment at [Link] (IFS) (83.3%, 5/6 runs): The therapist ex-
app/. The ketamine art therapy interview plicitly uses the IFS model, employing art as
transcript used for evaluation is available in the a primary tool to help clients identify, exter-
repository under /[Link]. We thank nalize, and build relationships with their in-
the participating therapist for sharing insights on ternal ’parts.’
integrating expressive arts with ketamine-assisted 2. Overcoming Creative and Emotional
psychotherapy. Blocks (83.3%, 5/6 runs): A significant ben-
AA designed and supervised project, wrote and efit is the unlocking of creative energy previ-
edited mansucript. NA developed model, wrote ously blocked by internal critics or emotional
and implemented code, and drafted manuscript. states like depression.
SA collected and provided qualitative data 3. Synergy of Therapeutic Modalities
(66.7%, 4/6 runs): The interview highlights
the natural and powerful synergy between art
Supplementary Materials therapy, psychedelic-assisted therapy, and In-
ternal Family Systems (IFS).
Supplementary File 1: Gemini 2.5 Pro 4. Client-Centered and Invitational Ap-
Analysis proach (50%, 3/6 runs): The therapist con-
Reliability Metrics: κ = 0.907 (Range: 0.745- sistently offers art as an option rather than
0.977), Cosine Similarity: 95.3% a requirement, respecting the client’s willing-
Consensus Themes (6 total): ness and readiness.
5. Process Over Product Philosophy (50%,
1. Overcoming Creative Blocks (83.3%, 5/6 3/6 runs): The focus is placed squarely on
runs): One client overcame perfectionist and the creative process and the feelings it evokes,
depressive parts through ketamine therapy rather than the aesthetic quality of the final
and began painting extensively, reconnecting artwork.
with a playful and peaceful creative process.
2. Challenges in Articulation (83.3%, 5/6 Supplementary File 3: Claude 3.5 Son-
runs): Some clients, especially those who are net Analysis
concrete thinkers or neurodiverse, struggle
with abstract art prompts or deeper integra- Reliability Metrics: κ = 0.842 (Range: 0.604-
tion of their experiences into daily life. 1.000), Cosine Similarity: 92.1%
3. Eco Art Therapy (66.7%, 4/6 runs): Eco Consensus Themes (4 total):
art therapy as an emerging opportunity.
4. Integration of Art Therapy and 1. Integration of Art and Psychedelics
Psychedelic Therapy (50%, 3/6 runs): (100%, 6/6 runs): The therapist empha-
The therapist emphasizes the natural pairing sizes the natural synergy between art therapy
of art therapy with psychedelic therapy, and psychedelic experiences, viewing them as
highlighting how visual art can deepen complementary modalities that enhance ther-
internal experiences. apeutic outcomes.
5. Integration of Internal Family Systems 2. Integration Challenges (83.3%, 5/6 runs):
(IFS) (50%, 3/6 runs): The therapist in- Some clients struggle with deeper meaning-
tegrates IFS into her approach, using parts making in their artwork, particularly those
work to externalize clients’ internal experi- who are more concrete thinkers.
ences through visual art and metaphors. 3. Integration Framework (66.7%, 4/6 runs):
6. Group Work and Collective Unburden- Uses a combination of Internal Family Sys-
ing (50%, 3/6 runs): Group work and collec- tems (IFS), art therapy, and ketamine-
tive unburdening as future opportunities. assisted psychotherapy, offering art as an op-
tional but encouraged component.
Supplementary File 2: GPT-4o Analysis 4. Creative Liberation (66.7%, 4/6 runs):
Clients experiencing breakthrough in creative
Reliability Metrics: κ = 0.853 (Range: 0.672- expression and overcoming perfectionist bar-
0.988), Cosine Similarity: 92.6% riers through the combination of ketamine
Consensus Themes (5 total): and art therapy.

9
 Note: Complete reports available at guage Model: An Exploration and Provoca-
[Link] tion on the Limits of the Approach. Qualita-
LLM-Thematic-Analysis-Tool. tive Research.

[10] Turobov, A., Coyle, D., & Harding, V.

References (2024). Using ChatGPT for thematic analy-
sis. Bennett Institute for Public Policy, Uni-
[1] Braun, V., & Clarke, V. (2006). Using the-
versity of Cambridge.
matic analysis in psychology. Qualitative Re-
search in Psychology, 3(2), 77-101. [11] Fulgencio, F. F. J. (2024). Thematic Anal-
[2] Bhaduri, S., Gil, A., Mittal, A., & ysis through Artificial Intelligence (AI). The
Mulkar, R. (2024). Qualitative Insights Tool Qualitative Report, 29(2), 566-584.
(QualIT): LLM Enhanced Topic Modeling.
[12] Zhang, S., et al. (2025). LATA: A Pilot Study
arXiv preprint arXiv:2409.15626.
on LLM-Assisted Thematic Analysis of On-
[3] Bennis, I., & Mouwafaq, S. (2025). Ad- line Social Network Data Generation Experi-
vancing AI-driven thematic analysis in qual- ences. ACM Digital Library.
itative research: a comparative study
of nine generative models on Cutaneous [13] Gupta, A. K., Li, S., Lee, J. A., & Nepal,
Leishmaniasis data. BMC Medical Infor- S. K. (2025). Utilizing Large Language Mod-
matics and Decision Making, 25(1), 124. els to Conduct Thematic Analysis: A Case
[Link] Study on Focus Group Transcripts. SSRN
Electronic Journal.
[4] Sakaguchi, K., Sakama, R., & Watari, T.
(2025). Evaluating ChatGPT in Qualita- [14] Braun, V., & Clarke, V. (2023). Large Lan-
tive Thematic Analysis With Human Re- guage Models in Thematic Analysis: Prompt
searchers in the Japanese Clinical Con- Engineering, Evaluation, and Guidelines for
text and Its Cultural Interpretation Chal- Qualitative Software Engineering Research.
lenges: Comparative Qualitative Study. Semantic Scholar.
Journal of Medical Internet Research, 27,
e71521. [Link] [15] Sanford, K., et al. (2024). Enhancing The-
matic Analysis with Local LLMs: A Scien-
[5] Rana, Z., & Asad, R. (2023). LLM- tific Evaluation of Prompt Engineering Tech-
in-the-loop: Leveraging Large Language niques. CMS Conferences.
Model for Thematic Analysis. arXiv preprint
arXiv:2310.09893. [16] Nelson, M. R. (2025). Reflecting on LLM
Support in Reflexive Thematic Analysis: An
[6] Schlagwein, D. (2024). Role Play: Conver- Exploratory Study. Qualitative Research.
sational Roles as a Framework for Reflexive
Practice in AI-Assisted Qualitative Research. [17] Patel, R., et al. (2024). Reflections on In-
Qualitative Research. ductive Thematic Saturation as a potential
metric for measuring the validity of an in-
[7] Landers, R. N., & Behrend, T. S. (2024).
ductive Thematic Analysis with LLMs. arXiv
Reconciling Methodological Paradigms: Em-
preprint.
ploying Large Language Models as Novice
Qualitative Research Assistants in Talent
[18] Chen, Y., et al. (2025). Codebook Re-
Management Research. arXiv preprint.
duction and Saturation: Novel observa-
[8] Alhakeem, M., et al. (2025). TAMA: A tions on Inductive Thematic Saturation for
Human-AI Collaborative Thematic Anal- Large Language Models and initial cod-
ysis Framework Using Multi-Agent LLMs ing in Thematic Analysis. arXiv preprint
for Clinical Interviews. arXiv preprint arXiv:[Link].
arXiv:[Link].
[19] Kumar, R., et al. (2025). Large Lan-
[9] Lindh, M., & Messina, R. (2023). Perform- guage Model for Qualitative Research – A
ing an Inductive Thematic Analysis of Semi- Systematic Mapping Study. arXiv preprint
Structured Interviews With a Large Lan- arXiv:[Link].

10
[20] Shrout, P. E., & Fleiss, J. L. (1979). Intr-
aclass correlations: Uses in assessing rater
reliability. Psychological Bulletin, 86(2), 420-
428.

[21] Hartmann, J. (2022). j-hartmann/emotion-

english-distilroberta-base. Hug-
ging Face. Retrieved from https:
//[Link]/j-hartmann/
emotion-english-distilroberta-base

11