Zero-to-One Guide: Information Retrieval with
Lucene & Luke (Step by Step)
Audience: Absolute beginners who have never studied Information Retrieval (IR).
Goal: By the end, you will (1) understand the core IR ideas, (2) build and query a Lucene
index, and (3) inspect that index with Luke. All steps are tested on Ubuntu.
0) What is Information Retrieval (IR)?
IR is about finding relevant documents among many. Google Search is a classic example.
We’ll build a tiny search engine locally using Apache Lucene.
Document: one item to search (e.g., an article, PDF, or product).
Field: a named part of a document (e.g., title, content).
Token: a word-like unit extracted during analysis (e.g., 'search', 'engine').
Index: a data structure optimized for fast search.
Analyzer: decides how text is split, normalized, lowercased, stemmed, etc.
Query: how users ask for information (free text or structured).
Lucene scores results using BM25 by default: roughly, documents that use the query terms
more often and in shorter documents tend to rank higher.
1) Environment Setup (Ubuntu)
1.1 Install Java (JDK)
Lucene 8 works smoothly on Java 11 or Java 17. Java 21 often works, but if you hit issues,
use Java 17.
# Option A: Install Temurin JDK 17 (recommended)
sudo apt-get update
sudo apt-get install -y wget unzip
sudo apt-get install -y openjdk-17-jdk
# Check
java -version
1.2 Download Lucene 8.11.2 (includes Luke 8)
Luke became an official Lucene module in 8.x, so you get Luke when you download Lucene
8.x.
mkdir -p ~/lucene-luke-demo && cd ~/lucene-luke-demo
# Download Lucene 8.11.2 from Apache archive (contains luke/ folder)
wget [Link]
[Link]
unzip [Link]
cd lucene-8.11.2
# Verify important folders exist
ls -1
# Expected: core/ analysis/ queryparser/ demo/ luke/ ...
2) Quick Win: Use the Built-in Demo to Index & Search
Lucene ships with a tiny demo. We’ll index two sample files and search them.
2.1 Create demo docs
cd ~/lucene-luke-demo/lucene-8.11.2/demo
mkdir -p docs
printf "Lucene is a powerful Java search library.\n" > docs/[Link]
printf "Luke helps you inspect Lucene indexes.\n" > docs/[Link]
2.2 Index the docs
IndexFiles is inside [Link]. We must add lucene-core to the classpath.
# From the demo folder
java -cp "[Link]:../core/[Link]"
[Link] -docs docs -index index
2.3 Search the index
SearchFiles needs queryparser and analyzers jars on the classpath.
java -cp
"[Link]:../core/[Link]:../queryparser/
[Link]:../analysis/common/lucene-analyzers-
[Link]" [Link] -index index
# When prompted, type a query, e.g.:
# Lucene
# or
# Luke
3) Inspect the Index with Luke (GUI)
Luke is a desktop app to open a Lucene index and explore its internals.
# Still inside ~/lucene-luke-demo/lucene-8.11.2
cd ~/lucene-luke-demo/lucene-8.11.2/luke
# Make the launcher executable (first time only)
chmod +x [Link]
# Run Luke (on Linux desktop)
./[Link]
In Luke:
File → Open Index → select: ~/lucene-luke-demo/lucene-8.11.2/demo/index
Overview: see number of docs, fields, segments, etc.
Documents tab: browse individual stored fields.
Search tab: try queries and see matched docs.
Analysis tab: type text to see tokenization (how the analyzer splits and normalizes text).
Terms tab: see the vocabulary and document frequencies.
4) Write Your Own Minimal Java Code (No IDE required)
We’ll create two Java classes: SimpleIndexer and SimpleSearcher. This helps students see
the core APIs.
4.1 [Link]
import [Link];
import [Link].*;
import [Link].*;
import [Link].*;
import [Link];
import [Link];
public class SimpleIndexer {
public static void main(String[] args) throws IOException {
String indexPath = "my_index";
Directory dir = [Link]([Link](indexPath));
StandardAnalyzer analyzer = new StandardAnalyzer();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
try (IndexWriter writer = new IndexWriter(dir, config)) {
Document doc1 = new Document();
[Link](new TextField("title", "Hello Lucene",
[Link]));
[Link](new TextField("content", "This is a simple Lucene
indexing demo.", [Link]));
[Link](doc1);
Document doc2 = new Document();
[Link](new TextField("title", "Meet Luke",
[Link]));
[Link](new TextField("content", "Luke lets you inspect
Lucene indexes.", [Link]));
[Link](doc2);
}
[Link]("Index created at: " + indexPath);
}
}
4.2 [Link]
import [Link];
import [Link];
import [Link];
import [Link];
import [Link].*;
import [Link].*;
import [Link];
public class SimpleSearcher {
public static void main(String[] args) throws Exception {
String indexPath = "my_index";
Directory dir = [Link]([Link](indexPath));
try (DirectoryReader reader = [Link](dir)) {
IndexSearcher searcher = new IndexSearcher(reader);
StandardAnalyzer analyzer = new StandardAnalyzer();
String userQuery = ([Link] > 0) ? [Link](" ",
args) : "Lucene";
Query query = new QueryParser("content",
analyzer).parse(userQuery);
TopDocs results = [Link](query, 10);
[Link]("Query: " + userQuery);
[Link]("Hits: " + [Link]);
for (ScoreDoc sd : [Link]) {
Document d = [Link]([Link]);
[Link]("- " + [Link]("title") + " (score="
+ [Link] + ")");
}
}
}
}
4.3 Compile & Run (using Lucene jars)
# From ~/lucene-luke-demo/lucene-8.11.2 (root of the unzipped dist)
# Save the two .java files in a new folder, e.g.
~/lucene-luke-demo/code
mkdir -p ~/lucene-luke-demo/code && cd ~/lucene-luke-demo/code
# Copy/paste the two files here as [Link] and
[Link]
# Compile: add core, queryparser, and analyzers-common to classpath
javac -cp "../core/[Link]:../queryparser/lucene-
[Link]:../analysis/common/lucene-analyzers-common-
[Link]" [Link] [Link]
# Run indexer
java -cp ".:../core/[Link]:../queryparser/lucene-
[Link]:../analysis/common/lucene-analyzers-common-
[Link]" SimpleIndexer
# Run searcher
java -cp ".:../core/[Link]:../queryparser/lucene-
[Link]:../analysis/common/lucene-analyzers-common-
[Link]" SimpleSearcher Lucene
5) (Optional) Open the Same Index in Luke
Start Luke (see step 3) and open the folder my_index. Explore the fields, terms, and run test
queries in Luke’s UI.
6) Doing It in IntelliJ IDEA (Maven Project)
If you prefer an IDE setup, use Maven to manage Lucene dependencies.
6.1 Create a Maven project
Open IntelliJ → New Project → Maven → Create project
Create src/main/java and src/main/resources if needed.
Add the following to [Link]:
<project xmlns="[Link]
xmlns:xsi="[Link]
xsi:schemaLocation="[Link]
[Link]
<modelVersion>4.0.0</modelVersion>
<groupId>[Link]</groupId>
<artifactId>lucene-luke-lab</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<[Link]>11</[Link]>
<[Link]>11</[Link]>
</properties>
<dependencies>
<dependency>
<groupId>[Link]</groupId>
<artifactId>lucene-core</artifactId>
<version>8.11.2</version>
</dependency>
<dependency>
<groupId>[Link]</groupId>
<artifactId>lucene-analyzers-common</artifactId>
<version>8.11.2</version>
</dependency>
<dependency>
<groupId>[Link]</groupId>
<artifactId>lucene-queryparser</artifactId>
<version>8.11.2</version>
</dependency>
</dependencies>
</project>
Create classes SimpleIndexer and SimpleSearcher under src/main/java (use the code
above). Then Run → Edit Configurations → add application runs with the appropriate main
class.
7) Analyzer Experiments (Bonus)
Swap the StandardAnalyzer for others to see differences in
tokenization/stopwords/stemming.
// Example: using EnglishAnalyzer
// import [Link];
// Analyzer analyzer = new EnglishAnalyzer();
// Example: using ArabicAnalyzer (requires lucene-analyzers-common)
/*
import [Link];
Analyzer analyzer = new ArabicAnalyzer();
*/
Use Luke → Analysis tab to visualize how text is tokenized with different analyzers.
8) Mini-Lesson: Query Language Basics
Term query: lucene
Phrase query: "lucene index"
Boolean: lucene AND luke, lucene OR luke, lucene NOT luke
Fielded: title:lucene content:index
Wildcard: luc* (use sparingly)
Try these in the demo SearchFiles program or in Luke’s Search tab.
9) Evaluation (Very Short Intro)
Precision: Of the retrieved docs, how many are relevant?
Recall: Of the relevant docs in the collection, how many did we retrieve?
Class exercise: decide which returned titles are relevant for the query 'lucene', compute
precision at 5.
10) Troubleshooting
NoClassDefFoundError (e.g., QueryParser): Add the missing jar to your classpath
(queryparser, analyzers-common).
IndexFormatTooNew/TooOld: Luke 8 reads Lucene 8 indexes. Rebuild with Lucene 8 if
needed.
Java 21 oddities: prefer JDK 17/11 for Lucene 8 projects.
X11 display issues when running Luke in Docker: run with DISPLAY and mount
/tmp/.X11-unix, or run Luke directly on your host.
‘no main manifest attribute’ when running a jar: many Lucene jars are libraries; run the
correct main class with -cp.
11) Student Tasks (Homework)
1. Create 5 small text files about different topics and index them.
2. Write a new field 'tags' and query only that field.
3. Swap StandardAnalyzer with EnglishAnalyzer and compare tokens in Luke → Analysis.
4. Measure precision@5 for two queries you choose.
Appendix A) Docker (Optional, GUI-enabled Luke)
If you prefer Docker, use a base image with JDK 17, install wget/unzip, download Lucene
8.11.2, and run [Link] on the host X server.
# Example (partial) Dockerfile
FROM eclipse-temurin:17-jdk
RUN apt-get update && apt-get install -y wget unzip && rm -rf
/var/lib/apt/lists/*
WORKDIR /opt/lucene8
RUN wget [Link]
[Link] && unzip [Link] && rm [Link]
# To run Luke with Linux desktop:
# docker run --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix
-it <image> bash
# Then: cd /opt/lucene8/lucene-8.11.2/luke && ./[Link]
Appendix B) Useful Commands Recap
# Build demo index
cd ~/lucene-luke-demo/lucene-8.11.2/demo
java -cp "[Link]:../core/[Link]"
[Link] -docs docs -index index
# Search demo index (interactive)
java -cp
"[Link]:../core/[Link]:../queryparser/
[Link]:../analysis/common/lucene-analyzers-
[Link]" [Link] -index index
# Launch Luke
cd ~/lucene-luke-demo/lucene-8.11.2/luke
chmod +x [Link]
./[Link]