Scribd
Upload
4 views3 pages

Structuring Unstructured Data with LLMs

The document outlines an assignment focused on structuring unstructured data using LLMs, which convert messy text into clean, queryable formats. It emphasizes the importance of bridging theory and practice, handling LLM uncertainty, and building production-ready data pipelines. Deliverables include Mermaid diagrams, a structured CSV file, and a Colab notebook with code and comments.

Uploaded by

shrayan.mishra07
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views3 pages

Structuring Unstructured Data with LLMs

The document outlines an assignment focused on structuring unstructured data using LLMs, which convert messy text into clean, queryable formats. It emphasizes the importance of bridging theory and practice, handling LLM uncertainty, and building production-ready data pipelines. Deliverables include Mermaid diagrams, a structured CSV file, and a Colab notebook with code and comments.

Uploaded by

shrayan.mishra07
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Structuring Unstructured Data with LLMs: DSPPractical

Assignment

Why This Matters

Unstructured data (text, PDFs, web content) makes up 80-90% of data, yet it’s unusable

for analysis without costly manual processing. LLMs solve this by acting as "structured

data compilers" – converting messy text into clean, queryable formats (entities, relations,

graphs). This assignment tests your ability to:

• Bridge theory and practice (real-world data ≠ textbook examples)

• Handle LLM uncertainty (confidence loops, error resilience)

• Build production-ready pipelines (not just one-off scripts)

The provided DSPy codebase demonstrates exactly what modern data engineering

teams need:

1. Entity extraction → Turning text into typed objects (e.g., "pelletized frass" → Drug)
2. Intelligent deduplication → Solving real-world noise (e.g., "PB IC", "pea-barley

intercrop", "pea-barley intercrops" → 1 entity)

3. Knowledge graph generation → Creating visual, queryable relationships (Mermaid

diagrams)

Code Walkthrough (Key Concepts for Applicants)

Code Sample: [Link]


w8EBRn_bRCysFnnBNUXCqg05KR8VsX?usp=sharing

1. Entity Extraction (ExtractEntities Signature)

class EntityWithAttr(BaseModel):
entity: str = Field(description="the named entity")

attr_type: str = Field(description="semantic type (e.g. Drug, Disease)")

class ExtractEntities([Link]):

paragraph: str = [Link]()

entities: List[EntityWithAttr] = [Link]()

ed-65-years-2025a1000ro7

[Link]://[Link]/global-development/2025/oct/13/astro-

ambassadors-stargazers-himalayas-hanle-ladakh-india

Assignment Deliverables

1. Mermaid Diagrams (10 total)

• Save as mermaid_{i}.md (e.g., mermaid_1.md)

• Must include:

o Valid Mermaid syntax (test in Mermaid Live Editor)

o Only entities from your deduplicated list as nodes

o Edge labels trimmed to 40 chars (as in example)

2. Structured CSV ([Link])

link tag tag_type

[Link] sustainable agriculture Concept

[Link] nitrogen uptake Process

... ... ...

Rules:

• tag: Exact entity string (e.g., "pea-barley intercrop", not "intercrop")


• tag_type: Semantic category (e.g., Crop, Process, Measurement)

• No duplicates per URL (use your deduplication logic!)

3. Colab Notebook

• Must include:

o Full code with comments explaining key steps

o Output csv

How to get Free LLM API key to use with DSPY?

1. Follow steps using your own account to get Free LLM api keys from here:

[Link]

You might also like