Skip to content

feat: add FirecrawlChannel for deep web scraping and search#502

Open
nankingjing wants to merge 1 commit into
Panniantong:mainfrom
nankingjing:feat/firecrawl-channel
Open

feat: add FirecrawlChannel for deep web scraping and search#502
nankingjing wants to merge 1 commit into
Panniantong:mainfrom
nankingjing:feat/firecrawl-channel

Conversation

@nankingjing

@nankingjing nankingjing commented Jul 13, 2026

Copy link
Copy Markdown

Add a new Firecrawl-based channel for Agent-Reach that provides:

  • read(): Scrape any URL to clean markdown via Firecrawl API
  • crawl(): Deep crawl websites with configurable depth/limits
  • search(): Web search with structured results
  • extract(): AI-powered structured data extraction

The channel is tier 1 (needs free API key from firecrawl.dev).
Gracefully reports missing dependencies and API key status in check().

Changed files:

  • agent_reach/channels/firecrawl.py - New FirecrawlChannel
  • agent_reach/channels/init.py - Register FirecrawlChannel
  • agent_reach/config.py - Add firecrawl to FEATURE_REQUIREMENTS
  • tests/test_firecrawl_channel.py - 25 unit tests

Tests: 25/25 passing + all contract tests pass.

@nankingjing
nankingjing force-pushed the feat/firecrawl-channel branch from cbe7779 to 4d1de20 Compare July 13, 2026 16:15
@nankingjing nankingjing changed the title feat(channels): add Firecrawl deep web scraping channel feat: add FirecrawlChannel for deep web scraping and search Jul 13, 2026
@nankingjing
nankingjing force-pushed the feat/firecrawl-channel branch 2 times, most recently from fb80111 to 2b20c37 Compare July 14, 2026 01:33
- Add FirecrawlChannel with read/search/crawl/extract methods.
- Register in channel registry and Config.FEATURE_REQUIREMENTS.
- Add comprehensive unit tests using mocks.
- Co-Authored-By: Claude <noreply@anthropic.com>
@nankingjing
nankingjing force-pushed the feat/firecrawl-channel branch from 2b20c37 to 9b57617 Compare July 14, 2026 01:37
@nankingjing

Copy link
Copy Markdown
Author

I cleaned the CRLF-only churn from the existing files, added a firecrawl optional dependency (also included in all), and verified the implementation against firecrawl-py 4.17.0 method signatures. The 25 Firecrawl tests pass; the broader channel-contract run has one pre-existing Windows YouTube config-path failure unrelated to this PR.

@nankingjing

Copy link
Copy Markdown
Author

Ready for review. This adds a FirecrawlChannel for deep web scraping and search (268 lines + 337 lines of contract tests). 25 tests pass. The Firecrawl dependency is optional, included under the all extras group alongside existing channels. @moxxxskaifa @Panniantong PTAL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant