Test Layers (what runs where)
Layer 1 — Endpoint-Level (ALL endpoints)
Purpose: “Does this API obey its contract and stay secure?”
Applies to every endpoint in all services.
Test types:
● Smoke
● Contract / schema
● Functional (isolated)
● Negative + Auth
How it’s generated:
● Auto-discovered from Swagger
● Payloads auto-generated
● Negative cases auto-mutated
This gives 70–80% coverage with zero business knowledge.
Layer 2 — Flow-Level (Critical paths only)
Purpose: “Do services work together correctly?”
Applies only to state-creating flows.
Critical flows (derived mechanically):
Auth Service
Login → Token issued → Token accepted by other services
Asset Service
Create Site → Create Camera → Retrieve Camera
Camera Metadata Service
Camera exists → Metadata fragment ingested → Fragment retrievable
Cross-Service
Camera created (Asset)
→ Metadata accepted (Metadata)
→ Analytics/queries reflect camera
Only ~10–12 flows total.
These are high-value, low-count tests.
Layer 3 — UI Validation (Critical UI journeys)
Purpose: “Is UI wired correctly to APIs?”
UI tests are thin, not exhaustive.
Included:
● Login via UI
● Create Site via UI
● Add Camera via UI
● File upload via UI
Excluded:
● Dashboards
● Read-only views
● Filters / charts
Rule:
If API tests validate logic, UI tests validate wiring only.
Layer 4 — System-Level (Selected APIs)
Load Testing
Goal: Ensure performance under expected traffic.
Selected APIs only:
● Login / token refresh
● Event / fragment ingestion
● Analytics queries
● File upload
Run profiles:
● Expected traffic
● Peak traffic (1–2×)
Stress Testing
Goal: Find breaking point, not correctness.
Same APIs as load, but at 5–10× traffic.
Outcome:
● Graceful degradation
● No data corruption
● No cascading failures
Layer 5 — Security (Entire platform)
Purpose: Horizontal risk detection.
Not endpoint-specific. Pattern-based.
Attack classes applied across all services:
● Auth bypass
● Token replay
● Cross-tenant access
● Privilege escalation
● Injection (SQL/JSON)
● Rate abuse
Security tests run:
● On staging
● Nightly / scheduled
● Not blocking PRs (signal, not noise)
Custom Python API Automation Framework built primarily on top of pytest.
Here is the breakdown of the technology stack based on your codebase:
1. Core Framework
● Runner:
● pytest (The main engine for discovering and running tests).
● language: Python 3.12+.
2. HTTP Clients (API Interaction)
● requests: For synchronous API calls (likely used in older or simpler tests).
● httpx: For asynchronous API calls (seen in
● [Link] as
● async_client) and SSE (Server-Sent Events) stream testing.
3. Reporting
● pytest-html: Generates the HTML report (
● [Link]).
● Custom CSV Logger: We recently implemented a hook in
● [Link] to write results to
● reports/test_history.csv.
4. specialized Testing
● Performance: locust (Files present in
● tests/performance/[Link]).
● UI / Browser: playwright (Listed in
● [Link] and markers, though we haven't actively run these yet).
5. Architecture Pattern
● Data-Driven: Layer 1 tests read from
● contracts/endpoint_inventory.json rather than having hardcoded test functions for every
endpoint.
● Environment Agnostic: Uses
● .env to switch between DEV/QA/PROD URLs dynamically.