Next-gen SIEM & observability.
Search years of logs in seconds.
Reduce log costs by 80-90%.
[Link] 2024 SALES@[Link]
Table of Contents
01 User Challenges 02 About Scanner 03 Features 04 Pricing
quotes
overview
detection rules as code
cost comparison
legacy siems
how it works: indexing
scanner api
detailed cost breakdown
SQL based tools how it works: search
scanner for grafana
performance scanner for splunk
upcoming features
product roadmap
05 Resources 06 Appendix
jupyter notebooks
Scanner for splunk
performance comparisons
low vs high volume log data
Challenges
Next-gen SIEM & observability [Link]
What We Hear
Our Existing SIEM Only getting 7-30 days of retention, which isn’t
enough to search and investigate historical data
Expensive and our log data growth is outpacing
our budget.
including testing new detections.
Vendor lock-in.
Unable to query historical data in our S3
archives natively or with an API.
Athena, Snowflake and Searches, especially over historical data, are
painfully slow, take additional engineering work
Need peace of mind that we can access all of
our historical data and search all of it fast when
other SQL-based tools to be able to perform, and/or crash. investigating an incident.
Next-gen SIEM & observability [Link]
User Challenges:
Legacy SIEMs
01 02 03
Short Retention High Costs Undetected Threats
With only 7-30 days of retention, you’re unable to Log volumes are always increasing, and legacy Attackers who’ve gained access to systems more
test new detections against, or perform threat SIEMs charge $3-$5 per GB ingested because of than 7-30 days ago, and have maintained a
hunting through, historical data. their architecture. persistent method of access, may go undetected.
Modern systems like Scanner use a cloud-native This scenario is fairly common: the average time to
design to reduce costs up to 80-90% identify and contain a breach is 277 days.
(IBM Report)
Next-gen SIEM & observability [Link]
User Challenges:
SQL-based tools: Athena,
Snowflake, and Tools Built
on Snowflake
01 02 03 04
Long Query Times Strict Schemas Upfront Work Maintenance
Queries often take hours to complete SQL is designed for business data Loading semi-structured data into Heavy maintenance requirement for
or crash when data set size reaches with strict schemas, which can be a relational database tables is a lot of transformations, mapping fields, data
10+TB. mismatch for semi-structured security work for each new log source. pipeline, etc.
data (eg. JSON).
Next-gen SIEM & observability [Link]
About Scanner
Next-gen SIEM & observability [Link]
Scanner is a SIEM and
observability platform
for logs in S3.
We provide log search and threat
detection. Search years of logs in
seconds. Reduce costs by 80-90%.
Next-gen SIEM & observability [Link]
Why Scanner
12+ 100%
Months of Data Retention Data ownership – no vendor lock-in
Under The Hood
Purpose-built, cloud-native log search designed
90%
for petabyte scale.
Reduction in logging costs test detections on historical data
Indexes maintained in cloud storage instead of
in-memory, 10x cost reduction.
50x 85%
Fast advanced queries via serverless compute -
search years of logs in seconds.
Faster soc investigations reduction in data engineering time
Next-gen SIEM & observability [Link]
how it works:
Indexing Scanner Instance
SQS
Indexers
Indexers
Scanner indexes raw log files directly Indexers
in users S3 buckets, avoiding time
consuming data engineering work. Notify Read Save index file
This gives teams 100% data ownership
Your AWS Account
and avoids vendor lock-in.
S3 Bucket S3 Bucket
Source Log Files Scanner Index
e.g. Cloudtrail Files Bucket
Next-gen SIEM & observability [Link]
Scanner
how it works: Instance
Search
Web Browser
Aggregator
Query Workers
When a user runs a log search, Scanner
launches Lambda functions to traverse
the index files in S3 massively in parallel.
Your AWS Account Your AWS Account
Lambda functions use Scanner’s inverted Scanner Index Files Scanner Index Files
index to rapidly reduce the search space
to find hits.
Users will see search speeds of up to
Scanner Instance
10TB per second over petabytes of log
data.
Your AWS Account
Scanner Index Files
Next-gen SIEM & observability [Link]
Scanner vs. Athena Performance
Data Set Query Time range 1 day 1 Week 1 Month 1 Year
CloudTrail JSON logs spanning 1 year, Query Time Crashed after
250TB uncompressed. AWS Athena 3m 14s 21m 5s 1h 18m 33s 1h 31m 1s
Query Time
Use Case 3.4s 3.6s 6.4s 20.6s
Scanner
Search for indicator of compromise.
Specifically, look up all activity for a Engineering Time
3m 10s 21m 1s 1h 18m 26s
specific AWS Access Key ID. ∞
Saved per Query (57x) (353x) (736x)
Next-gen SIEM & observability [Link]
Features
Next-gen SIEM & observability [Link]
Feature
Detection Rules name: AWS EC2 Startup Shell Script Change
enabled: true
as Code
description: |-
Detects changes to the EC2 instance startup script. The shell
script will be executed as root/SYSTEM every time the specific
instances are booted up.
References:
Tests File Name
* [Link] Rule Name
blob/866376cd711666c775bbfcde0524c817f2c5b181/pacu/modules/
detection_rules/unusual_service_ins
ec2__startup_shell_script/[Link]#L9
Unusual service installation
False Positives:
Manage detection rules directly in your team’s detection_rules/suspicious_process Suspicious process execution
* Valid changes to the startup script
G it Hub repositories for collaboration, change severity: High
management, and continuous deliver y of query_text: |-
detection_rules/aws_ec2_startup_sc AWS EC2 Startup Shell Script Change
%ingest.source_type="aws:cloudtrail"
detections. eventSource="[Link]"
eventName="ModifyInstanceAttribute"
detection_rules/system_config_cha Changes in system configurations
[Link]: 'userData'
| stats
Streamline t hreat detection development and detection_rules/multiple_failed_login
min(timestamp) as firstTime,
Multiple failed login attempts
backtest against years of logs in seconds. max(timestamp) as lastTime,
count() as eventCount
detection_rules/unusual_outbound_t Unusual outbound traffic
by
[Link],
eventSource,
detection_rules/vpn_access_unauth Unauthorized VPN access
Using familiar pull request workflows and our built- eventName,
in CI/CD. awsRegion
time_range_s: 300
detection_rules/abnormal_resource Abnormal resource usage
run_frequency_s: 60
event_sink_keys:
detection_rules/unusual_email_activ Unusual email activity
- high_severity_alerts
Next-gen SIEM & observability [Link]
detection_rules/unusual_database_ Unusual database access
Feature
Scanner API
S3
Load data into S3 using Cribl, Vector, or Logstash.
Scanner idexes the data in S3 and sends detection
alerts to endpoints like Slack, Webhooks, and Tines.
Query Scanner via our API with Splunk, Jupyter, and
Grafana.
Next-gen SIEM & observability [Link]
Feature
Scanner for
Grafana
Search S3 logs via Scanner API directly from Grafana.
Connect traces to logs for fast application debugging.
Next-gen SIEM & observability [Link]
Feature
Scanner for
Splunk
Search S3 logs via Scanner API directly from Splunk.
Correlate logs ingested into Splunk with logs in S3.
| scanner q="eventSource: GetObject
| stats count() by [Link]"
Next-gen SIEM & observability [Link]
Upcoming features
October 2024
Dashboards
November 2024
Integration with
Threat Intelligence
Feeds
Next-gen SIEM & observability [Link]
Product Velocity and Roadmap
Q1 2024 Detections as Code - YAML + CLI - Sigma Q2 2024 Self-setup for AWS and import rules
Ad-hoc Query and Detection Rule API Jupyter notebooks for threat hunting and manual response
Audit Logs Custom Content (ie Detections) in Scanner for Splunk
Detection event sinks - Slack and Webhooks SSO Support: SAML, OIDC, Okta Workforce
Scanner for Splunk custom search commands RBAC
Multi-tenant env for easy trial deployments Exporting
Q3 2024 Grafana plugin to support Traces to Logs Q4 2024 Threat Intelligence Feeds
Detection Rules with Change Management and Tests Log enrichment from threat intel
Dashboards UI Advanced query language features
Normalization to OCSF, CIM, and ECS Dashboards-as-code
Query language features: eval, iplocation, more
Anomaly Detection
Next-gen SIEM & observability [Link]
Pricing
Next-gen SIEM & observability [Link]
Pricing Comparison
Competitors' prices are based on companies' websites.
100 GB / Day 250 GB / Day 500 GB / Day 1 TB / Day 4 TB / Day
Scanner $14.1k $30.5k $71.8k $109.2k $333.5k
12+ Months Retention
Datadog $90.5k $227.4k $453.8k $921k $3.65m
30 Day retention
Splunk $89.4k $215.3k $395k $824.8k $3.4m
30 Day retention
Panther $125.6k $198.9k $316.9k $560.3 $700.4k
12 month retention
Azure Sentinel $123.4k $281.7k $525.9k $1.05m $3.98m
90 day retention
Elastic SIEM $73k $132.5k $261.5k $444.6k $1.6m
30 Day retention
Loki / Cloudwatch $28.2k $61k $143.6k $218.4k $432k
30 Day retention
price shown per year
Next-gen SIEM & observability [Link]
Detailed Pricing Breakdown
100 GB / Day 250 GB / Day 500 GB / Day 1 TB / Day 4 TB / Day
Scanner $14.1k $30.5k $71.8k $109.2k $333.5k
LOgs indexed in s3
S3 Storage $1.5k $3.8k $7.7k $15.4k $61.9k
12 Months retention
Indexing S3 Requests $835 $2k $4.2k $8.3k $33.4k
Paid to aws
Querying S3 Requests $15 $38 $76 $151 $605
Paid to aws
Total $16.5k $36.3k $83.8k $133k $429k
Scanner + CUstomer s3 COsts
price shown per year
Next-gen SIEM & observability [Link]
Next Steps:
Proof of Concept
No credit card or contract required
Concierge onboarding
Private Slack channel with dedicated engineering
support
Training session
Weekly working sessions
Next-gen SIEM & observability [Link]
Resources
links Contact
[Link] sales@[Link]
Documentation
Blog
Book a Demo
Next-gen SIEM & observability [Link]
Appendix
Next-gen SIEM & observability [Link]
beta
Jupyter Notebook Threat Hunting & Response
Access the data in the detection event.
Run Scanner queries to fetch context
data from logs.
Next-gen SIEM & observability [Link]
beta
Jupyter Notebook Threat Hunting & Response
Set the severity, send results to a specific event sink - in this case,
a Slack event sink.
Use Cases
Phase 1: Interactive Jupyter Phase 2: Jupyter notebooks running
notebooks in the frontend in the backend
Threat Hunting - Run Scanner Automated Response - When a
queries, correlate sources detection rule is triggered, run
interactively, explore, create notebooks, which can send data
charts with matplotlib, etc. to additional destinations /
notebooks.
Manual Response - Click a link in
a Slack alert or Jira ticket to open
up a notebook. Run the notebook
manually, follow the steps in
response.
Next-gen SIEM & observability [Link]
Takeaways From .conf24:
Splunk’s Federated S3
Splunk users are moving logs
into data lakes to help control
costs.
Splunk’s Federated Search for S3 allows you to Splunk’s Federated Analytics allows you to
query your data in S3. ingest a subset of your data lake from S3 back
into Splunk.
Must use SQL instead of SPL (Splunk Processing Language).
Live demo took > 1 min for a 15GB search. Expensive to ingest a subset of data lake into Splunk. Negates
Returns 100k rows. the point of storing logs in the data lake.
Expensive with unpredictable costs. All data must be in OCSF format in Amazon Security Lake.
The data is retained for 30 days or less.
Next-gen SIEM & observability [Link]
currently available in splunkbase
Scanner for Splunk
Instead of ingesting logs into Splunk, teams can move These logs can be retained in S3 for as long as needed,
logs into object storage and index them with Scanner for with most users retaining them for 12 months or more,
search speeds of up to 10TB per second. which is plenty coverage for advanced persistent threats.
250 GB / Day 500 GB / Day 1 TB / Day 4 TB / Day
Splunk $215.3k $395k $824.8k $3.4m
logs ingested directly
Splunk + Scanner* $38.6k $79.9k $117.3k $341.6k
Logs ingested by Scanner
* Scanner's ingestion price + base Splunk Cloud price of $8.1K/year for 5GB/day
price shown per year
Next-gen SIEM & observability [Link]
currently available in splunkbase
Scanner for Splunk
scanner
An event generating command that takes a Scanner
query and uses the time range specified in Splunk
to run searches. Returns log events. Users can also
analyze data from S3 alongside data in Splunk.
Next-gen SIEM & observability [Link]
currently available in splunkbase
Scanner for Splunk
scannertable
A report generating command that takes a Scanner
query and uses the time range specified in Splunk
to run searches. It returns a table, so it is often
useful with stats queries or to generate
visualizations in dashboards.
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Snowflake
Scanner (250TB) vs. Athena (250TB)
and Snowflake (15TB)
Watch a recorded demo where we run queries on a
large log data set using Scanner, Amazon Athena,
and Snowflake to look for indicators of compromise.
Scanner is far faster, and it does not require data
Go to Demo
engineering work to onboard logs.
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Athena
Data Set CloudTrail JSON logs spanning 1 year,
250TB uncompressed.
Athena Configuration Scanner Configuration
Partitioned by 6 fields derived from S3 file paths: Timestamp extracted from
organization_id, account_id, region, year, month, `eventTime` field.
day.
Index files automatically generated
Athena using the maximum of 124 DPUs per from raw data, no partitioning or
query. special configuration necessary.
Total partitions: 16,425 Total index files: 21,304
Use Case Search for indicator of compromise.
Specifically, look up all activity for a
4 different queries using different time spans:
1 day, week, month, year.
specific AWS Access Key ID.
Outcome Scanner is between 57x and 736x faster than Athena for query time spans between 1 day
and 1 month. The 1 year time span is unusable in Athena: it crashes because of S3 request
rate errors.
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Athena 1 Day Timespan – 3m 10s saved per query
Athena Scanner
select * from demo_cloudtrail_json
%ingest.source_type: "aws:cloudtrail" and
where year = 2023
[Link]: "ASIA5GB5Y4BKT6NCJIPO"
Query and month = 7
and day = 14
Start time: 2023-07-14T00:00:00.000Z
and [Link] = 'ASIA5GB5Y4BKT6NCJIPO'; End time: 2023-07-15T00:00:00.000Z
45 partitions
166 index files
Data Set
60.39 GB scanned 13.9 GB scanned
Runtime 3m 14s 3.4s (57x faster than Athena)
*Free - Using monthly free capacity
Cost $0.29
$0.03 - After free capacity exhausted
* F or e v er y 1 GB in d e x e d by Scanner , c u s t omers recei v e 20 GB of free qu er y capaci ty .
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Athena 1 Week Timespan – 21m 1s saved per query
Athena Scanner
select * from demo_cloudtrail_json
%ingest.source_type: "aws:cloudtrail" and
where year = 2023
[Link]: "ASIA5GB5Y4BKT6NCJIPO"
Query and month = 7
and day >= 14 and day < 22
Start time: 2023-07-14T00:00:00.000Z
and [Link] = 'ASIA5GB5Y4BKT6NCJIPO'; End time: 2023-07-22T00:00:00.000Z
315 partitions
610 index files
Data Set
480 GB scanned 108 GB scanned
Runtime 21m 5s 3.6s (353x faster than Athena)
*Free - Using monthly free capacity
Cost $2.34
$0.21 - After free capacity exhausted
* F or e v er y 1 GB in d e x e d by Scanner , c u s t omers recei v e 20 GB of free qu er y capaci ty .
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Athena 1 Month Timespan – 1h 18m 26s saved per query
Athena Scanner
select * from demo_cloudtrail_json
%ingest.source_type: "aws:cloudtrail" and
where year = 2023
[Link]: "ASIA5GB5Y4BKT6NCJIPO"
Query and month = 7
and [Link] = 'ASIA5GB5Y4BKT6NCJIPO';
Start time: 2023-07-01T00:00:00.000Z
End time: 2023-08-01T00:00:00.000Z
1395 partitions
2,080 index files
Data Set
1.81 TB scanned 375 GB scanned
Runtime 1h 18m 33s 64 . s (736x faster than Athena)
*Free - Using monthly free capacity
Cost $9.05
$0.73 - After free capacity exhausted
* F or e v er y 1 GB in d e x e d by Scanner , c u s t omers recei v e 20 GB of free qu er y capaci ty .
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Athena 1 Year Timespan – ∞ faster than Athena
Athena Scanner
select * from demo_cloudtrail_json
%ingest.source_type: "aws:cloudtrail" and
where year = 2023
[Link]: "ASIA5GB5Y4BKT6NCJIPO"
Query and [Link] = 'ASIA5GB5Y4BKT6NCJIPO';
Start time: 2023-01-01T00:00:00.000Z
End time: 2024-01-01T00:00:00.000Z
16,425 partitions
21,304 index files
Data Set
*Crashed after 2.28 TB scanned 3.84 TB scanned
Runtime *Crashed after 1h 31m 1s 32.6s (∞ faster than Athena)
*Free - Using monthly free capacity
Cost $11.40
$7.68 - After free capacity exhausted
*Cras h e d du e t o S3 Errors: “ P l ease re du ce y o u r re qu es t ra t e. S t a tu s Co d e: 503 ”
* F or e v er y 1 GB in d e x e d by Scanner , c u s t omers recei v e 20 GB of free qu er y capaci ty .
Next-gen SIEM & observability [Link]
performance comparison
Scanner vs. Panther
Scanner Panther
Runs on a proprietary query engine designed for S3 and petabyte-scale log data sets Runs on top of Snowflake
Low cost Expensiv
250GB/day: $30.5 Slo
500GB/day: $71.8 High data engineering work to introduce new log sources
1TB/day: $109.2 Must transform each log source into a SQL table schema
4TB/day: $333.5 Log retention 12+ months
Low data engineering work to introduce new log sources Supports Detections-as-Code
Logs are read by Scanner in raw format directly from S3. No need to transform raw logs into Written as Python code
SQL table schemas Rules only examine individual log events
Log retention 12+ months Sync code from Github repo
Query speed up to 10TB/sec Unit testing
Supports Detections-as-Code Detection historical backtest limited to past 15 days, at most 20GB of data.
Written as simple queries in YAML
Can look at trends over time ranges
Sync code from Github repo
Unit testing
Detection historical backtest on unlimited time range, eg. 12+ months, over all your data
Supports Response-as-Code
Link detection alerts to Jupyter Notebooks. Run notebooks with Scanner Python SDK to run
analysis.
Next-gen SIEM & observability [Link]
Example
Low vs High Volume Logs
# Log Sources SIEM Ingest Volume SIEM Ingest Cost
Low Volume Logs 25-100 500 GB / day $500k / year
500 GB / day $500k / year
High Volume Logs 3-5
1 TB / day $1m / year
# Log Sources SIEM Ingest Volume SIEM Ingest Cost
Low Volume Logs in SIEM 25-100 500 GB / day $500k / year
AWS WAF, Cloudflare,
500 GB / day $100k / year
High Volume Logs in S3 + Scanner Cloudtrail, VPC Flow Logs
1 TB / day $600k / year
$400k Savings
Next-gen SIEM & observability [Link]