Prometheus Overview
Network Monitoring and Management Resources
Network Startup Resource Center
[Link]
These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license
([Link]
Open Source / Community Driven
Available starting at [Link]
All components are available under the Apache 2 license on GitHub
Telemetry Data for Prometheus
Exporters Prometheus AlertManager
Node exporter
Remote Grafana
Storage
Prometheus Datastore and Server
Exporters Prometheus AlertManager
Node exporter
Remote Grafana
Storage
Prometheus Datastore and Server Details
Prometheus fundamentally stores all data as time series:
streams of timestamped values belonging to the same
metric and the same set of labeled dimensions.
– Every time series is uniquely identified by its metric name and optional
key-value pairs called labels:
[Link]
– Prometheus includes a local on-disk time series database, but also
optionally integrates with remote storage systems:
[Link]
– Data in Linux physically located at /var/lib/prometheus/data
Prometheus Dashboard
Exporters Prometheus AlertManager
Node exporter
*Grafana was designed to work as a UI for Remote Grafana*
analyzing metrics. As such, it can work with Storage
multiple time-series data stores, including
built-in integrations with Graphite, Prometheus,
InfluxDB, MySQL, PostgreSQL, and Elasticsearch,
and additional data sources using plugins. For
each data source, Grafana has a specific query
editor that is customized for the features and
capabilities that are included in that data source
([Link]
Sample Grafana Dashboard
Prometheus Alerting (1)
Exporters Prometheus AlertManager
Node exporter
Remote Grafana
Storage
Or…è
Prometheus Alerting (2)
Exporters Prometheus AlertManager
Node exporter
Remote Grafana
Storage
node_exporter
Promtail
Prometheus
Thank you Dean Pemberton for the next 7 slides
SYSTEM METRICS
Prometheus
& ALERTS
node_exporter
NETWORK
AND APP
METRICS
LOGS
Promtail
node_exporter COLLECT Promtail
STORE
Prometheus
VIEW
Generate alerts for
reachability and
metrics
node_exporter
Prometheus
Build Your NMM Environment (1)
Around Prometheus… What pieces do you want?
– Prometheus itself. The “server”
• Provides a data collection engine
• Time series storage engine
• Basic web interface for querying (promql)
– node_exporter configured on the collector and installed on
clients to send data to Prometheus.
– snmp_exporter configured on the server (collector). If you
have devices using snmp in your network.
Build Your NMM Environment (2)
– Grafana on the collector
• Provides a robust, pretty dashboard using Prometheus as a data
source
• Supports alerting
• Has many preconfigured dashboards you can import
– alert_manager works with (alongside) Prometheus to send
alerts via email, sms, slack, etc.
– custom_metrics in node_exporter to plug-in to your own
scripts.
Build Your NMM Environment (3)
– blackbox_exporter on the collector box. Allows for service
testing integrated in to the Prometheus ecosystem.
– PromQL – the Prometheus functional query language used in
the Prometheus and Grafana interfaces.
– Securing, Scaling and
Relabeling:
• Apply logic to scraped data before ingestion
• Allows for dropping of unwanted data (time-series, sensitive data, etc.)
• A few examples in provided exercises
[Link]
What’s Missing?
– Log information
• Consider using Grafana Loki
• Consider the ELK stack with Logstash (store) and Filebeat exporter
– Network flows
• ElastiFlow on top of ELK stack
– Projects are in flux due to licensing issues
– ElastiFlow is a very nice and comes with preconfigured queries
– Accepts Netflow, sFlow and IPFIX (IPv4, IPv6, TCP, UDP flows)
We will go in to further details on these items in this event.
Thanks!
Questions?
References
• Prometheus
[Link]
• TimescaleDB
[Link]
To docker-compose stacks
– [Link]
Other
– [Link]
– [Link]