Every internal metric on a service can be green while a user, outside, simply cannot reach the site at all. A misconfigured load balancer, an expired TLS certificate, an overly strict firewall rule: none of that shows up in application metrics, because these problems sit exactly between the user and the point where those metrics get generated. Blackbox monitoring exists to close that exact blind spot: probing the system from the outside, the way a real user would, without knowing anything about how it works internally.

What internal metrics cannot see

An application exporter measures what happens once a request has already arrived at the process: processing latency, application error rate, resource usage. It can say nothing about what kept the request from arriving in the first place. A DNS record pointing at a stale IP, a reverse proxy that responds but serves the wrong page, a certificate that expired at midnight: every one of these makes a service unreachable without any application metric ever finding out, because from its point of view, no request arrived at all.

blackbox_exporter: a prober, not an application metrics collector

blackbox_exporter is the official Prometheus ecosystem component built for exactly this role: it collects nothing inside the application, it runs HTTP, TCP, DNS, or ICMP probes from an external vantage point and exposes the result as standard Prometheus metrics.

# blackbox.yml — HTTP probe module with TLS verification
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      fail_if_ssl: false
      fail_if_not_ssl: true
# prometheus.yml — target probed through the module above
- job_name: 'blackbox-http'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://jm-dev.it
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

The most valuable result is not just probe_success (0 or 1), but probe_ssl_earliest_cert_expiry, which gives the TLS certificate’s expiry date as seen from the outside, in Unix seconds. That specific metric is what lets you alert days in advance instead of discovering the expiry at the exact moment users start seeing security warnings in their browser.

The network false-positive trap

The external probe introduces its own point of failure: if the node hosting blackbox_exporter loses network connectivity, or a firewall specifically blocks its outbound IP, every probe fails at once, which looks exactly like a full-blown outage across every monitored service. Telling a real outage apart from a problem with the prober itself needs either several probe points in distinct network zones, or a health metric for the prober itself (its own ability to reach a known, stable control point) checked before trusting the rest of its results.

An alerting rule that actually separates outage from expiry

TLS certificate expiry and HTTP availability deserve separate alerts with different urgency levels: a certificate expiring in seven days is not a 3am page, an HTTP outage is.

groups:
  - name: blackbox
    rules:
      - alert: ProbeFailing
        expr: probe_success == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} unreachable from the outside"

      - alert: CertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        labels:
          severity: ticket
        annotations:
          summary: "Certificate for {{ $labels.instance }} expires in under 14 days"

Where this fits

Blackbox monitoring does not replace application golden signals or a well-structured dashboard: it adds the one perspective neither can have, the view from outside the system. It is the layer closest to what a user actually experiences, and often the first one that would have caught an incident before anyone internal noticed. Setting up this external monitoring layer is part of what a reliability and observability engagement covers.

What to remember

Internal metrics never see what keeps a request from arriving; blackbox monitoring probes from the outside the way a real user would. blackbox_exporter exposes probe_success for availability and probe_ssl_earliest_cert_expiry to catch a certificate expiry ahead of time, but the prober itself is a failure point worth watching separately to avoid false-positive outage alerts.