Skip to content

monitoring

Monitoring stack gathering metrics and logs with a small resource footprint.

Uncategorized

Experimental

This service is experimental and will change in the future.

Usage

inventory.instances = {
  monitoring = {
    module = {
      name = "monitoring";
      input = "clan-core";
    };

    roles = {
      client = {
        # Enable monitoring for all machines in the clan.
        tags = [ "all" ];
        # Decide whether or not your server is reachable via https.
        settings.useSSL = true;
      };

      # Select one machine as the central monitoring server.
      # Hint: This is currently limited to exactly one server.
      server.machines.<machine>.settings = {
        # Optionally enable grafana for dashboards and alerts.
        grafana.enable = true;
      };
    };
  };
};

Architecture Overview

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace server {
        class `Visualization & Alerting` {<<Grafana>>}
        class `Log Storage` {<<Grafana Loki>>}
        class `Metrics Storage` {<<Grafana Mimir>>}
    }

    namespace client {
        class `Log & Metrics Collector` {<<Grafana Alloy>>}
        class `systemd services`
        class `system metrics`
    }

    `Visualization & Alerting` --> `Metrics Storage` : metrics
    `Visualization & Alerting` --> `Log Storage` : logs
    `Log Storage` <-- `Log & Metrics Collector` : logs
    `Metrics Storage` <-- `Log & Metrics Collector` : metrics
    `Log & Metrics Collector` --> `system metrics` : metrics
    `Log & Metrics Collector` --> `systemd services` : metrics & logs

Roles

Client

Clients are machines that create metrics and logs. Those are sent to the central monitoring server for storage and visualization.

Server

Servers store metrics and logs. They also provide optional dashboards for visualization and an alerting system.

Roles

The monitoring service has the following roles:

  • client
  • server

Options for the client role

monitoredSystemdServices

List of systemd services which are shown in the clan infrastructure grafana dashboard. Logs sent to the monitoring server are filtered using this list.

Options: "all" - all systemd services "nixos" (default) - services that have been explicitly enabled through nixos config listOf str - custom list of systemd services

Type: one of "all", "nixos" or list of string

Default:

"nixos"
Example 
[
  "alloy.service"
  "grafana.service"
  "loki.service"
  "mimir.service"
  "nginx.service"
]

Declared in: clanServices/monitoring/default.nix

useSSL

Whether to send metrics data via http or https. Enable this if your monitoring server is addressable using https.

Type: boolean

Default:

false
Example 
true

Declared in: clanServices/monitoring/default.nix

Options for the server role

grafana.enable

Whether to enable grafana.

Type: boolean

Default:

false
Example 
true

Declared in: clanServices/monitoring/default.nix

host

Hostname or address of the monitoring server (e.g. "qube.email"). The protocol (http/https) is controlled by the client's useSSL option. If null, derived automatically from the server machine name and meta.domain.

Type: null or string

Default:

null
Example 
"monitoring.example.com"

Declared in: clanServices/monitoring/default.nix