Monitoring Block
Prev  Blocks  Next

Monitoring Block

Table of Contents

Configuration
Provisioning
Errors Dashboard
Performance Dashboard
Nextcloud Dashboard
Deluge Dashboard
Requests Error Budget Alert
Options Reference

Defined in /modules/blocks/monitoring.nix.

This block sets up the monitoring stack for Self Host Blocks. It is composed of:

Configuration 

shb.monitoring = {
  enable = true;
  subdomain = "grafana";
  inherit domain;
  contactPoints = [ "me@example.com" ];
  adminPassword.result = config.sops.secrets."monitoring/admin_password".reuslt;
  secretKey.result = config.sops.secrets."monitoring/secret_key".result;
};

sops.secrets."monitoring/admin_password" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "grafana";
  group = "grafana";
  restartUnits = [ "grafana.service" ];
};
sops.secrets."monitoring/secret_key" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "grafana";
  group = "grafana";
  restartUnits = [ "grafana.service" ];
};

With that, Grafana, Prometheus, Loki and Promtail are setup! You can access Grafana at grafana.example.com with user admin and password ``.

I recommend adding a STMP server configuration so you receive alerts by email:

shb.monitoring.smtp = {
  from_address = "grafana@$example.com";
  from_name = "Grafana";
  host = "smtp.mailgun.org";
  port = 587;
  username = "postmaster@mg.example.com";
  passwordFile = config.sops.secrets."monitoring/smtp".path;
};

sops.secrets."monitoring/secret_key" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "grafana";
  group = "grafana";
  restartUnits = [ "grafana.service" ];
};

Since all logs are now stored in Loki, you can probably reduce the systemd journal retention time with:

# See https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=
services.journald.extraConfig = ''
SystemMaxUse=2G
SystemKeepFree=4G
SystemMaxFileSize=100M
MaxFileSec=day
'';

Provisioning

Self Host Blocks will create automatically the following resources:

  • For Grafana:

    • datasources

    • dashboards

    • contact points

    • notification policies

    • alerts

  • For Prometheus, the following exporters and related scrapers:

    • node

    • smartctl

    • nginx

  • For Loki, the following exporters and related scrapers:

    • systemd

Those resources are namespaced as appropriate under the Self Host Blocks namespace:

Errors Dashboard

This dashboard is meant to be the first stop to understand why a service is misbehaving.

The yellow and red dashed vertical bars correspond to the Requests Error Budget Alert firing.

Performance Dashboard

This dashboard is meant to be the first stop to understand why a service is performing poorly.

Performance Dashboard Top Part
Performance Dashboard Middle Part
Performance Dashboard Bottom Part

Nextcloud Dashboard

See Nextcloud service manual.

Deluge Dashboard

This dashboard is used to monitor a deluge instance.

Deluge Dashboard Top Part
Deluge Dashboard Bottom Part

Requests Error Budget Alert

This alert will fire when the ratio between number of requests getting a 5XX response from a service and the total requests to that service exceeds 1%.

Error Dashboard Top Part
Error Dashboard Bottom Part

Options Reference

shb.monitoring.enable

Whether to enable selfhostblocks.monitoring.

Type: boolean

Default: false

Example: true

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword

Initial admin password.

Type: submodule

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.request

Request part of the secret contract.

Options set by the requester module enforcing some properties the secret should have.

Type: submodule

Default: ""

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.request.group

Linux group owning the secret file.

Type: string

Default: "grafana"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.request.mode

Mode of the secret file.

Type: string

Default: "0400"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.request.owner

Linux user owning the secret file.

Type: string

Default: "grafana"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.request.restartUnits

Systemd units to restart after the secret is updated.

Type: list of string

Default:

[
  "grafana.service"
]

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.result

Result part of the secret contract.

Options set by the provider module that indicates where the secret can be found.

Type: submodule

Default:

{
  path = "/run/secrets/secret";
}

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.adminPassword.result.path

Path to the file containing the secret generated out of band.

This path will exist after deploying to a target host, it is not available through the nix store.

Type: path

Default: "/run/secrets/secret"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.contactPoints

List of email addresses to send alerts to

Type: list of string

Default: [ ]

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.debugLog

Set to true to enable debug logging of the infrastructure serving Grafana.

Type: boolean

Default: false

Example: true

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.domain

domain under which home-assistant will be served.

Type: string

Example: "mydomain.com"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.grafanaPort

Port where Grafana listens to HTTP requests.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default: 3000

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.lokiMajorVersion

Switching from version 2 to 3 requires manual intervention https://grafana.com/docs/loki/latest/setup/upgrade/#main–unreleased. So this let’s the user upgrade at their own pace.

Type: one of 2, 3

Default: 2

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.lokiPort

Port where Loki listens to HTTP requests.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default: 3002

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.orgId

Org ID where all self host blocks related config will be stored.

Type: signed integer

Default: 1

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.prometheusPort

Port where Prometheus listens to HTTP requests.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default: 3001

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.provisionDashboards

Provision Self Host Blocks dashboards under ‘Self Host Blocks’ folder.

Type: boolean

Default: true

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey

Secret key used for signing.

Type: submodule

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.request

Request part of the secret contract.

Options set by the requester module enforcing some properties the secret should have.

Type: submodule

Default: ""

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.request.group

Linux group owning the secret file.

Type: string

Default: "grafana"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.request.mode

Mode of the secret file.

Type: string

Default: "0400"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.request.owner

Linux user owning the secret file.

Type: string

Default: "grafana"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.request.restartUnits

Systemd units to restart after the secret is updated.

Type: list of string

Default:

[
  "grafana.service"
]

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.result

Result part of the secret contract.

Options set by the provider module that indicates where the secret can be found.

Type: submodule

Default:

{
  path = "/run/secrets/secret";
}

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.secretKey.result.path

Path to the file containing the secret generated out of band.

This path will exist after deploying to a target host, it is not available through the nix store.

Type: path

Default: "/run/secrets/secret"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp

SMTP options.

Type: null or (submodule)

Default: null

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.from_address

SMTP address from which the emails originate.

Type: string

Example: "vaultwarden@mydomain.com"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.from_name

SMTP name from which the emails originate.

Type: string

Default: "Vaultwarden"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.host

SMTP host to send the emails to.

Type: string

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.passwordFile

File containing the password to connect to the SMTP host.

Type: string

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.port

SMTP port to send the emails to.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default: 25

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.smtp.username

Username to connect to the SMTP host.

Type: string

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.ssl

Path to SSL files

Type: null or (anything)

Default: null

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.ssl.paths

Paths where the files for the certificate will be located.

This option is the contract output of the shb.certs.certs SSL block.

Type: anything

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.ssl.paths.cert

Path to the cert file.

Type: path

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.ssl.paths.key

Path to the key file.

Type: path

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.ssl.systemdService

Systemd oneshot service used to generate the certificate. Ends with the .service suffix.

Use this if downstream services must wait for the certificates to be generated before starting.

Type: string

Example: "cert-generator.service"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
shb.monitoring.subdomain

Subdomain under which home-assistant will be served.

Type: string

Example: "grafana"

Declared by:

<selfhostblocks/modules/blocks/monitoring.nix>
Prev  Up  Next
Restic Block  Home  Demos