Monitoring Block
Table of Contents
Defined in /modules/blocks/monitoring.nix.
This block sets up the monitoring stack for Self Host Blocks. It is composed of:
Grafana as the dashboard frontend.
Prometheus as the database for metrics.
Loki as the database for logs.
Configuration
shb.monitoring = {
enable = true;
subdomain = "grafana";
inherit domain;
contactPoints = [ "me@example.com" ];
adminPassword.result = config.sops.secrets."monitoring/admin_password".result;
secretKey.result = config.sops.secrets."monitoring/secret_key".result;
};
shb.sops.secret."monitoring/admin_password".request = config.shb.monitoring.adminPassword.request;
shb.sops.secret."monitoring/secret_key".request = config.shb.monitoring.secretKey.request;
With that, Grafana, Prometheus, Loki and Promtail are setup! You can access Grafana at
grafana.example.com with user admin and password ``.
I recommend adding a STMP server configuration so you receive alerts by email:
shb.monitoring.smtp = {
from_address = "grafana@$example.com";
from_name = "Grafana";
host = "smtp.mailgun.org";
port = 587;
username = "postmaster@mg.example.com";
passwordFile = config.sops.secrets."monitoring/smtp".path;
};
sops.secrets."monitoring/secret_key" = {
sopsFile = ./secrets.yaml;
mode = "0400";
owner = "grafana";
group = "grafana";
restartUnits = [ "grafana.service" ];
};
Since all logs are now stored in Loki, you can probably reduce the systemd journal retention time with:
# See https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=
services.journald.extraConfig = ''
SystemMaxUse=2G
SystemKeepFree=4G
SystemMaxFileSize=100M
MaxFileSec=day
'';
Other options are accessible through the upstream services modules. You might for example want to update the metrics retention time with:
services.prometheus.retentionTime = "60d";
Provisioning
Self Host Blocks will create automatically the following resources:
For Grafana:
datasources
dashboards
contact points
notification policies
alerts
For Prometheus, the following exporters and related scrapers:
node
smartctl
nginx
For Loki, the following exporters and related scrapers:
systemd
Those resources are namespaced as appropriate under the Self Host Blocks namespace:
Errors Dashboard
This dashboard is meant to be the first stop to understand why a service is misbehaving.
The yellow and red dashed vertical bars correspond to the Requests Error Budget Alert firing.
Performance Dashboard
This dashboard is meant to be the first stop to understand why a service is performing poorly.
Nextcloud Dashboard
See Nextcloud service manual.
Requests Error Budget Alert
This alert will fire when the ratio between number of requests getting a 5XX response from a service and the total requests to that service exceeds 1%.
Options Reference
-
shb.monitoring.enable -
Whether to enable selfhostblocks.monitoring.
Type: boolean
Default:
falseExample:
trueDeclared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword -
Initial admin password.
Type: submodule
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.request -
Request part of the secret contract.
Options set by the requester module enforcing some properties the secret should have.
Type: submodule
Default:
""Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.request.group -
Linux group owning the secret file.
Type: string
Default:
"grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.request.mode -
Mode of the secret file.
Type: string
Default:
"0400"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.request.owner -
Linux user owning the secret file.
Type: string
Default:
"grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.request.restartUnits -
Systemd units to restart after the secret is updated.
Type: list of string
Default:
[ "grafana.service" ]Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.result -
Result part of the secret contract.
Options set by the provider module that indicates where the secret can be found.
Type: submodule
Default:
{ path = "/run/secrets/secret"; }Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.adminPassword.result.path -
Path to the file containing the secret generated out of band.
This path will exist after deploying to a target host, it is not available through the nix store.
Type: absolute path
Default:
"/run/secrets/secret"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.contactPoints -
List of email addresses to send alerts to
Type: list of string
Default:
[ ]Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.debugLog -
Set to true to enable debug logging of the infrastructure serving Grafana.
Type: boolean
Default:
falseExample:
trueDeclared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.domain -
domain under which home-assistant will be served.
Type: string
Example:
"mydomain.com"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.grafanaPort -
Port where Grafana listens to HTTP requests.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
3000Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.lokiMajorVersion -
Switching from version 2 to 3 requires manual intervention https://grafana.com/docs/loki/latest/setup/upgrade/#main–unreleased. So this let’s the user upgrade at their own pace.
Type: one of 2, 3
Default:
2Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.lokiPort -
Port where Loki listens to HTTP requests.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
3002Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.orgId -
Org ID where all self host blocks related config will be stored.
Type: signed integer
Default:
1Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.prometheusPort -
Port where Prometheus listens to HTTP requests.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
3001Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.provisionDashboards -
Provision Self Host Blocks dashboards under ‘Self Host Blocks’ folder.
Type: boolean
Default:
trueDeclared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey -
Secret key used for signing.
Type: submodule
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.request -
Request part of the secret contract.
Options set by the requester module enforcing some properties the secret should have.
Type: submodule
Default:
""Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.request.group -
Linux group owning the secret file.
Type: string
Default:
"grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.request.mode -
Mode of the secret file.
Type: string
Default:
"0400"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.request.owner -
Linux user owning the secret file.
Type: string
Default:
"grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.request.restartUnits -
Systemd units to restart after the secret is updated.
Type: list of string
Default:
[ "grafana.service" ]Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.result -
Result part of the secret contract.
Options set by the provider module that indicates where the secret can be found.
Type: submodule
Default:
{ path = "/run/secrets/secret"; }Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.secretKey.result.path -
Path to the file containing the secret generated out of band.
This path will exist after deploying to a target host, it is not available through the nix store.
Type: absolute path
Default:
"/run/secrets/secret"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp -
SMTP options.
Type: null or (submodule)
Default:
nullDeclared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.from_address -
SMTP address from which the emails originate.
Type: string
Example:
"vaultwarden@mydomain.com"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.from_name -
SMTP name from which the emails originate.
Type: string
Default:
"Grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.host -
SMTP host to send the emails to.
Type: string
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.passwordFile -
File containing the password to connect to the SMTP host.
Type: string
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.port -
SMTP port to send the emails to.
Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)
Default:
25Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.smtp.username -
Username to connect to the SMTP host.
Type: string
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.ssl -
Path to SSL files
Type: null or (open submodule of anything)
Default:
nullDeclared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.ssl.paths -
Paths where the files for the certificate will be located.
This option is the contract output of the
shb.certs.certsSSL block.Type: open submodule of anything
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.ssl.paths.cert -
Path to the cert file.
Type: absolute path
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.ssl.paths.key -
Path to the key file.
Type: absolute path
Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.ssl.systemdService -
Systemd oneshot service used to generate the certificate. Ends with the
.servicesuffix.Use this if downstream services must wait for the certificates to be generated before starting.
Type: string
Example:
"cert-generator.service"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix> -
shb.monitoring.subdomain -
Subdomain under which home-assistant will be served.
Type: string
Example:
"grafana"Declared by:
<selfhostblocks/modules/blocks/monitoring.nix>