Proxmox Setup with Netdata, ZFS and TLS

Introduction

This guide will help you set up a Proxmox server with ZFS storage, netdata monitoring, and an nginx reverse proxy with TLS for the web interface and netdata.

Install Proxmox

https://www.proxmox.com/en/

During the installation, choose the following options:

Filesystem: xfs

Run post-install script

This script provides options for managing Proxmox VE repositories, including disabling the Enterprise Repo, adding or correcting PVE sources, enabling the No-Subscription Repo, adding the test Repo, disabling the subscription nag, updating Proxmox VE, and rebooting the system.

This script is provided by the community-scripts project: https://community-scripts.github.io/ProxmoxVE/scripts?id=post-pve-install

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/post-pve-install.sh)"

Install netdata

Netdata is an open-source monitoring tool that provides real-time performance metrics for systems, applications, and infrastructure. It offers a highly interactive web interface to visualize and analyze metrics such as CPU, memory, disk, network, and application performance.

Installation script is provided by the community-scripts project: https://community-scripts.github.io/ProxmoxVE/scripts?id=netdata

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/netdata.sh)"

Create TLS certificates

I’m using Cloudflare as my DNS provider, so I will use the ‘Cloudflare Managed DNS’ as the DNS API. You should pick whichever DNS API you are using.

Create a new ACME DNS Plugin: Go to Datacenter -> ACME -> Challange Plugins -> Add
- Plugin ID: cloudflare
- Validation Delay: 30 (default)
- DNS API: Cloudflare Managed DNS
- CF_Account_ID: <your Cloudflare account ID>
- CF_Token: <your Cloudflare API token>
Create a new ACME account: Go to Datacenter -> ACME -> Accounts -> Add
- Account Name: default
- Email: <your email>
- ACME Directory: Let’s Encrypt V2
- Accept TOS: checked
Add domain name to host Go to YOURNODE -> System -> Certificates -> ACME -> Add
- Challange Type: DNS
- Plugin: cloudflare (the plugin you created earlier)
- Domain: YOUR.FQDN.HOSTNAME.HERE
Select ACME account Go to YOURNODE -> System -> Certificates -> ACME Select the account you created earlier. (next to the ‘Order Certificates Now’ button)
Request certificate Go to YOURNODE -> System -> Certificates -> ACME Click ‘Order Certificates Now’.

You should now have a valid TLS certificate for YOUR.FQDN.HOSTNAME.HERE.

Install neovim

Neovim is my preferred text editor, but you can use any text editor you like.

apt install neovim
export EDITOR=$(which nvim) # Set neovim as the default editor for the current session

Install and configure nginx

Install nginx. This is the web server that will act as a reverse proxy for the Proxmox web interface and netdata.

apt install nginx

Edit the nginx systemd service and add the lines below. This will make sure that nginx starts after the Proxmox cluster service, which provides the TLS certificates for the Proxmox web interface.

systemctl edit nginx.service

[Unit]
Requires=pve-cluster.service
After=pve-cluster.service

Disable the default nginx configuration. This is the default configuration that listens on port 80 and 443, which we don’t want to use.

rm /etc/nginx/sites-enabled/default

Edit the new configuration, see config below.

nvim /etc/nginx/sites-available/proxmox

nginx config:

upstream proxmox {
    server 127.0.0.1:8006;
}

upstream netdata {
    server 127.0.0.1:19999;
    keepalive 1024;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name YOUR.FQDN.HOSTNAME.HERE;

    ssl_certificate /etc/pve/local/pveproxy-ssl.pem;
    ssl_certificate_key /etc/pve/local/pveproxy-ssl.key;

    proxy_redirect off;

    location = /netdata {
        return 301 /netdata/;
    }

    location ~ /netdata/(?<ndpath>.*) {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
        proxy_pass http://netdata/$ndpath$is_args$args;

        gzip on;
        gzip_proxied any;
        gzip_types *;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://proxmox;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }
}

Enable the new nginx site configuration.

ln -s /etc/nginx/sites-available/proxmox /etc/nginx/sites-enabled/proxmox

Check the nginx configuration for errors.

nginx -t

If the configuration is correct, restart the nginx service.

systemctl restart nginx

Expand the root partition

Remove proxmox data lvm-thin pool. We will create a new ZFS pool for storage.

lvremove /dev/pve/data

Extend the lvm partition to use the free space that was previously used by the data pool.

lvextend -l +100%FREE /dev/pve/root

Grow the filesystem to use the new space.

xfs_growfs /dev/pve/root

Configure ZFS ARC

Configure the minimum and maximum amount of ARC ZFS should use. The general rule of thumb is to allow 2GiB + 1GiB/TiB of storage.
So for a scenario where we have 8TiB of usable storage, we should have at least 10GiB of ARC configured. If you have spare RAM, you can increase this value to allow more data to be cached in RAM.

Default values for ZFS ARC:

zfs_arc_min: TOTAL_SYSYEM_RAM / 32
zfs_arc_max: TOTAL_SYSYEM_RAM / 2

The following sets the minimum ARC size to 10GiB - 1B and max to 10GiB. This ensures that ZFS will always use 10GiB of ARC.
We subtract one byte from the min-value because zfs_arc_min needs to be lower than zfs_arc_max.

echo "options zfs zfs_arc_min=$[10 * 1024*1024*1024 - 1]" >> /etc/modprobe.d/zfs.conf
echo "options zfs zfs_arc_max=$[10 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf

This example sets the minimum amount of used ARC to 32GiB and the maximum to 64GiB.

echo "options zfs zfs_arc_min=$[32 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf
echo "options zfs zfs_arc_max=$[64 * 1024*1024*1024]" >> /etc/modprobe.d/zfs.conf

Update your initramfs:

update-initramfs -u -k all

Create a new ZFS pool and datasets

Create a new ZFS pool. Replace the device names with the ones you have.

zpool create -o atime=off zfspool01 raidz /dev/sd{b,c,d,e} raidz /dev/sd{f,g,h,i}
# or with compression disabled, this will also disable compression for child data sets unless you manually enable it when creating datasets
zpool create -o compression=off -o atime=off zfspool01 raidz /dev/sd{b,c,d,e} raidz /dev/sd{f,g,h,i}

Create datasets for different types of data.

# Backups (vzdump)
zfs create -o mountpoint=/mnt/vzdump zfspool01/vzdump

# ISO images (compression disabled as ISOs compress really poorly)
zfs create -o compression=off -o mountpoint=/mnt/iso zfspool01/iso

# Container templates (compression disabled as templates are already zstd compressed)
zfs create -o compression=off -o mountpoint=/mnt/ctt zfspool01/ctt

# VMs and Containers if if you want file-based (qcow2) storage.
# Depending on your CPU and RAM, you might want to disable compression here '-o compression=off'.
zfs create -o mountpoint=/mnt/ct zfspool01/ct # Containers
zfs create -o mountpoint=/mnt/vm zfspool01/vm # Virtual machines

… or create a single dataset for all data

zfs create -o mountpoint=/mnt/vzdata zfspool01/vzdata

Create storage in Proxmox UI

Add each dataset as a storage in Proxmox UI. Go to Datacenter -> Storage -> Add -> Directory.

The ID-field should be unique for each storage type but can be anything you like. You may want to use ‘Container Templates’ instead of ‘ctt’ for example.

For backups, use the following settings: - ID: vzdump - Directory: /mnt/vzdump - Content: VZDump backup file

For ISO images, use the following settings: - ID: iso - Directory: /mnt/iso - Content: ISO image

For container templates, use the following settings: - ID: ctt - Directory: /mnt/ctt - Content: Container template

For containers, use the following settings: - ID: ct - Directory: /mnt/ct - Content: Container

For virtual machines, use the following settings: - ID: vm - Directory: /mnt/vm - Content: Disk image

… or

For the single dataset, use the following settings: - ID: vzdata - Directory: /mnt/vzdata - Content: Disk image, ISO image, Container template, VZDump backup file, Container, Snippets, Import

Enable Proxmox firewall

Go to YOURNODE -> Firewall -> Options -> Firewall -> Add
- Add a rule for the web interface:
  - Direction: IN
  - Enable: checked
  - Action: ACCEPT
  - Macro: blank
  - Interface: blank or your network interface of choice
  - Protocol: TCP
  - Source: blank or allowed IP range
  - Source port: blank
  - Destination: <your proxmox IP>
  - Destination port: 443
  - Comment: nginx proxy for web interface and netdata (tcp/443)
- Add a rule to deny access to the Proxmox web interface directly:
  - Direction: IN
  - Enable: unchecked (enable after you have verified that the proxy works)
  - Action: REJECT
  - Macro: blank
  - Interface: blank
  - Protocol: TCP
  - Source: blank
  - Source port: blank
  - Destination: blank
  - Destination port: 8006
  - Comment: deny default proxmox interface
Go to YOURNODE -> Firewall -> Options -> Firewall -> Enable (checked)
Go to Datacenter -> Firewall -> Options -> Firewall -> Enable (checked)

And we’re done!

You should now have access to the Proxmox web interface via https://YOUR.FQDN.HOSTNAME.HERE and netdata via https://YOUR.FQDN.HOSTNAME.HERE/netdata. (assuming you have a DNS record pointing to the IP of your Proxmox server)

When you have verified that the proxy works, you can enable the rule that denies access to the Proxmox web interface directly.