pmxcfs (Proxmox Cluster File System) restore a deleted node

avatar

The /etc/pve directory in Proxmox is a cluster-wide configuration filesystem that stores all VM/container configurations and cluster settings. It is SQLite-backed and the database is actually stored in /var/lib/pve-cluster/config.db and mounted as FUSE filesystem.

You can write to /etc/pve but not create directories, so if you delete a node from /etc/pve/nodes/ you cannot recreate it by simply making a directory. The only way to restore it is to restore the entire database from backup.

A discord user had etckeeper backup of the files but not of the underlying database, I thought it was an interesting challenge to try and recreate the files in the sqlite database from the files. Below is a script that does just that.

$ tree
.
├── config.db
├── mynode
│   ├── host.fw
│   ├── lrm_status
│   ├── lxc
│   │   ├── 102.conf
│   │   ├── 103.conf
│   │   └── 104.conf
│   ├── pve-ssl.key
│   ├── pve-ssl.pem
│   ├── qemu-server
│   │   ├── 100.conf
│   │   ├── 101.conf
│   │   ├── 105.conf
│   │   └── 106.conf
│   └── ssh_known_hosts
└── restore.sh
./restore.sh mynode
Restoring from: mynode
Node name: mynode
Database: config.db

Creating nodes/mynode directory structure...
Creating directory: mynode (inode=229338, parent=229322)

Processing files and directories...
Inserting file: host.fw (inode=229339, parent=229338)
Inserting file: lrm_status (inode=229340, parent=229338)
Inserting file: pve-ssl.key (inode=229341, parent=229338)
Inserting file: pve-ssl.pem (inode=229342, parent=229338)
Inserting file: ssh_known_hosts (inode=229343, parent=229338)
Creating directory: lxc (inode=229344, parent=229338)
Inserting file: 102.conf (inode=229345, parent=229344)
Inserting file: 103.conf (inode=229346, parent=229344)
Inserting file: 104.conf (inode=229347, parent=229344)
Creating directory: qemu-server (inode=229348, parent=229338)
Inserting file: 100.conf (inode=229349, parent=229348)
Inserting file: 101.conf (inode=229350, parent=229348)
Inserting file: 105.conf (inode=229351, parent=229348)
Inserting file: 106.conf (inode=229352, parent=229348)

Restore complete!

Database verification:
229322|0|4|DIR|nodes
229323|229322|4|DIR|proxmoxhost
229324|229323|4|DIR|lxc
229325|229323|4|DIR|qemu-server
229326|229323|8|FILE|host.fw
229327|229323|8|FILE|lrm_status
229328|229323|8|FILE|pve-ssl.key
229329|229323|8|FILE|pve-ssl.pem
229330|229323|8|FILE|ssh_known_hosts
229331|229324|8|FILE|102.conf
229332|229324|8|FILE|103.conf
229333|229324|8|FILE|104.conf
229334|229325|8|FILE|100.conf
229335|229325|8|FILE|101.conf
229336|229325|8|FILE|105.conf
229337|229325|8|FILE|106.conf
229338|229322|4|DIR|mynode
229339|229338|8|FILE|host.fw
229340|229338|8|FILE|lrm_status
229341|229338|8|FILE|pve-ssl.key
229342|229338|8|FILE|pve-ssl.pem
229343|229338|8|FILE|ssh_known_hosts
229344|229338|4|DIR|lxc
229345|229344|8|FILE|102.conf
229346|229344|8|FILE|103.conf
229347|229344|8|FILE|104.conf
229348|229338|4|DIR|qemu-server
229349|229348|8|FILE|100.conf
229350|229348|8|FILE|101.conf
229351|229348|8|FILE|105.conf
229352|229348|8|FILE|106.conf

Stop PVE services, restore config.db to /var/lib/pve-cluster/config.db and start services again or reboot.

You can find the restore script here: restore-etc-pve-node.sh