Minetest Live Backup, mtlbak for short, is a Bash script that uses the sqlite CLI tool and rsync to copy the contents of a Minetest server directory that is currently running. It is meant to be scheduled as a cron job or systemd service that runs every 30 minutes or every hour, and so on.
This script is aimed at server admins that want to make backups that are as recent as possible without taking the server down to use them in case of some sort of server failure while losing very little changes so your players do not get angry at you. This makes it especially well suited for survival and creative servers, or any other server that saves player progress.
Requirements
-You need to have the sqlite CLI tool and rsync available on your system, the packages on Debian/Ubuntu are called sqlite3 and rsync respectively.
-Your Minetest server needs to use sqlite as backend for all three databases (map, auth and players). If you do not want to use sqlite for auth and players you can modify the script to use cp or rsync instead of sqlite, but the map database is very hard to reliably copy without some tool like the one sqlite provides, so this script will not be useful to you.
-A UPS is recommended but not really necessary (only if you run your server at home), as you will make Minetest write changes to the map database as little as possible you run into the danger of a power outage happening while a backup is running and Minetest has not written data to disk yet.
Instructions to set up
1)Edit your Minetest server configuration file and set these settings:
sqlite_synchronous = 0
server_unload_unused_data_timeout = 900
server_map_save_interval = 900.0
The first setting will make Minetest keep data in memory for longer and write to disk less often, the two other settings are measured in seconds and control the time interval of Minetest writing a burst of data to the map database (greater values increase the amount of memory Minetest uses, be careful), but keep in mind that is not 100% accurate and it might be less or more time than what was set. Also remember that the last setting is a float and not an integer. It is recommended to set both time settings to the same value, however the values in this example are only to get you started, you may have to adjust this to your own needs, raising them if you get many failures while making backups.
2)After configuring your server and placing the script somewhere, you must modify the control variables in the script to set the paths for the source and destination that suit your system, as well as other settings, if you do not want to modify the script you can comment the lines that set the variables and uncomment the next 5 lines to pass parameters to the script and run it like this:
Code: Select all
mtlbak.sh SOURCE_DIR... BACKUP_DIR... LOG_FILE... MAX_RETRIES... SLEEP_INTERVAL...
BACKUP_DIR: directory where you want to store your backup
LOG_FILE: file used to log the results of backups
MAX_RETRIES: number of times to attempt the backup before giving up
SLEEP_INTERVAL: time in seconds to wait before the next attempt
3)You can now perform a trial run of the script, I recommend turning off the Minetest server the first time you run this and use the time utility to measure how much time it takes for a backup to be made with your hardware configuration, this will be useful to diagnose why your backups are failing, try to set server_unload_unused_data_timeout and server_map_save_interval to a value at least 5 times larger than what it takes for your system to perform a backup.
4)When you get settings that work for you, then you can set a cron job to run this periodically and keep an eye on the log file to make sure backups are working. You do not have to worry about two backups running at the same time as the script makes sure that there is no more than one instance backing up to the same directory.
Known issues and limitations
-The Minetest settings used to make this work are not reliable and sometimes Minetest just writes data randomly making backups fail. I found this behaviour by experimenting and not reading the source code of Minetest. There probably is a limit to how many players a server can keep online while still being able to make live backups, because more players means more activity and more disk writes; on my current server the map database is 4 GiB and it takes around 4 minutes for a backup to complete, setting server_unload_unused_data_timeout to 3600 is sometimes not enough to make backups when there are 6 or more players online.
-This script stresses your drives a lot, if you host on your own hardware, I would recommend using a hard drive for the backup directory or an SSD with very high write capacity, while a normal SSD for the source directory is fine. Keep in mind you will be writing the whole database around max_retries times every time you run this script.
-The reason this script is limited to sqlite is because sqlite implements a backup API that makes it easy to make sure you can reliably copy a running database without corrupting or losing data. While I tried using LevelDB, I did not find a way to make backups as with sqlite, and RedisDB and PostgreSQL are too complicated and seem to not offer a live backup functionality neither.
SOURCE CODE: (I may make a repository in the future)
LICENSE: GPLv3
Spoiler
Code: Select all
#! /bin/bash
#mtlbak (Minetest Live Backup) script for backing up a running Minetest server directory
#Copyright (C) 2021 Minix from FreedomTest (freedomtest@protonmail.com)
#This program is free software: you can redistribute it and/or modify
#it under the terms of the GNU General Public License as published by
#the Free Software Foundation, either version 3 of the License, or
#(at your option) any later version.
#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
#GNU General Public License for more details.
#You should have received a copy of the GNU General Public License
#along with this program. If not, see <https://www.gnu.org/licenses/>.
#DESCRIPTION: this bash script utilizes the sqlite CLI tool and rsync to copy the world directory of a live (i.e. running) Minetest server into a backup directory
#FEATURES:
#Abort execution in case another backup is still running for the same backup directory
#Properly backup the sqlite databases (map, auth and players) using the sqlite backup tool
#Log to a defined log file
#INSTRUCTIONS: set the control variables to values appropriate for your system, then edit your minetest.conf file to set sqlite_synchronous to 0 and server_unload_unused_data_timeout and server_map_save_interval to a high value, preferably set both to the same value, remember the first one is an integer and the last one is a float variable. Determining which values to use will require experimentation and timing how much a backup usually takes with your hardware (you can test this by running this script with the Minetest server off). Usually 900 for both the last settings is enough but you may need higher values. After that you can configure a cron job to run this script periodically.
#Worst case scenario backup duration assuming constant sqlite3 .backup execution duration: t = n * tb + (n - 1) * ts
#t: worst case scenario backup duration, n: number of retries, tb: duration of sqlite3 backup, ts: sleep interval
#If tb is not constant (when something else is using the drive and it becomes busy), then t could be greater than expected, in that case you need to make sure backups execute when the drives are idle or that the load is always the same to get a consistent t
#Control variables
#DO NOT append a slash at the end of the path
#Make sure the backup dir already exists before executing this script
#Comment these 5 lines and uncomment the next 5 to pass parameters to the script instead of using hard-coded values
source_dir="/var/games/minetest-server/.minetest/worlds/world"
backup_dir="/var/world-backup"
log_file="/var/log/mt_local_backup.log"
max_retries=5
sleep_interval=60 #seconds
#source_dir=$1
#backup_dir=$2
#log_file=$3
#max_retries=$4
#sleep_interval=$5
if [ -f /tmp/$(basename $source_dir)_local_backup_in_progress ]
then
echo "Backup for $(basename $source_dir) on $(date -R) aborted because another backup is already in progress, this may be caused by a backup that is taking longer than expected" >> $log_file
exit 1
fi
touch /tmp/$(basename $source_dir)_local_backup_in_progress
rm $backup_dir/map.sqlite.tmp* 2> /dev/null #Cleaning in case of crashed attempts
echo -e "\n$(date -R) backup started" >> $log_file
try=0
while [ $try -lt $max_retries ]
do
echo "Starting backup attempt #$(expr $try + 1) on $(date -R)" >> $log_file
sqlite3 $source_dir/map.sqlite ".backup $backup_dir/map.sqlite.tmp" 2> /dev/null
if [ $? -eq 0 ]
then
mv $backup_dir/{map.sqlite.tmp,map.sqlite}
#Backup auth.sqlite and players.sqlite files properly
until sqlite3 $source_dir/auth.sqlite ".backup $backup_dir/auth.sqlite" 2> /dev/null
do
continue
done
until sqlite3 $source_dir/players.sqlite ".backup $backup_dir/players.sqlite" 2> /dev/null
do
continue
done
rsync -ptrW --delete --exclude "*.sqlite" $source_dir/ $backup_dir/
echo "Backup finished succesfully on $(date -R)" >> $log_file
rm /tmp/$(basename $source_dir)_local_backup_in_progress 2> /dev/null
exit 0
else
rm $backup_dir/map.sqlite.tmp 2> /dev/null
try=$(expr $try + 1)
if [ $try -eq $max_retries ]
then
echo "Backup failed $max_retries times, aborting on $(date -R)" >> $log_file
rm /tmp/$(basename $source_dir)_local_backup_in_progress 2> /dev/null
exit 1
else
echo "map.sqlite backup attempt #$try failed, retrying in $sleep_interval seconds" >> $log_file
sleep $sleep_interval
fi
fi
done