Minetest Live Backup Bash script

Post Reply
User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Minetest Live Backup Bash script

by Minix » Post

Description

Minetest Live Backup, mtlbak for short, is a Bash script that uses the sqlite CLI tool and rsync to copy the contents of a Minetest server directory that is currently running. It is meant to be scheduled as a cron job or systemd service that runs every 30 minutes or every hour, and so on.

This script is aimed at server admins that want to make backups that are as recent as possible without taking the server down to use them in case of some sort of server failure while losing very little changes so your players do not get angry at you. This makes it especially well suited for survival and creative servers, or any other server that saves player progress.

Requirements

-You need to have the sqlite CLI tool and rsync available on your system, the packages on Debian/Ubuntu are called sqlite3 and rsync respectively.
-Your Minetest server needs to use sqlite as backend for all three databases (map, auth and players). If you do not want to use sqlite for auth and players you can modify the script to use cp or rsync instead of sqlite, but the map database is very hard to reliably copy without some tool like the one sqlite provides, so this script will not be useful to you.
-A UPS is recommended but not really necessary (only if you run your server at home), as you will make Minetest write changes to the map database as little as possible you run into the danger of a power outage happening while a backup is running and Minetest has not written data to disk yet.

Instructions to set up

1)Edit your Minetest server configuration file and set these settings:
sqlite_synchronous = 0
server_unload_unused_data_timeout = 900
server_map_save_interval = 900.0

The first setting will make Minetest keep data in memory for longer and write to disk less often, the two other settings are measured in seconds and control the time interval of Minetest writing a burst of data to the map database (greater values increase the amount of memory Minetest uses, be careful), but keep in mind that is not 100% accurate and it might be less or more time than what was set. Also remember that the last setting is a float and not an integer. It is recommended to set both time settings to the same value, however the values in this example are only to get you started, you may have to adjust this to your own needs, raising them if you get many failures while making backups.

2)After configuring your server and placing the script somewhere, you must modify the control variables in the script to set the paths for the source and destination that suit your system, as well as other settings, if you do not want to modify the script you can comment the lines that set the variables and uncomment the next 5 lines to pass parameters to the script and run it like this:

Code: Select all

mtlbak.sh SOURCE_DIR... BACKUP_DIR... LOG_FILE... MAX_RETRIES... SLEEP_INTERVAL...
SOURCE_DIR: directory where your Minetest server runs
BACKUP_DIR: directory where you want to store your backup
LOG_FILE: file used to log the results of backups
MAX_RETRIES: number of times to attempt the backup before giving up
SLEEP_INTERVAL: time in seconds to wait before the next attempt

3)You can now perform a trial run of the script, I recommend turning off the Minetest server the first time you run this and use the time utility to measure how much time it takes for a backup to be made with your hardware configuration, this will be useful to diagnose why your backups are failing, try to set server_unload_unused_data_timeout and server_map_save_interval to a value at least 5 times larger than what it takes for your system to perform a backup.

4)When you get settings that work for you, then you can set a cron job to run this periodically and keep an eye on the log file to make sure backups are working. You do not have to worry about two backups running at the same time as the script makes sure that there is no more than one instance backing up to the same directory.

Known issues and limitations

-The Minetest settings used to make this work are not reliable and sometimes Minetest just writes data randomly making backups fail. I found this behaviour by experimenting and not reading the source code of Minetest. There probably is a limit to how many players a server can keep online while still being able to make live backups, because more players means more activity and more disk writes; on my current server the map database is 4 GiB and it takes around 4 minutes for a backup to complete, setting server_unload_unused_data_timeout to 3600 is sometimes not enough to make backups when there are 6 or more players online.
-This script stresses your drives a lot, if you host on your own hardware, I would recommend using a hard drive for the backup directory or an SSD with very high write capacity, while a normal SSD for the source directory is fine. Keep in mind you will be writing the whole database around max_retries times every time you run this script.
-The reason this script is limited to sqlite is because sqlite implements a backup API that makes it easy to make sure you can reliably copy a running database without corrupting or losing data. While I tried using LevelDB, I did not find a way to make backups as with sqlite, and RedisDB and PostgreSQL are too complicated and seem to not offer a live backup functionality neither.

SOURCE CODE: (I may make a repository in the future)
LICENSE: GPLv3
Spoiler

Code: Select all

#! /bin/bash

#mtlbak (Minetest Live Backup) script for backing up a running Minetest server directory
#Copyright (C) 2021 Minix from FreedomTest (freedomtest@protonmail.com)

    #This program is free software: you can redistribute it and/or modify
    #it under the terms of the GNU General Public License as published by
    #the Free Software Foundation, either version 3 of the License, or
    #(at your option) any later version.

    #This program is distributed in the hope that it will be useful,
    #but WITHOUT ANY WARRANTY; without even the implied warranty of
    #MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    #GNU General Public License for more details.

    #You should have received a copy of the GNU General Public License
    #along with this program.  If not, see <https://www.gnu.org/licenses/>.

#DESCRIPTION: this bash script utilizes the sqlite CLI tool and rsync to copy the world directory of a live (i.e. running) Minetest server into a backup directory
#FEATURES:
#Abort execution in case another backup is still running for the same backup directory
#Properly backup the sqlite databases (map, auth and players) using the sqlite backup tool
#Log to a defined log file
#INSTRUCTIONS: set the control variables to values appropriate for your system, then edit your minetest.conf file to set sqlite_synchronous to 0 and server_unload_unused_data_timeout and server_map_save_interval to a high value, preferably set both to the same value, remember the first one is an integer and the last one is a float variable. Determining which values to use will require experimentation and timing how much a backup usually takes with your hardware (you can test this by running this script with the Minetest server off). Usually 900 for both the last settings is enough but you may need higher values. After that you can configure a cron job to run this script periodically.
#Worst case scenario backup duration assuming constant sqlite3 .backup execution duration: t = n * tb + (n - 1) * ts
#t: worst case scenario backup duration, n: number of retries, tb: duration of sqlite3 backup, ts: sleep interval
#If tb is not constant (when something else is using the drive and it becomes busy), then t could be greater than expected, in that case you need to make sure backups execute when the drives are idle or that the load is always the same to get a consistent t
#Control variables
#DO NOT append a slash at the end of the path
#Make sure the backup dir already exists before executing this script
#Comment these 5 lines and uncomment the next 5 to pass parameters to the script instead of using hard-coded values
source_dir="/var/games/minetest-server/.minetest/worlds/world"
backup_dir="/var/world-backup"
log_file="/var/log/mt_local_backup.log"
max_retries=5
sleep_interval=60 #seconds
#source_dir=$1
#backup_dir=$2
#log_file=$3
#max_retries=$4
#sleep_interval=$5

if [ -f /tmp/$(basename $source_dir)_local_backup_in_progress ]
then
	echo "Backup for $(basename $source_dir) on $(date -R) aborted because another backup is already in progress, this may be caused by a backup that is taking longer than expected" >> $log_file
	exit 1
fi
touch /tmp/$(basename $source_dir)_local_backup_in_progress
rm $backup_dir/map.sqlite.tmp* 2> /dev/null #Cleaning in case of crashed attempts
echo -e "\n$(date -R) backup started" >> $log_file
try=0
while [ $try -lt $max_retries ]
do

	echo "Starting backup attempt #$(expr $try + 1) on $(date -R)" >> $log_file
	sqlite3 $source_dir/map.sqlite ".backup $backup_dir/map.sqlite.tmp" 2> /dev/null
	if [ $? -eq 0 ]
	then
		mv $backup_dir/{map.sqlite.tmp,map.sqlite}
		#Backup auth.sqlite and players.sqlite files properly
		until sqlite3 $source_dir/auth.sqlite ".backup $backup_dir/auth.sqlite" 2> /dev/null
		do
			continue
		done
		until sqlite3 $source_dir/players.sqlite ".backup $backup_dir/players.sqlite" 2> /dev/null
		do
			continue
		done
		rsync -ptrW --delete --exclude "*.sqlite" $source_dir/ $backup_dir/ 
		echo "Backup finished succesfully on $(date -R)" >> $log_file
		rm /tmp/$(basename $source_dir)_local_backup_in_progress 2> /dev/null
		exit 0
	else
		rm $backup_dir/map.sqlite.tmp 2> /dev/null
		try=$(expr $try + 1)
		if [ $try -eq $max_retries ]
		then
			echo "Backup failed $max_retries times, aborting on $(date -R)" >> $log_file
			rm /tmp/$(basename $source_dir)_local_backup_in_progress 2> /dev/null
			exit 1
		else
			echo "map.sqlite backup attempt #$try failed, retrying in $sleep_interval seconds" >> $log_file
			sleep $sleep_interval
		fi
	fi
done
Last edited by Minix on Sat Sep 25, 2021 21:34, edited 2 times in total.

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: Minetest Live Backup Bash script

by AmyMoriyama » Post

Unless I am oblivious, there is one major flaw with this, you can only backup from the latest backup. If something major happens but the server keeps running, you may end up with a bad backup and nothing else to work with.

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Minetest Live Backup Bash script

by Minix » Post

AmyMoriyama wrote:
Sat Aug 28, 2021 13:37
Unless I am oblivious, there is one major flaw with this, you can only backup from the latest backup. If something major happens but the server keeps running, you may end up with a bad backup and nothing else to work with.
Yes you are right, I developed this so that I can keep a recent as possible copy of the server files on a remote backup server to run the Minetest server there if something happens to the local server. If you wanted to keep safe backups to rollback in case of something undesirable happening in the Minetest server, then you could use some tool like rsnapshot to keep copies of the backups made by this script from different points in time.

The goal of this script is just making backups without turning the server off.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Minetest Live Backup Bash script

by lag01 » Post

Live backup sounds promising. Then i will be able to restart server less often.

User avatar
Miniontoby
Member
Posts: 616
Joined: Fri Mar 01, 2019 19:25
GitHub: Miniontoby
IRC: Miniontoby
In-game: Miniontoby
Location: The Netherlands

Re: Minetest Live Backup Bash script

by Miniontoby » Post

What about adding this into my mtctl???

Would be nice if I would not have to write a the whole code to backup a server....
Working on mtctl ---- Check my mod "Doorbell" -- Stay safe

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Minetest Live Backup Bash script

by Minix » Post

Miniontoby wrote:
Wed Sep 22, 2021 17:48
What about adding this into my mtctl???

Would be nice if I would not have to write a the whole code to backup a server....
You can merge this code into your project, there is absolutely no problem. You may want to comment lines 32 to 36 and uncomment lines 37 to 41 to be able to pass parameters to the script as explained in the first post.

User avatar
Miniontoby
Member
Posts: 616
Joined: Fri Mar 01, 2019 19:25
GitHub: Miniontoby
IRC: Miniontoby
In-game: Miniontoby
Location: The Netherlands

Re: Minetest Live Backup Bash script

by Miniontoby » Post

Minix wrote:
Thu Sep 23, 2021 03:27
Miniontoby wrote:
Wed Sep 22, 2021 17:48
What about adding this into my mtctl???

Would be nice if I would not have to write a the whole code to backup a server....
You can merge this code into your project, there is absolutely no problem. You may want to comment lines 32 to 36 and uncomment lines 37 to 41 to be able to pass parameters to the script as explained in the first post.
Thanks, what kind of credits and copyright do I have to include?

Btw please keep me up to date with the code, whould be nice ;)

'You may want to comment lines 32 to 36 and uncomment lines 37 to 41' -> I probarly don't need to, I have already these kinds of variables set in my main source....
Working on mtctl ---- Check my mod "Doorbell" -- Stay safe

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Minetest Live Backup Bash script

by Minix » Post

Miniontoby wrote:
Sat Sep 25, 2021 09:57
Thanks, what kind of credits and copyright do I have to include?

Btw please keep me up to date with the code, whould be nice ;)

'You may want to comment lines 32 to 36 and uncomment lines 37 to 41' -> I probarly don't need to, I have already these kinds of variables set in my main source....
The license is GPLv3, you can merge this into your project as long as you provide the source code to your project, I think credits are not necessary but I would not complain. I should upload this to a repository for easier access, but I have not changed the code since release because there have been no issues on my server, otherwise I would update this post.

User avatar
Miniontoby
Member
Posts: 616
Joined: Fri Mar 01, 2019 19:25
GitHub: Miniontoby
IRC: Miniontoby
In-game: Miniontoby
Location: The Netherlands

Re: Minetest Live Backup Bash script

by Miniontoby » Post

Minix wrote:
Sat Sep 25, 2021 21:32
The license is GPLv3, you can merge this into your project as long as you provide the source code to your project, I think credits are not necessary but I would not complain. I should upload this to a repository for easier access, but I have not changed the code since release because there have been no issues on my server, otherwise I would update this post.
I have implemented it into my test code and i edited the code a bit so it can fit in my code with multiple servers and I added the LICENSE that was at the top of your code into my LICENSE.txt

Check the forum topic in the link in the signature
Working on mtctl ---- Check my mod "Doorbell" -- Stay safe

User avatar
Andarius68
Member
Posts: 18
Joined: Mon Feb 19, 2024 22:34
GitHub: Andarius68
IRC: Andarius68
In-game: Andarius68
Location: Near Dallas, Texas, U.S.A.
Contact:

Re: Minetest Live Backup Bash script

by Andarius68 » Post

EDIT: Please disregard this Bash Script rookie mistake. I switched out the ~s for $HOMEs and it's working great now. Doh!

I'm not sure what I've done wrong here but it looks like the driectory and file name variables I have given are not being accepted by either the script or by sqlite3 maybe. I have attempted this on my system:

source_dir="~/testserver/serverfiles/worlds/world"
backup_dir="~/testserver/serverfiles/world-backup"
log_file="~/testserver/serverfiles/backup.log"
max_retries=5
sleep_interval=60


But this yields the response on the log_file line that the directory or file does not exist. Although I can copy that exact line (sans the quotation marks of course) and paste it after nano and open the backup.log file and read it. So, it looks like the path is correct and the file exists? Now, if I remove that path and just put backup.log there by itself the scrip runs but it fails repeatedly on backing up map.sqlite and gives up after the 5th try as the script says to do. I am only guessing that the reason it keeps failing must relate again to those path variables. But, how should I format them so that they work properly?

Fri, 15 Mar 2024 15:39:32 +0000 backup started
Starting backup attempt #1 on Fri, 15 Mar 2024 15:39:32 +0000
map.sqlite backup attempt #1 failed, retrying in 60 seconds
Starting backup attempt #2 on Fri, 15 Mar 2024 15:40:32 +0000
map.sqlite backup attempt #2 failed, retrying in 60 seconds
Starting backup attempt #3 on Fri, 15 Mar 2024 15:41:32 +0000
map.sqlite backup attempt #3 failed, retrying in 60 seconds
Starting backup attempt #4 on Fri, 15 Mar 2024 15:42:32 +0000
map.sqlite backup attempt #4 failed, retrying in 60 seconds
Starting backup attempt #5 on Fri, 15 Mar 2024 15:43:32 +0000
Backup failed 5 times, aborting on Fri, 15 Mar 2024 15:43:32 +0000
See my Minetest related content on my website.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest