Daily Worlds Backups Script

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Post

wziard wrote:But that does mean that mod_storage, or any mod file writing, can be out of sync with the map if you backup both at separate times while minetest is running? Or do I overlook something.
yes, they can be out of sync.

You could schedule your backups when no players are online to avoid any of those issues. I myself think it's more intesting to have e.g. hourly backups in case it's needed (and have 40 of them) than to have a backup once a week that is going to make players cry...

wziard
Member
Posts: 127
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Post

I make daily backups, at a time when it's unlikely any of the players are awake (they are all school friends of my kids :-) ). But when I have to do my backups when nobody's online, I can just as easily just shutdown the server and restart it and be 100% certain the backup is consistent. I'm going to change my backupscript to just stop & start the server.

I think the opposite of you :-). Having 40 backups which might all be inconsistent is imho worse than having one backup which is guaranteed to be good.

micheal65536
Member
Posts: 167
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Post

BuckarooBanzay wrote:But i do think the database should be consistent:

The map save method: https://github.com/minetest/minetest/bl ... 1840-L1853

<!-- snip -->

As far as i understand a transaction gets started and commited in the database driver:
https://github.com/minetest/minetest/bl ... 3.cpp#L125
Thanks for checking. Yes, that does seem to indicate that Minetest is using transactions properly to ensure atomic updates of multiple blocks in SQLite. I have checked and PostgreSQL and Redis also appear to use a transaction. LevelDB is missing an implementation for the "beginSave" and "endSave" methods as is the files backend, so no transaction is used (obviously transactions don't really apply to files but I don't know if LevelDB should be using transactions).
BuckarooBanzay wrote:I can live with that inaccuracy, the world should still be usable though (no corruptions)
A sheep more or less does not matter anyway :D
Hold on a second, this is about more than just sheep. Consider a mod that stores metadata across multiple nodes. For example if a node is placed next to an existing node, the existing node might be updated to say "there is a connected node on the left-hand side". (Not referring to Minetest's connected drawtypes, this is about e.g. a tech mod that has its own concept of "connected".) And when a node is removed the remaining node might be updated to say "there is no connected node on the left-hand side". If the nodes are in different map blocks, these nodes may be in an inconsistent state. Unlikely, yes. Bad design, yes (and no, depending on the situation and exactly how it's implemented). A real PITA to troubleshoot/debug when it keeps crashing your server or breaks a large and important machine, yes.

Obviously this is mostly a hypothetical discussion at this point as we've already seen that Minetest is using database transactions but I'm putting this out there to emphasise the importance of consistent backups, that this is an issue that should always be considered when backing up (or otherwise maintaining) a server and it can be about more than a few missing sheep.
Last edited by micheal65536 on Thu Jan 03, 2019 19:06, edited 4 times in total.

micheal65536
Member
Posts: 167
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Post

wziard wrote:I think the opposite of you :-). Having 40 backups which might all be inconsistent is imho worse than having one backup which is guaranteed to be good.
Agreed. The inconvenience of shutting down the server far outweighs the potential consequences of a bad backup procedure. This is the kind of corruption that may go unnoticed for quite some time, and I wouldn't mess around with that. Transactions are intended to avert corruption in the event of a crash or other unexpected circumstances, not to be relied on to get away with lazy/bad habits.

You could even combine some of the snapshotting techniques suggested in this thread with actually shutting the server down. You could shut the server down, snapshot the database/storage volume, and bring the server up again in probably around 10 seconds, and the server can continue running while the backup takes place from the snapshot. Even if the backup takes a few minutes or hours to complete, the server doesn't need to be down for longer than the time it takes to create the snapshot. Minetest even has an option to prompt the players to reconnect when the server is shut down, for which this is a perfect use case.

But then, even if you are shutting the server down for longer, any decent player should hopefully be understanding of the importance of backups. I played on a server that was shut down for about an hour to back up every night (and update mods if required) and not once did I see anyone complaining. If anything, it tended to turn into a race to see who could be the first one back on when the server came back up.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Post

micheal65536 wrote:Agreed. The inconvenience of shutting down the server far outweighs the potential consequences of a bad backup procedure.
Maybe on most servers this would be a better priority. But not for my server.

On ITB, the worst thing that can happen is that someone builds for 4 hours and loses their box. I'd rather have a backup every hour so the max time lost for a player is ... 1 hour, than that the player has to start from scratch again since the last backup was a day or so ago.

Of course, on ITB, being a build/solve puzzle server, inventory consistency is completely irrelevant, chests have no value, players are always reset to spawn at login anyway, and so *consistency* of node timers/entities/inventories is entirely worthless. The only thing of value is the puzzle builds and the time people put in that.

Note that ITB backups are very small, 20-30 mb or so, due to the game design and restrictions. Running many backups is extremely cheap because of that.

micheal65536
Member
Posts: 167
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Post

sofar wrote:
micheal65536 wrote:Agreed. The inconvenience of shutting down the server far outweighs the potential consequences of a bad backup procedure.
Maybe on most servers this would be a better priority. But not for my server.

On ITB, the worst thing that can happen is that someone builds for 4 hours and loses their box. I'd rather have a backup every hour so the max time lost for a player is ... 1 hour, than that the player has to start from scratch again since the last backup was a day or so ago.

Of course, on ITB, being a build/solve puzzle server, inventory consistency is completely irrelevant, chests have no value, players are always reset to spawn at login anyway, and so *consistency* of node timers/entities/inventories is entirely worthless. The only thing of value is the puzzle builds and the time people put in that.

Note that ITB backups are very small, 20-30 mb or so, due to the game design and restrictions. Running many backups is extremely cheap because of that.
Ah, that makes a lot more sense. I was assuming that all the servers being discussed were "regular" servers, not unusual ones with different requirements.

dawgdoc
Member
Posts: 260
Joined: Mon Feb 27, 2017 01:10
GitHub: dawgdoc

Re: Daily Worlds Backups Script

by dawgdoc » Post

@joe7575 Depending on how long ago you had the 2Gb file size limit, there have been several possible causes of that limit.
  • ext3 file size limit, also fat32 and nvfs
  • glibc/libc limit on kernels prior to 2.4.0
  • samba was built with 32bit integers
  • Nautilus/Gnome support for large files
  • 32bit system issues
  • other LFS (Large File Support issues
Give a man a fish, feed him for a day. Give a kid a fish, it's going to die.

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Daily Worlds Backups Script

by Linuxdirk » Post

ext3 supports up to 2 Terabyte depending on other circumstances. But 16 Gigabytes is absolutely possible and likely won’t ever be an issue when backing up a normal Minetest server.

dawgdoc
Member
Posts: 260
Joined: Mon Feb 27, 2017 01:10
GitHub: dawgdoc

Re: Daily Worlds Backups Script

by dawgdoc » Post

Linuxdirk wrote:ext3 supports up to 2 Terabyte depending on other circumstances.
I agree.... NOW. But not since its origin, which is why I stated "Depending on how long ago you had the 2Gb file size limit..."
Give a man a fish, feed him for a day. Give a kid a fish, it's going to die.

User avatar
Mantar
Member
Posts: 584
Joined: Thu Oct 05, 2017 18:46
Contact:

Re: Daily Worlds Backups Script

by Mantar » Post

Even ext2 supported 16gb files.
Lead dev of Exile, git repo: https://codeberg.org/Mantar/Exile

User avatar
joe7575
Member
Posts: 851
Joined: Mon Apr 24, 2017 20:38
GitHub: joe7575
In-game: JoSto wuffi
Location: Germany, in the deep south

Re: Daily Worlds Backups Script

by joe7575 » Post

dawgdoc wrote:@joe7575 Depending on how long ago you had the 2Gb file size limit, there have been several possible causes of that limit.
  • ext3 file size limit, also fat32 and nvfs
  • glibc/libc limit on kernels prior to 2.4.0
  • samba was built with 32bit integers
  • Nautilus/Gnome support for large files
  • 32bit system issues
  • other LFS (Large File Support issues
No, this was 2 month ago. In the meantime I changed nothing on the system but the issue is gone.
I can't reproduce it. I have no idea what the cause was.
Sent from my Commodore 64. Some of my Mods: Tech Age, TechPack, Hyperloop, Tower Crane, Lumberjack, vm16, Minecart, Signs Bot.

User avatar
captpete
Member
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Re: Daily Worlds Backups Script

by captpete » Post

micheal65536 wrote:
captpete wrote:Now, has anyone any suggestions to this non-programmer on how to do what I want to do.
Ignoring the debate about compression utilities and what the best way to run a service is, the real issue with your script lies in these lines:

Code: Select all

wall Minetest is going down for maintenance shortly.  Please logout now.
sleep 3m
skill -KILL /dev/pts/*
sleep 20s
Obviously what you are trying to do here is warn everyone playing on your server that the server is going down, then you're waiting 3 minutes for everyone to log out before forcing them to log out, then you're forcing everyone who's still logged in to log out, then you're waiting another 20 seconds for their logout to complete before stopping the server.

That's not actually what this does. The commands that you're using target users on your Linux system, not the Minetest server. The people logging in to your Minetest server are not users on the computer itself. They won't see the message from the "wall" command and they won't be logged out by the "skill" command.

What you actually need to do is to use a block of code similar to the following in a Minetest mod, within the Minetest game itself:

Code: Select all

minetest.register_chatcommand("shutdown_for_maintenance", {
    params = "",
    description = "Warns players and then shuts down the server after a delay",
    privs = { server = true },
    func = function(name, param)
        -- warn players
        minetest.chat_send_all("Minetest is going down for maintenance shortly. Please log out now.")

        -- shut down server after 3 minutes
        minetest.request_shutdown("Minetest is going down for maintenance.", false, 3 * 60)
    end
})
This registers a chat command to shutdown the server with the appropriate warnings and delays (the command can only be executed by someone with admin privileges on the server). Note that there's no need to explicitly kick the players that are still logged in, they are automatically logged out when the server shuts down and shown the message passed to minetest.request_shutdown. Also I had originally written this to use minetest.after but minetest.request_shutdown includes a delay parameter. Either method is probably fine.

If you want to do this automatically (without logging in to the server to run the command), you will need to pass the command to the server from your script and then wait the three minutes for the server to shut down before continuing. I don't know how to pass the command to the server from a script but maybe someone else can suggest that part. Your script would then look something like this:

Code: Select all

# warn users, stop server
<execute /shutdown_for_maintenance command on server, however that's done>
sleep 3.5m
killall minetestserver
This waits 3 minutes and 30 seconds for the server to shut down, which covers the 3-minute delay and includes a bit of extra time in case it is slow for some reason. Then it kills the server in case it hasn't shut down yet (unlikely but possible if something goes wrong).
joe7575 wrote:gzip is no good idea. If your 'map.sqlite' is larger than 2 GB, gzip fails (as on my server)
I only use "tar -cf ...""
wziard wrote:For the record, it is the first time I hear of this 2GB limit in gzip. And I have never experienced it myself, though I'm quite sure I have gzipped larger (disk-image) files in the past. Maybe it is a filesystem limitation in joe7575's system?
Never heard of it either. Normally I use tar and then manually pipe into gzip, I've never used tar's built-in gzip option. Perhaps that imposes an extra limit. I've gzipped many files larger than 2 GB. Might also depend on if your system is 32-bit or 64-bit (just throwing this out there to experiment with).
Thanks, that gives me a lot to think about and some good insight into the process.

User avatar
captpete
Member
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Re: Daily Worlds Backups Script

by captpete » Post

sofar wrote:You can backup your world without shutting it down entirely. All you need to do is run something like this:

Code: Select all

#!/bin/sh
for db in map players sauth; do
    sqlite3 $db.sqlite ".timeout 1000" ".backup $db-backup.sqlite"
done
And then moving the backup files to a different location, of course.

Note: this assumes you're using sqlite player database, and sauth mod. If not leave 'sauth' out of the for loop.
Thanks Sofar, I'll have to checkout [sauth] and see if it works on my setup. If it does I'll put it in crontab -e.

User avatar
captpete
Member
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Re: Daily Worlds Backups Script

by captpete » Post

BuckarooBanzay wrote:
micheal65536 wrote:
joe7575 wrote:But in the past I had problems to extract archives from my server on my Desktop PC, when using gzip or bzip2.
The problem did not occur as long as the file 'map.sqlite' was smaller than 2GB.
I have to look deeper...
Was it Windows? Or using a different filesystem? Some operating systems and/or filesystems impose lower limits on file sizes (or used to in the past). Different file compression utilities might also impose limits.
Linuxdirk wrote:What seems to be lost within the discussion on how to properly shut down a server is the fact that it is not necessary to shut down a Minetest server to perform a backup.

viewtopic.php?p=339487#p339487
The server that I used to play on regularly was shut down for backing up every night, as is every other reputable server that I have come across. In theory, the database engine and/or Minetest engine should be able to handle any inconsistencies introduced through live backups, but I would not rely on that in production. I can imagine a few situations where this may introduce strange artefacts in Minetest if Minetest does not use transactions (AFAIK it doesn't) to ensure atomic update of multiple blocks that are being modified at the same in-game instant (an entity moving from one map block to another is one example, the entity may be duplicated or vanish entirely if the backup is carried out at the instant at which the entity moves from one map block to another and one of the blocks has been updated in the database and the other one hasn't).
If we are on the topic of backups, here is how i do it (https://pandorabox.io and a few private servers):
My servers don't need to restart to create a consistent backup.

SQlite3 backed world

Here i rely on LVM Storage and snapshots (https://www.thomas-krenn.com/de/wiki/LVM_Snapshots)

Code: Select all

# create a snapshot volume
/sbin/lvcreate -L5G -s -n backup /dev/magellan-vg/root

# mount the snapshot-volume for backup
mount -o ro /dev/magellan-vg/backup /mnt/snapshot/

# TODO: do your backup here (tar/zip/etc)

# unmount the volume
umount /mnt/snapshot

# and remove the snapshot
/sbin/lvremove /dev/magellan-vg/backup -f
Postgres backed world

Here i use a combination of file-backup (for the most mod-data, travelnet, etc) and Postgres WAL replication ( https://www.postgresql.org/docs/10/cont ... iving.html)
There is a main server instance and a standby server instance (of postgres)

Config files:

Main Server postgres.conf

Code: Select all

wal_level = replica
archive_mode = on
archive_command = 'cp %p /backup/%f'
Standby Server recovery.conf

Code: Select all

restore_command = 'cp /restore/%f %p'
The running/main server produces periodical WAL files with the DB-Data in it. Those
get rsynced with the standby server.
The standby server consumes those WAL files and replicates the main database this way.

I do a daily backup of the standby server and restore it sometimes on another host for mod-testing.

(If anyone is interested in online postgres-backup i can do a writeup of my setup)
I didn't take into account that some mods have there own data files. I'll have to check this out - I have a few mods in the "mods" directory for general administration and other uses. Most of my mods are customized to their worlds and are in their respective "worldmods" directories under the "worlds/" hierarchy.

wziard
Member
Posts: 127
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Post

If you just want to keep it simple, you might want to do what I do now. It's not the best system on a very busy server, but as all my players are schoolfriends of my children I can safely shutdown/restart the server at 2:00h A.M.

I just do (from a cron job, scheduled at 02:00h in the night)

Code: Select all

#!/bin/sh
killall minetestserver
tar -zcf backup-`date +"%Y%m%d"`.tgz .minetest/worlds/myworld
minetestserver
It's not elegant or fast or particularly efficient. But it gives you a guaranteed consistent snapshot of your complete world. And it's (extremely) simple.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 25 guests