Daily Worlds Backups Script

User avatar
captpete
Member
 
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Daily Worlds Backups Script

by captpete » Sat Dec 29, 2018 04:31

Here's a simple shell script I'm putting together for a daily backup of the "worlds" directory on my (private) server. Does anyone see a potential problem with it? The minetestserver runs off an SD so no mechanical hard drive issues. Thanks for any input...

Code: Select all
#!/bin/sh
########################################
# Minetest Server Worlds Backup "mtwbu"
# Do "chmod 755" as root and add to cron
# created by Peter Cumminsky "captpete"
# Copyright 2018  *  License GPLv3
########################################

# warn users, kill and logout users, stop server
wall Minetest is going down for maintenance shortly.  Please logout now.
sleep 3m
skill -KILL /dev/pts/*
sleep 20s
killall minetestserver

# directory to backup
dir-to-backup="/home/captpete/.minetest/worlds"

# destination of backup
dest="/home/captpete/.minetest/backups/"

# create archive filename
archive_file="mtworlds_$(date +%y%m%d).tgz"

# backup the directory using tar and gzip
tar czf $dest/$archive_file $dir_to_backup

# restart the server
nohup /usr/games/minetestserver &
 

User avatar
Linuxdirk
Member
 
Posts: 1867
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Daily Worlds Backups Script

by Linuxdirk » Sun Dec 30, 2018 20:43

captpete wrote:Does anyone see a potential problem with it?

Since you asked for it …

Code: Select all
wall Minetest is going down for maintenance shortly.  Please logout now.

This does not do what you expect it to do.

Code: Select all
skill -KILL /dev/pts/*

The fact aside that skill is officially obsolete this is not how to safely log out users.

Code: Select all
sleep 20s

No need to wait because the command is blocking so commands after skill will only execute when skill is done. Actually sleeping for 20 seconds allows users to re-login manually or automatically.

Code: Select all
killall minetestserver

This is under no circumstances the correct way to end a service.

Code: Select all
tar czf $dest/$archive_file $dir_to_backup

You might want to use pczf to preserve permissions as well.

Code: Select all
nohup /usr/games/minetestserver &

This is not how to properly start a service.
 

User avatar
joe7575
Member
 
Posts: 291
Joined: Mon Apr 24, 2017 20:38
Location: Germany, in the deep south
GitHub: joe7575
In-game: JoSto

Re: Daily Worlds Backups Script

by joe7575 » Sun Dec 30, 2018 22:19

Code: Select all
# backup the directory using tar and gzip
tar czf $dest/$archive_file $dir_to_backup


gzip is no good idea. If your 'map.sqlite' is larger than 2 GB, gzip fails (as on my server)
I only use "tar -cf ...""
Sent from my Commodore 64. My Mods: TechPack, Hyperloop, Tower Crane, Lumberjack, Iron Age, Entrance.
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Mon Dec 31, 2018 01:23

You can backup your world without shutting it down entirely. All you need to do is run something like this:

Code: Select all
#!/bin/sh
for db in map players sauth; do
    sqlite3 $db.sqlite ".timeout 1000" ".backup $db-backup.sqlite"
done


And then moving the backup files to a different location, of course.

Note: this assumes you're using sqlite player database, and sauth mod. If not leave 'sauth' out of the for loop.
 

IhrFussel
Member
 
Posts: 62
Joined: Tue Nov 22, 2016 12:54
GitHub: IhrFussel
IRC: IhrFussel
In-game: IhrFussel

Re: Daily Worlds Backups Script

by IhrFussel » Mon Dec 31, 2018 15:10

Linuxdirk wrote:
Code: Select all
killall minetestserver

This is under no circumstances the correct way to end a service.


You don't need a service to auto-restart Minetest.

It works perfectly fine with a bash script with an endless loop. That is how I do it for almost 3 years now and not once did something break.

'killall minetestserver' is 100% safe to use and if it wasn't Minetest could be considered crappy according to Freenode's #linux.
 

User avatar
Linuxdirk
Member
 
Posts: 1867
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Daily Worlds Backups Script

by Linuxdirk » Mon Dec 31, 2018 15:20

IhrFussel wrote:It works perfectly fine with a bash script with an endless loop.

It may work, yes. But it is still the wrong way to crudely run a stupid bash script.

Setting up a a service is done within minutes and gives you proper start/stop/restart/crash/reboot handling, gives you proper technical logging and is just better than slapping on a dumb userland script doing the system’s job in a worse way.
 

wziard
Member
 
Posts: 115
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Mon Dec 31, 2018 15:45

LinuxDirk wrote:> killall minetestserver
This is under no circumstances the correct way to end a service.

As minetest handles sigint just fine, this is a perfectly normal way to shutdown minetest. As long as you want to shutdown all instances of course.

You prefer services. I prefer old-fashioned bash scripts. Don't mistake your personal preferences for *the truth*(tm).

I have had more trouble with systemd not running my services for (to me) mysterious reasons in the last 2 years, than I've ever had using simple bash scripts in the last 20 years.

For the rest I largely agree with your post.
 

User avatar
Linuxdirk
Member
 
Posts: 1867
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Daily Worlds Backups Script

by Linuxdirk » Mon Dec 31, 2018 15:50

wziard wrote:I have had more trouble with systemd not running my services for (to me) mysterious reasons in the last 2 years, than I've ever had using simple bash scripts in the last 20 years.

Fortunately I never had any issues with systemd. Neither in setting up the system nor in setting up timers or services.
 

User avatar
captpete
Member
 
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Re: Daily Worlds Backups Script

by captpete » Mon Dec 31, 2018 16:00

Linuxdirk wrote:
captpete wrote:Does anyone see a potential problem with it?

Since you asked for it …

Code: Select all
wall Minetest is going down for maintenance shortly.  Please logout now.

This does not do what you expect it to do.

Code: Select all
skill -KILL /dev/pts/*

The fact aside that skill is officially obsolete this is not how to safely log out users.

Code: Select all
sleep 20s

No need to wait because the command is blocking so commands after skill will only execute when skill is done. Actually sleeping for 20 seconds allows users to re-login manually or automatically.

Code: Select all
killall minetestserver

This is under no circumstances the correct way to end a service.

Code: Select all
tar czf $dest/$archive_file $dir_to_backup

You might want to use pczf to preserve permissions as well.

Code: Select all
nohup /usr/games/minetestserver &

This is not how to properly start a service.


Not really helpful as you did not offer alternatives/corrections for most of it. Calling it "Stupid" and "Dumb" (in a further post) and by inference calling me stupid/dumb is an unwarranted ad-hominem attack. I could report you for that, but I'll refrain this time.

Now, has anyone any suggestions to this non-programmer on how to do what I want to do. Thanks to Solar on the backup script without shutting down the server though I am not clear on the singleplayer reference as this is a headless server so I don't have a singleplayer running ever. Will this work for such an instance? Also, this is on a Raspberry Pi and how do I set it up as a "service" on it. I currently just start it at bootup from a crontab. Thanks for all efforts to help.

PS. The reason I'm setting this up on a RPI is that my 3-year old Gaming computer seems to have died and being an impoverished disabled veteran, with a fixed income, after Christmas gift-buying, leaves me without much in the way of funds and I had a RPI laying around after creating a Retro-Pi for my Daughter that I used as a test machine.
 

User avatar
captpete
Member
 
Posts: 134
Joined: Fri Nov 25, 2016 03:02

Re: Daily Worlds Backups Script

by captpete » Mon Dec 31, 2018 16:08

joe7575 wrote:
Code: Select all
# backup the directory using tar and gzip
tar czf $dest/$archive_file $dir_to_backup


gzip is no good idea. If your 'map.sqlite' is larger than 2 GB, gzip fails (as on my server)
I only use "tar -cf ...""


Is Bzip a better option? I have that available even though I don't think the map.sqlite will get that big though maybe so. How about the the various LZ compression algorithms?
 

User avatar
joe7575
Member
 
Posts: 291
Joined: Mon Apr 24, 2017 20:38
Location: Germany, in the deep south
GitHub: joe7575
In-game: JoSto

Re: Daily Worlds Backups Script

by joe7575 » Mon Dec 31, 2018 17:02

captpete wrote:
joe7575 wrote:
Code: Select all
# backup the directory using tar and gzip
tar czf $dest/$archive_file $dir_to_backup


gzip is no good idea. If your 'map.sqlite' is larger than 2 GB, gzip fails (as on my server)
I only use "tar -cf ...""


Is Bzip a better option? I have that available even though I don't think the map.sqlite will get that big though maybe so. How about the the various LZ compression algorithms?


I don't know, I have tried it with bzip2, but it also fails. 2 GB is not much for a public server, this can be reached in weeks.
Sent from my Commodore 64. My Mods: TechPack, Hyperloop, Tower Crane, Lumberjack, Iron Age, Entrance.
 

wziard
Member
 
Posts: 115
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Mon Dec 31, 2018 19:38

captpete wrote:Is Bzip a better option? I have that available even though I don't think the map.sqlite will get that big though maybe so. How about the the various LZ compression algorithms?

For the record, it is the first time I hear of this 2GB limit in gzip. And I have never experienced it myself, though I'm quite sure I have gzipped larger (disk-image) files in the past. Maybe it is a filesystem limitation in joe7575's system?
Joe7575? are you sure you don't mean 'zip'. zip has a 2GB filesize limit IRRC. In any case I'm quite sure the gzip version on raspbian has no problems zipping 8GB files.
bzip2 is quite slow on a raspberry pi.
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Tue Jan 01, 2019 05:52

captpete wrote:Is Bzip a better option? I have that available even though I don't think the map.sqlite will get that big though maybe so. How about the the various LZ compression algorithms?


All compression algorithms are the best. It just matters what your idea of $BEST is:

If you like "it works everywhere", gzip/zlib is best
If you think "everywhere includes windows" zip is better
if you like better compression, in some cases bzip2 is best
if you like better compression, in a lot of cases xz is best
if you like faster compression, zstd is best

Note: there are some generalizations above. If you want to discuss compression algorithms, perhaps best to make a new thread in the "off-topic" forum.

For sqlite files, most likely xz will make the smallest files, and zstd will be fastest while beating zlib/gzip/bzip2 on size.
 

micheal65536
Member
 
Posts: 95
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Tue Jan 01, 2019 12:12

captpete wrote:Now, has anyone any suggestions to this non-programmer on how to do what I want to do.

Ignoring the debate about compression utilities and what the best way to run a service is, the real issue with your script lies in these lines:
Code: Select all
wall Minetest is going down for maintenance shortly.  Please logout now.
sleep 3m
skill -KILL /dev/pts/*
sleep 20s

Obviously what you are trying to do here is warn everyone playing on your server that the server is going down, then you're waiting 3 minutes for everyone to log out before forcing them to log out, then you're forcing everyone who's still logged in to log out, then you're waiting another 20 seconds for their logout to complete before stopping the server.

That's not actually what this does. The commands that you're using target users on your Linux system, not the Minetest server. The people logging in to your Minetest server are not users on the computer itself. They won't see the message from the "wall" command and they won't be logged out by the "skill" command.

What you actually need to do is to use a block of code similar to the following in a Minetest mod, within the Minetest game itself:
Code: Select all
minetest.register_chatcommand("shutdown_for_maintenance", {
    params = "",
    description = "Warns players and then shuts down the server after a delay",
    privs = { server = true },
    func = function(name, param)
        -- warn players
        minetest.chat_send_all("Minetest is going down for maintenance shortly. Please log out now.")

        -- shut down server after 3 minutes
        minetest.request_shutdown("Minetest is going down for maintenance.", false, 3 * 60)
    end
})

This registers a chat command to shutdown the server with the appropriate warnings and delays (the command can only be executed by someone with admin privileges on the server). Note that there's no need to explicitly kick the players that are still logged in, they are automatically logged out when the server shuts down and shown the message passed to minetest.request_shutdown. Also I had originally written this to use minetest.after but minetest.request_shutdown includes a delay parameter. Either method is probably fine.

If you want to do this automatically (without logging in to the server to run the command), you will need to pass the command to the server from your script and then wait the three minutes for the server to shut down before continuing. I don't know how to pass the command to the server from a script but maybe someone else can suggest that part. Your script would then look something like this:
Code: Select all
# warn users, stop server
<execute /shutdown_for_maintenance command on server, however that's done>
sleep 3.5m
killall minetestserver

This waits 3 minutes and 30 seconds for the server to shut down, which covers the 3-minute delay and includes a bit of extra time in case it is slow for some reason. Then it kills the server in case it hasn't shut down yet (unlikely but possible if something goes wrong).

joe7575 wrote:gzip is no good idea. If your 'map.sqlite' is larger than 2 GB, gzip fails (as on my server)
I only use "tar -cf ...""

wziard wrote:For the record, it is the first time I hear of this 2GB limit in gzip. And I have never experienced it myself, though I'm quite sure I have gzipped larger (disk-image) files in the past. Maybe it is a filesystem limitation in joe7575's system?

Never heard of it either. Normally I use tar and then manually pipe into gzip, I've never used tar's built-in gzip option. Perhaps that imposes an extra limit. I've gzipped many files larger than 2 GB. Might also depend on if your system is 32-bit or 64-bit (just throwing this out there to experiment with).
 

User avatar
joe7575
Member
 
Posts: 291
Joined: Mon Apr 24, 2017 20:38
Location: Germany, in the deep south
GitHub: joe7575
In-game: JoSto

Re: Daily Worlds Backups Script

by joe7575 » Tue Jan 01, 2019 12:53

Sorry, this is not a general problem.
I executed the following script on my server (Ubuntu Server 18.04) and on my Desktop PC (Xubuntu 18.04) without any issues on both.
Code: Select all
#!/usr/bin/env bash

echo create tar
tar -cf test.tar ./world/
echo extract tar
mkdir ~/temp/world1
tar -xf test.tar -C ~/temp/world1

echo create bzip2
tar -cjf test.tar.bz2 ./world/
echo extract bzip2
mkdir ~/temp/world2
tar -xjf test.tar.bz2 -C ~/temp/world2

echo create xz
tar -cJf test.tar.xz ./world/
echo extract xz
mkdir ~/temp/world3
tar -xJf test.tar.xz -C ~/temp/world3

echo create gzip
tar -czf test.tar.gz ./world/
echo extract gzip
mkdir ~/temp/world4
tar -xzf test.tar.gz -C ~/temp/world4.


But in the past I had problems to extract archives from my server on my Desktop PC, when using gzip or bzip2.
The problem did not occur as long as the file 'map.sqlite' was smaller than 2GB.
I have to look deeper...
Sent from my Commodore 64. My Mods: TechPack, Hyperloop, Tower Crane, Lumberjack, Iron Age, Entrance.
 

User avatar
Linuxdirk
Member
 
Posts: 1867
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Daily Worlds Backups Script

by Linuxdirk » Tue Jan 01, 2019 14:43

What seems to be lost within the discussion on how to properly shut down a server is the fact that it is not necessary to shut down a Minetest server to perform a backup.

viewtopic.php?p=339487#p339487
 

micheal65536
Member
 
Posts: 95
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Tue Jan 01, 2019 16:21

joe7575 wrote:But in the past I had problems to extract archives from my server on my Desktop PC, when using gzip or bzip2.
The problem did not occur as long as the file 'map.sqlite' was smaller than 2GB.
I have to look deeper...

Was it Windows? Or using a different filesystem? Some operating systems and/or filesystems impose lower limits on file sizes (or used to in the past). Different file compression utilities might also impose limits.

Linuxdirk wrote:What seems to be lost within the discussion on how to properly shut down a server is the fact that it is not necessary to shut down a Minetest server to perform a backup.

viewtopic.php?p=339487#p339487

The server that I used to play on regularly was shut down for backing up every night, as is every other reputable server that I have come across. In theory, the database engine and/or Minetest engine should be able to handle any inconsistencies introduced through live backups, but I would not rely on that in production. I can imagine a few situations where this may introduce strange artefacts in Minetest if Minetest does not use transactions (AFAIK it doesn't) to ensure atomic update of multiple blocks that are being modified at the same in-game instant (an entity moving from one map block to another is one example, the entity may be duplicated or vanish entirely if the backup is carried out at the instant at which the entity moves from one map block to another and one of the blocks has been updated in the database and the other one hasn't).
 

BuckarooBanzay
Member
 
Posts: 189
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai

Re: Daily Worlds Backups Script

by BuckarooBanzay » Tue Jan 01, 2019 18:14

micheal65536 wrote:
joe7575 wrote:But in the past I had problems to extract archives from my server on my Desktop PC, when using gzip or bzip2.
The problem did not occur as long as the file 'map.sqlite' was smaller than 2GB.
I have to look deeper...

Was it Windows? Or using a different filesystem? Some operating systems and/or filesystems impose lower limits on file sizes (or used to in the past). Different file compression utilities might also impose limits.

Linuxdirk wrote:What seems to be lost within the discussion on how to properly shut down a server is the fact that it is not necessary to shut down a Minetest server to perform a backup.

viewtopic.php?p=339487#p339487

The server that I used to play on regularly was shut down for backing up every night, as is every other reputable server that I have come across. In theory, the database engine and/or Minetest engine should be able to handle any inconsistencies introduced through live backups, but I would not rely on that in production. I can imagine a few situations where this may introduce strange artefacts in Minetest if Minetest does not use transactions (AFAIK it doesn't) to ensure atomic update of multiple blocks that are being modified at the same in-game instant (an entity moving from one map block to another is one example, the entity may be duplicated or vanish entirely if the backup is carried out at the instant at which the entity moves from one map block to another and one of the blocks has been updated in the database and the other one hasn't).


If we are on the topic of backups, here is how i do it (https://pandorabox.io and a few private servers):
My servers don't need to restart to create a consistent backup.

SQlite3 backed world

Here i rely on LVM Storage and snapshots (https://www.thomas-krenn.com/de/wiki/LVM_Snapshots)
Code: Select all
# create a snapshot volume
/sbin/lvcreate -L5G -s -n backup /dev/magellan-vg/root

# mount the snapshot-volume for backup
mount -o ro /dev/magellan-vg/backup /mnt/snapshot/

# TODO: do your backup here (tar/zip/etc)

# unmount the volume
umount /mnt/snapshot

# and remove the snapshot
/sbin/lvremove /dev/magellan-vg/backup -f


Postgres backed world

Here i use a combination of file-backup (for the most mod-data, travelnet, etc) and Postgres WAL replication ( https://www.postgresql.org/docs/10/cont ... iving.html)
There is a main server instance and a standby server instance (of postgres)

Config files:

Main Server postgres.conf
Code: Select all
wal_level = replica
archive_mode = on
archive_command = 'cp %p /backup/%f'


Standby Server recovery.conf
Code: Select all
restore_command = 'cp /restore/%f %p'


The running/main server produces periodical WAL files with the DB-Data in it. Those
get rsynced with the standby server.
The standby server consumes those WAL files and replicates the main database this way.

I do a daily backup of the standby server and restore it sometimes on another host for mod-testing.

(If anyone is interested in online postgres-backup i can do a writeup of my setup)
 

wziard
Member
 
Posts: 115
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Tue Jan 01, 2019 18:51

I don't completely shutdown my server, but I don't want it running while I create my backup. I'm afraid the mod metadata files will be out of sync with the map as it takes a substantial amount of time to replicate the map and I don't know if minetest's sqlite3 updates are atomic (I guess probably not).

So I just send it a sigstop (killall -STOP minetest) and a sigcontinue when done (killall -CONT minetest). This way my server is paused while I do the work. It does cause most players to timeout though, so it's not much better than just shutting down and restarting.

the LVM snapshots is an interesting approach, I'm going to look into that. But it would mean re-installing my server with LVM enabled.
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Tue Jan 01, 2019 20:28

wziard wrote:I don't completely shutdown my server, but I don't want it running while I create my backup. I'm afraid the mod metadata files will be out of sync with the map as it takes a substantial amount of time to replicate the map and I don't know if minetest's sqlite3 updates are atomic (I guess probably not).


They are. Each mapblock is stored as a single BLOB row in a table. Each mapwrite writes the entire BLOB at once, including all metadata and entities, timers, etc..

The sqlite3 backup method is 100% safe, since sqlite3 knows when it needs to wait for a row to be committed (which is atomic) before it rolls the backup. You are shutting your server down for no reason.
 

wziard
Member
 
Posts: 115
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Wed Jan 02, 2019 17:38

sofar wrote:The sqlite3 backup method is 100% safe, since sqlite3 knows when it needs to wait for a row to be committed (which is atomic) before it rolls the backup. You are shutting your server down for no reason.

But what about mod storage? Those sometimes only written at shutdown I think (which makes me think my START/STOP method is no good either). And some mods (I'm certain about hyperloop) periodically write data to the mod_storage folder. Also world.sql and players.sql are separate files which could get out of sync if for example a new player gets added while you are copying the map.

Maybe those files don't *need* to be synchronized, I haven't looked into the minetest code enough yet to know for sure. But it screams "heisenbug" to me to make a backup while the server is running, with all those different files being copied at different times...
 

micheal65536
Member
 
Posts: 95
Joined: Mon May 22, 2017 20:27

Re: Daily Worlds Backups Script

by micheal65536 » Wed Jan 02, 2019 20:49

BuckarooBanzay wrote:SQlite3 backed world

Here i rely on LVM Storage and snapshots (https://www.thomas-krenn.com/de/wiki/LVM_Snapshots)

<!-- snip -->

Postgres backed world

Here i use a combination of file-backup (for the most mod-data, travelnet, etc) and Postgres WAL replication ( https://www.postgresql.org/docs/10/cont ... iving.html)

wziard wrote:So I just send it a sigstop (killall -STOP minetest) and a sigcontinue when done (killall -CONT minetest). This way my server is paused while I do the work. It does cause most players to timeout though, so it's not much better than just shutting down and restarting.

sofar wrote:The sqlite3 backup method is 100% safe, since sqlite3 knows when it needs to wait for a row to be committed (which is atomic) before it rolls the backup. You are shutting your server down for no reason.

All of these methods are making the same assumption, which I believe to be inaccurate (or, put it this way, no one has investigated to confirm whether or not it is accurate). The assumption is that multiple map blocks that have been modified in the same game step are being committed atomically. I do not believe that Minetest takes any steps to ensure that this is the case. Whatever snapshotting mechanism or database engine you use won't handle this itself, because as far as they are concerned each map block update is a separate transaction.

Consider this situation:
  1. Sheep moves, crossing map block boundary from block A to block B, invalidating both blocks
  2. Block A is committed to database (atomically)
  3. Database is suspended/snapshotted for backup (backup will contain the newly-committed block A and the original block B)
  4. Database is resumed
  5. Block B is committed to database
If this backup was restored from, the sheep would have vanished. Similar situations could occur with mods that "move" a node from one place to another (e.g. pistons), mods that update surrounding nodes in response to a change in one mod (many tech mods), and so on.

Of course, this is also working on the assumption that Minetest commits map blocks every time they are changed. I do not believe this is the case, otherwise multiple database rows would be committed on every single game step whenever a moving entity is present and this would create a significant performance impact (yes, I'm referring to committing, not syncing to disk).

I'd be interested if someone with more experience with the Minetest internals could offer more insight into either of these two issues. I'm not familiar enough with the Minetest codebase to investigate further myself but assuming that "the database engine handles atomic commits and snapshots, therefore a live backup is safe" seems foolish to me.
 

BuckarooBanzay
Member
 
Posts: 189
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai

Re: Daily Worlds Backups Script

by BuckarooBanzay » Wed Jan 02, 2019 21:13

micheal65536 wrote:
Consider this situation:
  1. Sheep moves, crossing map block boundary from block A to block B, invalidating both blocks
  2. Block A is committed to database (atomically)
  3. Database is suspended/snapshotted for backup (backup will contain the newly-committed block A and the original block B)
  4. Database is resumed
  5. Block B is committed to database
If this backup was restored from, the sheep would have vanished. Similar situations could occur with mods that "move" a node from one place to another (e.g. pistons), mods that update surrounding nodes in response to a change in one mod (many tech mods), and so on.


I can live with that inaccuracy, the world should still be usable though (no corruptions)
A sheep more or less does not matter anyway :D

But i do think the database should be consistent:

The map save method: https://github.com/minetest/minetest/bl ... 1840-L1853

Code: Select all
...
            if(!save_started) {
               beginSave();
               save_started = true;
            }

            modprofiler.add(block->getModifiedReasonString(), 1);

            saveBlock(block);
            block_count++;
         }
      }
   }

   if(save_started)
endSave();
...


As far as i understand a transaction gets started and commited in the database driver:
https://github.com/minetest/minetest/bl ... 3.cpp#L125
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Daily Worlds Backups Script

by sofar » Wed Jan 02, 2019 21:31

wziard wrote:But what about mod storage? Those sometimes only written at shutdown I think


mod_storage is committed periodically, after changes are needed to be committed. The changes are kept for a small amount of time in memory and then saved, within a few seconds.
 

wziard
Member
 
Posts: 115
Joined: Mon Oct 29, 2018 19:12

Re: Daily Worlds Backups Script

by wziard » Wed Jan 02, 2019 22:14

But that does mean that mod_storage, or any mod file writing, can be out of sync with the map if you backup both at separate times while minetest is running? Or do I overlook something.
 

Next

Return to Servers



Who is online

Users browsing this forum: No registered users and 4 guests