What does server_unload_unused_data_timeout really do ?

Post Reply
User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

What does server_unload_unused_data_timeout really do ?

by Minix » Post

Let me explain, I run a server and I am constantly running out of memory because a lot of mapblocks are loaded in memory, so I was tweaking minetest.conf, specifically server_unload_unused_data_timeout, however I do not understand what it really does and I have not found a post that talks about this.

Expected functionality: the integer value you set in this setting should be the number of seconds it takes for the server to unload mapblocks from memory after ABM finishes working on that mapblock (all players leave the area). This way only mapblocks that are under ABM and those that have not timed out yet are still in memory so performance improves and memory is not wasted.

From experimenting that is not what that setting does. I did the following experiment:

Created a new default minetest 5.3.0 stable server and started exploring spawn and different areas recording memory usage with different timeout values:

With the timeout set to 1, memory usage started from 50 MB and only went to 60 MB after exploring an area with radius of 300 nodes.

Timeout (default) 29 started at 50 MB and maxed at 150 MB after exploring, even 30 minutes after logging out of the server, memory usage was the same 150 MB.

Then I set the timeout to 100 and explored the X axis from node 0 to 3000, memory usage went from 50 MB to a maximum of 250 MB. Even after 3 hours memory usage was the same, and it did not get any bigger so that leads me to think that value is just a memory cap for the server so that it uses as much memory as it can for mapblocks but unloading those that are not in use to not exceed that value.

Another weird thing, in my main server I noticed that when a player explores a large area I get warnings such as:

WARNING[Server]: active block modifiers took 201ms (processed 236 of 251 active blocks)

I know that is just because my cpu and i/o are crap but the weird thing is that the number of active blocks sometimes reaches 500 even though active_block_range is set to 3 and only one player is on the server. So I think that for some reason mapblocks are just kept loaded in memory and ABMs are always running on those mapblocks ?? I am confused, please explain this to me, and thanks for reading this wall of text.
Last edited by Minix on Tue Jul 13, 2021 15:58, edited 1 time in total.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: What does server_unload_unused_data_timeout really do ?

by Festus1965 » Post

one thing is the time mt keep loaded map-blocks in memory

other thing is what is the maximum memory it was allowed to use,
what might be useless after some time not visited (activated) and might be allowed to disappear from memory (from HD reload if needed)

(here with 32 GB RAM about - htop 33 GB virtual, 27 GB reserved, SHR 20 GB - real Mem 7.6 GB in use)

maybe a release will happen, if you start something else, what want mem and so mt is forced then to free it ?
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: What does server_unload_unused_data_timeout really do ?

by Minix » Post

Festus1965 wrote:
Sat Nov 14, 2020 11:48
one thing is the time mt keep loaded map-blocks in memory

other thing is what is the maximum memory it was allowed to use,
what might be useless after some time not visited (activated) and might be allowed to disappear from memory (from HD reload if needed)

(here with 32 GB RAM about - htop 33 GB virtual, 27 GB reserved, SHR 20 GB - real Mem 7.6 GB in use)

maybe a release will happen, if you start something else, what want mem and so mt is forced then to free it ?
Yes, for example if minetestserver is using 50% of the system memory, then if other programs need memory usually some of minetest's data is sent to swap memory but that fills the swap memory really quick. So far what I have done to keep physical memory and swap usage in acceptable levels is to decrease server_unload_unused_data_timeout to 800, previously it was 1800 and minetest was using around 2 GB between physical memory and swap.

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: What does server_unload_unused_data_timeout really do ?

by Minix » Post

Yesterday I was experimenting with some backup solutions and found out that server_unload_unused_data timeout correlates exactly to the interval of Minetest writing changes to the world database. However that interval also gets affected by the settings server_map_save_interval and sqlite_synchronous. If one sets sqlite_synchronous to 0 and the other two to the same value then that is the time it will take between changes in the world database file, but that is not entirely consistent, as some times Minetest writes out data randomly.

In conclusion, if using sqlite as world backend setting sqlite_synchronous = 0 and server_map_save_interval = server_unload_unused_data_timeout = x makes Minetest wait x seconds between each burst of writing data to the database, but that is not 100 % consistent and can vary.

server_unload_unused_data_timeout also affects how much memory Minetest will use, with lower values meaning less memory being used.

Mineminer
Member
Posts: 325
Joined: Mon Mar 05, 2018 04:05

Re: What does server_unload_unused_data_timeout really do ?

by Mineminer » Post

Minix wrote:
Tue Jul 13, 2021 16:09
Yesterday I was experimenting with some backup solutions and found out that server_unload_unused_data timeout correlates exactly to the interval of Minetest writing changes to the world database. However that interval also gets affected by the settings server_map_save_interval and sqlite_synchronous. If one sets sqlite_synchronous to 0 and the other two to the same value then that is the time it will take between changes in the world database file, but that is not entirely consistent, as some times Minetest writes out data randomly.

In conclusion, if using sqlite as world backend setting sqlite_synchronous = 0 and server_map_save_interval = server_unload_unused_data_timeout = x makes Minetest wait x seconds between each burst of writing data to the database, but that is not 100 % consistent and can vary.

server_unload_unused_data_timeout also affects how much memory Minetest will use, with lower values meaning less memory being used.


That what I don't get, In Minecraft there's a command called /save which can either be /save on meaning default (what's Minetest does), /save off (when you want changes NOT to go to disk) which makes backing up a breeze. Since you are telling the server "holds data in memory right now" then finally /save all which acts like a "dump" and spills what's in memory onto the disk right there and then.

Without these functions I don't think you are able to do LIVE backups securely and reliability without going into "exotic" solutions like LVM Live Snapshots and EVEN then you still required to shutdown the MT instance for the time it's take LVM to create it's Live Snapshot.

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: What does server_unload_unused_data_timeout really do ?

by Minix » Post

Mineminer wrote:
Tue Jul 13, 2021 16:17

That what I don't get, In Minecraft there's a command called /save which can either be /save on meaning default (what's Minetest does), /save off (when you want changes NOT to go to disk) which makes backing up a breeze. Since you are telling the server "holds data in memory right now" then finally /save all which acts like a "dump" and spills what's in memory onto the disk right there and then.

Without these functions I don't think you are able to do LIVE backups securely and reliability without going into "exotic" solutions like LVM Live Snapshots and EVEN then you still required to shutdown the MT instance for the time it's take LVM to create it's Live Snapshot.
I am developing a live backup solution which relies on the online backup API of sqlite, of course a live backup will never be up to date because a Minetest world is constantly being modified, but it is possible to have consistent backups that are at least 10 minutes old.

Mineminer
Member
Posts: 325
Joined: Mon Mar 05, 2018 04:05

Re: What does server_unload_unused_data_timeout really do ?

by Mineminer » Post

Minix wrote:
Tue Jul 13, 2021 16:22
Mineminer wrote:
Tue Jul 13, 2021 16:17

That what I don't get, In Minecraft there's a command called /save which can either be /save on meaning default (what's Minetest does), /save off (when you want changes NOT to go to disk) which makes backing up a breeze. Since you are telling the server "holds data in memory right now" then finally /save all which acts like a "dump" and spills what's in memory onto the disk right there and then.

Without these functions I don't think you are able to do LIVE backups securely and reliability without going into "exotic" solutions like LVM Live Snapshots and EVEN then you still required to shutdown the MT instance for the time it's take LVM to create it's Live Snapshot.
I am developing a live backup solution which relies on the online backup API of sqlite, of course a live backup will never be up to date because a Minetest world is constantly being modified, but it is possible to have consistent backups that are at least 10 minutes old.
That would be wonderful indeed since I would gladly swap uptime for backups that are offsetted by a mere 10 minutes since initiated. Since you DO need to lose what's in RAM at the time of the backup plus the time it take to completely finish the backing up processes. :)

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: What does server_unload_unused_data_timeout really do ?

by Minix » Post

I will post my solution here in the forums when I get something functional. My goal is to make two Minetest servers get synchronized over the internet so that if one of them fails, the other one takes over.

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: What does server_unload_unused_data_timeout really do ?

by ShadMOrdre » Post

Two different topics here.

The first, and simplest to address, backups. I've successfully used minetestmapper while emerging areas of the map. As long as the SQLite file / table is not locked at the time, simple Select SQL queries called by a backup program should get you up to date data backups. Other options include a chron script that simply copies the world folder into a compressed file.

As for the other topic, server data loads, I've done some experimenting in other ways. I frequently recreate the same map to test new content or code. I frequently emerge large areas of the map, sometimes 100s of square kilometers. Since I run a headless server in the background, and a client Minetest that connects to it, I usually tinker with these two settings: client_unload_unused_data_timeout and client_mapblock_limit. I set the former to 6000, so that explored areas stay loaded in RAM for large panoramic screenshots and epic scenes. I set the later to -1, so that I can emerge large areas of the map.

I've not had issues as of late regarding server side warnings, but this is mainly due to tracking to memory leaks in my custom mapgen as well as tweaking numerous ABMs. On older PCs and older versions of MT, I would get server side warnings from certain mod combinations. Neither is related to exploring and emerging the map, IMHO.

Having not set any of the server side settings, and using my content heavy GAL project, I've successfully emerged 144sqkm. Since mapgen is all server side, I'd venture to say that the default settings are fine.

While this might seem irrelevant, the mapgen code seems to generate blocks around the user based on a sphere with a radius of the users set view distance. If your view distance is set high, around 240, then the server must generate a sphere 480m in diameter around the user. This places load on the server, and induces "lag" in the client which now must wait for the server to emerge, package, and send all those blocks. By reducing view distance to the minimum 20m, and then using 'r' to view all, I can sometimes improve performance client side. By using a chat command that lets me emerge areas of the map, and showing patience while a 4sqkm area is emerged, I then am able to fly around the map faster, with terrain being drawn faster, thus improving the whole experience.


Why the long winded explanation?

It seems that there is a disconnect between server side data limits, client side data limits, and mismatched settings for both. Client side settings that affect the server, and have a potential impact on performance.

By your description of your experiments, I should not be able to emerge, and thus load, such large areas of the map. I only four clients have each fully drawn a 2kmX2km section of the map in their client, the server will actually be hosting 8sqkm, and would crash. It doesn't, even when running on the same PC as one of the clients connecting to it.

My current rigs are:
i5 8GB nVidia GT 1030 2GB Win10 Desktop
Pentium J2900 8GB onboard Intel graphics Lubuntu 18 Desktop
AMD APU 8GB Radeon 6320 Win 10 Laptop
Older laptops with 2GB and onboard 128MB graphics.

Interesting side note, I get different performace on the top two rigs, which also act as headless servers at times. Depending on which custom Lua mapgen I use, Win10 outperforms Lubuntu on noise based lua mapgens, while Lubuntu outshines Win10 everywhere else.

Shad

Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests