storage space used per 100th nodes squared

Post Reply
c56
Member
Posts: 307
Joined: Wed Apr 21, 2021 03:05
GitHub: tigercoding56
In-game: bm5 or bemo5 also sell_her_on55

storage space used per 100th nodes squared

by c56 » Post

what is he storage space used (generally ) per 100th nodes squared of mapsize in minetest
?
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55

User avatar
joe7575
Member
Posts: 851
Joined: Mon Apr 24, 2017 20:38
GitHub: joe7575
In-game: JoSto wuffi
Location: Germany, in the deep south

Re: storage space used per 100th nodes squared

by joe7575 » Post

4 bytes * 100 * 100 * 100 = 4 MB
Sent from my Commodore 64. Some of my Mods: Tech Age, TechPack, Hyperloop, Tower Crane, Lumberjack, vm16, Minecart, Signs Bot.

User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

Re: storage space used per 100th nodes squared

by LMD » Post

Worst case (excluding meta) is Joe's calculation.

Minetest always uses 4 bytes per node (2 bytes content ID ("nodename"), 2 bytes param1 & param2) in memory for a nodes in a mapblock (which is 16^3). There will also be some mapblock overhead. Meta takes significantly more bytes, but usually meta is very sparse (only a few nodes in the world have little metadata).

On disk, mapblocks are compressed (used to be Zlib, now it's Zstd). Both compressions use run-length encoding which allows them to greatly reduce the size of the same blocks "in a row". Mapblocks consisting out of air will only be a few bytes large; mapblocks with lots of stone will also be smaller than a mapblock consisting of randomly arranged random blocks.
My stuff: Projects - Mods - Website

User avatar
TenPlus1
Member
Posts: 3715
Joined: Mon Jul 29, 2013 13:38
In-game: TenPlus1
Contact:

Re: storage space used per 100th nodes squared

by TenPlus1 » Post

Would still be handy to have a feature to not only backup maps but shrink/clean them in some way to reduce size. Xanadu mapfile is already over 90gb in size.

User avatar
Desour
Member
Posts: 1469
Joined: Thu Jun 19, 2014 19:49
GitHub: Desour
IRC: Desour
In-game: DS
Location: I'm scared that if this is too exact, I will be unable to use my keyboard.

Re: storage space used per 100th nodes squared

by Desour » Post

Zstd dictionaries could help decreasing size, if you also store them in the database, and group multiple similar mapblocks to one dictionary.

Also, if you have deterministic mapgen, you could try to only store diffs of mapblocks that weren't loaded for a long time:
* Mapgen the chunk again.
* Create a diff of the stored mapblock compared with the generated one. I.e. a mapblock that has some special node everywhere where the nodes match.
* If the diff is smaller than the actual mapblock (after compression), remove the mapblock from the db, and add the diff to another db, (otherwise mark the mapblock as "diff is bigger" in worldstorage, to avoid trying it again). Also store in worldstorage that there's a diff for the mapblock.
* Use a custom mapgen that looks into worldstorage if there's a diff and if so, it applies the diff on the mapblock after usual mapgen.
* => If mapblocks nearly weren't touched at all, and mapgen is deterministic, you need to store almost nothing.
he/him; Codeberg; GitHub; ContentDB; public personal TODO list; "DS" is preferred (but often too short)

User avatar
Blockhead
Member
Posts: 1622
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: storage space used per 100th nodes squared

by Blockhead » Post

This is a hard question to answer because it depends, as LMD and Desour have pointed out, about what's in each 16x16x16 block:
  • The air above the ground level is tiny in storage size
  • A solid block of underground with no caves and only a few ore/mineral types is pretty small
  • The normal surface level terrain is pretty small if it's only sand, dirt, stone, air. But if it's got more types of foliage it will take more space.
  • A player base has a quite large space requirement because it will have a lot of chests. Even more so if there are mods like techpack, technic, pipeworks, mesecons in play. Not all of this data lives in the database file, but it all counts on your hard drive.
I suggest using /emergeblocks to emerge the terrain for the area you need e.g. /emergeblocks (-2000,-48,-2000) (2000,128,2000). That will take a while so be patient and do something else while it runs. That will tell you approximately how much space it would take to store 4 km x 4 km just around the surface, with your given mods. If you run several mapgens and mapgen-affecting mods it will also show you how slow each one is. Bear in mind that if you have a lot of people going mining underground and building bases it will take even more storage. A policy of deleting everything below -200 or so periodically may be quite effective for a server.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

c56
Member
Posts: 307
Joined: Wed Apr 21, 2021 03:05
GitHub: tigercoding56
In-game: bm5 or bemo5 also sell_her_on55

Re: storage space used per 100th nodes squared

by c56 » Post

Blockhead wrote:
Sat Oct 01, 2022 02:44
This is a hard question to answer because it depends, as LMD and Desour have pointed out, about what's in each 16x16x16 block:
  • The air above the ground level is tiny in storage size
  • A solid block of underground with no caves and only a few ore/mineral types is pretty small
  • The normal surface level terrain is pretty small if it's only sand, dirt, stone, air. But if it's got more types of foliage it will take more space.
  • A player base has a quite large space requirement because it will have a lot of chests. Even more so if there are mods like techpack, technic, pipeworks, mesecons in play. Not all of this data lives in the database file, but it all counts on your hard drive.
I suggest using /emergeblocks to emerge the terrain for the area you need e.g. /emergeblocks (-2000,-48,-2000) (2000,128,2000). That will take a while so be patient and do something else while it runs. That will tell you approximately how much space it would take to store 4 km x 4 km just around the surface, with your given mods. If you run several mapgens and mapgen-affecting mods it will also show you how slow each one is. Bear in mind that if you have a lot of people going mining underground and building bases it will take even more storage. A policy of deleting everything below -200 or so periodically may be quite effective for a server.
ok (for clarification i meant on average ) , but i am still interested so i will probably test it out
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55

c56
Member
Posts: 307
Joined: Wed Apr 21, 2021 03:05
GitHub: tigercoding56
In-game: bm5 or bemo5 also sell_her_on55

Re: storage space used per 100th nodes squared

by c56 » Post

i determined that a world of 10000 * 2000 * 10000 will only take around 2.5 gigabytes to store (32 megabytes for 4k * 4k * 4k cube ) (assuming normal world with about the upper 1000th nodes air ) but after that the results increase kind of exponentially so yeah the mapgen-diff store method is looking attractive
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests