what is he storage space used (generally ) per 100th nodes squared of mapsize in minetest
?
storage space used per 100th nodes squared
-
- Member
- Posts: 307
- Joined: Wed Apr 21, 2021 03:05
- GitHub: tigercoding56
- In-game: bm5 or bemo5 also sell_her_on55
storage space used per 100th nodes squared
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55
- joe7575
- Member
- Posts: 851
- Joined: Mon Apr 24, 2017 20:38
- GitHub: joe7575
- In-game: JoSto wuffi
- Location: Germany, in the deep south
Re: storage space used per 100th nodes squared
4 bytes * 100 * 100 * 100 = 4 MB
Sent from my Commodore 64. Some of my Mods: Tech Age, TechPack, Hyperloop, Tower Crane, Lumberjack, vm16, Minecart, Signs Bot.
- LMD
- Member
- Posts: 1400
- Joined: Sat Apr 08, 2017 08:16
- GitHub: appgurueu
- IRC: appguru[eu]
- In-game: LMD
- Location: Germany
- Contact:
Re: storage space used per 100th nodes squared
Worst case (excluding meta) is Joe's calculation.
Minetest always uses 4 bytes per node (2 bytes content ID ("nodename"), 2 bytes param1 & param2) in memory for a nodes in a mapblock (which is 16^3). There will also be some mapblock overhead. Meta takes significantly more bytes, but usually meta is very sparse (only a few nodes in the world have little metadata).
On disk, mapblocks are compressed (used to be Zlib, now it's Zstd). Both compressions use run-length encoding which allows them to greatly reduce the size of the same blocks "in a row". Mapblocks consisting out of air will only be a few bytes large; mapblocks with lots of stone will also be smaller than a mapblock consisting of randomly arranged random blocks.
Minetest always uses 4 bytes per node (2 bytes content ID ("nodename"), 2 bytes param1 & param2) in memory for a nodes in a mapblock (which is 16^3). There will also be some mapblock overhead. Meta takes significantly more bytes, but usually meta is very sparse (only a few nodes in the world have little metadata).
On disk, mapblocks are compressed (used to be Zlib, now it's Zstd). Both compressions use run-length encoding which allows them to greatly reduce the size of the same blocks "in a row". Mapblocks consisting out of air will only be a few bytes large; mapblocks with lots of stone will also be smaller than a mapblock consisting of randomly arranged random blocks.
Re: storage space used per 100th nodes squared
Would still be handy to have a feature to not only backup maps but shrink/clean them in some way to reduce size. Xanadu mapfile is already over 90gb in size.
- Desour
- Member
- Posts: 1473
- Joined: Thu Jun 19, 2014 19:49
- GitHub: Desour
- IRC: Desour
- In-game: DS
- Location: I'm scared that if this is too exact, I will be unable to use my keyboard.
Re: storage space used per 100th nodes squared
Zstd dictionaries could help decreasing size, if you also store them in the database, and group multiple similar mapblocks to one dictionary.
Also, if you have deterministic mapgen, you could try to only store diffs of mapblocks that weren't loaded for a long time:
* Mapgen the chunk again.
* Create a diff of the stored mapblock compared with the generated one. I.e. a mapblock that has some special node everywhere where the nodes match.
* If the diff is smaller than the actual mapblock (after compression), remove the mapblock from the db, and add the diff to another db, (otherwise mark the mapblock as "diff is bigger" in worldstorage, to avoid trying it again). Also store in worldstorage that there's a diff for the mapblock.
* Use a custom mapgen that looks into worldstorage if there's a diff and if so, it applies the diff on the mapblock after usual mapgen.
* => If mapblocks nearly weren't touched at all, and mapgen is deterministic, you need to store almost nothing.
Also, if you have deterministic mapgen, you could try to only store diffs of mapblocks that weren't loaded for a long time:
* Mapgen the chunk again.
* Create a diff of the stored mapblock compared with the generated one. I.e. a mapblock that has some special node everywhere where the nodes match.
* If the diff is smaller than the actual mapblock (after compression), remove the mapblock from the db, and add the diff to another db, (otherwise mark the mapblock as "diff is bigger" in worldstorage, to avoid trying it again). Also store in worldstorage that there's a diff for the mapblock.
* Use a custom mapgen that looks into worldstorage if there's a diff and if so, it applies the diff on the mapblock after usual mapgen.
* => If mapblocks nearly weren't touched at all, and mapgen is deterministic, you need to store almost nothing.
he/him; Codeberg; GitHub; ContentDB; public personal TODO list; "DS" is preferred (but often too short)
- Blockhead
- Member
- Posts: 1698
- Joined: Wed Jul 17, 2019 10:14
- GitHub: Montandalar
- IRC: Blockhead256
- In-game: Blockhead Blockhead256
- Location: Land Down Under
- Contact:
Re: storage space used per 100th nodes squared
This is a hard question to answer because it depends, as LMD and Desour have pointed out, about what's in each 16x16x16 block:
- The air above the ground level is tiny in storage size
- A solid block of underground with no caves and only a few ore/mineral types is pretty small
- The normal surface level terrain is pretty small if it's only sand, dirt, stone, air. But if it's got more types of foliage it will take more space.
- A player base has a quite large space requirement because it will have a lot of chests. Even more so if there are mods like techpack, technic, pipeworks, mesecons in play. Not all of this data lives in the database file, but it all counts on your hard drive.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂
-
- Member
- Posts: 307
- Joined: Wed Apr 21, 2021 03:05
- GitHub: tigercoding56
- In-game: bm5 or bemo5 also sell_her_on55
Re: storage space used per 100th nodes squared
ok (for clarification i meant on average ) , but i am still interested so i will probably test it outBlockhead wrote: ↑Sat Oct 01, 2022 02:44This is a hard question to answer because it depends, as LMD and Desour have pointed out, about what's in each 16x16x16 block:I suggest using /emergeblocks to emerge the terrain for the area you need e.g. /emergeblocks (-2000,-48,-2000) (2000,128,2000). That will take a while so be patient and do something else while it runs. That will tell you approximately how much space it would take to store 4 km x 4 km just around the surface, with your given mods. If you run several mapgens and mapgen-affecting mods it will also show you how slow each one is. Bear in mind that if you have a lot of people going mining underground and building bases it will take even more storage. A policy of deleting everything below -200 or so periodically may be quite effective for a server.
- The air above the ground level is tiny in storage size
- A solid block of underground with no caves and only a few ore/mineral types is pretty small
- The normal surface level terrain is pretty small if it's only sand, dirt, stone, air. But if it's got more types of foliage it will take more space.
- A player base has a quite large space requirement because it will have a lot of chests. Even more so if there are mods like techpack, technic, pipeworks, mesecons in play. Not all of this data lives in the database file, but it all counts on your hard drive.
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55
-
- Member
- Posts: 307
- Joined: Wed Apr 21, 2021 03:05
- GitHub: tigercoding56
- In-game: bm5 or bemo5 also sell_her_on55
Re: storage space used per 100th nodes squared
i determined that a world of 10000 * 2000 * 10000 will only take around 2.5 gigabytes to store (32 megabytes for 4k * 4k * 4k cube ) (assuming normal world with about the upper 1000th nodes air ) but after that the results increase kind of exponentially so yeah the mapgen-diff store method is looking attractive
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55
Who is online
Users browsing this forum: No registered users and 6 guests