Ores are waste of HDD

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 06:54

I noticed one thing on my server. While there was a bug which caused the ores not to generate, the world size was increasing way slower. The reason is that the lack ores decreases entropy, so the map can be compressed more effectively.

I also noticed that like 99% ores will be never mined out and won't be even spotted by anyone. The idea is: Do not generate any ores until they can be seen. Has anyone been experimenting with such mechanism? Actually, it will be also good against hackers so they can't know where the ores are until they make them visible. The disadvantages are that it slows down the server and prospectors from technic mod will no longer work.
If you lack the reality, go on a trip or find a job.
 

Xudo
Member
 
Posts: 138
Joined: Wed Nov 09, 2016 16:43
GitHub: akryukov92
In-game: Xudo

Re: Ores are waste of HDD

by Xudo » Wed Aug 14, 2019 10:07

Can you provide more accurate measurements? What was your world settings?
How many players have been online simultaneously?

It might be a good catch. Though, I'm not sure that it will be easy to change. Map generation is performed only once. Splitting it to two phases will greatly increase complexity of code.
May be another compression algorithm might help.
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 10:37

Unfortunately, I don't have any accurate data. The amount of players is more or less constant. The size growth is about 3-5 times faster when ores generate properly.
If you lack the reality, go on a trip or find a job.
 

Sokomine
Member
 
Posts: 3827
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: Ores are waste of HDD

by Sokomine » Wed Aug 14, 2019 10:58

Hard to tell if there wheren't any further factors involved. If players notice that there are no ores present, they will stop mining for them. Less terrain will have to be generated. But you're also right that less diffrent nodes in a chunk will likely cause smaller compressed data.

Hume2 wrote:I also noticed that like 99% ores will be never mined out and won't be even spotted by anyone. The idea is: Do not generate any ores until they can be seen. Has anyone been experimenting with such mechanism? Actually, it will be also good against hackers so they can't know where the ores are until they make them visible. The disadvantages are that it slows down the server and prospectors from technic mod will no longer work.

This has been discussed in at least one thread about client side mods as a way of protection against local cheats. The trouble with your idea about "can be seen" is that that might be very cost-intensive at runtime. Much more expensive than some additional hard disk space. And: When can ores be seen? Does it depend on light level? Players present? What about walls of caves?

I still wish for some additional columns in the SQL table that holds map information: Time the mapblock was generated, time it was last modified, total number of nodes digged in this mapblock by players, total number of nodes placed in this mapchunk by players *excluding torches*. That would increase the map database size even further. But it'd also allow to identify mapblocks that are of no further intrest - mapblocks where players just walked past on the surface, mapblocks where players mined once and likely won't come back. Servers could delete such unused mapblocks and save space. Backups of the map could become incremental backups if necessary (just save what was changed). I've some years ago done a local backup of all I could see and find on Redcrabs server. The size of that database was about 150 MB. I'm sure the real database was way larger.
A list of my mods can be found here.
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 11:50

Actually, only two players noticed that something was wrong.

By "can be seen" I mean that its adjacent nodes are see-through. So the cave walls will be generated while the bulk will stay consistent until it gets exposed.

I think, adding a few numbers to each mapblock won't increase the size much.
If you lack the reality, go on a trip or find a job.
 

User avatar
Linuxdirk
Member
 
Posts: 2050
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 12:22

Linuxdirk wrote:What happens if the ore node is hidden again? Will it be removed or will it stay?

It will stay. I think, there's no need to remove them again because that will happen in a very few cases.
If you lack the reality, go on a trip or find a job.
 

User avatar
TumeniNodes
Member
 
Posts: 2741
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: Ores are waste of HDD

by TumeniNodes » Wed Aug 14, 2019 15:44

The problem you may actually be noticing, is the generation of cave systems, which is a constant process
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am
 

User avatar
Krock
Developer
 
Posts: 4296
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: Ores are waste of HDD

by Krock » Wed Aug 14, 2019 17:08

Freeminer has/had (?) a system to detect player-made mapblock changes. This way most of the generated stuff could simply be deleted from the memory without saving.
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 19:22

Deleting unused mapblocks is also a perspective option.
If you lack the reality, go on a trip or find a job.
 

Sokomine
Member
 
Posts: 3827
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: Ores are waste of HDD

by Sokomine » Wed Aug 14, 2019 19:53

TumeniNodes wrote:The problem you may actually be noticing, is the generation of cave systems, which is a constant process

Not that constant, but...caves may always eat into generated mapchunks if not all mapchunks around the current one have been generated. I don't know how expensive it would be to trigger ore generation in all cave walls only.

Krock wrote:Freeminer has/had (?) a system to detect player-made mapblock changes. This way most of the generated stuff could simply be deleted from the memory without saving.

Maybe that's a bit extreme. In most cases the current system works fine. It's just that maps grow rapidly into the gigabyte space without player-built areas justifying that. Beeing able to manually clean up the map database once in a while while still just reading already expensively generated mapblocks from storage most of the time could be helpful.
A list of my mods can be found here.
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Wed Aug 14, 2019 20:00

I think, caves don't cause much entropy. If you take a mk2 prospector from technic and set it to a common ore, you can notice that the veins are scattered everywhere.
If you lack the reality, go on a trip or find a job.
 

User avatar
TumeniNodes
Member
 
Posts: 2741
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: Ores are waste of HDD

by TumeniNodes » Wed Aug 14, 2019 22:05

caves are constantly generated (even unexplored caves) this is, as far as I recall a known issue.
I could be mistaken though
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Thu Aug 15, 2019 05:56

TumeniNodes wrote:caves are constantly generated (even unexplored caves) this is, as far as I recall a known issue.
I could be mistaken though

What do you mean by caves being constantly generated? Do they generate in already generated mapblocks?
If you lack the reality, go on a trip or find a job.
 

User avatar
TumeniNodes
Member
 
Posts: 2741
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: Ores are waste of HDD

by TumeniNodes » Thu Aug 15, 2019 06:15

yes, this is some of what causes a hit to FPS as well as adding to the issue you are talking about here
at least I believe it is related. or creates a similar issue

As I said, I could be wrong but, I think I recall this being mentioned some time ago.
The constant generation of cave systems, add to lowered fps and an increase in the size of the map.sqlite
paramat would be able to correct me if I'm wrong

Looking back and re-reading your topic though, I am embarrassed to realize I am way off topic so I apologize.
I should have waited to comment until I was able to read more thoroughly
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am
 

User avatar
Hume2
Member
 
Posts: 313
Joined: Tue Jun 19, 2018 08:24
Location: Czech Republic
GitHub: Hume2
In-game: Hume2

Re: Ores are waste of HDD

by Hume2 » Thu Aug 15, 2019 17:35

I noticed that caves with water or lava pool might exceed to neighbour mapblocks but I never noticed any new cave being generated during gameplay.
If you lack the reality, go on a trip or find a job.
 

Sokomine
Member
 
Posts: 3827
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: Ores are waste of HDD

by Sokomine » Thu Aug 15, 2019 17:51

Caves are usually generated at mapgen time. When a mapchunk (usually 5x5x5 blocks, each 16x16x16 nodes large) is generated, it is generated with a "shell" of 16 blocks in each direction. Mapgen doesn't fill that shell space with normal map nodes - but caves may extend into it. And if the mapblocks forming the shell have already been generated, the new mapchunk may create caves in those already generated mapblocks. As soon as all neighbouring mapblocks have been generated, no more caves will be created.
A list of my mods can be found here.
 

ShadMOrdre
Member
 
Posts: 412
Joined: Mon Dec 29, 2014 08:07
Location: USA
GitHub: ShadMOrdre
In-game: shadmordre

Re: Ores are waste of HDD

by ShadMOrdre » Thu Aug 15, 2019 19:33

Regarding the ores issue, I've considered some alternative ways to "hide" ores.

In my lib_materials mod, I have 150+ of stone types and all default, technic, moreores, and real_minerals ores, as well as others. (Don't think overkill, think content depth and possibility.) In the real world, ores are usually associated with certain types of stone. So I have considered simply having the stone type drop the relevant ores, based on the rarity param for drops. This will most certainly "hide" the ores from the client side cheat mods.

I'm sure that a mechanism could be added, to accommodate technic, to allow the prospector to still find ores. If we use the idea from above, where the ore is only generated when discovered, this could be handled with a simple ABM/LBM/LVM function, I think. It may just be better to have the prospector also look for certain stone types that have a higher propensity to drop the relevant ores.

All configured mapgen noises are loaded and ran at each mapgen. This includes caves, ores, decos, and any other mapgen related noises. Even in air, the mapgen needs to check if the relevant conditions are met to the generate the appropriate parts. So while we are in air, the mapgen still needs to determine if caves and ores should be generated. I know from running a "caveless" configuration of mgv7, that the caves certainly affect mapgen times, because terrain generates slightly faster when both standard caves and mgv7 caverns are disable.

Per paramat, in the mapgen questions thread, the fastest mapgen would be v7 with floatlands, mountatins, rivers, caverns disable as well as the basic mapgen setting of nocaves. I can see where disabling, or not generating ores, would/should speed this up, but if now, the compression algorithm comes into play due to increased compression then this should be balanced better.

Paramat has been working on a new mapgen, with little to no extra noises beyond terrain and rivers. It be interesting to see whether ores exhibit the same type of issue on this newer mapgen.

Shad
MY MODS: lib_ecology lib_materials lib_clouds lib_node_shapes ---- Inspired By: Open Source Virtual World Simulator Opensimulator.
 

User avatar
paramat
Developer
 
Posts: 3396
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: Ores are waste of HDD

by paramat » Sat Aug 17, 2019 23:51

> The idea is: Do not generate any ores until they can be seen.

In my opinion this is a horrible idea, as a solution to a non-problem. Worlds should obviously have an interesting structure and it is simplest and optimal by far to generate them that way from the start, we just have to accept this fine detail and the way it reduces the ability to compress.
Memory is exponentially growing in capacity and becoming cheaper, this is not a priority concern.
Adding fine detail to worlds afterwards would cause so many problems and complexities.
Besides, ores are not the only valuable nodes in a world.

> The problem you may actually be noticing, is the generation of cave systems, which is a constant process

> What do you mean by caves being constantly generated? Do they generate in already generated mapblocks?

Caves are only generated at mapgen time, just like everyhitng else. And yes some of the caves (only the flooded caves) do extend beyond mapchunk borders by up to 1 mapblock (16 nodes), but this is still 'at the mapgen time of the parent mapchunk' and so is mostly irrelevant.
Dungeons extend beyond the borders in the same way.
 


Return to Feature Discussion



Who is online

Users browsing this forum: No registered users and 0 guests