Ores are waste of HDD

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Ores are waste of HDD

by Hume2 » Post

I noticed one thing on my server. While there was a bug which caused the ores not to generate, the world size was increasing way slower. The reason is that the lack ores decreases entropy, so the map can be compressed more effectively.

I also noticed that like 99% ores will be never mined out and won't be even spotted by anyone. The idea is: Do not generate any ores until they can be seen. Has anyone been experimenting with such mechanism? Actually, it will be also good against hackers so they can't know where the ores are until they make them visible. The disadvantages are that it slows down the server and prospectors from technic mod will no longer work.
If you lack the reality, go on a trip or find a job.

Xudo
Member
Posts: 162
Joined: Wed Nov 09, 2016 16:43
GitHub: akryukov92
In-game: Xudo

Re: Ores are waste of HDD

by Xudo » Post

Can you provide more accurate measurements? What was your world settings?
How many players have been online simultaneously?

It might be a good catch. Though, I'm not sure that it will be easy to change. Map generation is performed only once. Splitting it to two phases will greatly increase complexity of code.
May be another compression algorithm might help.

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

Unfortunately, I don't have any accurate data. The amount of players is more or less constant. The size growth is about 3-5 times faster when ores generate properly.
If you lack the reality, go on a trip or find a job.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Ores are waste of HDD

by Sokomine » Post

Hard to tell if there wheren't any further factors involved. If players notice that there are no ores present, they will stop mining for them. Less terrain will have to be generated. But you're also right that less diffrent nodes in a chunk will likely cause smaller compressed data.
Hume2 wrote: I also noticed that like 99% ores will be never mined out and won't be even spotted by anyone. The idea is: Do not generate any ores until they can be seen. Has anyone been experimenting with such mechanism? Actually, it will be also good against hackers so they can't know where the ores are until they make them visible. The disadvantages are that it slows down the server and prospectors from technic mod will no longer work.
This has been discussed in at least one thread about client side mods as a way of protection against local cheats. The trouble with your idea about "can be seen" is that that might be very cost-intensive at runtime. Much more expensive than some additional hard disk space. And: When can ores be seen? Does it depend on light level? Players present? What about walls of caves?

I still wish for some additional columns in the SQL table that holds map information: Time the mapblock was generated, time it was last modified, total number of nodes digged in this mapblock by players, total number of nodes placed in this mapchunk by players *excluding torches*. That would increase the map database size even further. But it'd also allow to identify mapblocks that are of no further intrest - mapblocks where players just walked past on the surface, mapblocks where players mined once and likely won't come back. Servers could delete such unused mapblocks and save space. Backups of the map could become incremental backups if necessary (just save what was changed). I've some years ago done a local backup of all I could see and find on Redcrabs server. The size of that database was about 150 MB. I'm sure the real database was way larger.
A list of my mods can be found here.

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

Actually, only two players noticed that something was wrong.

By "can be seen" I mean that its adjacent nodes are see-through. So the cave walls will be generated while the bulk will stay consistent until it gets exposed.

I think, adding a few numbers to each mapblock won't increase the size much.
If you lack the reality, go on a trip or find a job.

User avatar
Linuxdirk
Member
Posts: 3217
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Ores are waste of HDD

by Linuxdirk » Post

What happens if the ore node is hidden again? Will it be removed or will it stay?

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

Linuxdirk wrote:What happens if the ore node is hidden again? Will it be removed or will it stay?
It will stay. I think, there's no need to remove them again because that will happen in a very few cases.
If you lack the reality, go on a trip or find a job.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: Ores are waste of HDD

by TumeniNodes » Post

The problem you may actually be noticing, is the generation of cave systems, which is a constant process
A Wonderful World

User avatar
Krock
Developer
Posts: 4649
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Ores are waste of HDD

by Krock » Post

Freeminer has/had (?) a system to detect player-made mapblock changes. This way most of the generated stuff could simply be deleted from the memory without saving.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

Deleting unused mapblocks is also a perspective option.
If you lack the reality, go on a trip or find a job.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Ores are waste of HDD

by Sokomine » Post

TumeniNodes wrote: The problem you may actually be noticing, is the generation of cave systems, which is a constant process
Not that constant, but...caves may always eat into generated mapchunks if not all mapchunks around the current one have been generated. I don't know how expensive it would be to trigger ore generation in all cave walls only.
Krock wrote: Freeminer has/had (?) a system to detect player-made mapblock changes. This way most of the generated stuff could simply be deleted from the memory without saving.
Maybe that's a bit extreme. In most cases the current system works fine. It's just that maps grow rapidly into the gigabyte space without player-built areas justifying that. Beeing able to manually clean up the map database once in a while while still just reading already expensively generated mapblocks from storage most of the time could be helpful.
A list of my mods can be found here.

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

I think, caves don't cause much entropy. If you take a mk2 prospector from technic and set it to a common ore, you can notice that the veins are scattered everywhere.
If you lack the reality, go on a trip or find a job.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: Ores are waste of HDD

by TumeniNodes » Post

caves are constantly generated (even unexplored caves) this is, as far as I recall a known issue.
I could be mistaken though
A Wonderful World

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

TumeniNodes wrote:caves are constantly generated (even unexplored caves) this is, as far as I recall a known issue.
I could be mistaken though
What do you mean by caves being constantly generated? Do they generate in already generated mapblocks?
If you lack the reality, go on a trip or find a job.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: Ores are waste of HDD

by TumeniNodes » Post

yes, this is some of what causes a hit to FPS as well as adding to the issue you are talking about here
at least I believe it is related. or creates a similar issue

As I said, I could be wrong but, I think I recall this being mentioned some time ago.
The constant generation of cave systems, add to lowered fps and an increase in the size of the map.sqlite
paramat would be able to correct me if I'm wrong

Looking back and re-reading your topic though, I am embarrassed to realize I am way off topic so I apologize.
I should have waited to comment until I was able to read more thoroughly
A Wonderful World

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: Ores are waste of HDD

by Hume2 » Post

I noticed that caves with water or lava pool might exceed to neighbour mapblocks but I never noticed any new cave being generated during gameplay.
If you lack the reality, go on a trip or find a job.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Ores are waste of HDD

by Sokomine » Post

Caves are usually generated at mapgen time. When a mapchunk (usually 5x5x5 blocks, each 16x16x16 nodes large) is generated, it is generated with a "shell" of 16 blocks in each direction. Mapgen doesn't fill that shell space with normal map nodes - but caves may extend into it. And if the mapblocks forming the shell have already been generated, the new mapchunk may create caves in those already generated mapblocks. As soon as all neighbouring mapblocks have been generated, no more caves will be created.
A list of my mods can be found here.

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: Ores are waste of HDD

by ShadMOrdre » Post

Regarding the ores issue, I've considered some alternative ways to "hide" ores.

In my lib_materials mod, I have 150+ of stone types and all default, technic, moreores, and real_minerals ores, as well as others. (Don't think overkill, think content depth and possibility.) In the real world, ores are usually associated with certain types of stone. So I have considered simply having the stone type drop the relevant ores, based on the rarity param for drops. This will most certainly "hide" the ores from the client side cheat mods.

I'm sure that a mechanism could be added, to accommodate technic, to allow the prospector to still find ores. If we use the idea from above, where the ore is only generated when discovered, this could be handled with a simple ABM/LBM/LVM function, I think. It may just be better to have the prospector also look for certain stone types that have a higher propensity to drop the relevant ores.

All configured mapgen noises are loaded and ran at each mapgen. This includes caves, ores, decos, and any other mapgen related noises. Even in air, the mapgen needs to check if the relevant conditions are met to the generate the appropriate parts. So while we are in air, the mapgen still needs to determine if caves and ores should be generated. I know from running a "caveless" configuration of mgv7, that the caves certainly affect mapgen times, because terrain generates slightly faster when both standard caves and mgv7 caverns are disable.

Per paramat, in the mapgen questions thread, the fastest mapgen would be v7 with floatlands, mountatins, rivers, caverns disable as well as the basic mapgen setting of nocaves. I can see where disabling, or not generating ores, would/should speed this up, but if now, the compression algorithm comes into play due to increased compression then this should be balanced better.

Paramat has been working on a new mapgen, with little to no extra noises beyond terrain and rivers. It be interesting to see whether ores exhibit the same type of issue on this newer mapgen.

Shad

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: Ores are waste of HDD

by paramat » Post

> The idea is: Do not generate any ores until they can be seen.

In my opinion this is a horrible idea, as a solution to a non-problem. Worlds should obviously have an interesting structure and it is simplest and optimal by far to generate them that way from the start, we just have to accept this fine detail and the way it reduces the ability to compress.
Memory is exponentially growing in capacity and becoming cheaper, this is not a priority concern.
Adding fine detail to worlds afterwards would cause so many problems and complexities.
Besides, ores are not the only valuable nodes in a world.

> The problem you may actually be noticing, is the generation of cave systems, which is a constant process

> What do you mean by caves being constantly generated? Do they generate in already generated mapblocks?

Caves are only generated at mapgen time, just like everyhitng else. And yes some of the caves (only the flooded caves) do extend beyond mapchunk borders by up to 1 mapblock (16 nodes), but this is still 'at the mapgen time of the parent mapchunk' and so is mostly irrelevant.
Dungeons extend beyond the borders in the same way.

Astrobe
Member
Posts: 571
Joined: Sun Apr 01, 2018 10:46

Re: Ores are waste of HDD

by Astrobe » Post

paramat wrote: Memory is exponentially growing in capacity and becoming cheaper, this is not a priority concern.
Parkinson's law.

That's interesting to use Moore's law for a game that inherited its procgen from ancestors (Elite, Roguelikes) that were struggling with hardware limitations.

Don't use that argument. It is a creativity killer. It is also too often used to sweep problems under the carpet.

Actually, I'm not arguing that something should be done about it in the engine right now. Perhaps the creativity should be in mods. Perhaps one could remove stone_with_x nodes and ore generation, and replace it with stone nodes that randomly drop ores. It solves the problem stated by OP, right?

Ok, it creates new problems - and that's where things get interesting. How the frack players know where to dig, for instance. Then maybe one could say "the chance to get gold is higher if you dig in river beds". Then I think a game could manually handle drops so that the relative chances of getting some ore depends on the environment.
Or maybe one could introduce a metal detector device that actually creates fake stone nodes that drop ore. Or it could just mutate stone nodes into ore-stone nodes deep enough that the player can't see the trick happening.

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: Ores are waste of HDD

by duane » Post

Astrobe wrote:Don't use that argument. It is a creativity killer. It is also too often used to sweep problems under the carpet.
Paramat's right. Nobody operating a big server is going to trade more cpu time for a tiny amount of map compression. However, if you just want to think of new ways to handle ores, that's a different story.
Astrobe wrote:Perhaps one could remove stone_with_x nodes and ore generation, and replace it with stone nodes that randomly drop ores.
Tried that. Nobody liked it. (Not even me.)
Astrobe wrote:Then maybe one could say "the chance to get gold is higher if you dig in river beds".
This has also been done before, but nobody was interested enough to come up with a good set of rules. The only thing I could think of was the old saw about ferns growing near iron ore, but most people expect iron and better minerals to be deep underground, so that doesn't work well either.

Putting ore next to lava is popular, but the real problem with generating on the fly is caves. People want to see ore in their cave walls. Most of them could care less about the rest of the volume in any given chunk. If you generate the walls on the fly, you're likely to burn almost as much cpu time as it took to generate the caves to begin with, which is kind of wasteful
Believe in people and you don't need to believe anything else.

Astrobe
Member
Posts: 571
Joined: Sun Apr 01, 2018 10:46

Re: Ores are waste of HDD

by Astrobe » Post

duane wrote: This has also been done before, but nobody was interested enough to come up with a good set of rules. The only thing I could think of was the old saw about ferns growing near iron ore, but most people expect iron and better minerals to be deep underground, so that doesn't work well either.
That's a way of thinking that needs to be sent to the trash can. It makes everything above level zero mostly uninteresting, makes people spend most of their time in mines and then they complain that all they see is grey walls and that underground biomes are needed. From the supposedly "realistic" claim that ores are deep in the ground, you end up with even less realistic underground features (that add to the generation time).

Just put ores at all level (they'll have to be way more sparse, which should help both with compression and generation time) and then a hole at the top of a mountain suddenly becomes damn interesting. And scuba diving for ores too.
Putting ore next to lava is popular, but the real problem with generating on the fly is caves. People want to see ore in their cave walls. Most of them could care less about the rest of the volume in any given chunk. If you generate the walls on the fly, you're likely to burn almost as much cpu time as it took to generate the caves to begin with, which is kind of wasteful
Or you can hack the lava cooling mechanism so that cooling lava sources creates stones with ore (you may need additional hacks to make sure that the lava cooling material is consumed in the process).

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: Ores are waste of HDD

by duane » Post

Astrobe wrote:It makes everything above level zero mostly uninteresting, makes people spend most of their time in mines and then they complain that all they see is grey walls and that underground biomes are needed.
On the other hand, some people like caves, and without ore down there, you have to come up with another incentive to explore. Otherwise, why have them at all?
Or you can hack the lava cooling mechanism so that cooling lava sources creates stones with ore (you may need additional hacks to make sure that the lava cooling material is consumed in the process).
Consuming water would be pointless -- it's too easy to regenerate. Consuming the lava would be problematic since the flowing lava makes the stone, and you don't have any sure way of telling where it came from. Besides, random ores from lava is just as dull as random ores from stone.
Just put ores at all level (they'll have to be way more sparse, which should help both with compression and generation time) and then a hole at the top of a mountain suddenly becomes damn interesting.
Making them sparse would just make it more frustratingly random (and still dull). I already put ore on my mountain tops, but it's not very challenging, apart from the monsters that show up.

The only idea I've heard so far that I like is being able to go somewhere or do something specific to find specific types of ore. Say, you have to find a particular type of water/terrain combination for gold and pan for it -- and the panning process tells you immediately whether you've found a good location or a bad one, so you don't have as much tedious repetition -- then it peters out after a bit. If you're looking for coal, you have to drill into rock to take samples; if you find fossils, you know a coal bed is nearby. You have to click your compass to locate iron ore by deflection. Mese is very rare, but it creates a loud vibration in the rock, or poisons the plants above it. Diamonds only occur in active lava; it's easy to find, but the lava explodes frequently.

If there are specific ways to find the ore, you don't have to plant nearly as much. There still needs to be more to it than that, but you get the idea.
Believe in people and you don't need to believe anything else.

Astrobe
Member
Posts: 571
Joined: Sun Apr 01, 2018 10:46

Re: Ores are waste of HDD

by Astrobe » Post

duane wrote:
Astrobe wrote:It makes everything above level zero mostly uninteresting, makes people spend most of their time in mines and then they complain that all they see is grey walls and that underground biomes are needed.
On the other hand, some people like caves, and without ore down there, you have to come up with another incentive to explore. Otherwise, why have them at all?
The problem is that game mechanics and features don't work in isolation. They form a network of interactions. That's why they might make no sense in your game despite the fact they work (as far as I can tell) in my game.

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: Ores are waste of HDD

by duane » Post

Astrobe wrote:The problem is that game mechanics and features don't work in isolation. They form a network of interactions. That's why they might make no sense in your game despite the fact they work (as far as I can tell) in my game.
Well, if you want a general solution, it needs to work for the default game.
Believe in people and you don't need to believe anything else.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests