jungletree overwriting generated area

[Kilarin]

So, I've got a mod.
It soft depends on default (default?)
it has a register_on_generated function that clears a certain area, setting the nodes to air.
I can see it clear that area, in debug.txt I can watch a particular node get set to air and saved. As several other chunks are generated, that node is still air. Then, suddenly, a jungle gets written over that area and the node changes to jungletree.

now, the only thing in this game that has jungletrees is default. And since I soft depend on default, that means that my mod's "on_generated" function should run AFTER any default on_generated functions, right?

So, since I can watch my mod clear the area, what on earth is suddenly deciding to write jungle trees over it later?

[duane]

now, the only thing in this game that has jungletrees is default. And since I soft depend on default, that means that my mod's "on_generated" function should run AFTER any default on_generated functions, right?

There are a lot of qualifiers to that statement. Bear in mind that I haven't done a serious mapgen in a while, so anything I say may be out of date, but let me give you some.

The jungle trees planted by the game's built-in mapgens are planted by C++ code in the game itself. The default game's lua puts tree descriptions into a table, and the minetest game does the actual planting before your mod is even thought of.

However, each chunk the game generates wants to overgenerate into any adjacent chunks that haven't been generated yet. How does the game know they haven't? If there's nothing but air there, of course. So it generates terrain into any open air spaces up to 16 blocks into the next chunk. That makes the forests blend together better.

It's very difficult to get around overgeneration. I gave up trying and went to singlenode mapgens when I absolutely needed to control the boundaries of a chunk.

Also, the register_ongenerated function does not guarantee your mod's place in the ongenerated queue. You may be first, you may be last. Ideally, if your mod and default are the only mods running, and you depend on default, you'll be last, but there's no way to be absolutely sure. You can make yours first by pushing your function to the start of the table, but making it last is problematic.

What's more, if your computer has multiple cores, minetest will try to use each of them, assigning part of a chunk's terrain to each core. Very efficient. Except that the game has no idea how to tell your mod to work on part of a chunk. So it generates part of a chunk, calls your mod, generates another part, calls your mod, generates... etc. If you have four cores, your mod can get called four times and be given four different sets of terrain data (some of which will be empty) to work with. In theory, it will always get called last, but I've seen cases where that didn't seem to happen.

Now imagine you're using multiple cores and there are several mods (including default) using ongenerated callbacks. Some of them take longer than others to finish what they're doing and save their data to the map. Whoever happens to finish last will overwrite everyone else basing their data on what they were given when they were called.

Since I don't know exactly what your mod is doing, I can't say which of these applies, if any. If the area you're interested in happens to be on a chunk boundary, you're in for a serious challenge. Your best bet may be to drop diamond blocks there and replace them later, but even that won't always work.

Now, before anybody gets mad at this confusing state of affairs, let me assure you that it works really well for generating pretty terrain very quickly, which is the developers' goal.

[sorcerykid]

However, each chunk the game generates wants to overgenerate into any adjacent chunks that haven't been generated yet. How does the game know they haven't? If there's nothing but air there, of course. So it generates terrain into any open air spaces up to 16 blocks into the next chunk. That makes the forests blend together better.

Is this always true? When I enable local map saves, this overgeneration phenomenon seems to occur everywhere (regardless of whether any adjacent map chunks were not-generated.

[Kilarin]

Now, before anybody gets mad at this confusing state of affairs, let me assure you that it works really well for generating pretty terrain very quickly, which is the developers' goal.

Nothing to get mad about. But it does seem like there ought tho be some way around this. Thank you so VERY much for the detailed explanation.

[duane]

Is this always true? When I enable local map saves, this overgeneration phenomenon seems to occur everywhere (regardless of whether any adjacent map chunks were not-generated.

The way to be sure is to read the mapgen source. As I recall, it just checks for air and null blocks, so it could very well overgenerate into existing chunks (but only at the borders).

Of course the reason for this is the problem of placing trees at the border. If you put half a tree down, the next chunk has no way of knowing to place the other half. (Bear in mind that the next chunk may not be generated for days, if ever.) If you avoid the borders, you end up with really jarring empty spots in a regular grid. The only sensible way to do it is to go ahead and put the whole tree down and let the next chunk work around it.

[duane]

\But it does seem like there ought tho be some way around this. Thank you so VERY much for the detailed explanation.

No problem, and there is a way to work around it. Unfortunately, it generally involves doing the mapgen work in lua, which is complicated and slow.

You could clear all the trees out of the decoration table and plant them yourself, but it's going to take a lot more cpu time and you may not be happy with the result.

[Kilarin]

We have a register_on_generated. Seems like we need a register_after_decorations. :)

I'm thinking that one, rather crazy, way to get around this would be to put an invisible block in each chunk that would clean up any decorations the first time it is loaded after generation, then delete itself.

Unfortunately, there is no guarantee that the special block itself would not be overwritten with trees or grass, although I would think being at the center of the chunk would make that a lot less likely.

[paramat]

If this is caused by tree overgeneration, the trees will be appearing only in the bottom 16 nodes of a mapchunk, that's a way to check the cause.

[Kilarin]

here is one place that matches your description. The canyon is being generated seemingly correctly, I can fly to this spot and see that everything had been removed and replaced with air, then suddenly, boom! the trees flash into being:


I added a mod to show the chunk outlines and it shows that yes, these trees are at the bottom of the chunk.

But I'm also seeing a lot of stuff like this:

leaves, bits of trunk, scattered randomly around the chunk. Like I said above, I can actually put logging statements in the mod code that shows the mod replacing a bit of trunk with air, saving it, and it STAYS air for a while, then a few chunks later it suddenly converts back to tree. It's obviously not all of the trees in examples like this, just odd random bits.

I'm working on writing an abm that will run 10 seconds later and clear out this junk, (then self destruct) but that seems a very inefficient way to do this.

I'll post the code updates soon so anyone who is interested can play around with it and help me figure out a better way to do this. :)

[sorcerykid]

The way to be sure is to read the mapgen source. As I recall, it just checks for air and null blocks, so it could very well overgenerate into existing chunks (but only at the borders).

Thanks of the tip! I took a peek at the C++ source code, but couldn't make heads nor tails of the biome handling routines. However, I think there is more involved than checking for air and null blocks.

Case in point: Here are two screencaps of my house on Banana Land facing toward spawn. The first was taken on the live server a few hours ago, the second is from a local map save earlier this week. Apart from the gigantic tree in the background (a since demolished player build) I can find no explanation for why my entire homestead is covered in a layer of snow and overrun with pine trees. Even the lake directly behind my house (just out of frame) is half-filled with ice. None of this area is anywhere near the edge of the local map save.

[duane]

I can find no explanation for why my entire homestead is covered in a layer of snow and overrun with pine trees. Even the lake directly behind my house (just out of frame) is half-filled with ice. None of this area is anywhere near the edge of the local map save.

The snow and ice you're looking at are "dust", which is applied at the very end of the mapgen process by a separate method, possibly after mods. It looks like your existing chunk was generated twice. I can't imagine why that would happen unless a mod does it. I seem to remember a lua function that recreates all of the decorations from a biome, which would be a likely culprit if you were trying some mod that ran it. Note that decorations should be completely repeatable, so you'd only notice a change where someone had cleared an area.

[Kilarin]

for anyone who is interested in the overgeneration issue, I wrote a VERY simple testing mod:
https://github.com/Kilarin/flatland
All this does is for ANY chunk that contains nodes from y=0 to y=200, it replaces all content with air, except for the outline of the chunk, which it replaces with cobble or brick (to make it easy to see the chunks)

With this, you can see the overgeneration, it doesn't stay at the bottom of the chunk, but it certainly does seem to be staying at the edges.


And when everything lines up just right, I suspect when the original surface was just a few nodes below the bottom of a chunk, you get whole forests like this:

[duane]

With this, you can see the overgeneration, it doesn't stay at the bottom of the chunk, but it certainly does seem to be staying at the edges.

This makes it really fun to try to build walls around a chunk for a city. Sometimes you get more tree than wall. I took to generating my own terrain in singlenode or abandoning the chunk boundaries and using abms to try to clean up the mess (which burned up so much cpu time that it probably wasn't worth it).

I wonder what would happen if you turned off decorations in the mapgen with the nodecorations flag, used generate_decorations with specific boundaries to create them in your mod, then removed them. Unfortunately, although there are parameters for the boundaries, the documentation suggests that they're ignored.

[paramat]

Overgeneration works sideways too. So a tree will be placed with it's trunk near the edge of a mapchunk and some leaves will extend into the neighbour mapchunk.
Dungeons and the liquid-type caves overgenerate horizontally and vertically.

[Kilarin]

So, couple of questions
1: Is there a good reason that biome generation is based on the original generated surface, not on the existing surface after mod's run? I'm assuming that there is some important reason why it is done like this and it would be impractical to recalculate after all of the on_generated functions have run?
2: I don't suppose there is any way to flag a chunk for no overgeneration? I'm not certain there are even such things as chunk specific flags, so probably not.

[duane]

1: Is there a good reason that biome generation is based on the original generated surface, not on the existing surface after mod's run? I'm assuming that there is some important reason why it is done like this and it would be impractical to recalculate after all of the on_generated functions have run?

I suspect the primary reason is that most mods work better after all the terrain generation is done. Only a fraction make major changes to the landscape, and those tend to differ on when would be best to interrupt generation. Some would prefer to get only the base stone, before any water, biomes or decorations are placed. Others would prefer after everything except trees (which would be complicated, since you'd have to run two cycles of decorations). Who do you accommodate?

At the moment, on_generate callbacks are done in EmergeThread::finishGen. Note the "thread", indicating that several of these should run simultaneously. The actual mapgen terrain generation doesn't appear to be threaded, but the lighting and liquid handling (which take up a lot of cpu time) are. After that, in the same thread, the callbacks are made. (Personally, I think callbacks should NOT be threaded for the reasons I discussed above.)

Most mods are going to want to know where light and water/lava are at before they begin, so you can't just move the callbacks. Someone would have to add code to separate them based on when they want to occur, and all of the mapgens would have to have code to execute the few that wanted to be called in the middle of the mapgen process. The mapgens obviously generate terrain in different ways, so it would be hard to standardize this.

In all the modding done on Minetest, I've only noticed a few people complain about this problem, but if any developer slows down terrain generation, even a little, you can be sure that there'll be a mob with torches and pitchforks at his virtual door almost instantly.

2: I don't suppose there is any way to flag a chunk for no overgeneration? I'm not certain there are even such things as chunk specific flags, so probably not.

I haven't found one. To do that, you'd have to first, make a list of all chunks that a mod didn't want overgenerated, then you'd have to check it every time a chunk is generated to see what to do. You'd have to add code to all the mapgens and decoration methods to toggle overgeneration. Also, you can't specify one chunk -- it's got to toggle every adjacent chunk, so in 27 chunks you'd have problems with forests looking ugly, light leaking into caves, breaks in water, etc.

Since we've already discussed ways to get around this, and we haven't even touched on gennotify, I don't expect this to happen. Still, if you're really interested in this, you could always get into the code on github and make the changes yourself. If it works, you might get it pulled into the main project.


A bit off-topic:

It looks to my highly untrained eye, that moving the lua callbacks (and possibly the m_server->m_env->activateBlock call) to the EmergeThread::run method would solve some of the problem. on_generated callbacks would only get called once (assuming mapgen::makeChunk only gets called once), and everything the mapgen does would be finished by that point, including the light and water handling (I think).

Doing it that way, the terrain wouldn't change half-way through a mod. However, you'd still have overgeneration, it just might be easier to deal with. Mods using on_generated would actually take less cpu time, but there might be other complications I can't see. I'm under outside pressure to avoid any serious programming, but it took me five minutes to compile the change in. I may be able to run some simple tests later.

But, as I say, this won't help your situation much.

[paramat]

1. Yes, several.
2. There isn't.

[FaceDeer]

I just whipped up a simple mapgen mod that I think is running into some of these problems. I've got it generating large mountains scattered randomly around the map and I was hoping that it would be a simple process - place a bunch of default:stone and a bit of dirt, then call minetest.generate_decorations(vm, minp, maxp) and minetest.generate_ores(vm, minp, maxp) to make it blend into the landscape.

It's not working though (otherwise I wouldn't be here) and from what I read in this thread it doesn't sound promising. I actually don't mind the "overgeneration", but it's not being done *consistently* - I wind up with trees half-buried in the mountainside sometimes, sometimes I've got strips of trees and snow on my mountain with an obvious chunk boundary, and so forth. minetest.generate_decorations(vm, minp, maxp) doesn't seem to do anything, adding that to my on_generated function has no visible effect.

Perhaps an approach to some of the problems mentioned in this thread would be to give register_on_generated a parameter to allow the callback to be placed at one of several different "milestones" in the generation process. One after base terrain is generated, one after vegetation and other decorations are placed, etc. Being able to choose would make it much easier to write modular mapgen mods that can work seamlessly with existing mapgens.