Large schematics are spawned incompletely

Post Reply
User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Large schematics are spawned incompletely

by MirceaKitsune » Post

https://github.com/minetest/minetest/issues/8232

This is an old problem that's affected my Structures mod for years. Now that its city generator is nearing completion and I'm in the process of designing a new city with large buildings, this bug is the last thing standing in the way and posing a huge blocker to my project.

Minetest doesn't seem to properly handle spawning large schematics as part of newly generated chunks. I tried using both minetest.place_schematic and minetest.place_schematic_on_vmanip from the minetest.register_on_generated function but both come with a huge problem: The schematics are always incomplete with large cubic chunks ripped out of them. This results in every city being completely mangled to the point of structures making no sense. The bug doesn't seem to affect smaller structures at all (roughly 20 x 20 x 20 and under) even when they're being spawned rapidly, it's primarily just big schematics.

Here are a few screenshots showing this issue with the new apartment blocks I'm working on. You can see the interiors of apartments as they're randomly sliced open. Instead of the weirdly cut out cubes, we should only be seeing solid walls with windows from the outside.

Image
Image
Image
Image

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

Is this caused by the structure being partly in chunks that hadn't been emerged when the schematic was placed? And were then generated afterwards.

You might already know if that's the cause, but if not you might be able to determine it if you create a testing world with backend = dummy in the world.mt file - that way the world is recreated every time you start and you can see how approaching the same schematic from different directions causes it to be missing from different chunks.

- If that turns out to be the problem, then workarounds should be possible.

Also, if you need a mod that shows the chunk boundaries...

Code: Select all

local data          = {} -- reuse the massive VoxelManip memory buffers instead of creating on every on_generate()

local function on_generated(minp, maxp, blockseed)

  local vm, emerge_min, emerge_max = minetest.get_mapgen_object("voxelmanip")
  vm:get_data(data)        -- put all nodes except the ground surface in this array
  local area = VoxelArea:new{MinEdge=emerge_min, MaxEdge=emerge_max}

  local nodeId_debug1 = minetest.get_content_id("default:clay")

  -- v7 rivers appear to be cutting off the top and bottom of the view area
  for x = minp.x, maxp.x do 
    data[area:index(x, minp.y, minp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, minp.z)] = nodeId_debug1
    data[area:index(x, minp.y, maxp.z)] = nodeId_debug1
    data[area:index(x, maxp.y, maxp.z)] = nodeId_debug1
  end
  for y = minp.y, maxp.y do 
    data[area:index(minp.x, y, minp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, minp.z)] = nodeId_debug1
    data[area:index(minp.x, y, maxp.z)] = nodeId_debug1
    data[area:index(maxp.x, y, maxp.z)] = nodeId_debug1
  end
  for z = minp.z, maxp.z do 
    data[area:index(minp.x, minp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, minp.y, z)] = nodeId_debug1
    data[area:index(minp.x, maxp.y, z)] = nodeId_debug1
    data[area:index(maxp.x, maxp.y, z)] = nodeId_debug1
  end

  vm:set_data(data)    
  vm:set_lighting({day=0, night=0})
  vm:calc_lighting()
  vm:write_to_map()
end

minetest.register_on_generated(on_generated)

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

I should clarify that I'm already using the vmanip to spawn structures with on_generate. I understand this function should now only attempt to spawn a piece of schematic that intersects the chunk in cause. So this shouldn't be a case of chunk overflow normally, since I use the vm provided by the generator which should know what to do.

That is an interesting test, I didn't know about this parameter. I will keep in mind.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

DrFrankenstone wrote:FWIW, I didn't mean that drawing beyond the bounds of the chunk might cause it, I meant that being limited to within the bounds of the on_generate chunk may cause it if you don't also successfully spawn the structure in every other chunk that the structure touches.

Are the missing parts aligned with chunk boundaries?
Oh: I am calling minetest.place_schematic_on_vmanip for the same structure multiple times, in case it intersects multiple chunks. Each minetest.on_generated call should be addressing every remaining piece of a schematic.

It doesn't seem like they're aligned. The large cubic chunks ripped out of buildings appear to be random and their size seems to vary. I still suspect it has something to do with chunks but we need to dig deeper.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

Yeah, I've seen it a couple of times now too, and it doesn't seem to be aligned with chunk boundaries. It also appears to be deterministic - reappearing every time the map is generated, even with the backend set to dummy.

Edit: the position of the player when the map is being generated can sometimes affect where holes appear.
(player position changes the order chunks are generated in)

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

Very interesting. Thank you for testing and mentioning those details, it may help in finding a solution.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Large schematics are spawned incompletely

by Sokomine » Post

I had some strange behaviour with my skyplatform mod as well. Apart from the roads and the dirt with grass in the middle, there ought to be only sand as ground. Each platform is 100x100 nodes wide and placed using minetest.place_schematic(..):
Image
Attachments
skyplatform_strange.jpg
skyplatform_strange.jpg (88.23 KiB) Viewed 2126 times
A list of my mods can be found here.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

I suspect a concurrency bug.

I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.

My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).
Last edited by DrFrankenstone on Sun Feb 24, 2019 14:54, edited 1 time in total.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Large schematics are spawned incompletely

by Sokomine » Post

DrFrankenstone wrote: I've been using large schematics to create giant trees. The replacement table passed to place_schematic() is different for each type of tree, and I've noticed that if two trees are near each other (like in the cherry blossom screenshot), the first replacement table ends up being applied to both schematics, even though a very different table was passed to the second place_schematic() call.
The replacement table is definitely cached. When I noticed that some time ago, I made sure lua_api.txt was adjusted to mention that behaviour.
DrFrankenstone wrote: My thinkings is that the replacement table might not be the only thing inadvertently being shared by invocations of place_schematic() (which might be running in parallel).
Possibly. Borders between mapchunks seem to cause some problems. Perhaps it's really mapgen running in parallel that causes it. Except that in the example above the sourrounding mapchunks ought to have long been created.
A list of my mods can be found here.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

damn, I saw that in the lua_api.txt (thank you) and forgot. Working around this is going to be interesting.

User avatar
texmex
Member
Posts: 1753
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: tacotexmex

Re: Large schematics are spawned incompletely

by texmex » Post

Does schemlib solve the issue at mapchunk borders? IIRC it dealt with such problems.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: Large schematics are spawned incompletely

by paramat » Post

What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

paramat wrote:What value are you using for 'num_emerge_threads'? The default is automatic behaviour which is 'number of processors' - 2. Also note that a sqlite world database is not 'thread safe' so 'num_emerge_threads' should be set to '1'.
Try '1' to see if this improves anything.
I just tried setting to to 1 (it was 0), and that hasn't fixed it.

The pieces that go missing seem to be 16x16, though usually they are bunched together as bigger missing pieces.

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: Large schematics are spawned incompletely

by rubenwardy » Post

I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

Confirming that num_emerge_threads = 1 in minetest.conf does not seem to affect the problem, it still occurs the same way.

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

rubenwardy wrote:I had this bug in CTF, and found that you need to emerge the area first using minetest.emerge_area
My lua code's in the on_generate, so partly is the emerge code, but it does improve if I move the place_schematic() call inside a minetest.after(0.1, ...) call to help it happen after the emerge.

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

Has there been any progress on this issue recently? I'm concerned about my Structures mod and the plans I made for it, which rely on the hope that this breaking problem can be solved.

I still suspect it must have something to do with the engine "choking" on large amounts of data and forgetting to register nodes in certain chunks. In this case it should be easy to fix theoretically, by making sure there's a queue and the engine doesn't drop any node changes before they go in effect. Can anyone help confirm if this is indeed the case, so we can know the exact cause for starters?

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: Large schematics are spawned incompletely

by ShadMOrdre » Post

So over the last year, I've spent time converting many of the mg_villages village schematics into lua tables. I've also done this with the various tree schems that I've collected. I did this so that I could use the schem with different nodes.

What I discovered, in many cases the various nodes are placed with random face_dirs. When converted to a lua table, this contributes to bloat. Try converting the schems to lua tables, and use the format below as a guide.

Example:

Code: Select all

local _____ = {name = "air",param2 = 0,prob = 0}

local Mat00 = {name = "default:stone", param2 = 0, prob = 254}
local Mat01 = {name = "default:cobble", param2 = 0, prob = 254}

{
  size = {x=10,y=10,z=10},
  yslice_prob = {
    {ypos = 0, prob = 254},
    {ypos = 1, prob = 254},
    {ypos = 2, prob = 254},
    {ypos = 3, prob = 254},
    {ypos = 4, prob = 254},
    {ypos = 5, prob = 254},
    {ypos = 6, prob = 254},
    {ypos = 7, prob = 254},
    {ypos = 8, prob = 254},
    {ypos = 9, prob = 254}
  }
  data = {
    Mat00, Mat00, Mat00, Mat00, Mat00, Mat00, Mat00, Mat00, Mat00, Mat00,
    _____, Mat00, Mat00, _____, _____, _____, _____, Mat00,  _____, _____,
    ...
  }
  }
Nodes like default:stone really don't have a face_dir, they are a cube. They are the same regardless of the face_dir. Why does this matter? In a lua table, how many variables are you accomodating? The less, the better. By streamlining this, I am able to successfully spawn Peaks small Church mts file, which is x=37, y=51, z=49 nodes. The spawning uses place_schematic in a minetest.after(5 seconds) call.

Maybe it is a combo of both actions, but the only time I see schematics cut out like in your screenshots, is when I am flying around, generating the map as I go, and I switch between Unlimited View and Normal view at the time when a schem is being loaded for generating. In my 8GB computer, this seems to be a memory deficiency, as they spawn normal if I stop flying long enough for the schem to load, and they always load in my unpublished Towns mod, using both streamlined lua schems and the minetest.after call.

Yesterday, I uploaded an indev game world which includes the aforementioned mod, spawning Peaks church. If you are willing to cope with a zip file for the time being, you can download the lib_additions modpack, here, that includes Towns_Royal, where this is being done. Both a somewhat streamlined lua schematic of Peaks church, and the code that spawns it inworld are in there. In addition, in the system.zip file, in the lib_develop modpack, you can find my Instabuild mod, which includes a simple hand tool in which you can import and spawn a schematic, or convert from mts to lua schem file. There does seem to be an upper limit on the size of mts files, but this is in the megabytes, so means either an extremely large build, or a build that just includes lots of node defs. Both can still be saved to lua tables, if not spawnable inworld. Once a lua table, you open and edit to the above mentioned format for better results.

How much memory does your computer have? Are you using LuaJIT? Can you spawn the schematic in question with either WorldEdit or HandleSchematics? What other mods are you currently loading that might be consuming memory?

Hope this gives you some food for thought, as well as, maybe a tool or two.

Shad

User avatar
DrFrankenstone
Member
Posts: 231
Joined: Tue May 24, 2016 05:36
GitHub: treer
Location: Australia
Contact:

Re: Large schematics are spawned incompletely

by DrFrankenstone » Post

MirceaKitsune wrote:Has there been any progress on this issue recently? I'm concerned about my Structures mod and the plans I made for it, which rely on the hope that this breaking problem can be solved.
ShadMOrdre wrote:The spawning uses place_schematic in a minetest.after(5 seconds) call.

Maybe it is a combo of both actions
I meant to reply to MirceaKitsune before... when I said it improved after moving the place_schematic() call inside a minetest.after(0.1, …), I should clarify that I've never seen the problem occur since.

Like with ShadMOrdre, It's hard to know whether there are other necessary parts to this solution (for example my minetest.after(0.1, …) call is being made from inside the on_generate/emerge code, perhaps that matters, perhaps it doesn't), but as hacky as mysterious it is, the minetest.after() solution does appear to be a solution.
Last edited by DrFrankenstone on Sun Jun 02, 2019 06:03, edited 1 time in total.

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: Large schematics are spawned incompletely

by ShadMOrdre » Post

I should also note, since I failed to clarify this point.

The Towns_Royal mod always generates the schematic when I pregenerate the area.

In my lib_materials and lib_ecology mods, there is an /emerge_area x y z x y z chat command that I frequently use to pregenerate the map, to test that things are generating as expected, without my player flying around being a determining factor. It also speeds up the client render when I do fly around, since the world is generated, all the client/server have to worry about is actually sending data, rather than generating and sending data. On my github, the emerge_area chat command also exists in a standalone mod called lib_emerge.

User avatar
MirceaKitsune
Member
Posts: 924
Joined: Sat May 21, 2011 22:31
GitHub: MirceaKitsune
IRC: Taoki
In-game: MirceaKitsune
Location: Romania, Bucharest

Re: Large schematics are spawned incompletely

by MirceaKitsune » Post

A few clarifications I should add:
  • The schematics I'm spawning are always mts files. They are not stored as Lua tables.
  • In my own test, when moving the minetest.place_schematic_on_vmanip function to minetest.after, the problem still persists. This is the case even if I use a very long delay, like 10 seconds or more after on_generated makes the call.
  • I don't know if this has any relevance. But before spawning the schematic I also clear the cubic area in the vmanip object, as well as generating the ground under it. I strongly doubt this has any relevance however... especially since I only fill and clear the area below and above the structure, not through it in the zone where it's spawned.

User avatar
Miniontoby
Member
Posts: 616
Joined: Fri Mar 01, 2019 19:25
GitHub: Miniontoby
IRC: Miniontoby
In-game: Miniontoby
Location: The Netherlands

Re: Large schematics are spawned incompletely

by Miniontoby » Post

You can use the Worldedit mod
Working on mtctl ---- Check my mod "Doorbell" -- Stay safe

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests