[Mod] Dwarf Fortress style caverns [dfcaverns]

[Germain]

Don't waste your time. I will try again.

I ever had subterannée mod.

So, il will reinstall this, and mapgen_helper.

Thanks for these indications.
I will try tomorow.

See you later

[argyle]

This mod is absolutely gorgeous! Well done.

[Germain]

Hi!

I installed the mod subterragen refactor, and uncommented my lines, and all seems works fine!
https://github.com/minetest-mods/subter ... e/refactor

[ThorfinnS]

Absolutely stunning! Pretty much exactly what I was picturing in my mind while playing DF. Except these cave spiders are pansies. ;-) Looking forward to mining some goblinite.

My only "complaint" is that in some of the caverns there is so much flowing lava that it slows down my sacrificial laptop. (An old pentium that I use to connect to public wifi and don't really care if it gets ransomwared.) But on my desktops, and all but a few of the older machines in the computer lab, works great.

[ThorfinnS]

Quick question. When I load using the minetest single player launcher, and I enable the three mods dfcaverns, mapgen_helper, and subterrane, then save, if I go back and look, it has disabled mapgen_helper and subterrane.

I see you've put the latter two into the modpack, but there's the same behavior there. Enable modpack does not enable those two mods, and while I can enable them manually, it won't save those changes. I don't think it's causing problems; the caverns are clearly different, so the mapgen must be working. It is throwing errors, though:

Code: Select all

2019-10-22 10:46:19: ERROR[Main]: mod "df_caverns" has unsatisfied dependencies:  "subterrane" "df_mapitems"
2019-10-22 10:46:19: ERROR[Main]: mod "df_mapitems" has unsatisfied dependencies:  "subterrane"

Any guesses?

[ThorfinnS]

And now I don't understand what happened. The machine that was running fine now has all the dfcaverns stuff as unknowns. All that's happened with that machine is I've run a script to update all the gits. Now even with base MTG and ONLY dfcaverns, it's malfunctioning as in my last post. And the new servers I've started won't load, either. Weird, since dfcaverns does not have any updates since then.

I've tried new install with fresh download of the official release of mt and a new clone of dfcaverns, same thing.

Any guesses?

[Nathan.S]

If you have a mod in a modpack that has the same name as a mod not in the modpack they sometimes have problems loading. I've experienced this myself.

[ThorfinnS]

If you have a mod in a modpack that has the same name as a mod not in the modpack they sometimes have problems loading. I've experienced this myself.

Thanks!

Turns out that was it. My update script downloaded everything in the git, including the blank entries for the two mods within the dfcaverns modpack git. Using git clone ignores them. Turns out git pull and git fresh-->git merge might operate slightly differently.

Anyway deleting the two blank subdirectories seemed to have fixed it. Haven't dug deep enough to be sure, but the error messages are gone.

[FaceDeer]

A heads up, I'm gearing up for a major new release of dfcaverns. It adds some new content beneath the Underworld layer, along with a new mechanism for penetrating the Slade floor of the Underworld to get at it. This is not at all Dwarf Fortress "canon", but I was given a very nice pile of art assets by ClockGen and was inspired to run with it.

It currently lives in this branch of dfcaverns, make sure to get the latest mapgen_helper and subterrane code to go along with it. DFCaverns now also makes use of namegen and named_waypoints if they're installed, it uses them to add some landmarks in the Underworld. If you also happen to be using magma_conduits (and with a Dwarf Fortress theme you really should :) then those mods will also give names to the peaks of the volcanoes it generates.

It's all starting to come together. :) I still want to do some testing before I merge that branch into master, I'll post a more detailed breakdown of the update here when I do.

[FreeGamers]

Seeing major CPU usage on the new biome. Trying to check it out but I have to step away for awhile apparently. Unusually high. just an early warning. Still testing. Looks very cool so far though :) Maybe add some fireflies if you feel they would fit in? It could illuminate things above the flowers a bit more.

[FaceDeer]

During my local testing I've been starting new worlds and then emerging (-2000, -4000, -2000) (2000, 200, 2000), and I haven't been seeing significantly higher mapblock generation times for the primordial layer compared to the others. However, there are a number of additional post-mapgen processes that I'm doing in the primordial layers that only trigger when the player actually enters the mapblock and it becomes "active". Those wouldn't be accounted for in my emergeblocks testing approach. The jungle biome in particular - I'm using node timers to convert the interiors of the giant plant material columns into "packed root" material and some map chunks are going to have a large amount of that stuff in them. Are you testing these areas by noclipping your way down through them at high speed? That might be causing a worst-case scenario that wouldn't be nearly so bad for a real-world situation where players are climbing around more slowly. I can't think of any unusual CPU activity in those layers beyond that, I'll do a little profiling and see if I can spot anything.

I could spread the node timers out a lot more, which should even the load out over time, or perhaps I should ponder a different approach. Hm.

Fireflies sound like an excellent addition to the jungle biome, that should be straightforward enough.

[FreeGamers]

I pushed it into my master branch anyways. It was pretty damn slow but I can't pinpoint what caused it since i updated a bunch of mods at the same time. Subterrane, magen helper, and dfcaverns, nether. Nether and primordial biomes were intersecting due to my configs so I've added a bit more buffer between them to try to prevent this.

[FaceDeer]

Primordial-specific code will only run if the player is in the primordial stratum, so hopefully if that is the code at fault it will at least not get in the way most of the time.

I'll continue to profile and optimize dfcaverns code where I can over time. Hopefully I'll find some clever tricks.

[FreeGamers]

Still getting a really weird anomaly with dfcaverns_primordial. The CPU gets pegged at 100% for a long period of time, mapgen seems to stop, nothing appears in the primordial biome (I wasn't willing to wait longer than 10 minutes), and even after the player leaving the game, the server was still at 100% utilization for another 5 minutes. Eventually it petered out when I left. But I'm going to mention this again because I can't see this boding well in a server game which will lag a lot of players out or just have them leave. We'll see when it happens I guess. I've been dealing with some lag problems in my game since I've added this, but it could just be time to find more optimizations. I need to find a good way to see cpu utilization and schedules. I think the MT client as a built in profiler but i havent seen anything server side to use yet. I'd like something simple to use instead of a advanced web ui and hooks system. I'll be looking for that soon here.



Also, don't worry about finding workarounds to your unbreakable nodes, I updated the staff of earth to include the df_underworld_items on its blacklist. Its fixed in the context of my game/server. Not sure if you really need to include that in your scope, it seems more like a rare special item case that just adds stuff to player inventory. Up to you if you want to include those types of situations.

EDIT: even after taking 10 minutes to write this, the biome in the testing server was not loaded fully. the CPU was still churning away at it. My CPU is not a slouch. Its a 25W 8th Gen quad core with eight threads. SSD too. The stable server system is about similar performance. 45W 3rd Gen quad core with 4 threads (HT off) and SSD, 8GB RAM. It should be more than sufficient hardware wise. Are you seeing the same problems as me? Maybe its a stale version of MT 5.2.0-dev? from a couple weeks ago.

[FaceDeer]

I've been doing some mapgen profiling in an effort to track down problems like this, and honestly, I'm just not seeing it. At least not in terms of the work that mapgen itself is doing. In the test runs I did yesterday the primordial cavern layer had a somewhat fatter "long tail" of map chunk generation times than the other cavern layers but not to a drastic degree. Certainly not in the tens of seconds, I maxed out at 2.6 seconds for the longest-running map chunk (compared to a max of 2.9 seconds for cavern layer 3, the most expensive cavern layer). Chunks taking that long were quite rare, there were only a handful in each case.

If you want to try out the metrics I've added in the past few days, update to the latest dfcaverns/mapgen_helper/subterrane versions and then set "mapgen_helper_record_time" and "mapgen_helper_persist_recorded_time" to true. Then start up a world and "/emergeblocks (-2000, -4000, -2000) (2000, 200, 2000)" to generate the standard 16 cubic kilometer block that I've been using as my standard test run. Or shrink its X/Z extent if that takes too long, a full 16 cubic kilometers can be an all-day thing.

I'll rig up some test parameters you can tweak to turn off various different features of Primordial to see if we can narrow down what's going on. I'm still thinking it's some post-mapgen process that's doing it, such as the giant mycelium growth or the plant-matter-to-packed-root conversion steps, since those are things that are unique to Primordial. My computer is comparable in specs to yours, though I'm running Windows 10 so there's going to be a lot of differences in the OS and the compiled build. I'll grab a latest build of MT 5.2 and see if I can provoke these problems locally.

The built-in profiler does have server-side aspects to it, it can track the cost of ABMs so it'd be worth turning that on and seeing if any leap out as candidates.

As for the unbreakable node thing, I think I've already fixed it. I didn't have the nodes' diggability properties set up very well.

Edit: Since I continue to be most suspicious of the giant mycelium mapgen growth as the source of this CPU spike, I've changed it to trigger off of node timers that are spread out over a period of time instead of triggering off an ABM immediately. So if you update to latest and can't reproduce the problem any more, I declare victory. I hope.

[FreeGamers]

Noticeably improved. Still a bit weird though. I'll have to dig into a profiler to get feedback. Thank you.

[FaceDeer]

Well, that's promising.

I've just added a setting you can try fiddling with, "dcaverns_giant_mycelium_timer_spread", which defaults to 10 seconds. When the fungal caverns get generated a timer is started for each giant mycelium starting node with a random delay of 1 to this many seconds, which spreads out the computational load for generating those fungal networks across a greater range of time and allows other processes to interject. Setting that higher might help more.

[FaceDeer]

And I just revamped the code that the giant mycelium nodes use to actually perform their "growth" operation and hopefully made it a lot more efficient. It's still a pretty complicated routine and I didn't find a way to reduce the map info reading/writing part but there's a lot fewer tables and dereferences now. So the CPU load of actually running the giant mycelium code should be less now.

I think unless there are any more showstoppers I'll likely merge the Primordial branch into master soon and tag a release. I could continue tinkering with little details forever but I want to move on to other mods for a bit now. :)

Oh, and speaking of which, last night I banged together a mod that might be of interest for folks using giant cavern mods like this one: torch_bomb. It's an exploding block that sprays torches into the walls (and ceiling and floor) all around it. The mega-bomb version has a torch-embedding radius of 120 meters, should be enough to handle everything within reasonable visual range of the player inside one of the giant caverns of dfcaverns. Thought I'd mention it here.

[FaceDeer]

Rather than continue endlessly fiddling, I've gone and merged the Primordial branch into master, and updated the content database with the latest. Version 2.0: Primordial Caverns has been released.

Whew. Now to await the deluge of bug reports. :) Though much thanks to FreeGamers for testing this out on his server, their bug reports (and the bug reports of their players) have been immensely helpful already.

And thanks to ClockGen, without whose giant pile of art assets I wouldn't have been driven to do this new cavern layer in the first place.

[MisterE]

Please give us a hint as to how to solve the puzzle seal. In the code I can see that it has to do with color groups, and it is obviously a combo. It looks like there is supposed to be a way to get it to tell you the combo in chat, but that command is commented out.

[FaceDeer]

It's as easy as 1, 2, 3. Or rather, as easy as 0, 1, 2, 3, 4, 5, 6, 7.

Count the sparks. The decorative art in the formspec has a clue as to what that means.

[FreeGamers]

Still getting massive performance drops on new cave generation. I think its happening for any new dfcavern generated. Could this be from subterranean or mapgen_helper?

[FaceDeer]

There is one change I made to subterrane that might account for an across-the-board increase in resource usage. Previously, subterrane would only "carve out" cavern within the boundaries of the map chunk it was currently generating. I updated it so that it would carve out cavern spaces in the edges overlapping adjacent map chunks as well. This is how most mapgens operate, it allows for things like trees to be placed on the edge of a map chunk without it getting cut off by an imaginary wall.

I made the change because this was much more apparent in the Primordial fungal layers with its giant 16-node-tall mushrooms, they were getting sliced off quite frequently and blatantly. But it also fixed that issue on all the other layers, it used to be that you'd find other large fungi cut in half by map chunk boundaries in a similar manner. It was just a bit less obvious because the fungal trees in the upper layers are smaller.

The net result is that parts of mapgen are run twice on regions around the edges of a map chunk, increasing the number of nodes that are processed with each chunk. I can't think of an easy way around this extra cost, though, while still fixing the bug that this extra work is used to fix. I'll take another look at the code and see if there's more optimizing I can do in that area.

Ironically, the Covid-19 lockdown has come at a time when I actually have a lot _less_ free time for Minetest modding. You'd think this would be the ideal time for idle stuff like this. Alas, life is complicated sometimes. :)

Oh, MisterE, did my hint help? Should I give more? It's always difficult figuring out how hard a puzzle is when you're designing it from the inside, I was worried I might have made this one too tough. I didn't want it to be a walk in the park either, though - on the journey through life one should slow down a bit to smell the flowers. Getting through the Slade should be an _accomplishment_. But if nobody's figuring it out that's no fun either, I'll try to think of more ways to insert clues into the game.

[FreeGamers]

FaceDeer, I understand about the drop in productivity despite the stay at home stuff going on. I'm facing a similar situation as well. I've had a huge increase in work for myself and for my job the last week or two. Hasn't left much time for my modding obligation. I basically don't stop working until I pass out at night. Its a very good thing though for my life though. Glad you are doing well.

[Hamlet]

All-in-one download: DFCaverns Modpack v2.0 (get this if you just want to install it and run)

Minetest + Minetest Game v5.2.0

Code: Select all

WARNING[Main]: Undeclared global variable "S" accessed at ...quest/mods/MAP/dfcaverns/df_primordial_items/doc.lua:5
ERROR[Main]: ModError: Failed to load and run script from /minetest/bin/../games/hamlets_quest/mods/MAP/dfcaverns/df_primordial_items/init.lua:
ERROR[Main]: ...quest/mods/MAP/dfcaverns/df_primordial_items/doc.lua:5: attempt to call global 'S' (a nil value)
ERROR[Main]: stack traceback:
ERROR[Main]: 	...quest/mods/MAP/dfcaverns/df_primordial_items/doc.lua:5: in main chunk
ERROR[Main]: 	[C]: in function 'dofile'
ERROR[Main]: 	...uest/mods/MAP/dfcaverns/df_primordial_items/init.lua:6: in main chunk
ACTION[Main]: Server: Shutting down