ABM, optimization and dynamic_liquid [core dev wanted! ;)]

[cronvel]

Hello,
Like many have already asked, I need some clarification on how ABM run in the engine side, and how to make them efficient when modding.

Context: I'm really fan of @FaceDeer mode dynamic_liquid, but as a side-effect, it produces nasty lags. I already tried few optimizations, but it didn't do much (in fact, I see no difference at all). Since I'm addict to this mod, I just can't turn it off, the world seems so boring without it...

The questions (core dev wanted!):

• What have the most impact on the performance?
  • The number of node having ABM even if the ABM actually change nothing (exit condition triggered)?
  • The number of actual changes produced by ABM?
  • Rather than the number of node, the number of map block that have changed? (i.e. when a node change, is that only node that is transferred to clients, or the whole block it is part of)
  • Is the lag server-side (internal computing) or due to client-server communications (lot of things to send over the network to synchronize clients)?
• Why dynamic_liquid is so intensive for the server (the mod is already optimized), while at the same time, flowing liquid (which is an engine-stuff) does not need much resource? To me, making flowing liquid is even more complicated than dynamic_liquid behavior, and flowing liquid is a sort of ABM (even if it's technically not). Why flowing is so much optimized compared to a an optimized Lua ABM?
• Is there any chance/plan that some kind of dynamic liquid behavior could be implemented directly inside the engine?

What I have done to optimize dynamic_liquid (without success): To optimize this mod even better that it was, I have added another node type, that I named block_water. An ABM with a very high delay (20 seconds) turns source_water into block_water if the source_water have no air or flowing_water as its neighbors. Block_water does not have the dynamic ABM. Once in a while (every 30 seconds), I check if those block_water have air or source_water nearby to turn them again into source_water. The idea was to reduce the number of dynamic candidate. For ocean, it probably remove 80% of dynamic node. Alas, it does not do much improvements...

[Krock]

So you summon the witch who knows about the black magic spells... alright.

but as a side-effect, it produces nasty lags.

In which cases does it lag really hard? ABMs are run each second unless you tweak the settings. Same for liquid updates. You can set both to a lower value to get a faster "flow".

[*]What have the most impact on the performance?

A few basics for understanding what's happening in the code (entire function).
1) There's some node caching, but the final effect is negligible (why? see below).
2) ABMHandler::apply is called for each active mapblock (next to players, usually)
3) Each node is checked whether it has ABMs assigned to it (3x for-loop)
4) Each ABM is run and checked for its probability (chance value) and neighbour nodes (3x3x3-1 checks)
5) If everything's fine, the Lua callback from the ABM definition is run

So in the worst case there are 1 + 3 + 1 + 3 loops that have to be executed each second. In a test world this takes ~60us to execute it each second. See F6 profiler "SEnv: modify in blocks avg per interval" (average value).

What likely matters here is the amount of code executed in the Lua callback.

Things that make ABMs faster (top = most effective)

• Either use LBMs or node timers to replace ABMs (saves up to 8 loops)
• Decrease active_block_range to 2 (saves 98 active mapblocks)
• Increased "chance" or "interval" (saves up to 3 loops, skipped neighbour lookup)
• Slim "action" callback Lua function (saves time for executed ABMs)
• Less node replacements in Lua (saves client mapblock mesh creation time and server-side map saving)

[/list]

No, neither network nor client lag have an impact on ABM execution.

[cronvel]

Thanks for the reply ;)

Also I'm still surprise that flowing liquid is so easily handled by the engine when dynamic liquid cause so much computing. I was told that Lua was really fast (something like 50% of C, which should be good enough).
Also, is there any chance something like dynamic liquid make it to the engine?

[Krock]

Also, is there any chance something like dynamic liquid make it to the engine?

There was a finite water and rudimentary weather system in 0.4.7. It was removed later on due to lack of maintenance, API functions and various bugs. Some water functionalities could be added to the engine, but these would unlikely be accepted on public servers due to heavy mapblock modifications (database writes) and rather slow fluid calculations.

[FreeGamers]

I read in the code of dynamic_liquid, in the comments, that FaceDeer basically hooked into the MT engine calculations for water ABM to determine where to move nodes. I haven't looked at it recently, but I think it was rather efficient for what it did. I've been running it on my server for months and have been quite happy with it. I don't think performance is too bad with it as the ABM and liquid interval are normally set to 1sec anyways.

[paramat]

> Also, is there any chance something like dynamic liquid make it to the engine?

Highly unlikely, as it causes so much problematic behaviour. Any mapgens with sloping rivers would drain away and become empty. Lakes would eventually drain away into the tunnels below. There is also no need for it. Best optional and left to mods.

> There was a finite water and rudimentary weather system in 0.4.7. It was removed later on due to lack of maintenance, API functions and various bugs.

But mostly because core dev 'hmmmm' (our previous mapgen dev) rightly realised it was a proller nightmare that would create a lot of mess and complexity. The Freeminer MT fork was the proof of the mess it causes in worlds.

[Lemente]

Hi !
I am also very interested in the simulation part of Minetest, and I also tried to "improve" dynamic_liquid.

Here was my idea: when emptying a lake in a cave below it, it goes quite fast at the beginning, but slows down. This is due to the way water nodes move randomly horizontally.
I decided to add "momentum" to water nodes. Basically, it a nodes start moving in a direction, it will continue in the same direction until it reaches an obstacle. Result : water nodes always start moving in the direction of a newly made hole at a bottom of a lake, and water settles a lot faster. The cost is a more complexe ABM (but it also cut down some calculation, since water nodes first try to go in the same direction), and a lot of metadatas.

I made some more improvements :
- if a water node goes back on its track (or any water node track actually), it tries to move sideways. This to avoid water nodes getting stuck in a loop. This means water nodes also leave an invisible trail of metadatas behind them
- falling water nodes erase metadata above them, because that information isn't relevant anymore. So a lake emptying itself will remove most of the metadatas.

This method allows to slow down the ABM, while getting a better result for large bodies of water. But doing so would also slow down small simulations.

We could solve that by using your method of creating a different water node when it is "isolated" (less than 3 water neighbor?), and make its ABM much faster.

Another idea I had was based on the concepts of lossless/lossy (thing png versus jpg format). Instead of making a lossless water simulation, we could make a rougher one, that moves "chunks" of water at a time, with the risk of loosing some of them. This could be limited to the mapblocks surrounding the player, while keeping the lossless one for the chunk the player is currently in. (is it posible to mix ABMs that way?)

Another idea to test: make water nodes fall like sand.

I haven't shared my code yet. I am thinking of doing that if anyone is interested (and if I am allowed to do so, I haven't checked yet)

edit: here is my fork of the dynamic liquid mod https://github.com/Lemente/dynamic_liquid