Advice on creating highly efficient mods

[calakin]

Hello,

I've known of Minetest for perhaps a year or so. I'm a student in software engineering, as well as a fan of games like Minecraft. I'm interested in working on mods of my own, both for fun and as practical experience.

I started this topic to discuss the best practices for making efficient mods. As I've been looking around the forums, I've seen a lot of talk about how some things like ABMs really decrease performance, for example. I haven't really seen a lot of explanation. Hopefully, some wise people will help to explain the efficiency of the different tools (like ABMs) available to modders, common mistakes that hurt performance, how to test mod performance, and so on.

I did a few brief searches, but I didn't see another topic like this one. If I have missed something, please include links here. One helpful page from this wiki I have found is below, but it doesn't really answer my questions specific to Minetest.
http://dev.minetest.net/Lua_Optimization_Tips

[Krock]

Hello,

The probably fastest way straight forward for more efficient mods is using LuaJIT, which increases the execution speed of the scripts overall. Later Lua versions (5.2/5.3) now also have a better performance than 5.1, but those are not supported in Minetest.

We have a profiler which can be enabled in order to measure the execution speed of a mod. It gives accurate information about the used callbacks, including average execution time and peak values. Sadly I could not find any better profiler documentation, so also try running Minetest from the terminal/console window if there's no other output when running the "/profiler <args>" command.

[calakin]

Thanks Krock!

I have installed LuaJIT on my computer now. It is added to my path (Windows). Does anything need to be changed in Minetest to utilize LuaJIT?

The profiler was very helpful!

[sofar]

viewtopic.php?f=47&t=19289 is relevant. You can also google for `site:forum.minetest.net mod performance` to get a few more related threads.

The reason ABM's are inefficient is because ABM's incur a non-avoidable CPU cost for zero changes made to the world.

To put it in english: ABMs require processing time, even though no actual Lua code will be run in many of the cases.

Most ABM handlers specify a `chance` value that is not `1`. This means that on each mapblock that is loaded, the engine needs to identify all blocks that match the ABM, and perform a statistical chance operation - roll a dice. If the chance roll fails, nothing is changed in the world, but the CPU has been used to perform a random number selection, which uses the RNG, and then some basic math. RNG processing is non-trivial. Matching nodes is non-trivial, doing this regularly for almost every loaded node is non-trivial. And worst of all, we have not even ran any lua code yet. This is all still within C++.

The overhead is exposed easily if you use a profiling method that exposes C++ function calls on top of Lua calls. Mod Profiling will only show you the overhead of the executed lua code, but it hides the C++ processing needed that happens for many more nodes than the lua code is run for.

Conclusion: mod profiling is incomplete and does not show the true cost of some mods.
Conclusion: ABMs incur a hidden overhead that may be significant

Now, I use ABMs in many of my projects. You can use them and use them safely and be aware of the potential cost. If you are not using **data** to do performance analysis, then just code first and optimize later. There are definitely designs that work better and can save you a lot of CPU time, such as replacing ABMs with nodetimers, but they may not cover all ABM use cases easily or even at all.

[Krock]

Does anything need to be changed in Minetest to utilize LuaJIT?

Sorry, I was being unclear there. LuaJIT is a library which must be built into your Minetest binary. If you are on Windows, then you're already using it. Check it easily by running the following command in your terminal/console window: (run in the directory where your "minetest" binary is)

Code: Select all

./minetest --version

If "USE_LUAJIT=1" shows up, then you've got nothing more to do. Otherwise check out how to build Minetest.

[xeranas]

Not sure if LuaJIT is what minetest mods developers should rely on if it is case that not every official build includes it. The only thing what matters is how well mod perform on user build not on developer (If you know what I mean).

From my experience you cannot control performance of minetest lua api e.g. minetest particles does not perform well (does not matter LuaJIT or not its up to game engine implementation) and as mod developer you only can limit usage of particles. Sometimes compromises needs to be done (e.g. throw out some fancy but optional features), limit invocations of heavy tasks, update state in bigger intervals and etc.

I suggest first write mod proof-of-concept with minimal effort, to check if minetest api is capable of creating functionality which you are looking, then create wip mod, ask people of feedback and only then invest time into optimizing if you feel that mod could perform better. Do not let your-inner-perfectionist stop you from creating your first mod ;)

[sofar]

update state in bigger intervals

ouch

Delaying work and batching it is, from a performance point of view, a terrible thing to do. You're actually postponing work and making the lua thread do more, causing a much more significant delay than if you had done a tiny bit of work right when it was relevant.

You should "race to idle" instead - don't wait for caches to get invalidated and forcing your CPU to load the needed content all the way from RAM, and possibly do this for 100+ items that need processing, while you had each item in the l2 cache when it last became active already.

Bottom line is that postponing work is almost always the wrong thing to do.

[xeranas]

I talking about how often you perform global step, not every situation needs update state 100 times per seconds.. Your recent posts are quite salty, you may need some vocations and chill after all this is just game and people crating mods purely for fun..

[sofar]

I talking about how often you perform global step, not every situation needs update state 100 times per seconds.. Your recent posts are quite salty, you may need some vocations and chill after all this is just game and people crating mods purely for fun..

You were reading unsalted engineering advice, straight from my own bucket of learning experiences doing this sort of thing for fun, and weirdly enough there's a company that pays me a salary for it.

Nothing personal about it, nor do I "need to chill". The thread was about discussing highly efficient mods, and I provided my professional insights for free. If you think they're too heated, that's your perception, but in no way my intent for you to perceive it that way. I'd love to go on vacation though!

Back to the topic, since that's actually interesting to me:

Nothing in minetest runs 100 times per second. The globalstep is 10, maybe 20x per second. Things like player objects are in hot cache and calling checks on player controls is cheap, so yeah, maybe that can be toned down a notch here or there without affecting game lag, but that's -from my experience- not where problems come from anyway.

Everything is a trade off when it comes to performance. Ultimately the most important metric is player enjoyment, and that correlates heavily with server responsiveness, not with server slowness, so no matter how you slice it, you want the server to respond to interaction quickly, not slower.

If there's anything I've learned about performance, it's that data is the only thing that matters. I've been shown to be wrong many times, and every time I'm wrong and the data shows it, I get excited because there is something new to be learned.

[calakin]

Hmm, I definitely appreciate all the discussion. I think it's possible xeranas was referring to situations where the function has a high probability of occurring, close to 1. In that case, the amount of work done by the server each call would be near the maximum amount anyway, so less frequent function calls would mean less work overall. Like sofar said, most functions has some context (probability or condition or something) that makes them more relevant, and delaying them sort of accumulates the work that must be done, which makes bigger delays when the function IS called. Pretty much it would be a smoother workload to do it more often. Am I understanding this correctly? It is probably oversimplified.

If that's true, would I be correct in saying the optimal interval for a function depends on the chance / what it actually does?

On a different topic related to performance, does it slow things down to register a ton of new nodes? By a ton, let's say 300 or something. I know it may slow it down slightly while loading the mod, but is there any other effect?

[Skulls]

... Things like player objects are in hot cache and calling checks on player controls is cheap ...

If there's anything I've learned about performance, it's that data is the only thing that matters. I've been shown to be wrong many times, and every time I'm wrong and the data shows it, I get excited because there is something new to be learned.

Question for sofar, if you have the time. Currently I'm trying to build an entity-component system for mobs. I've shifted my data in a bunch of Lua tables, the entities get an ID when activated, etc etc. The difference is that instead of putting "handling" logic in entity onstep function, I call one master "update" function which is registered as as global step. One dip into the mob code per server tick to do all the whatevers for all entities rather than each entity having its own onstep. Of course I have to handle loading and unloading but my thought was that switching from the C++ world to the Lua world was expensive enough to care.

Wet finger in the air opinion, if the logic of the code is basically the same and has to be done per loaded entity anyway, does global step vs entity onstep make any difference?

[Lejo]

If "USE_LUAJIT=1" shows up, then you've got nothing more to do. Otherwise check out how to build Minetest.

What does I have to do on Linux?
Does I have to install LUAJIT itself first?
EDIT: My server already uses it.

[sofar]

Wet finger in the air opinion, if the logic of the code is basically the same and has to be done per loaded entity anyway, does global step vs entity onstep make any difference?

The difference is that onstep is only called for active entities. You will need to run at least a check to see that you're not running your custom code for entities that have been saved to the map.

It likely is in the margins. I don't think that this is where the core spends most of its time, so any optimization in this area is likely not going to yield a large difference. My wet finger says it's probably more extra work coding.