Threading Lua API

[Starbeamrainbowlabs]

I'm pretty sure that Minetest doesn't present a threading API to mods in Lua. For mods like my WorldEditAdditions though, such an API would be really quite useful to run CPU intensive tasks without blocking the main server thread.

To this end, I propose a threading API. To spawn a new thread, you might do this:

Code: Select all

local mydata
-- .....
minetest.thread("path/to/file.lua", mydata, local function(result)
	-- process result in callback here
end);

Then, in the new thread, it might look like this:

Code: Select all

local mydata = thread.get_data() -- Unsure about this - any better ideas would be great!

local result = {}

-- do stuff here

return result

This would come with a number of limitations:

• The main server thread cannot access any data inside the newly thread
• Likewise, the new thread cannot access anything in the main thread
• The minetest API is unavailable in the new thread (except for thread.get_data() or however the new thread gets the data passed to it
• Data is copied when being transferred from one thread to another (zero-copy would be nice, but I recognise that would probably be very difficult to implement)

A potential future direction would be a messaging style-system (allowing for mod to keep state / run tasks on a persistent background thread, and message the main server thread to interact with the Minetest world).

Thoughts?

[rubenwardy]

for reference, the main menu has a multithreading API for Lua and it works like this:

Code: Select all

local function func_to_run_in_thread(params)
    print(dump(params))
    --> { foo = "bar" }

    return "ok"
end

minetest.handle_async(func_to_run_in_thread, { foo = "bar" }, function(result)
    print(dump(result))
    --> "ok"
end)

[sorcerykid]

I think so long as the thread is only concerned with data processing (thus running within a sandbox, only having access to the builtin Lua functions and classes), then should be 100% viable.

More often than not, this would gain the most practical benefit from being able to manipulate the map itself. I believe that would require exposing a new API for mapblock mutexes. Not sure how complex that would be to implement, but it's sure to open up a lot of possibilities for maintenance-type mods, possibly eliminating the need for LBMs.

[Hybrid Dog]

I have created the function_delayer mod: https://github.com/HybridDog/function_delayer
It is some kind of scheduler which allows the modder to queue functions' executions so that the players do not notice a lag.
For example, I use it for a TNT mod where, when a lot of TNT nodes should explode all at once, the single explosions are enqueued instead of calculating all explosions at once. Testing has shown that it works well.
It does not use other processor cores but nonetheless allows computation-intensive map change operations. It does not require mutexes but a very big computation task needs to be split into subtasks; for example, a task which generates a big forest needs to be split into subtasks where each subtasks generates a part of the forest (e.g. a single tree or a small set of the trees).

[Starbeamrainbowlabs]

for reference, the main menu has a multithreading API for Lua and it works like this:

Oh wow nice! It's a shame that it's usable in the main game though. That looks like just the thing!

I think so long as the thread is only concerned with data processing (thus running within a sandbox, only having access to the builtin Lua functions and classes), then should be 100% viable.

More often than not, this would gain the most practical benefit from being able to manipulate the map itself. I believe that would require exposing a new API for mapblock mutexes. Not sure how complex that would be to implement, but it's sure to open up a lot of possibilities for maintenance-type mods, possibly eliminating the need for LBMs.

Yeah, data processing is the idea. Indeed, map manipulation is my primary use-case here. In WorldEdit / WorldEditAdditions, often you want to perform complex operations on large chunks of the world, so doing that in a separate thread would be awesome!

That idea for a map mutex api sounds very cool indeed too - I definitely agree on the possibilities there. Multi-threaded stuff like that might be somewhat complicated I'd guess (not a core dev, so I wouldn't know) and perhaps best considered as a separate feature for later on, but awesome nonetheless.

If LBMs could be parallelised this a paradigm like this, then perhaps the same could go for initial mapgen too. Again a separate feature, but it would really speed up initial mapgen if that kinda thing could be offloaded too - with the same restrictions of course.

[Sokomine]

Multi-threaded stuff like that might be somewhat complicated I'd guess

Yes, I'm afraid so. It isn't even easy to do that on the modding level.

If LBMs could be parallelised this a paradigm like this, then perhaps the same could go for initial mapgen too.

AFAIK mapgen was one of the first things that was parallelized.

As useful as all that is, it does get complicated when trying to create mods that are not limited to one mapchunk alone or that depend on the heightmap. I'm currently trying out ways to automaticly place a road at mapgen time that goes straight in one direction and adjusts to the height of the terrain. This mostly works (at least with mapgens where this approach does make some sense and the terrain isn't too extreme), but getting the connection to the next mapchunk smooth is rather tricky and involves more thinking and planning in the future.

[Starbeamrainbowlabs]

Yes, I'm afraid so. It isn't even easy to do that on the modding level.

Hrm, I thought so.

AFAIK mapgen was one of the first things that was parallelized.

As useful as all that is, it does get complicated when trying to create mods that are not limited to one mapchunk alone or that depend on the heightmap. I'm currently trying out ways to automaticly place a road at mapgen time that goes straight in one direction and adjusts to the height of the terrain. This mostly works (at least with mapgens where this approach does make some sense and the terrain isn't too extreme), but getting the connection to the next mapchunk smooth is rather tricky and involves more thinking and planning in the future.

Nice! Yeah, when mods come into it it's going to get complex I guess. I can see a use-case for mods that create custom biomes having a method by which they can e.g. place trees etc in parallel for example - but as you say something like that would require planning.

Perhaps sticking to a simple minetest.handle_async first would be a good place to start?