[Mod]FFIopt [ffiopt]

EvidenceBKidscode
New member
 
Posts: 5
Joined: Tue Feb 05, 2019 13:33

[Mod]FFIopt [ffiopt]

by EvidenceBKidscode » Wed Apr 17, 2019 14:41

FFIopt optimizes access to large amounts of data via the Lua API and reduces Lua memory footprint.

License: LGPL v3

Dependancies: None of mods but needs engine to be patched

Source: https://github.com/EvidenceBKidscode/ffiopt

Description

Lua API methods that transmit large amount of data (VoxelManip:get_data, PerlinNoiseMap:get3dMap_flat) copies data from C++ structures to Lua tables. This copy is quite slow and consumes a lot of Lua memory.

FFIopt replaces this copy by a direct access from Lua to C++ data using FFI library. The engine has to be patched to expose some C++ pointers to Lua API.

If engine correctly patched, FFIopt hacks VoxelManip and PerlinNoiseMap to use FFI rather than Lua tables.

Result is :
  • Faster data retreiving and storing ;
  • Slower table access (essencially due to use of metatable avoiding mods code change) ;
  • Lower Lua memory footprint (avoiding many OOM errors) ;
How to use it

  • Use LuaJIT.
  • Patch the engine to add VoxelManip:get_data_ptr, PerlinNoiseMap:get_result_size and PerlinNoiseMap:get_result_ptr methods to the API.
  • Install and enable ffiopt mod.
  • On starting Minetest, ffiopt will tell if it is able to optimise voxel manips and perlin noise.
  • Nothing to change in mods code.

Experiments

FFIopt can be tested through ffiopt_demo mod. Here is what we obtained.

Voxel Manip

Without FFIopt, in the demo, 30 % of the time is spend in get_data and another 30 % is spent in set_data. Lua processing takes 20 % and read/write the remaining 20 %.

So most of the time is spent in converting C++ data to Lua table.

With FFIopt, Lua takes 68 % of the time and read/write 32 % (get_data and set_data don’t do anything).
Lua processing time is increased by 75 %, but total time is reduced by 50 %.

Furthermore, without FFIopt, launching demo 9 times leaded to 3 OOM crash. With FFIopt, we have been able to launch more than 20 times the demo with no crash.

Perlin noise

With FFIopt, Lua processing increased by 50 % but total time reduced by 15 %.

Lua time is increased mostly because of the use of a metatable. This can be avoided but the mod code has to be changed (C++ indexes starts from 0, not 1) and secured to avoid out of range indexes.
 

User avatar
runs
Member
 
Posts: 372
Joined: Sat Oct 27, 2018 08:32
GitHub: runsy

Re: [Mod]FFIopt [ffiopt]

by runs » Wed Apr 17, 2019 16:07

Why is not this in the official engine yet?

Are you a https://kidscode.com developer?
 

User avatar
Pyrollo
Member
 
Posts: 260
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj
 

User avatar
Festus1965
Member
 
Posts: 957
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: [Mod]FFIopt [ffiopt]

by Festus1965 » Thu Apr 18, 2019 09:37

I am using it actual, got the patch done, mod active
thanks, I was reading that about the FFI, but no good enough to make something ...
I own minetest.one. Server Asia Thailand Minetest.one (5.0.1) via server.minetest.one:30001 using [Mod]s and build some own Mods.
 

User avatar
runs
Member
 
Posts: 372
Joined: Sat Oct 27, 2018 08:32
GitHub: runsy

Re: [Mod]FFIopt [ffiopt]

by runs » Thu Apr 18, 2019 10:02

Festus1965 wrote:I am using it actual, got the patch done, mod active
thanks, I was reading that about the FFI, but no good enough to make something ...


You said that consumes much more memory...
 

User avatar
Festus1965
Member
 
Posts: 957
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: [Mod]FFIopt [ffiopt]

by Festus1965 » Thu Apr 18, 2019 10:21

runs wrote:You said that consumes much more memory...
Nope,
I wrote that compare to just before used configuration for 6 days, the minimum was 66 MB,
then the one just for a view hours used 51 MB at start (no [Mod] time, updated [Mod] EUban),
and this new one used once (mean not confirmed, as also [Mod] time is back) was using more, as 78 MB
but I updated the collect-garbage result after one hour and there are 52 MB used now.

So as there are only 2 changes left: update [mod] EUban and new [Mod] ffiopt - what is the reason ?
I think that might be more about better EUban, but how often I have to start both versions (with/-out ffiopt) to be sure ?
I own minetest.one. Server Asia Thailand Minetest.one (5.0.1) via server.minetest.one:30001 using [Mod]s and build some own Mods.
 

User avatar
Festus1965
Member
 
Posts: 957
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: [Mod]FFIopt [ffiopt]

by Festus1965 » Fri Apr 19, 2019 01:23

but I found another problem
description here: server ffiopt and worldedit seam have a problem with other

WorldEdit didn't work when ffiopt true
I own minetest.one. Server Asia Thailand Minetest.one (5.0.1) via server.minetest.one:30001 using [Mod]s and build some own Mods.
 

User avatar
Pyrollo
Member
 
Posts: 260
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: [Mod]FFIopt [ffiopt]

by Pyrollo » Fri Apr 19, 2019 06:40

Mmmhh..

Worldedit uses voxelmanip in a non usual way and does not use VoxelManip:get_data. Some change required to make it compatible.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
Pyrollo
Member
 
Posts: 260
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: [Mod]FFIopt [ffiopt]

by Pyrollo » Fri Apr 19, 2019 06:53

New issue opened: https://github.com/EvidenceBKidscode/ffiopt/issues/1

If you have any issue, please don't hesitate to report them in Github.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
pgimeno
New member
 
Posts: 2
Joined: Sun Apr 21, 2019 16:18
IRC: p_gimeno
In-game: pgmine

Re: [Mod]FFIopt [ffiopt]

by pgimeno » Sun Apr 21, 2019 17:37

Sorry, I'm no longer registered in Microsoft GitHub and I'm not planing to return, so I can't submit issues there.

I see several issues with this mod.

First and foremost, security. I see two issues in this respect:

- It's exposing the FFI interface publicly, which basically defeats the purpose of having mod security. Any mod can, through the exposed FFI interface, call any C library function including, but not limited to, fopen and fwrite.
- Besides that, the pointer to the data is available in the table itself. This makes it possible (through use of large or negative indexes) to access any arbitrary memory position, which could potentially expose passwords or other stuff to untrusted mods.

Next is the use of a long int, and subsequently a Lua number (double precision float), to hold a pointer. This is not portable, and it may cause issues in some platform. It may turn out, by pure luck, to work fine in all platforms that Minetest supports, though.

Then, the benchmarks do not use the buffer version of the functions, which are designed to prevent object creation and internal reallocation, therefore it's not really a very fair comparison for well-written mods.

Lastly, it does not patch the functions with the new spellings: get_2d_map_flat et al.

I wrote a PR for Minetest that exposed C functions that could be called through FFI, to let modders handle this manually, but it was not taken with a lot of enthusiasm by the core developers. See https://github.com/minetest/minetest/pull/6863 (all appearances of 'ghost' I can see as of this date are my posts).
 

User avatar
rubenwardy
Moderator
 
Posts: 5731
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [Mod]FFIopt [ffiopt]

by rubenwardy » Sun Apr 21, 2019 18:05

pgimeno wrote:I wrote a PR for Minetest that exposed C functions that could be called through FFI, to let modders handle this manually, but it was not taken with a lot of enthusiasm by the core developers. See https://github.com/minetest/minetest/pull/6863 (all appearances of 'ghost' I can see as of this date are my posts).


I really like this feature and it had the potential reduce OOMs and improve performance - but you yourself said that the performance improvement was negligible
 

User avatar
pgimeno
New member
 
Posts: 2
Joined: Sun Apr 21, 2019 16:18
IRC: p_gimeno
In-game: pgmine

Re: [Mod]FFIopt [ffiopt]

by pgimeno » Sun Apr 21, 2019 18:16

rubenwardy wrote:I really like this feature and it had the potential reduce OOMs and improve performance - but you yourself said that the performance improvement was negligible

Not negligible, but small in comparison with the processing time required in most use cases (Lua mapgens that process noise for landscaping).

That was actually paramat's point: https://github.com/minetest/minetest/pu ... -354859130 and I agreed (edit: https://github.com/minetest/minetest/pu ... -355646707), but even then I see value in this, because of moving the allocation outside of Lua memory.
 

User avatar
Pyrollo
Member
 
Posts: 260
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: [Mod]FFIopt [ffiopt]

by Pyrollo » Sun Apr 21, 2019 20:43

Hi pgimeno, thanks a lot for your feedback, many interesting things :)

We'll study that closely, trying make mod evolve.

pgimeno wrote:Then, the benchmarks do not use the buffer version of the functions, which are designed to prevent object creation and internal reallocation, therefore it's not really a very fair comparison for well-written mods.


I did not take care of this aspect in the benchmark because in various studied options, it turned out that every time passing values through Lua stack was used to retrieve map data, things were slow (tryed to implemet something like get(index) on the C++ side but the get argument value/returning map value took almost the same time than populating the table).

For our purpose (changing many nodes at once) using FFI had a noticeable change on performances (but at first we disabled the light calculation on save, way to slow and not necessary for us).

On a memory point of view I totally agree with you. Data should not be duplicated to Lua memory.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 


Return to WIP Mods



Who is online

Users browsing this forum: No registered users and 5 guests