3 Memory use optimisations for Lua mapgens

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

3 Memory use optimisations for Lua mapgens

by paramat » Thu Dec 15, 2016 05:57

Many players are reporting OOM (Out Of Memory) errors when using LuaJIT, which has a low memory use limit.
Lua mapgens can use excessive memory if not using these 3 optimisations, and most Lua mapgens, even most of mine, are not using these.

Examples of these optimisations are from my mod 'stability' https://github.com/paramat/stability which is currently the best mod of mine to demonstrate good practice in Lua mapgen.

//////////////////////////////////////////////////////////////////////////////

1. Perlin noise objects: Only create once

The noise object is created by 'minetest.get_perlin_map()'
It has to be created inside 'register_on_generated()' to be usable, but only needs to be created once, many mapgen mods create it for every mapchunk, this consumes memory unnecessarily.

Localise the noise object outside 'register_on_generated()' and initialise it to 'nil'.
See the code below for how to create it once only.
The creation of the perlin noise tables with 'get3dMap_flat()' etc. is done per mapchunk.

Code: Select all
-- Initialize noise objects to nil

local nobj_terrain = nil
local nobj_biome = nil

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   nobj_terrain = nobj_terrain or minetest.get_perlin_map(np_terrain, chulens3d)
   nobj_biome = nobj_biome or minetest.get_perlin_map(np_biome, chulens2d)
   
   nobj_terrain:get3dMap_flat(minpos3d, nvals_terrain)
   nobj_biome:get2dMap_flat(minpos2d, nvals_biome)

...
...

end)


////////////////////////////////////////////////////////////////////////////////////

2. Perlin noise tables: Re-use a single table

The Lua table that stores the noise values for a mapchunk is big, especially for 3D noise (80 ^ 3 = 512000 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Code: Select all
For each of the functions with an optional `buffer` parameter:  If `buffer` is not
nil, this table will be used to store the result instead of creating a new table.

#### Methods
* `get2dMap_flat(pos, buffer)`: returns a flat `<size.x * size.y>` element array of 2D noise
  with values starting at `pos={x=,y=}`
* `get3dMap_flat(pos, buffer)`: Same as `get2dMap_flat`, but 3D noise


Localise the noise buffer outside 'register_on_generated()'
Use the buffer parameter in 'get3dMap_flat()' etc.

Code: Select all
-- Localise noise buffers

local nvals_terrain = {}
local nvals_biome = {}

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   nobj_terrain:get3dMap_flat(minpos3d, nvals_terrain)
   nobj_biome:get2dMap_flat(minpos2d, nvals_biome)

...
...

end)


/////////////////////////////////////////////////////////////////////////////////////////

3. Lua voxelmanip table: Re-use a single table

The Lua table that stores the node content ids for a mapchunk plus the mapblock shell is big (112 ^ 3 = 1404928 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Very recently a 'buffer' parameter was also added to 'get_param2_data()', so is only usable there in latest Minetest dev or in Minetest 0.4.15 stable or later.

Code: Select all
* `get_data([buffer])`: Retrieves the node content data loaded into the `VoxelManip` object
    * returns raw node data in the form of an array of node content IDs
    * if the param `buffer` is present, this table will be used to store the result instead
* `get_param2_data([buffer])`: Gets the raw `param2` data read into the `VoxelManip` object
    * Returns an array (indices 1 to volume) of integers ranging from `0` to `255`
    * If the param `buffer` is present, this table will be used to store the result instead


Localise the data buffer outside 'register_on_generated()'.
Use the buffer in 'get_data()' or 'get_param2_data()'.

Code: Select all
-- Localise data buffer

local data = {}

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
   vm:get_data(data)

...
...

end)
Last edited by paramat on Sat Mar 17, 2018 05:49, edited 12 times in total.
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Thu Dec 15, 2016 20:30

Thanks for this guide. I think it would be useful to add this text to the Developer Wiki:
http://dev.minetest.net/Main_Page
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat
 

User avatar
Spaghetti Developer
Member
 
Posts: 32
Joined: Sun Dec 04, 2016 14:01
Location: Rome, Italy
GitHub: SpaghettiDeveloper
In-game: Spaghetti Developer

Re: 3 Memory use optimisations for Lua mapgens

by Spaghetti Developer » Sun Dec 18, 2016 18:00

Thanks Paramat! a long time I noticed this problem, the chink are struggling to generate and often they crash, then I have to walk to get closer so that you generate the missing area. I noticed a strong lag, due to the excessive vegetation in the jungle biome, and biome valley. Too many trees my friend! should not lower the density of vegetation? often biomes are full of trees excessively. This improves the generation of the chunk and making the most beautiful biomes and enjoyable, especially in multiplayer. I hope I explained myself :D
 

User avatar
Hybrid Dog
Member
 
Posts: 2606
Joined: Thu Nov 01, 2012 12:46

by Hybrid Dog » Sun Dec 18, 2016 20:48

> *** Note a buffer is only usable in 'get_param2_data()' in latest Minetest dev or in Minetest 0.4.15 stable ***

The buffer doesn't work for 'get_data()', so using it there doesn't hone memory usage, does it?
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Dec 19, 2016 02:13

The buffer does work for 'get_data()' and 'get_param2_data()'.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:00

Note i have been corrected on point 3, the initialisation of dbuf must define it as a table:

Code: Select all
 local dbuf = {}


Otherwise memory usage is not reduced.

Code: Select all
    -- Localise data buffer

    local dbuf = {}

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
       local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
       local data = vm:get_data(dbuf)

    ...
    ...

    end)
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:45

Just realised this probably applies to point 2 also:

Code: Select all
    -- Localise noise buffers

    local nbuf_terrain = {}
    local nbuf_biome = {}

    ...
    ...

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
       local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

    ...
    ...

    end)
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Apr 03, 2017 22:46

^ It does apply to point 2 also, updated first post.
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Sun May 07, 2017 14:19

paramat wrote:Feel free to, i have no wiki access.

Why not?
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Tue May 09, 2017 14:30

@paramat: There's a small mistake in your first post: It should be “minetest.get_perlin”, not “minetest.get_perlin_map”.

And I have another question: What do you have to say about the usage of perlin:get2d, as opposed to perlin:get2dMap or perlin:get2dMap_flat (same for 3D variants)? Are there any special optimization rules I should follow?

And when does it make more sense to use get2d instead of get2dMap?

Final question: How do I measure the performance / memory usage? I want to know if any of these optimizations actually worked.
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Wed May 10, 2017 05:13

I'm fairly sure it should be 'get perlin map', the lines in the code blocks are copied from working mods.
However a similar optimisation can be done when using per-point perlin noise instead of noise maps, see latest technic mod worldgen file.
'get map flat' is probably better then 'get map' as it seems a flat array is the most efficient way to access bulk noise data. Not sure what 'get map' is for, that's a question for hmmmm.
'get2D' is not bulk noise but noise calculated per-point, like Lua voxelmanip bulk noise calculation becomes optimal for large amounts of noise.
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Wed May 10, 2017 23:20

Okay, thanks. Just FYI, I have applied memory optimizations for the map generator in MineClone 2 because someone reported OOM bugs.

But I have no idea how to verify by fix. :-(
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Thu Nov 30, 2017 09:51

What do I do if players still report OOM bugs even if I have applied all 3 optimizations? :-(

Is there any way to debug OOM errors properly?
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
duane
Member
 
Posts: 1258
Joined: Wed Aug 19, 2015 19:11
Location: Oklahoma City
GitHub: duane-r

Re: 3 Memory use optimisations for Lua mapgens

by duane » Fri Dec 01, 2017 01:16

Wuzzy wrote:What do I do if players still report OOM bugs even if I have applied all 3 optimizations? :-(

Is there any way to debug OOM errors properly?


I just use collectgarbage("count") to check memory use before and after a process. That should give you an idea of where your memory hogs are. You might also want to stop and restart the garbage collector, just to be sure.
Believe in people and you don't need to believe anything else.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Fri Dec 01, 2017 04:12

The optimisations reduce the chance of OOM but it could still happen in some situations. Maybe just don't use LuaJIT?
 

User avatar
Wuzzy
Member
 
Posts: 3069
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Fri Dec 01, 2017 08:26

I have the impression that many people are reporting OOM problems while having LuaJIT enabled.

Does Minetest ship with LuaJIT by default or does it offer this option? If yes, it seems counter-productive to offer an option which you just publicly recommended to not use at all .
My projects: MineClone 2. Hades Revisited. Help modpack. A ton of other mods, see here.
 

User avatar
duane
Member
 
Posts: 1258
Joined: Wed Aug 19, 2015 19:11
Location: Oklahoma City
GitHub: duane-r

Re: 3 Memory use optimisations for Lua mapgens

by duane » Sat Dec 02, 2017 05:20

Wuzzy wrote:I have the impression that many people are reporting OOM problems while having LuaJIT enabled.

Does Minetest ship with LuaJIT by default or does it offer this option? If yes, it seems counter-productive to offer an option which you just publicly recommended to not use at all .


LuaJit is very nice, if you don't overload it. It gives a noticeable speed increase with the default minetest game, which I suspect, is why it's included by default. If you add loads of mods (especially mapgens or inefficient designs), it becomes a bad thing.

It seems to me that if luajit is failing, you need to move more of your work into the game itself (unless you can make it much more efficient). If luajit crumbles, you're doing so much in lua that you'll slow down any servers using it anyway. Of course, I've only followed that advice myself once, and probably never will again.
Believe in people and you don't need to believe anything else.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Mar 17, 2018 05:42

Just realised i still have two of these optimisations wrong, points 2 and 3.
EDIT: Now corrected above.
I will soon make a news post with a new LVM example mod that shows the correct optimisations in use.
 

User avatar
Hybrid Dog
Member
 
Posts: 2606
Joined: Thu Nov 01, 2012 12:46

Re: 3 Memory use optimisations for Lua mapgens

by Hybrid Dog » Sun Mar 18, 2018 17:42

Thanks, I added it to the nether mod: https://github.com/HybridDog/nether-pac ... 7d1f8613c5
When using the buffers, their content is never garbage collected and kept in memory until shutdown, isn't it?

Your description of the third optimization tells that Minetest 0.4.15 happened very recently.
 

User avatar
paramat
Developer
 
Posts: 3102
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Mar 19, 2018 01:13

Yes Lua garbage collection does happen but as far as i know not very quickly, Duane is the expert on that. So reusing a table is always better.
 

User avatar
Pyrollo
Member
 
Posts: 50
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj
 


Return to News



Who is online

Users browsing this forum: No registered users and 3 guests