3 Memory use optimisations for Lua mapgens

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

3 Memory use optimisations for Lua mapgens

by paramat » Thu Dec 15, 2016 05:57

Many players are reporting OOM (Out Of Memory) errors when using LuaJIT, which has a low memory use limit.
Lua mapgens can use excessive memory if not using these 3 optimisations, and most Lua mapgens, even most of mine, are not using these.

Examples of these optimisations are from my mod 'stability' https://github.com/paramat/stability which is currently the best mod of mine to demonstrate good practice in Lua mapgen.

//////////////////////////////////////////////////////////////////////////////

1. Perlin noise objects: Only create once

The noise object is created by 'minetest.get_perlin_map()'
It has to be created inside 'register_on_generated()' to be usable, but only needs to be created once, many mapgen mods create it for every mapchunk, this consumes memory unnecessarily.

Localise the noise object outside 'register_on_generated()' and initialise it to 'nil'.
See the code below for how to create it once only.
The creation of the perlin noise tables with 'get3dMap_flat()' etc. is done per mapchunk.

Code: Select all
-- Initialize noise objects to nil

local nobj_terrain = nil
local nobj_biome = nil

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   nobj_terrain = nobj_terrain or minetest.get_perlin_map(np_terrain, chulens3d)
   nobj_biome = nobj_biome or minetest.get_perlin_map(np_biome, chulens2d)
   
   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


////////////////////////////////////////////////////////////////////////////////////

2. Perlin noise tables: Re-use a single table

The Lua table that stores the noise values for a mapchunk is big, especially for 3D noise (80 ^ 3 = 512000 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Code: Select all
For each of the functions with an optional `buffer` parameter:  If `buffer` is not
nil, this table will be used to store the result instead of creating a new table.

#### Methods
* `get2dMap_flat(pos, buffer)`: returns a flat `<size.x * size.y>` element array of 2D noise
  with values starting at `pos={x=,y=}`
* `get3dMap_flat(pos, buffer)`: Same as `get2dMap_flat`, but 3D noise


Localise the noise buffer outside 'register_on_generated()'
Use the buffer parameter in 'get3dMap_flat()' etc.

Code: Select all
-- Localise noise buffers

local nbuf_terrain = {}
local nbuf_biome = {}

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


/////////////////////////////////////////////////////////////////////////////////////////

3. Lua voxelmanip table: Re-use a single table

The Lua table that stores the node content ids for a mapchunk plus the mapblock shell is big (112 ^ 3 = 1404928 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Very recently a 'buffer' parameter was also added to 'get_param2_data()', so is only usable there in latest Minetest dev or in Minetest 0.4.15 stable or later.

Code: Select all
* `get_data([buffer])`: Retrieves the node content data loaded into the `VoxelManip` object
    * returns raw node data in the form of an array of node content IDs
    * if the param `buffer` is present, this table will be used to store the result instead
* `get_param2_data([buffer])`: Gets the raw `param2` data read into the `VoxelManip` object
    * Returns an array (indices 1 to volume) of integers ranging from `0` to `255`
    * If the param `buffer` is present, this table will be used to store the result instead


Localise the data buffer outside 'register_on_generated()'.
Use the buffer in 'get_data()' or 'get_param2_data()'.

Code: Select all
-- Localise data buffer

local dbuf = {}

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
   local data = vm:get_data(dbuf)

...
...

end)
Last edited by paramat on Mon Apr 03, 2017 22:45, edited 11 times in total.
 

User avatar
Wuzzy
Member
 
Posts: 2431
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Thu Dec 15, 2016 20:30

Thanks for this guide. I think it would be useful to add this text to the Developer Wiki:
http://dev.minetest.net/Main_Page
I'm creating MineClone 2, a Minecraft clone for Minetest.
I made the Help modpack, adding in-game help to Minetest.
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat
 

User avatar
Spaghetti Developer
Member
 
Posts: 32
Joined: Sun Dec 04, 2016 14:01
Location: Rome, Italy
GitHub: SpaghettiDeveloper
In-game: Spaghetti Developer

Re: 3 Memory use optimisations for Lua mapgens

by Spaghetti Developer » Sun Dec 18, 2016 18:00

Thanks Paramat! a long time I noticed this problem, the chink are struggling to generate and often they crash, then I have to walk to get closer so that you generate the missing area. I noticed a strong lag, due to the excessive vegetation in the jungle biome, and biome valley. Too many trees my friend! should not lower the density of vegetation? often biomes are full of trees excessively. This improves the generation of the chunk and making the most beautiful biomes and enjoyable, especially in multiplayer. I hope I explained myself :D
 

Hybrid Dog
Member
 
Posts: 2496
Joined: Thu Nov 01, 2012 12:46

by Hybrid Dog » Sun Dec 18, 2016 20:48

> *** Note a buffer is only usable in 'get_param2_data()' in latest Minetest dev or in Minetest 0.4.15 stable ***

The buffer doesn't work for 'get_data()', so using it there doesn't hone memory usage, does it?
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Dec 19, 2016 02:13

The buffer does work for 'get_data()' and 'get_param2_data()'.
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:00

Note i have been corrected on point 3, the initialisation of dbuf must define it as a table:

Code: Select all
 local dbuf = {}


Otherwise memory usage is not reduced.

Code: Select all
    -- Localise data buffer

    local dbuf = {}

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
       local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
       local data = vm:get_data(dbuf)

    ...
    ...

    end)
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:45

Just realised this probably applies to point 2 also:

Code: Select all
    -- Localise noise buffers

    local nbuf_terrain = {}
    local nbuf_biome = {}

    ...
    ...

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
       local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

    ...
    ...

    end)
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Apr 03, 2017 22:46

^ It does apply to point 2 also, updated first post.
 

User avatar
Wuzzy
Member
 
Posts: 2431
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Sun May 07, 2017 14:19

paramat wrote:Feel free to, i have no wiki access.

Why not?
I'm creating MineClone 2, a Minecraft clone for Minetest.
I made the Help modpack, adding in-game help to Minetest.
 

User avatar
Wuzzy
Member
 
Posts: 2431
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Tue May 09, 2017 14:30

@paramat: There's a small mistake in your first post: It should be “minetest.get_perlin”, not “minetest.get_perlin_map”.

And I have another question: What do you have to say about the usage of perlin:get2d, as opposed to perlin:get2dMap or perlin:get2dMap_flat (same for 3D variants)? Are there any special optimization rules I should follow?

And when does it make more sense to use get2d instead of get2dMap?

Final question: How do I measure the performance / memory usage? I want to know if any of these optimizations actually worked.
I'm creating MineClone 2, a Minecraft clone for Minetest.
I made the Help modpack, adding in-game help to Minetest.
 

paramat
Developer
 
Posts: 2834
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Wed May 10, 2017 05:13

I'm fairly sure it should be 'get perlin map', the lines in the code blocks are copied from working mods.
However a similar optimisation can be done when using per-point perlin noise instead of noise maps, see latest technic mod worldgen file.
'get map flat' is probably better then 'get map' as it seems a flat array is the most efficient way to access bulk noise data. Not sure what 'get map' is for, that's a question for hmmmm.
'get2D' is not bulk noise but noise calculated per-point, like Lua voxelmanip bulk noise calculation becomes optimal for large amounts of noise.
 

User avatar
Wuzzy
Member
 
Posts: 2431
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Wed May 10, 2017 23:20

Okay, thanks. Just FYI, I have applied memory optimizations for the map generator in MineClone 2 because someone reported OOM bugs.

But I have no idea how to verify by fix. :-(
I'm creating MineClone 2, a Minecraft clone for Minetest.
I made the Help modpack, adding in-game help to Minetest.
 


Return to News



Who is online

Users browsing this forum: No registered users and 4 guests