3 Memory use optimisations for Lua mapgens

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat

3 Memory use optimisations for Lua mapgens

by paramat » Thu Dec 15, 2016 05:57

Many players are reporting OOM (Out Of Memory) errors when using LuaJIT, which has a low memory use limit.
Lua mapgens can use excessive memory if not using these 3 optimisations, and most Lua mapgens, even most of mine, are not using these.

Examples of these optimisations are from my mod 'stability' https://github.com/paramat/stability which is currently the best mod of mine to demonstrate good practice in Lua mapgen.

//////////////////////////////////////////////////////////////////////////////

1. Perlin noise objects: Only create once

The noise object is created by 'minetest.get_perlin_map()'
It has to be created inside 'register_on_generated()' to be usable, but only needs to be created once, many mapgen mods create it for every mapchunk, this consumes memory unnecessarily.

Localise the noise object outside 'register_on_generated()' and initialise it to 'nil'.
See the code below for how to create it once only.
The creation of the perlin noise tables with 'get3dMap_flat()' etc. is done per mapchunk.

Code: Select all
-- Initialize noise objects to nil

local nobj_terrain = nil
local nobj_biome = nil

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   nobj_terrain = nobj_terrain or minetest.get_perlin_map(np_terrain, chulens3d)
   nobj_biome = nobj_biome or minetest.get_perlin_map(np_biome, chulens2d)
   
   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


////////////////////////////////////////////////////////////////////////////////////

2. Perlin noise tables: Re-use a single table

The Lua table that stores the noise values for a mapchunk is big, especially for 3D noise (80 ^ 3 = 512000 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Code: Select all
For each of the functions with an optional `buffer` parameter:  If `buffer` is not
nil, this table will be used to store the result instead of creating a new table.

#### Methods
* `get2dMap_flat(pos, buffer)`: returns a flat `<size.x * size.y>` element array of 2D noise
  with values starting at `pos={x=,y=}`
* `get3dMap_flat(pos, buffer)`: Same as `get2dMap_flat`, but 3D noise


Localise the noise buffer outside 'register_on_generated()'
Use the buffer parameter in 'get3dMap_flat()' etc.

Code: Select all
-- Localise noise buffers

local nbuf_terrain = {}
local nbuf_biome = {}

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


/////////////////////////////////////////////////////////////////////////////////////////

3. Lua voxelmanip table: Re-use a single table

The Lua table that stores the node content ids for a mapchunk plus the mapblock shell is big (112 ^ 3 = 1404928 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Very recently a 'buffer' parameter was also added to 'get_param2_data()', so is only usable there in latest Minetest dev or in Minetest 0.4.15 stable or later.

Code: Select all
* `get_data([buffer])`: Retrieves the node content data loaded into the `VoxelManip` object
    * returns raw node data in the form of an array of node content IDs
    * if the param `buffer` is present, this table will be used to store the result instead
* `get_param2_data([buffer])`: Gets the raw `param2` data read into the `VoxelManip` object
    * Returns an array (indices 1 to volume) of integers ranging from `0` to `255`
    * If the param `buffer` is present, this table will be used to store the result instead


Localise the data buffer outside 'register_on_generated()'.
Use the buffer in 'get_data()' or 'get_param2_data()'.

Code: Select all
-- Localise data buffer

local dbuf = {}

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
   local data = vm:get_data(dbuf)

...
...

end)
Last edited by paramat on Mon Apr 03, 2017 22:45, edited 11 times in total.
 

User avatar
Wuzzy
Member
 
Posts: 2181
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Thu Dec 15, 2016 20:30

Thanks for this guide. I think it would be useful to add this text to the Developer Wiki:
http://dev.minetest.net/Main_Page
I'm creating MineClone 2, a Minecraft clone for Minetest.
I made the Help modpack, adding in-game help to Minetest.
 

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
 

User avatar
Spaghetti Developer
Member
 
Posts: 32
Joined: Sun Dec 04, 2016 14:01
Location: Rome, Italy
GitHub: SpaghettiDeveloper
In-game: Spaghetti Developer

Re: 3 Memory use optimisations for Lua mapgens

by Spaghetti Developer » Sun Dec 18, 2016 18:00

Thanks Paramat! a long time I noticed this problem, the chink are struggling to generate and often they crash, then I have to walk to get closer so that you generate the missing area. I noticed a strong lag, due to the excessive vegetation in the jungle biome, and biome valley. Too many trees my friend! should not lower the density of vegetation? often biomes are full of trees excessively. This improves the generation of the chunk and making the most beautiful biomes and enjoyable, especially in multiplayer. I hope I explained myself :D
 

Hybrid Dog
Member
 
Posts: 2477
Joined: Thu Nov 01, 2012 12:46

by Hybrid Dog » Sun Dec 18, 2016 20:48

> *** Note a buffer is only usable in 'get_param2_data()' in latest Minetest dev or in Minetest 0.4.15 stable ***

The buffer doesn't work for 'get_data()', so using it there doesn't hone memory usage, does it?
 

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Dec 19, 2016 02:13

The buffer does work for 'get_data()' and 'get_param2_data()'.
 

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:00

Note i have been corrected on point 3, the initialisation of dbuf must define it as a table:

Code: Select all
 local dbuf = {}


Otherwise memory usage is not reduced.

Code: Select all
    -- Localise data buffer

    local dbuf = {}

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
       local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
       local data = vm:get_data(dbuf)

    ...
    ...

    end)
 

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Sat Apr 01, 2017 19:45

Just realised this probably applies to point 2 also:

Code: Select all
    -- Localise noise buffers

    local nbuf_terrain = {}
    local nbuf_biome = {}

    ...
    ...

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
       local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

    ...
    ...

    end)
 

paramat
Developer
 
Posts: 2703
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Mon Apr 03, 2017 22:46

^ It does apply to point 2 also, updated first post.
 


Return to News



Who is online

Users browsing this forum: No registered users and 1 guest