3 Memory use optimisations for Lua mapgens

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

3 Memory use optimisations for Lua mapgens

by paramat » Post

Many players are reporting OOM (Out Of Memory) errors when using LuaJIT, which has a low memory use limit.
Lua mapgens can use excessive memory if not using these 3 optimisations, and most Lua mapgens, even most of mine, are not using these.

Examples of these optimisations are from my mod 'stability' https://github.com/paramat/stability which is currently the best mod of mine to demonstrate good practice in Lua mapgen.

//////////////////////////////////////////////////////////////////////////////

1. Perlin noise objects: Only create once

The noise object is created by 'minetest.get_perlin_map()'
It has to be created inside 'register_on_generated()' to be usable, but only needs to be created once, many mapgen mods create it for every mapchunk, this consumes memory unnecessarily.

Localise the noise object outside 'register_on_generated()' and initialise it to 'nil'.
See the code below for how to create it once only.
The creation of the perlin noise tables with 'get3dMap_flat()' etc. is done per mapchunk.

Code: Select all

-- Initialize noise objects to nil

local nobj_terrain = nil
local nobj_biome = nil

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

	nobj_terrain = nobj_terrain or minetest.get_perlin_map(np_terrain, chulens3d)
	nobj_biome = nobj_biome or minetest.get_perlin_map(np_biome, chulens2d)
	
	nobj_terrain:get3dMap_flat(minpos3d, nvals_terrain)
	nobj_biome:get2dMap_flat(minpos2d, nvals_biome)

...
...

end)
////////////////////////////////////////////////////////////////////////////////////

2. Perlin noise tables: Re-use a single table

The Lua table that stores the noise values for a mapchunk is big, especially for 3D noise (80 ^ 3 = 512000 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Code: Select all

For each of the functions with an optional `buffer` parameter:  If `buffer` is not
nil, this table will be used to store the result instead of creating a new table.

#### Methods
* `get2dMap_flat(pos, buffer)`: returns a flat `<size.x * size.y>` element array of 2D noise
  with values starting at `pos={x=,y=}`
* `get3dMap_flat(pos, buffer)`: Same as `get2dMap_flat`, but 3D noise
Localise the noise buffer outside 'register_on_generated()'
Use the buffer parameter in 'get3dMap_flat()' etc.

Code: Select all

-- Localise noise buffers

local nvals_terrain = {}
local nvals_biome = {}

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

	nobj_terrain:get3dMap_flat(minpos3d, nvals_terrain)
	nobj_biome:get2dMap_flat(minpos2d, nvals_biome)

...
...

end)
/////////////////////////////////////////////////////////////////////////////////////////

3. Lua voxelmanip table: Re-use a single table

The Lua table that stores the node content ids for a mapchunk plus the mapblock shell is big (112 ^ 3 = 1404928 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Very recently a 'buffer' parameter was also added to 'get_param2_data()', so is only usable there in latest Minetest dev or in Minetest 0.4.15 stable or later.

Code: Select all

* `get_data([buffer])`: Retrieves the node content data loaded into the `VoxelManip` object
    * returns raw node data in the form of an array of node content IDs
    * if the param `buffer` is present, this table will be used to store the result instead
* `get_param2_data([buffer])`: Gets the raw `param2` data read into the `VoxelManip` object
    * Returns an array (indices 1 to volume) of integers ranging from `0` to `255`
    * If the param `buffer` is present, this table will be used to store the result instead
Localise the data buffer outside 'register_on_generated()'.
Use the buffer in 'get_data()' or 'get_param2_data()'.

Code: Select all

-- Localise data buffer

local data = {}

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

	local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
	local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
	vm:get_data(data)

...
...

end)
Last edited by paramat on Sat Mar 17, 2018 05:49, edited 12 times in total.

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

Thanks for this guide. I think it would be useful to add this text to the Developer Wiki:
http://dev.minetest.net/Main_Page

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

Feel free to, i have no wiki access.

User avatar
Spaghetti Developer
Member
Posts: 32
Joined: Sun Dec 04, 2016 14:01
GitHub: SpaghettiDeveloper
In-game: Spaghetti Developer
Location: Rome, Italy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Spaghetti Developer » Post

Thanks Paramat! a long time I noticed this problem, the chink are struggling to generate and often they crash, then I have to walk to get closer so that you generate the missing area. I noticed a strong lag, due to the excessive vegetation in the jungle biome, and biome valley. Too many trees my friend! should not lower the density of vegetation? often biomes are full of trees excessively. This improves the generation of the chunk and making the most beautiful biomes and enjoyable, especially in multiplayer. I hope I explained myself :D

User avatar
Hybrid Dog
Member
Posts: 2828
Joined: Thu Nov 01, 2012 12:46
GitHub: HybridDog

by Hybrid Dog » Post

> *** Note a buffer is only usable in 'get_param2_data()' in latest Minetest dev or in Minetest 0.4.15 stable ***

The buffer doesn't work for 'get_data()', so using it there doesn't hone memory usage, does it?

‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

The buffer does work for 'get_data()' and 'get_param2_data()'.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

Note i have been corrected on point 3, the initialisation of dbuf must define it as a table:

Code: Select all

 local dbuf = {}
Otherwise memory usage is not reduced.

Code: Select all

    -- Localise data buffer

    local dbuf = {}

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
       local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
       local data = vm:get_data(dbuf)

    ...
    ...

    end)

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

Just realised this probably applies to point 2 also:

Code: Select all

    -- Localise noise buffers

    local nbuf_terrain = {}
    local nbuf_biome = {}

    ...
    ...

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
       local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

    ...
    ...

    end)

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

^ It does apply to point 2 also, updated first post.

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

paramat wrote:Feel free to, i have no wiki access.
Why not?

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

@paramat: There's a small mistake in your first post: It should be “minetest.get_perlin”, not “minetest.get_perlin_map”.

And I have another question: What do you have to say about the usage of perlin:get2d, as opposed to perlin:get2dMap or perlin:get2dMap_flat (same for 3D variants)? Are there any special optimization rules I should follow?

And when does it make more sense to use get2d instead of get2dMap?

Final question: How do I measure the performance / memory usage? I want to know if any of these optimizations actually worked.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

I'm fairly sure it should be 'get perlin map', the lines in the code blocks are copied from working mods.
However a similar optimisation can be done when using per-point perlin noise instead of noise maps, see latest technic mod worldgen file.
'get map flat' is probably better then 'get map' as it seems a flat array is the most efficient way to access bulk noise data. Not sure what 'get map' is for, that's a question for hmmmm.
'get2D' is not bulk noise but noise calculated per-point, like Lua voxelmanip bulk noise calculation becomes optimal for large amounts of noise.

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

Okay, thanks. Just FYI, I have applied memory optimizations for the map generator in MineClone 2 because someone reported OOM bugs.

But I have no idea how to verify by fix. :-(

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

What do I do if players still report OOM bugs even if I have applied all 3 optimizations? :-(

Is there any way to debug OOM errors properly?

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by duane » Post

Wuzzy wrote:What do I do if players still report OOM bugs even if I have applied all 3 optimizations? :-(

Is there any way to debug OOM errors properly?
I just use collectgarbage("count") to check memory use before and after a process. That should give you an idea of where your memory hogs are. You might also want to stop and restart the garbage collector, just to be sure.
Believe in people and you don't need to believe anything else.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

The optimisations reduce the chance of OOM but it could still happen in some situations. Maybe just don't use LuaJIT?

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by Wuzzy » Post

I have the impression that many people are reporting OOM problems while having LuaJIT enabled.

Does Minetest ship with LuaJIT by default or does it offer this option? If yes, it seems counter-productive to offer an option which you just publicly recommended to not use at all .

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: 3 Memory use optimisations for Lua mapgens

by duane » Post

Wuzzy wrote:I have the impression that many people are reporting OOM problems while having LuaJIT enabled.

Does Minetest ship with LuaJIT by default or does it offer this option? If yes, it seems counter-productive to offer an option which you just publicly recommended to not use at all .
LuaJit is very nice, if you don't overload it. It gives a noticeable speed increase with the default minetest game, which I suspect, is why it's included by default. If you add loads of mods (especially mapgens or inefficient designs), it becomes a bad thing.

It seems to me that if luajit is failing, you need to move more of your work into the game itself (unless you can make it much more efficient). If luajit crumbles, you're doing so much in lua that you'll slow down any servers using it anyway. Of course, I've only followed that advice myself once, and probably never will again.
Believe in people and you don't need to believe anything else.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

Just realised i still have two of these optimisations wrong, points 2 and 3.
EDIT: Now corrected above.
I will soon make a news post with a new LVM example mod that shows the correct optimisations in use.

User avatar
Hybrid Dog
Member
Posts: 2828
Joined: Thu Nov 01, 2012 12:46
GitHub: HybridDog

Re: 3 Memory use optimisations for Lua mapgens

by Hybrid Dog » Post

Thanks, I added it to the nether mod: https://github.com/HybridDog/nether-pac ... 7d1f8613c5
When using the buffers, their content is never garbage collected and kept in memory until shutdown, isn't it?

Your description of the third optimization tells that Minetest 0.4.15 happened very recently.

‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

Yes Lua garbage collection does happen but as far as i know not very quickly, Duane is the expert on that. So reusing a table is always better.

User avatar
Pyrollo
Developer
Posts: 385
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: 3 Memory use optimisations for Lua mapgens

by Pyrollo » Post

Thanks a lot for these tips Paramat, I copied your post into the dev wiki : http://dev.minetest.net/Mapgen_memory_optimisations
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

Brian Gaucher
Member
Posts: 77
Joined: Wed Jan 10, 2018 01:56
GitHub: BrianGaucher
In-game: Camasia

Re: 3 Memory use optimisations for Lua mapgens

by Brian Gaucher » Post

I'm starting out in modding.
How hard is making a simple MapGen compared to making other mods?
Also, can you recommend some Youtubes, Books, tutorials, or strategies to learn algorithmic map generation.
Thanks.
Current projects: Making a CTF map, Learning C++, Learning Programmer's Dvorak

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: 3 Memory use optimisations for Lua mapgens

by paramat » Post

See my more recent lua mapgen mods as good examples. Also see the Lua Voxel Manipulator example in this news subforum.

User avatar
Hybrid Dog
Member
Posts: 2828
Joined: Thu Nov 01, 2012 12:46
GitHub: HybridDog

Re: 3 Memory use optimisations for Lua mapgens

by Hybrid Dog » Post

That example can be updated to use the buffer, can't it? https://dev.minetest.net/vmanip#Examples

‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests