Page 1 of 1

Lua PerlinNoiseMap memory efficiency improvements

PostPosted: Sun May 17, 2015 16:16
by hmmmm
Hello Minetest Modding Community,

Recently I've had some complaints about extremely high memory usage in mods that make extensive use of Perlin noise maps (core.get_perlin_map/PerlinNoiseMap). I investigated and found that the most memory-intensive part of noise is, by far, placing the resulting computation into a Lua table. Optimizing memory usage was possible, but taking advantage of the improvements required some additions to the Lua API.

The first (and simplest) of the changes is to pass along a pre-existing table to receive the results of the noise calculations. This not only saves CPU time as all entries have already been allocated, but it minimizes Lua object creation so memory usage stays stable throughout execution. It can be used like so:
Code: Select all
local noise_buffer = {}
local noise = nil
...
function do_noise_thing(pmin, pmax)
   ...
   -- Notice that we reuse the same noise object instead of creating a new one each call
   noise = noise or core.get_perlin_map(noise_params, chunk_size)

   -- Here, the variable 'noisevals' is merely a reference to the table noise_buffer, not a copy.
   local noisevals = noise:get3dMap_flat(pmin, noise_buffer)
   ...
end


The second change requires the use of new PerlinNoiseMap methods calc2dMap, calc3dMap, and getMapSlice.
First, calc2dMap or calc3dMap is called to compute the noise. Then, when a chunk of noise is needed, it can be retrieved from the internal buffer using getMapSlice(). getMapSlice() takes a slice of noise at the specified coordinates, relative to the position inside the buffer (not absolute map coordinates!) and starts at 1, as is standard for Lua. If a coordinate is not specified in the slice offset parameter, then all noise along that axis is written to the output table. Therefore, a horizontal plane would be {y=}, a vertical column would be {x=, z=}, a single row spanning the X axis would be {y=, z=}, the whole buffer would be {}, and a single point would be {x=, y=, z=}. getMapSlice's second parameter, the slice size, is the length along that specified axis to retrieve.

Here is an example of these new methods in use:
Code: Select all
local noise_buffer = {}
local noise = nil
...
function do_noise_thing(pmin, pmax)
   ...
   noise = noise or core.get_perlin_map(noise_params, chunk_size)

   -- Calculate the noise starting at pmin, storing the result internally.
   noise:calc3dMap(pmin)

   -- Generate the terrain
   local slice_count = 0
   local nvals
   local ni
   for z = pmin.z, pmax.z do
      -- Get an 80x80x2 slice of the noise result, if needed
      if slice_count == 0 then
         slice_count = 2
         nvals = noise:getMapSlice({z=z-pmin.z+1}, {z=slice_count}, noise_buffer)
         ni = 1
      end

      for y = pmin.y, pmax.y do
      for x = pmin.x, pmax.x do
         if nvals[ni] > 0.5 then
            ...
         end
         ni = ni + 1
      end
      end

      slice_count = slice_count - 1
   end
   ...
end


With the above code, execution time is 1.8% slower than the previous example, but uses a whole 40x less memory. The table noise_buffer consumes 200KB instead of 8000KB.

I hope you all find this information useful in minimizing the memory footprint of your mods making use of noise maps.

- hmmmmm

Re: Lua PerlinNoiseMap memory efficiency improvements

PostPosted: Sun May 17, 2015 20:47
by paramat
Excellent work and extremely useful, thanks so much.

Re: Lua PerlinNoiseMap memory efficiency improvements

PostPosted: Mon May 18, 2015 15:58
by Krock
Thanks for pointing to this. Does it count for Lua and LuaJIT?

Re: Lua PerlinNoiseMap memory efficiency improvements

PostPosted: Tue May 26, 2015 05:30
by paramat
Yes both.

See the issue at github for more detail on this subject https://github.com/minetest/minetest/issues/2665