Page 1 of 1

LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Wed Jan 16, 2019 11:57
by Festus1965
So, I put 32 GB of RAM into the MTS Server, but anyway it is useless, as the Server crashes latest up to 24 hours, before the last changed settings of ubuntu can use 80% of the RAM for writing cache ... most i just get up to 7 GB RAM using, and then gone ... All new and useless to put hardware in ... next i take the 32 GB back into my workpc, there i use just in 10 minutes near 10 GB ... as the server just reaches 7 GB, there 16 max is enough.

So to the topic:

How to detect LUAJIT using MODS causing OOM ?
* detect mod
* detect file with the code
* and how to change the code for better

or how to reduce LUAJIT amount ?
* mod [stop_lj_oom] tested, but lag ...


To check out weather I need them or not - to get less failure and longer runs of the MTS.

The OOM is not a fresh case, but I want to get rid of !

Links:
* Lua OOM crashes
* Minetest memory error(Lua OOM error not enough memory)
* Minetest with LuaJit GC64 mode (>2gb mem)
* Lua OOM errors
* Remove LuaJIT support?

* [mod] Stop LUAJIT OOM crashes [stop_lj_oom]

I will check the last link more now and learn again ...
edit: stop_lj_oom was running some days until value 1.450.000 but when active, getting about plus 0.8 lag

edit: detected now, mods which (even) crash, (when garbage collector is running)
* mod 'mcl_dungeons'
* mod worldview
* mod technic_worldgen
** here a possible solution - I just edited my mod "memory optimisation fix is simple"
* mod pipeworks
* mod 'biome_lib'
* mod fire
* mod 'televator'

Re: OOM - how to detect Mods using it - and rid off !

PostPosted: Thu Jan 17, 2019 01:06
by Festus1965
So it is running,
original - no changes

and since the night is activates 4 times a second with a value about near 400000
mean:
it does collectgarbage() ... without much clean up, as OOM full (> 307200) just in not a second again !

server itself is still good with lag 0.32, mhhh

the last crashes before using [stop_lj_oom] have been reported with:
* editing ...
* 1533 MB

now changed setting to 600000 instead of 307xxx and restart server.


:.edit: 17.01. - 17.15
just back from a transport, and didnt see the server for 1.5 h
now every about 2.5 sec the "collectgarbage()" is running with values about 600.000
thinking, but will rise up to 900.000 next

Update: OOM - how to detect Mods using it - and rid off !

PostPosted: Fri Jan 18, 2019 23:16
by Festus1965
the "collectgarbage()" is running with values about 1.200.000 in every near 6 seconds now
the lag is 1, even i am alone there ...
1 Core / 1 Thread CPU between 60-80%,
cache 21,9 GB

in the moment lag of 1 instead of 0.4 seams to be because of "collectgarbage()" ? ... as nothing else changed ... mhh, not funny

Re: OOM - how to detect Mods using it - and rid off !

PostPosted: Mon Jan 21, 2019 04:10
by Festus1965
the "collectgarbage()" was running with values about 1.300.000
at the end it collected froom first just 40 sec more fast each near 6 sec again,

(with value of 1.500.000 it crashed after near 20 hours again with OOM at 970 MB)

and then mod worldview crashed the server even (guess taking too much - free after collectgarbage too less until out of Mem)

solution until now
* "collectgarbage()" back to value 1.200.000
* mod worldview = false

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Sun Jan 27, 2019 04:50
by Festus1965
here the changed I did at technic/technic_worldgen/

Code: Select all
-- Sulfur
--local sulfur_buf = {}  -- this is old
local data = {}  --  this is new plus
local sulfur_noise = nil

minetest.register_on_generated(function(minp, maxp, seed)
   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local a = VoxelArea:new{
      MinEdge = {x = emin.x, y = emin.y, z = emin.z},
      MaxEdge = {x = emax.x, y = emax.y, z = emax.z},
   }
   --local data = vm:get_data(sulfur_buf) -- this isold
   vm:get_data(data) -- this is new
   local pr = PseudoRandom(17 * minp.x + 42 * minp.y + 101 * minp.z)
   sulfur_noise = sulfur_noise or minetest.get_perlin(9876, 3, 0.5, 100)


one more,
i am on to learn how to detect the code in the mod and solute it ...

2019-01-27 11:30:47: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'biome_lib' in callback environment_Step(): not enough memory
2019-01-27 11:30:47: ERROR[Main]: Current Lua memory usage: 1057 MB
2019-01-27 11:30:47: ERROR[Main]: stack traceback:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Tue Jan 29, 2019 23:21
by Festus1965
Code: Select all
2019-01-30 01:53:27: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod '[b]fire[/b]' in callback environment_Step(): not enough memory
2019-01-30 01:53:27: ERROR[Main]: Current Lua memory usage: 1274 MB
2019-01-30 01:53:27: ERROR[Main]: stack traceback:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Tue Feb 05, 2019 22:30
by Festus1965
Code: Select all
2019-02-04 17:20:59: ERROR[Main]: ServerError: AsyncErr: ServerThread::run Lua: OOM error from mod 'pipeworks' in callback LuaABM::trigger(): not enough memory
2019-02-04 17:20:59: ERROR[Main]: Current Lua memory usage: 1341 MB

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Tue Feb 05, 2019 22:30
by Festus1965
Code: Select all
2019-02-05 14:26:38: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'televator' in callback environment_Step(): not enough memory
2019-02-05 14:26:38: ERROR[Main]: Current Lua memory usage: 1133 MB
2019-02-05 14:26:38: ERROR[Main]: stack traceback:

LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Wed Mar 06, 2019 11:13
by Festus1965
Maybe I found something,

lua mem profiler

I will check it out next server starts.

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Wed Mar 06, 2019 15:59
by Pyrollo
Hi,

I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Thu Mar 07, 2019 01:08
by Festus1965
Pyrollo wrote:I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?


OOM
I was checking the git from my seen most OOM causing Mods (technic, pipeworks and mesecons) but cant really see if those are fixed with that code, I asked today morning, but downloaded and test when time

lua mem profiler, (was not running yet)
Code: Select all
2019-03-07 09:01:50: ERROR[Main]: ModError: Failed to load and run script from /home/thomas/.minetest/mods/luaprofiler/init.lua:
2019-03-07 09:01:50: ERROR[Main]: /home/thomas/.minetest/mods/luaprofiler/init.lua:458: '<eof>' expected near 'end'

that is added as mod and loaded without problem, but now I am missing a possibilities to start it
I need to build in some
Code: Select all
minetest.register_chatcommand("luamemcheck", {

what calls then the function
Code: Select all
ProFi:start( 'once' )

will see how to do

PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Thu Mar 07, 2019 12:19
by Pyrollo
Festus1965 wrote:PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this


I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Thu Mar 07, 2019 13:09
by Festus1965
sure, but that fore i supposed to try some software that tells us, what mod is using most of the memory until it OOM-crashes - and then go to this 2 or 4 mods and optimize like you supposed ...
otherwise we have to upgrade all of them, guessing that is a problem

So analyzing first, than act ... ?

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Tue Mar 12, 2019 07:08
by Festus1965
Pyrollo wrote:I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.


I am so sorry Pyrollo,
but both posts are from
* Sun May 17, 2015 16:16 - Lua PerlinNoiseMap memory efficiency improvements
* Wed May 11, 2016 20:26 - Perlin noise exceeds -1/1 range

so that I was quite sure that should have been solved ...

and so I am more locking in this direction Important oregen memory use optimization from Dec 2018 to understand ... I read a lot last 5 days, and will try out if have fun to do ...

as after that 5 days there was another 11 crashes, 7 of them by technic OOM hehe

I am looking for some tricks told
* https://stackoverflow.com/questions/27015150/how-to-get-past-1gb-memory-limit-of-64-bit-luajit-on-linux
* https://kvitajakub.github.io/2016/03/08/luajit-memory-limitations/
* https://github.com/cloudflare/luajit-mm
but still need to get back another (is in Celine's PC but she play Roblox - so I cant ...)

Re: LuaJit OOM - how to detect Mods using it - and rid off !

PostPosted: Sat Jun 22, 2019 06:25
by Festus1965
next step:

I just deleted my github.

so also my [mod] free_lj_mem

even I don't find anymore the thread ...

continue to go ... out here

just got mail confirm "This email is to confirm that you’ve deleted your account ‘Minetest-One’ from GitHub. Your repositories and content have been deleted from the system. If you were on a paid plan, you will not be billed again. We’re sorry to see you go. You can reply directly to this email if you have any questions or feedback, we’d love to hear from you. "

feeling more free already