LuaJit OOM - how to detect Mods using it - and rid off !

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Wed Jan 16, 2019 11:57

So, I put 32 GB of RAM into the MTS Server, but anyway it is useless, as the Server crashes latest up to 24 hours, before the last changed settings of ubuntu can use 80% of the RAM for writing cache ... most i just get up to 7 GB RAM using, and then gone ... All new and useless to put hardware in ... next i take the 32 GB back into my workpc, there i use just in 10 minutes near 10 GB ... as the server just reaches 7 GB, there 16 max is enough.

So to the topic:

How to detect LUAJIT using MODS causing OOM ?
* detect mod
* detect file with the code
* and how to change the code for better

or how to reduce LUAJIT amount ?
* mod [stop_lj_oom] tested, but lag ...


To check out weather I need them or not - to get less failure and longer runs of the MTS.

The OOM is not a fresh case, but I want to get rid of !

Links:
* Lua OOM crashes
* Minetest memory error(Lua OOM error not enough memory)
* Minetest with LuaJit GC64 mode (>2gb mem)
* Lua OOM errors
* Remove LuaJIT support?

* [mod] Stop LUAJIT OOM crashes [stop_lj_oom]

I will check the last link more now and learn again ...
edit: stop_lj_oom was running some days until value 1.450.000 but when active, getting about plus 0.8 lag

edit: detected now, mods which (even) crash, (when garbage collector is running)
* mod 'mcl_dungeons'
* mod worldview
* mod technic_worldgen
** here a possible solution - I just edited my mod "memory optimisation fix is simple"
* mod pipeworks
* mod 'biome_lib'
* mod fire
* mod 'televator'
Last edited by Festus1965 on Tue Feb 05, 2019 22:31, edited 13 times in total.
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Thu Jan 17, 2019 01:06

So it is running,
original - no changes

and since the night is activates 4 times a second with a value about near 400000
mean:
it does collectgarbage() ... without much clean up, as OOM full (> 307200) just in not a second again !

server itself is still good with lag 0.32, mhhh

the last crashes before using [stop_lj_oom] have been reported with:
* editing ...
* 1533 MB

now changed setting to 600000 instead of 307xxx and restart server.


:.edit: 17.01. - 17.15
just back from a transport, and didnt see the server for 1.5 h
now every about 2.5 sec the "collectgarbage()" is running with values about 600.000
thinking, but will rise up to 900.000 next
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Update: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Fri Jan 18, 2019 23:16

the "collectgarbage()" is running with values about 1.200.000 in every near 6 seconds now
the lag is 1, even i am alone there ...
1 Core / 1 Thread CPU between 60-80%,
cache 21,9 GB

in the moment lag of 1 instead of 0.4 seams to be because of "collectgarbage()" ? ... as nothing else changed ... mhh, not funny
Last edited by Festus1965 on Tue Jan 22, 2019 08:14, edited 1 time in total.
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Mon Jan 21, 2019 04:10

the "collectgarbage()" was running with values about 1.300.000
at the end it collected froom first just 40 sec more fast each near 6 sec again,

(with value of 1.500.000 it crashed after near 20 hours again with OOM at 970 MB)

and then mod worldview crashed the server even (guess taking too much - free after collectgarbage too less until out of Mem)

solution until now
* "collectgarbage()" back to value 1.200.000
* mod worldview = false
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Sun Jan 27, 2019 04:50

here the changed I did at technic/technic_worldgen/

Code: Select all
-- Sulfur
--local sulfur_buf = {}  -- this is old
local data = {}  --  this is new plus
local sulfur_noise = nil

minetest.register_on_generated(function(minp, maxp, seed)
   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local a = VoxelArea:new{
      MinEdge = {x = emin.x, y = emin.y, z = emin.z},
      MaxEdge = {x = emax.x, y = emax.y, z = emax.z},
   }
   --local data = vm:get_data(sulfur_buf) -- this isold
   vm:get_data(data) -- this is new
   local pr = PseudoRandom(17 * minp.x + 42 * minp.y + 101 * minp.z)
   sulfur_noise = sulfur_noise or minetest.get_perlin(9876, 3, 0.5, 100)


one more,
i am on to learn how to detect the code in the mod and solute it ...

2019-01-27 11:30:47: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'biome_lib' in callback environment_Step(): not enough memory
2019-01-27 11:30:47: ERROR[Main]: Current Lua memory usage: 1057 MB
2019-01-27 11:30:47: ERROR[Main]: stack traceback:
Last edited by Festus1965 on Sat Feb 02, 2019 23:16, edited 2 times in total.
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Tue Jan 29, 2019 23:21

Code: Select all
2019-01-30 01:53:27: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod '[b]fire[/b]' in callback environment_Step(): not enough memory
2019-01-30 01:53:27: ERROR[Main]: Current Lua memory usage: 1274 MB
2019-01-30 01:53:27: ERROR[Main]: stack traceback:
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Tue Feb 05, 2019 22:30

Code: Select all
2019-02-04 17:20:59: ERROR[Main]: ServerError: AsyncErr: ServerThread::run Lua: OOM error from mod 'pipeworks' in callback LuaABM::trigger(): not enough memory
2019-02-04 17:20:59: ERROR[Main]: Current Lua memory usage: 1341 MB
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Tue Feb 05, 2019 22:30

Code: Select all
2019-02-05 14:26:38: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'televator' in callback environment_Step(): not enough memory
2019-02-05 14:26:38: ERROR[Main]: Current Lua memory usage: 1133 MB
2019-02-05 14:26:38: ERROR[Main]: stack traceback:
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer
 

User avatar
Pyrollo
Member
 
Posts: 272
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Pyrollo » Wed Mar 06, 2019 15:59

Hi,

I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Thu Mar 07, 2019 01:08

Pyrollo wrote:I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?


OOM
I was checking the git from my seen most OOM causing Mods (technic, pipeworks and mesecons) but cant really see if those are fixed with that code, I asked today morning, but downloaded and test when time

lua mem profiler, (was not running yet)
Code: Select all
2019-03-07 09:01:50: ERROR[Main]: ModError: Failed to load and run script from /home/thomas/.minetest/mods/luaprofiler/init.lua:
2019-03-07 09:01:50: ERROR[Main]: /home/thomas/.minetest/mods/luaprofiler/init.lua:458: '<eof>' expected near 'end'

that is added as mod and loaded without problem, but now I am missing a possibilities to start it
I need to build in some
Code: Select all
minetest.register_chatcommand("luamemcheck", {

what calls then the function
Code: Select all
ProFi:start( 'once' )

will see how to do

PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this
 

User avatar
Pyrollo
Member
 
Posts: 272
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Pyrollo » Thu Mar 07, 2019 12:19

Festus1965 wrote:PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this


I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Thu Mar 07, 2019 13:09

sure, but that fore i supposed to try some software that tells us, what mod is using most of the memory until it OOM-crashes - and then go to this 2 or 4 mods and optimize like you supposed ...
otherwise we have to upgrade all of them, guessing that is a problem

So analyzing first, than act ... ?
 

User avatar
Festus1965
Member
 
Posts: 712
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
GitHub: Minetest-One
In-game: Thomas Explorer

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Tue Mar 12, 2019 07:08

Pyrollo wrote:I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.


I am so sorry Pyrollo,
but both posts are from
* Sun May 17, 2015 16:16 - Lua PerlinNoiseMap memory efficiency improvements
* Wed May 11, 2016 20:26 - Perlin noise exceeds -1/1 range

so that I was quite sure that should have been solved ...

and so I am more locking in this direction Important oregen memory use optimization from Dec 2018 to understand ... I read a lot last 5 days, and will try out if have fun to do ...

as after that 5 days there was another 11 crashes, 7 of them by technic OOM hehe

I am looking for some tricks told
* https://stackoverflow.com/questions/27015150/how-to-get-past-1gb-memory-limit-of-64-bit-luajit-on-linux
* https://kvitajakub.github.io/2016/03/08/luajit-memory-limitations/
* https://github.com/cloudflare/luajit-mm
but still need to get back another (is in Celine's PC but she play Roblox - so I cant ...)
 


Return to Problems



Who is online

Users browsing this forum: No registered users and 2 guests