LuaJit OOM - how to detect Mods using it - and rid off !

Post Reply
User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

So, I put 32 GB of RAM into the MTS Server, but anyway it is useless, as the Server crashes latest up to 24 hours, before the last changed settings of ubuntu can use 80% of the RAM for writing cache ... most i just get up to 7 GB RAM using, and then gone ... All new and useless to put hardware in ... next i take the 32 GB back into my workpc, there i use just in 10 minutes near 10 GB ... as the server just reaches 7 GB, there 16 max is enough.

So to the topic:

How to detect LUAJIT using MODS causing OOM ?
* detect mod
* detect file with the code
* and how to change the code for better

or how to reduce LUAJIT amount ?
* mod [stop_lj_oom] tested, but lag ...


To check out weather I need them or not - to get less failure and longer runs of the MTS.

The OOM is not a fresh case, but I want to get rid of !

Links:
* Lua OOM crashes
* Minetest memory error(Lua OOM error not enough memory)
* Minetest with LuaJit GC64 mode (>2gb mem)
* Lua OOM errors
* Remove LuaJIT support?

* [mod] Stop LUAJIT OOM crashes [stop_lj_oom]

I will check the last link more now and learn again ...
edit: stop_lj_oom was running some days until value 1.450.000 but when active, getting about plus 0.8 lag

edit: detected now, mods which (even) crash, (when garbage collector is running)
* mod 'mcl_dungeons'
* mod worldview
* mod technic_worldgen
** here a possible solution - I just edited my mod "memory optimisation fix is simple"
* mod pipeworks
* mod 'biome_lib'
* mod fire
* mod 'televator'
Last edited by Festus1965 on Tue Feb 05, 2019 22:31, edited 13 times in total.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

So it is running,
original - no changes

and since the night is activates 4 times a second with a value about near 400000
mean:
it does collectgarbage() ... without much clean up, as OOM full (> 307200) just in not a second again !

server itself is still good with lag 0.32, mhhh

the last crashes before using [stop_lj_oom] have been reported with:
* editing ...
* 1533 MB

now changed setting to 600000 instead of 307xxx and restart server.


:.edit: 17.01. - 17.15
just back from a transport, and didnt see the server for 1.5 h
now every about 2.5 sec the "collectgarbage()" is running with values about 600.000
thinking, but will rise up to 900.000 next
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Update: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

the "collectgarbage()" is running with values about 1.200.000 in every near 6 seconds now
the lag is 1, even i am alone there ...
1 Core / 1 Thread CPU between 60-80%,
cache 21,9 GB

in the moment lag of 1 instead of 0.4 seams to be because of "collectgarbage()" ? ... as nothing else changed ... mhh, not funny
Last edited by Festus1965 on Tue Jan 22, 2019 08:14, edited 1 time in total.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

the "collectgarbage()" was running with values about 1.300.000
at the end it collected froom first just 40 sec more fast each near 6 sec again,

(with value of 1.500.000 it crashed after near 20 hours again with OOM at 970 MB)

and then mod worldview crashed the server even (guess taking too much - free after collectgarbage too less until out of Mem)

solution until now
* "collectgarbage()" back to value 1.200.000
* mod worldview = false
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

here the changed I did at technic/technic_worldgen/

Code: Select all

-- Sulfur
--local sulfur_buf = {}  -- this is old
local data = {}  --  this is new plus
local sulfur_noise = nil

minetest.register_on_generated(function(minp, maxp, seed)
	local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
	local a = VoxelArea:new{
		MinEdge = {x = emin.x, y = emin.y, z = emin.z},
		MaxEdge = {x = emax.x, y = emax.y, z = emax.z},
	}
	--local data = vm:get_data(sulfur_buf) -- this isold
	vm:get_data(data) -- this is new
	local pr = PseudoRandom(17 * minp.x + 42 * minp.y + 101 * minp.z)
	sulfur_noise = sulfur_noise or minetest.get_perlin(9876, 3, 0.5, 100)
one more,
i am on to learn how to detect the code in the mod and solute it ...
2019-01-27 11:30:47: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'biome_lib' in callback environment_Step(): not enough memory
2019-01-27 11:30:47: ERROR[Main]: Current Lua memory usage: 1057 MB
2019-01-27 11:30:47: ERROR[Main]: stack traceback:
Last edited by Festus1965 on Sat Feb 02, 2019 23:16, edited 2 times in total.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Code: Select all

2019-01-30 01:53:27: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod '[b]fire[/b]' in callback environment_Step(): not enough memory
2019-01-30 01:53:27: ERROR[Main]: Current Lua memory usage: 1274 MB
2019-01-30 01:53:27: ERROR[Main]: stack traceback:
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Code: Select all

2019-02-04 17:20:59: ERROR[Main]: ServerError: AsyncErr: ServerThread::run Lua: OOM error from mod 'pipeworks' in callback LuaABM::trigger(): not enough memory
2019-02-04 17:20:59: ERROR[Main]: Current Lua memory usage: 1341 MB
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Code: Select all

2019-02-05 14:26:38: ERROR[Main]: ServerError: AsyncErr: environment_Step: OOM error from mod 'televator' in callback environment_Step(): not enough memory
2019-02-05 14:26:38: ERROR[Main]: Current Lua memory usage: 1133 MB
2019-02-05 14:26:38: ERROR[Main]: stack traceback:
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Maybe I found something,

lua mem profiler

I will check it out next server starts.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Pyrollo
Developer
Posts: 385
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Pyrollo » Post

Hi,

I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Pyrollo wrote:I'm very interested to know if it's always because of use of vm get_data (I'm working on performances on big map updates and it may solve some OOM problems).

Do you get some cases when PerlinNoiseMap:get3dMap is implied ?
OOM
I was checking the git from my seen most OOM causing Mods (technic, pipeworks and mesecons) but cant really see if those are fixed with that code, I asked today morning, but downloaded and test when time

lua mem profiler, (was not running yet)

Code: Select all

2019-03-07 09:01:50: ERROR[Main]: ModError: Failed to load and run script from /home/thomas/.minetest/mods/luaprofiler/init.lua:
2019-03-07 09:01:50: ERROR[Main]: /home/thomas/.minetest/mods/luaprofiler/init.lua:458: '<eof>' expected near 'end'
that is added as mod and loaded without problem, but now I am missing a possibilities to start it
I need to build in some

Code: Select all

minetest.register_chatcommand("luamemcheck", {
what calls then the function

Code: Select all

ProFi:start( 'once' )
will see how to do

PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Pyrollo
Developer
Posts: 385
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Pyrollo » Post

Festus1965 wrote:PerlinNoiseMap:get3dMap
what is this, where, ... never seen/read this
I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

sure, but that fore i supposed to try some software that tells us, what mod is using most of the memory until it OOM-crashes - and then go to this 2 or 4 mods and optimize like you supposed ...
otherwise we have to upgrade all of them, guessing that is a problem

So analyzing first, than act ... ?
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

Pyrollo wrote:I refer to this because I suspect large table transfer from C to LUA to be slow. And it seems also related to OOM you mentioned. One of the solution is to stop creating big table in LUA and use another way to access large collection data from C.
I am so sorry Pyrollo,
but both posts are from
* Sun May 17, 2015 16:16 - Lua PerlinNoiseMap memory efficiency improvements
* Wed May 11, 2016 20:26 - Perlin noise exceeds -1/1 range

so that I was quite sure that should have been solved ...

and so I am more locking in this direction Important oregen memory use optimization from Dec 2018 to understand ... I read a lot last 5 days, and will try out if have fun to do ...

as after that 5 days there was another 11 crashes, 7 of them by technic OOM hehe

I am looking for some tricks told
* https://stackoverflow.com/questions/270 ... t-on-linux
* https://kvitajakub.github.io/2016/03/08 ... mitations/
* https://github.com/cloudflare/luajit-mm
but still need to get back another (is in Celine's PC but she play Roblox - so I cant ...)
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: LuaJit OOM - how to detect Mods using it - and rid off !

by Festus1965 » Post

next step:

I just deleted my github.

so also my [mod] free_lj_mem

even I don't find anymore the thread ...

continue to go ... out here

just got mail confirm "This email is to confirm that you’ve deleted your account ‘Minetest-One’ from GitHub. Your repositories and content have been deleted from the system. If you were on a paid plan, you will not be billed again. We’re sorry to see you go. You can reply directly to this email if you have any questions or feedback, we’d love to hear from you. "

feeling more free already
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest