Lua OOM crashes

Post Reply
blert2112
Member
Posts: 244
Joined: Sat Apr 25, 2015 04:05
GitHub: blert2112

Lua OOM crashes

by blert2112 » Post

This is driving me flippin' crazy! I have been hammering on this all day and have gotten nowhere. I continually get Lua Out-Of-Memory crashes when using some of the mods I like the most.
Using a two day old build (or sfan5's latest):
running only Minetest_Game, mg_villages and handle_schematics results in an almost immediate crash,
running only Minetest_game, biome_lib, plantlife and moretrees results in a crash after a couple minutes.
So, I figured I would try the Valleys C mapgen to see if I could get around using these big mods. Worked fine until I installed the valleys_c support mod... OOM crash.
I tried recompiling many times with two different LuaJit libraries (the one you get from buildbot and the one installed in my MinGW setup). No joy.
I recompiled without Luajit and the OOM crashes go away but that opens up to other mods failing because, being written for LuaJit (which has a few enhancements found in Lua5.2), they crash when they come up against Lua5.1.
Any help would be greatly appreciated.
PC specs:
Core I5-5200U 2.2GHz
8GB RAM
Win10 x64
nothing of consequence running that would eat resources

edit: new info
Using sfan5's builds, I went back a few months to see if some recent change is causing my troubles. No luck. For grins I decided to try a 32bit build. Guess what? No crash! Ok, that was Krock's 32 bit MSVC build... What if it has something to do with MinGW builds not getting along with Win10? So I tried sfan5's latest 32bit build... No crash!
Something is going on here that is beyond my knowledge. For now I will have to go with a 32bit build and maybe do a bit of research.

edit2: more new info
Here's the thing... If I tell the video driver to use the NVidia 940M w/4GB there is no longer an OOM crash. So...
On my rig:
- 64bit build with the integrated Intel video = OOM crash
- 32bit build with the integrated Intel video = no crash
- 64 or 32bit build using the NVidia = no crash
It seems that memory sharing with the integrated Intel video is getting in Lua's way even though it can only grab at most 2GB out of 8 and there is plenty left over just sitting idle. I assume this won't be a problem once I move the game onto my server as graphics won't be an issue for it. But, I guess I am going to have to do some testing on my kid's laptops as they don't have the option to use dedicated graphics.

edit3: yet again
Spoke too soon. :(
OOM crash again with 64bit build of Valleys_c mapgen + valleys_c lua supplement using dedicated graphics. Beginning to wonder if this is worth the effort.

GeneticAlgorithm
New member
Posts: 3
Joined: Sat Jan 09, 2016 21:43

Re: Lua OOM crashes

by GeneticAlgorithm » Post

I have been having this problem too. I'll put more info on it soon. I've had to stay with 4.12.

I have used:
minetest-0.4.13-win64-msvc
(I think this is the main download)

Specs
Intel Pentium 2020M
4GB RAM
Integrated Intel graphics
Win 8.1 x64

EDIT: I downloaded the latest version from sfan5, build bd40ee2-win64. So far, the error hasn't occurred.

EDIT2: Occurs with new world made with recent release of Dreambuilder. '80 mb'

EDIT3: Tried 'Mod Profiling'. Appeared to speed things up for a bit, but got this
Lua: OOM error from mod 'technic_worldgen' in callback environment_OnGenerated(
): not enough memory
Current Lua memory usage: 52 MB
Last edited by GeneticAlgorithm on Sun Jan 10, 2016 00:16, edited 2 times in total.

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: Lua OOM crashes

by duane » Post

Is there really that much difference between the 32 and 64 bit builds? I never noticed much in the experimentation I did back when everyone was switching over; I just went to 64 bit by default.
Believe in people and you don't need to believe anything else.

GeneticAlgorithm
New member
Posts: 3
Joined: Sat Jan 09, 2016 21:43

Re: Lua OOM crashes

by GeneticAlgorithm » Post

Not sure if it does. I just received it again with the 64bit. Haven't tested the 32bit yet, will update. However, I think the error will appear again because Dreambuilder is a large subgame with lots of lua code and objects. I googled Lua memory adjustment. Found info adjusting Lua memory for Elder Scrolls Online. I think we may need that feature for Minetest.

I think the problem is literally a memory restriction for lua. So this error may be common for heavy subgames and those with lots of big mods (like technic)
Last edited by GeneticAlgorithm on Sun Jan 10, 2016 01:40, edited 2 times in total.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: Lua OOM crashes

by paramat » Post

It's the lower memory restriction for LuaJIT. Mods shouldn't be written to only work with LuaJIT.
Complex lua mapgens (like some of mine and valleys mapgen) use up a lot of memory when using noise, especially 3D noise, enough to hit the LuaJIT ~2GB limit.
For now just don't use LuaJIT.

GeneticAlgorithm
New member
Posts: 3
Joined: Sat Jan 09, 2016 21:43

Re: Lua OOM crashes

by GeneticAlgorithm » Post

Thanks!
PS: I'm a big fan of you're mods!

blert2112
Member
Posts: 244
Joined: Sat Apr 25, 2015 04:05
GitHub: blert2112

Re: Lua OOM crashes

by blert2112 » Post

I found a work-around. Instead of playing a single player game, I fire up a server and connect to it by running the client on the same machine, so two processes instead of one. It's a pain when testing mods but it's better than constantly crashing. Probably not what you want to do if you don't have at least four cores though.

User avatar
Don
Member
Posts: 1643
Joined: Sat May 17, 2014 18:40
GitHub: DonBatman
IRC: Batman
In-game: Batman

Re: Lua OOM crashes

by Don » Post

GeneticAlgorithm wrote:Thanks!
PS: I'm a big fan of you're mods!
Who isn't a fan? Paramat has done some amazing work.
Many of my mods are now a part of Minetest-mods. A place where you know they are maintained!

A list of my mods can be found here

User avatar
xeranas
Member
Posts: 162
Joined: Fri Feb 05, 2016 11:06

Re: Lua OOM crashes

by xeranas » Post

paramat wrote:Mods shouldn't be written to only work with LuaJIT
How does this looks like? How I can tell that mod only works with LuaJIT?
paramat wrote:For now just don't use LuaJIT.
Ok, how?
I downloaded windows build from minetest dowload page (0.4.14 64bit) and I getting OOM when hitting 12MB?

Code: Select all

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB
Do I miss put magic parameter somewhere?

User avatar
azekill_DIABLO
Member
Posts: 7507
Joined: Wed Oct 29, 2014 20:05
GitHub: azekillDIABLO
In-game: azekill_DIABLO
Location: OMICRON
Contact:

Re: Lua OOM crashes

by azekill_DIABLO » Post

12 MB??! that a lot man!

i have a world with 40 mods with no bugs (and very heavy mods like mymonths)
Gone, but not dead. Contact me on discord: azekill_DIABLO#6565
DMs are always open if you want to get in touch!

blert2112
Member
Posts: 244
Joined: Sat Apr 25, 2015 04:05
GitHub: blert2112

Re: Lua OOM crashes

by blert2112 » Post

xeranas wrote:
paramat wrote:Mods shouldn't be written to only work with LuaJIT
How does this looks like? How I can tell that mod only works with LuaJIT?
Unless you comb through the code you will not be able to tell. Most (if not all) mods seems to be made forgetting that someone may not be using LuaJIT. LuaJIT is more like Lua5.2 than Lua5.1: for instance, math.random() can break under Lua5.1 because it will wrap back around into negative numbers if the numbers fed to it are too large (Lua5.2 and JIT do not have this problem).
xeranas wrote:
paramat wrote:For now just don't use LuaJIT.
Ok, how?
I downloaded windows build from minetest dowload page (0.4.14 64bit) and I getting OOM when hitting 12MB?
You will need to compile MineTest without LuaJIT. I thought there might be a way to turn it on and off ( jit.on()/jit.off() ) but it doesn't work.
xeranas wrote:

Code: Select all

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB
Do I miss put magic parameter somewhere?
This means the memory usage was 12MB BEFORE something caused the OOM error. It does not reflect an actual accurate metric of memory usage.

What are you doing when the crash happens? Just playing a normal game or cruising around in fast mode? There are a few workarounds you can try: my Stop Lua OOM Crashes mod, remove some heavy mods, (my personal goto) run the server and client as separate processes.

One big thing that mod creators need to start doing... stop using globals for everything (globals are NOT tended to by the GC and are never cleared from memory even if they will never be used again). All those leftover somemod.register_something() functions and giant global tables that are no longer needed after the game loads and should be cleared. On my personal server I have totally re-written every mod in Minetest_Game (along with HomeDecor, Valleys_c helper, and a bunch of others) and have done away with almost every global function, table and var unless it is totally necessary.

anuser
Member
Posts: 22
Joined: Sun May 01, 2016 23:07

Re: Lua OOM crashes

by anuser » Post

I have the same problem. With 64 version of newest engine i have out of memory error just a few sec after begining. This make this game not playable. And it happens only when I have worldgen of technic mode turned on. Also the game load more slowly then. It is not related to how many other mods I run. But with the 32 version of the same engine this problem not exist: I can run all mods I have with full technic and there is no problem. But this not mean that 32 version is OK.

It can be also not playable an crash just after the world is loaded at begining. But this time this happened only when I have Sphax TestBDcraft textures turned on. Without them evrything seem to be all right. Of course I know that this is not related to technic or textures packages. With older engine release there was no problem with that modes/textures at all. In this case there is no message about out of memory, this look like some other error.

Details of 32 version problem
Spoiler
Windows 7, minetest-0.4.14-win32 sfan5's build
I do not have "/home/stefan/mt-build/build/minetest..." directory

Code: Select all

2016-05-24 21:59:57: ERROR[MeshUpdate]: In thread 179c:
2016-05-24 21:59:57: ERROR[MeshUpdate]: /home/stefan/mt-build/build/minetest/minetest/src/util/thread.h:e2: run: A fatal error occured: std::bad_alloc
2016-05-24 21:59:57: ERROR[MeshUpdate]: Debug stacks:
2016-05-24 21:59:57: ERROR[MeshUpdate]: DEBUG STACK FOR THREAD 358: 
2016-05-24 21:59:57: ERROR[MeshUpdate]: #0  main
2016-05-24 21:59:57: ERROR[MeshUpdate]: #1  renderMap
2016-05-24 21:59:57: ERROR[MeshUpdate]: DEBUG STACK FOR THREAD d60: 
2016-05-24 21:59:57: ERROR[MeshUpdate]: #0  run
2016-05-24 21:59:57: ERROR[MeshUpdate]: DEBUG STACK FOR THREAD e24: 
2016-05-24 21:59:57: ERROR[MeshUpdate]: #0  run
2016-05-24 21:59:57: ERROR[MeshUpdate]: DEBUG STACK FOR THREAD 1738: 
2016-05-24 21:59:57: ERROR[MeshUpdate]: #0  run
2016-05-24 21:59:57: ERROR[MeshUpdate]: #1  AsyncRunStep
2016-05-24 21:59:57: ERROR[MeshUpdate]: #2  step
2016-05-24 21:59:57: ERROR[MeshUpdate]: DEBUG STACK FOR THREAD 179c: 
2016-05-24 21:59:57: ERROR[MeshUpdate]: #0  run
I guess that this out of memory crashes were not present in earlier versions and it shows up only with recent versions. If I am right, maybe some people smarter than me could review the last changes in code to find the problem. It must be there, with recent changes. There is always possibility to compile engine without such single recent-code-update to see if the problem is still present. Maybe I do not understand how it works and I am wrong here, posting some stupid advices. But the problem exist and is really serious.

I am not coder but I think this is critical error that make the game not playable and reduce it to work-in-progress-without-public-release version. Sorry, if anyone will ask me now about Minetest, I can only tell him - in this moment - do not even try this, just forget about it. I mean this problem with out of memory crashes should make red alert here and be the main topic right now.

Edit: I am talking about the singleplayer mode and sorry for my language.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: Lua OOM crashes

by Fixer » Post

Ha! I've heared this "no OOM crash on 32bit builds" from maikerumine, interesting.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: Lua OOM crashes

by Fixer » Post

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB
I have a custom subgame (for testing) that gives this kind of error, but for 'darkage' mod, interesting. Here it is:
2016-05-25 16:11:28: ERROR[Main]: ServerError: Lua: OOM error from mod 'darkage' in callback environment_OnGenerated(): not enough memory
2016-05-25 16:11:28: ERROR[Main]: Current Lua memory usage: 22 MB

blert2112
Member
Posts: 244
Joined: Sat Apr 25, 2015 04:05
GitHub: blert2112

Re: Lua OOM crashes

by blert2112 » Post

@Fixerol: Which DarkAge are you using? One that is using the VoxelManipulator I would have to guess. The problem is not from the DarkAge mod itself. BTW, try my version of DarkAge, it doesn't use the VM, it uses the ore_gen.

I get these LuaJIT OOM crashes when using the VoxelManipulator. Especially if more than one mod is using it. I can't say if there is a memory leak in the code (because I am not good enough at C++ to tell what's going on), in fact, there probably isn't. The issue looks to be with LuaJIT. The Lua garbage collector is not running all the time. If I recall correctly, it runs about every two seconds by default. A lot can happen in two seconds and the VoxelManipulator object and associated tables are quite large and can take up a good chunk of LuaJIT's memory. If more objects and associated tables are created before some of the others are garbage collected then the memory usage can add up quick. Add to this the fact that the GarbageCollector is quite inefficient when it has to deal with a lot of objects. I have successfully moved the DATA and AREAINDEX tables outside of LuaJIT's memory limit but it doesn't seem to help much because the functions to create these tables consume the same amount of memory as the table itself does. Yes, that memory can be set to NIL but it doesn't make a difference because nothing is actually freed until the garbage collector runs again. "Run the garbage collector more often" you may say... Well, you can but doing so can actually slow everything down and cause other issues and trying to run the GC while it is currently running can cause real bad things to happen. LuaJIT3 is supposed to have a brand new GarbageCollector with no memory limit but as far as I have seen it is not even on the horizon and there is no beta or alpha of it either. The best things that can be done to help alleviate this problem is to: 1) stop using globals for everything, 2) if you have to use a global clear it when it is no longer needed (example: mymod:register_something() ... clear this in a minetest.after(0) ), 3) reuse local objects (create the object outside of the loop instead of declaring 'local somevar' every iteration). Hopefully some of this will help someone but there is no way around the LuaJIT memory limit until version3 comes out... unless a table is large and needs to be kept in memory all the time, then it can be moved outside the memory limit quite easily.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: Lua OOM crashes

by Fixer » Post

It uses VoxelManipulator.

Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests