Remove LuaJIT support?

Post Reply
User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Remove LuaJIT support?

by Wuzzy » Post

People frequently get Lua OOM errors. Devs frequently claim it has something to do with LuaJIT.

viewtopic.php?f=47&t=18820
viewtopic.php?f=3&t=18263
viewtopic.php?f=6&t=13826

Even sfan5 said a while ago:
So far I've had no luck getting that mode to run without segfaults under stress testing (on Linux).
It seems something is seriously broken here.

Would it be possible to just drop LuaJIT support from Minetest? It always seems the source of so many worries. I don't even understand why we have to support LuaJIT in the first place.
So the simplest solution seems to just get rid of it.
Or at least tuck it away in a branch until it's ready for prime time.

Note I do not understand the inner workings of Minetest, also, I am the opposite of a LuaJIT expert ;-) , so I might be mistaken here.

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: Remove LuaJIT support?

by ChimneySwift » Post

IIRC LuaJIT is much faster, which is pretty good for servers...
A spoon is basically a tiny bowl with a stick on it

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: Remove LuaJIT support?

by rubenwardy » Post

Neither I nor my server has ever had an OOM. I think it's partially hardware related, but mainly mod related - I don't use any mods which manipulate perlin noise, for example. The perlin noise bit can be fixed by making use of FFI - fast something interface - which allows Lua to use more than the limit

If this isn't fixable, then it should probably be disabled by default
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: Remove LuaJIT support?

by ExeterDad » Post

I've also never experienced a OOM. Even with machines with lowend specs.
I would be very disappointed if I didn't have the option to take advantage of the LuaJit performance.

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Remove LuaJIT support?

by twoelk » Post

This is the original issue where LuaJit was identified as one possible source of OutOfMemory crashes
https://github.com/minetest/minetest/issues/2988
notice it doesn't matter how much RAM your setup can provide, LuaJit has a built in memory limit of sorts by design and it seems this may not be changed anytime soon from the LuaJit side.

So if you intend to do something really memory intensive like doing large World-Editing (like importing 2 million nodes and above in a single action) you might be advised to do such in a non-luajit-version of minetest. Be prepared for slow processing at times though.
Last edited by twoelk on Tue Aug 21, 2018 18:16, edited 2 times in total.

User avatar
v-rob
Developer
Posts: 970
Joined: Thu Mar 24, 2016 03:19
GitHub: v-rob
IRC: v-rob
Location: Right behind you.

Re: Remove LuaJIT support?

by v-rob » Post

I used to get OOM errors all the time back when I used 0.4.14 and below, and that was from exploring the world, and not even a very big portion of it, with no mods. I haven't gotten any OOMs since 0.4.15 except for when I use extensive WorldEdit or some mods using perlin-noise.

So, I have to say that I'm against removing LuaJIT by default, but I'm always for adding more options. If there was an option for it, I would be more than happy because there will always be others who want it disabled.

There should be some page on the forums/wiki that explains OOM errors and what you can do to stop them before reporting one. Probably on the forums as a sticky topic in Problems.
Core Developer | My Best Mods: Bridger - Slats - Stained Glass

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: Remove LuaJIT support?

by Wuzzy » Post

Don't you think that LuaJIT is a problem when Minetest keeps crashing for large portion of players?
It's a bad idea to keep something like this enabled by default.

Don't expect players to fix all the problems in Minetest themselves. Most players are not coders. Many players don't even know what LuaJIT is and they will be completely stuck and locked out of playing Minetest.
That's why sane defaults are so important.

User avatar
Krock
Developer
Posts: 4649
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Remove LuaJIT support?

by Krock » Post

Wuzzy wrote:It's a bad idea to keep something like this enabled by default..
It's not like the LuaJIT developer's main intention is to provide the worst user experience. I'm pretty sure he's aware about this issue since the patch for Windows exists already (64 bit GC builds). Of course would it be nice to not have these memory limits - but either way should mods be designed so that they could also be used on low-end servers (includes Android, singleplayer).
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

Astrobe
Member
Posts: 570
Joined: Sun Apr 01, 2018 10:46

Re: Remove LuaJIT support?

by Astrobe » Post

> Don't you think that LuaJIT is a problem when Minetest keeps crashing for large portion of players?

I tend to blame the mods and remove them when I get OOMs. I'd rather sack a still-in-beta or poorly written mod than the performance boost LuaJIT provides to all of the other well-behaving mods.

As for players, they will have to learn how to deal with problematic mods anyway - be it because of OOMs, massive lags, mod incompatibilities or plain scripting bugs.

Players should actually prefer to stick to games. They do that selection for them, in addition to providing (hopefully) a consistent set of features.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: Remove LuaJIT support?

by sorcerykid » Post

Personally, I view LuaJIT as a crutch. It's nice to have, but shouldn't be in any way be deemed a necessityr. I think mods and games should be tuned and optimized enough so as not to require a LuaJIT environment, but if one is available then they will perform better than optimal.

This is at least one good argument for why LuaJIT as a default can be problematic, at least in the context of mod or game development.

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: Remove LuaJIT support?

by Wuzzy » Post

I also have the impression that the memory limit forced by LuaJIT is frustratingly small, much smaller than the system could offer. For complex games the only way to “fix” this is to dump lots of files. This can't be the solution. The memory limit forced by LuaJIT is just completely arbitrary.

I think it's unfair to blame games for getting OOM. Clearly there needs to be a better way.

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: Remove LuaJIT support?

by rubenwardy » Post

The memory limit is around 4GB, and is not arbitrary at all - the size of a 32 bit integer. The issue is that there is some bug which makes this limit a lot lower in practice, which may be linked to hardware or the OS, I'm unsure. It's also possible that minetest is wrongly reporting metrics

In any case, it shouldn't be possible to use up 4GB of RAM with normal mods. FFI can be used to store perlin noises on the c++ with no performance loss. Databasess should be used instead of massive in-memory tables, for example Auth, but this is only a problem on multiplayer

There's also the experimental gc64, which allows luajit to use a lot more memory. I'm not sure how reliable it is. sfan5's builds use it, I believe
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

Astrobe
Member
Posts: 570
Joined: Sun Apr 01, 2018 10:46

Re: Remove LuaJIT support?

by Astrobe » Post

rubenwardy wrote: In any case, it shouldn't be possible to use up 4GB of RAM with normal mods. FFI can be used to store perlin noises on the c++ with no performance loss. Databases should be used instead of massive in-memory tables, for example Auth, but this is only a problem on multiplayer
It's not just about Perlin noises. I've seen mods OOM-crash that don't do anything Perlin related (FFI stands for "foreign function interface" BTW).

I'm not familiar with LuaJIT, but in plain Lua one can write bindings to interface with native functions and memory structures. It's kind of like adding primitives to Lua. One typically resorts to this when scripting is not fast enough.

Stuff like basic mob handling (collision detection at least), falling nodes, and other functions found in MTG's default are good candidates for this.

We could have almost the best of both worlds - fast API functions, no OOM-crashes, availability on all platforms (LuaJIT is basically an on-the-fly compiler, which means you need an assembler guru to port it; if it's not supported on your CPU, no candy for you) and even maybe an update to Lua 5.2 or 5.3.

Of course, in addition of issues I'm not aware of, it would be more than a lot of work.

User avatar
Wuzzy
Member
Posts: 4786
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy
Contact:

Re: Remove LuaJIT support?

by Wuzzy » Post

So. The problem is:
Players continue to experience problems with LuaJIT frequently which locks them ouf of gamepla. This is a real problem. Remember, we are NOT talking about some minor bug here, it's about crashes which make games unplayable!
Obviously, players have and will blame games, mods and Minetest as being crashy, broken and unreliable. And rightfully so, I might add.

So, since removing LuaJIT is not really an option, what else could be done to fix the problem?

One first easy short-term solution might be to disable LuaJIT by default (if it isn't already) and declare it VERY CLEARLY as “experimental” and “for advanced users only” and say “abandon all hope, ye who proceeds here”. :-)
Maybe also write somewhere in the main menu when LuaJIT is enabled, as a warning.

User avatar
GoldFireUn
Member
Posts: 108
Joined: Sun Aug 21, 2016 13:30
GitHub: BluebirdGreycoat
In-game: GoldFireUn
Location: Wisconsin, USA
Contact:

Re: Remove LuaJIT support?

by GoldFireUn » Post

If my understanding is correct, even with buggy LuaJIT mods can use up to 2 GB of memory. Which is quite a lot. It seems the best solution right now is to just write better code and clean up the clutter that so many modders seem to make. ;-) The cost of the loss of performance in removing LuaJIT is just too high IMO.

My server runs an extremely complex survival game, with 2 Lua mapgens and a third planned. I haven't seen Lua OOM in over a year.

Byakuren
Member
Posts: 818
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

Re: Remove LuaJIT support?

by Byakuren » Post

LuaJIT dependency cannot be solved fully by optimizing the mod if the mod implements some potentially-expensive algorithm. For example, keeping track of power network connectivity.
Every time a mod API is left undocumented, a koala dies.

User avatar
Glory!
Member
Posts: 92
Joined: Thu Apr 30, 2015 17:45
GitHub: Glory7000
In-game: Glory7000
Location: Kernel Debugging Land <3

Re: Remove LuaJIT support?

by Glory! » Post

When we release Minetest Quantum, we'll remove not just LuaJIT, but also all Lua support.

All content for MT is written in Lua, LuaJIT is merely an improvement over the standard implementation, removing Lua support would practically kill all content and the community, what are you, nuts?
Behold the Razgriz, its wings of black sheath. :: My skin A competitor that unfortunately needs attention to stay afloat.

Post Reply

Who is online

Users browsing this forum: Andrey01 and 10 guests