Remove LuaJIT support?

User avatar
Wuzzy
Member
 
Posts: 3823
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Remove LuaJIT support?

by Wuzzy » Mon Aug 13, 2018 09:59

People frequently get Lua OOM errors. Devs frequently claim it has something to do with LuaJIT.

viewtopic.php?f=47&t=18820
viewtopic.php?f=3&t=18263
viewtopic.php?f=6&t=13826

Even sfan5 said a while ago:
So far I've had no luck getting that mode to run without segfaults under stress testing (on Linux).

It seems something is seriously broken here.

Would it be possible to just drop LuaJIT support from Minetest? It always seems the source of so many worries. I don't even understand why we have to support LuaJIT in the first place.
So the simplest solution seems to just get rid of it.
Or at least tuck it away in a branch until it's ready for prime time.

Note I do not understand the inner workings of Minetest, also, I am the opposite of a LuaJIT expert ;-) , so I might be mistaken here.
My creations. I gladly accept bitcoins: 17fsUywHxeMHKG41UFfu34F1rAxZcrVoqH
 

User avatar
ChimneySwift
Member
 
Posts: 320
Joined: Fri Sep 22, 2017 06:46
Location: 127.0.0.1
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift

Re: Remove LuaJIT support?

by ChimneySwift » Mon Aug 13, 2018 10:44

IIRC LuaJIT is much faster, which is pretty good for servers...
A spoon is basically a tiny bowl with a stick on it
 

User avatar
rubenwardy
Moderator
 
Posts: 6012
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Remove LuaJIT support?

by rubenwardy » Mon Aug 13, 2018 10:50

Neither I nor my server has ever had an OOM. I think it's partially hardware related, but mainly mod related - I don't use any mods which manipulate perlin noise, for example. The perlin noise bit can be fixed by making use of FFI - fast something interface - which allows Lua to use more than the limit

If this isn't fixable, then it should probably be disabled by default
 

User avatar
ExeterDad
Member
 
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
Location: New Hampshire U.S.A
In-game: ExeterDad

Re: Remove LuaJIT support?

by ExeterDad » Mon Aug 13, 2018 11:28

I've also never experienced a OOM. Even with machines with lowend specs.
I would be very disappointed if I didn't have the option to take advantage of the LuaJit performance.
 

twoelk
Member
 
Posts: 1352
Joined: Fri Apr 19, 2013 16:19
Location: northern Germany
GitHub: twoelk
IRC: twoelk
In-game: twoelk

Re: Remove LuaJIT support?

by twoelk » Mon Aug 13, 2018 13:12

This is the original issue where LuaJit was identified as one possible source of OutOfMemory crashes
https://github.com/minetest/minetest/issues/2988
notice it doesn't matter how much RAM your setup can provide, LuaJit has a built in memory limit of sorts by design and it seems this may not be changed anytime soon from the LuaJit side.

So if you intend to do something really memory intensive like doing large World-Editing (like importing 2 million nodes and above in a single action) you might be advised to do such in a non-luajit-version of minetest. Be prepared for slow processing at times though.
Last edited by twoelk on Tue Aug 21, 2018 18:16, edited 2 times in total.
 

User avatar
v-rob
Member
 
Posts: 729
Joined: Thu Mar 24, 2016 03:19
Location: Right behind you.
GitHub: v-rob

Re: Remove LuaJIT support?

by v-rob » Mon Aug 13, 2018 16:32

I used to get OOM errors all the time back when I used 0.4.14 and below, and that was from exploring the world, and not even a very big portion of it, with no mods. I haven't gotten any OOMs since 0.4.15 except for when I use extensive WorldEdit or some mods using perlin-noise.

So, I have to say that I'm against removing LuaJIT by default, but I'm always for adding more options. If there was an option for it, I would be more than happy because there will always be others who want it disabled.

There should be some page on the forums/wiki that explains OOM errors and what you can do to stop them before reporting one. Probably on the forums as a sticky topic in Problems.
 

User avatar
Wuzzy
Member
 
Posts: 3823
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Remove LuaJIT support?

by Wuzzy » Mon Aug 13, 2018 17:16

Don't you think that LuaJIT is a problem when Minetest keeps crashing for large portion of players?
It's a bad idea to keep something like this enabled by default.

Don't expect players to fix all the problems in Minetest themselves. Most players are not coders. Many players don't even know what LuaJIT is and they will be completely stuck and locked out of playing Minetest.
That's why sane defaults are so important.
My creations. I gladly accept bitcoins: 17fsUywHxeMHKG41UFfu34F1rAxZcrVoqH
 

User avatar
Krock
Developer
 
Posts: 4478
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: Remove LuaJIT support?

by Krock » Mon Aug 13, 2018 18:01

Wuzzy wrote:It's a bad idea to keep something like this enabled by default..

It's not like the LuaJIT developer's main intention is to provide the worst user experience. I'm pretty sure he's aware about this issue since the patch for Windows exists already (64 bit GC builds). Of course would it be nice to not have these memory limits - but either way should mods be designed so that they could also be used on low-end servers (includes Android, singleplayer).
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

Astrobe
Member
 
Posts: 303
Joined: Sun Apr 01, 2018 10:46

Re: Remove LuaJIT support?

by Astrobe » Mon Aug 13, 2018 18:01

> Don't you think that LuaJIT is a problem when Minetest keeps crashing for large portion of players?

I tend to blame the mods and remove them when I get OOMs. I'd rather sack a still-in-beta or poorly written mod than the performance boost LuaJIT provides to all of the other well-behaving mods.

As for players, they will have to learn how to deal with problematic mods anyway - be it because of OOMs, massive lags, mod incompatibilities or plain scripting bugs.

Players should actually prefer to stick to games. They do that selection for them, in addition to providing (hopefully) a consistent set of features.
 

User avatar
sorcerykid
Member
 
Posts: 1247
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: Remove LuaJIT support?

by sorcerykid » Mon Aug 13, 2018 19:31

Personally, I view LuaJIT as a crutch. It's nice to have, but shouldn't be in any way be deemed a necessityr. I think mods and games should be tuned and optimized enough so as not to require a LuaJIT environment, but if one is available then they will perform better than optimal.

This is at least one good argument for why LuaJIT as a default can be problematic, at least in the context of mod or game development.
 

User avatar
Wuzzy
Member
 
Posts: 3823
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Remove LuaJIT support?

by Wuzzy » Mon Aug 13, 2018 19:42

I also have the impression that the memory limit forced by LuaJIT is frustratingly small, much smaller than the system could offer. For complex games the only way to “fix” this is to dump lots of files. This can't be the solution. The memory limit forced by LuaJIT is just completely arbitrary.

I think it's unfair to blame games for getting OOM. Clearly there needs to be a better way.
My creations. I gladly accept bitcoins: 17fsUywHxeMHKG41UFfu34F1rAxZcrVoqH
 

User avatar
rubenwardy
Moderator
 
Posts: 6012
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Remove LuaJIT support?

by rubenwardy » Mon Aug 13, 2018 20:53

The memory limit is around 4GB, and is not arbitrary at all - the size of a 32 bit integer. The issue is that there is some bug which makes this limit a lot lower in practice, which may be linked to hardware or the OS, I'm unsure. It's also possible that minetest is wrongly reporting metrics

In any case, it shouldn't be possible to use up 4GB of RAM with normal mods. FFI can be used to store perlin noises on the c++ with no performance loss. Databasess should be used instead of massive in-memory tables, for example Auth, but this is only a problem on multiplayer

There's also the experimental gc64, which allows luajit to use a lot more memory. I'm not sure how reliable it is. sfan5's builds use it, I believe
 

Astrobe
Member
 
Posts: 303
Joined: Sun Apr 01, 2018 10:46

Re: Remove LuaJIT support?

by Astrobe » Wed Aug 15, 2018 15:24

rubenwardy wrote:In any case, it shouldn't be possible to use up 4GB of RAM with normal mods. FFI can be used to store perlin noises on the c++ with no performance loss. Databases should be used instead of massive in-memory tables, for example Auth, but this is only a problem on multiplayer


It's not just about Perlin noises. I've seen mods OOM-crash that don't do anything Perlin related (FFI stands for "foreign function interface" BTW).

I'm not familiar with LuaJIT, but in plain Lua one can write bindings to interface with native functions and memory structures. It's kind of like adding primitives to Lua. One typically resorts to this when scripting is not fast enough.

Stuff like basic mob handling (collision detection at least), falling nodes, and other functions found in MTG's default are good candidates for this.

We could have almost the best of both worlds - fast API functions, no OOM-crashes, availability on all platforms (LuaJIT is basically an on-the-fly compiler, which means you need an assembler guru to port it; if it's not supported on your CPU, no candy for you) and even maybe an update to Lua 5.2 or 5.3.

Of course, in addition of issues I'm not aware of, it would be more than a lot of work.
 

User avatar
Wuzzy
Member
 
Posts: 3823
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Remove LuaJIT support?

by Wuzzy » Tue Aug 21, 2018 10:45

So. The problem is:
Players continue to experience problems with LuaJIT frequently which locks them ouf of gamepla. This is a real problem. Remember, we are NOT talking about some minor bug here, it's about crashes which make games unplayable!
Obviously, players have and will blame games, mods and Minetest as being crashy, broken and unreliable. And rightfully so, I might add.

So, since removing LuaJIT is not really an option, what else could be done to fix the problem?

One first easy short-term solution might be to disable LuaJIT by default (if it isn't already) and declare it VERY CLEARLY as “experimental” and “for advanced users only” and say “abandon all hope, ye who proceeds here”. :-)
Maybe also write somewhere in the main menu when LuaJIT is enabled, as a warning.
My creations. I gladly accept bitcoins: 17fsUywHxeMHKG41UFfu34F1rAxZcrVoqH
 

User avatar
GoldFireUn
Member
 
Posts: 104
Joined: Sun Aug 21, 2016 13:30
Location: Wisconsin, USA
GitHub: BluebirdGreycoat
In-game: GoldFireUn

Re: Remove LuaJIT support?

by GoldFireUn » Tue Sep 04, 2018 01:19

If my understanding is correct, even with buggy LuaJIT mods can use up to 2 GB of memory. Which is quite a lot. It seems the best solution right now is to just write better code and clean up the clutter that so many modders seem to make. ;-) The cost of the loss of performance in removing LuaJIT is just too high IMO.

My server runs an extremely complex survival game, with 2 Lua mapgens and a third planned. I haven't seen Lua OOM in over a year.
 

Byakuren
Member
 
Posts: 816
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

Re: Remove LuaJIT support?

by Byakuren » Wed Sep 05, 2018 22:07

LuaJIT dependency cannot be solved fully by optimizing the mod if the mod implements some potentially-expensive algorithm. For example, keeping track of power network connectivity.
Every time a mod API is left undocumented, a koala dies.
 

User avatar
Glory!
Member
 
Posts: 92
Joined: Thu Apr 30, 2015 17:45
Location: Kernel Debugging Land <3
GitHub: Glory7000
In-game: Glory7000

Re: Remove LuaJIT support?

by Glory! » Sun Sep 09, 2018 23:56

When we release Minetest Quantum, we'll remove not just LuaJIT, but also all Lua support.

All content for MT is written in Lua, LuaJIT is merely an improvement over the standard implementation, removing Lua support would practically kill all content and the community, what are you, nuts?
Behold the Razgriz, its wings of black sheath. :: My skin A competitor that unfortunately needs attention to stay afloat.
 


Return to General Discussion



Who is online

Users browsing this forum: GreenDimond and 2 guests