Python as a modding language

[prestidigitator]

Oh, and the idea that Python constrains your choice of object-orientation is also not true. Python's metaclasses and custom attribute access, for example, make prototyping pretty damned easy if that's what you want to do. At least as easy as hacking other forms of object-orientation into Lua.

[v-rob]

But Python and Lua together? That's a big problem, and it isn't worth the trouble and confusion. Plus, the devs aren't going to do it because Python is unnecessary because Lua works just fine, all mods are made in Lua, and they already have enough to do.

[sofar]

Can we change the poll option "yes" to "yes but I understand it will never happen" ?

[Skulls]

Can we change the poll option "yes" to "yes but I understand it will never happen" ?

+1!

And add an Ada option, just for kicks.

[prestidigitator]

Eh. I certainly don't expect it'll happen in the actual game, given the history and attitude of the development team. But someone might be interested in doing it in a fork or in a similar project, so it's still pretty useful info.

[burli]

So in an apples-to-apples comparison Python is about 18% slower (66 seconds vs. 56 seconds) for this particular benchmark.

Did you run the python code on multiple cores? If yes, then YOU don't make an apples-to-apples comparison, because you removed the optimization on one language, but leave it to the other, and this makes the performance of LuaJIT even more impressive. An unoptimized Lua code running on a single core is still faster than an optimized python code running on multiple cores.

This "benchmark game" tries to compare languages in a certain way. Everyone can try to write optimized code for each language and compare with each other. In this case, Lua tries to optimize with caching, python tries to optimize with multicores.

In addition, none of these benchmarks compare various methods of transitioning between C and the particular scripting language in question, and that is likely to be far more interesting in the Minetest case than shuffling some array elements around like in the benchmark you chose. Comparing similar tasks involving bidirectional calls and data movement between the Lua API and, for example, CFFI for Python would be an interesting exercise. Maybe I'll play with that at some point.

You are right, but I don't think that there will be much difference.

There is a reason why Python is not used in most games. You can't use the benefit of Python. You need performance. And it must be safe (no/restricted OS access)

[prestidigitator]

So in an apples-to-apples comparison Python is about 18% slower (66 seconds vs. 56 seconds) for this particular benchmark.

Did you run the python code on multiple cores? If yes, then YOU don't make an apples-to-apples comparison, because you removed the optimization on one language, but leave it to the other, and this makes the performance of LuaJIT even more impressive. An unoptimized Lua code running on a single core is still faster than an optimized python code running on multiple cores.

I literally converted the Lua code to Python, line-by-line. It ran on a single core. The code is right there.

There is a reason why Python is not used in most games. You can't use the benefit of Python. You need performance. And it must be safe (no/restricted OS access)

Depends on how it is being used. And on how it is sandboxed. The same thing applies to Lua. Just a little while ago, you could destroy anything in the filesystem which the user running the game had write access to, not to mention pulling all kinds of confidential info from files and sending it over the network, all from a Minetest mod.

I mean, to be fair Lua isn't used in most games either. I'm not going to argue against the fact that Lua is very small and simple, and relatively easy to sandbox. However, the more of the application or game you implement in your scripting language, the more useful it is to make it a robust, mature, featureful language.

As for the performance argument, it's outdated. If you need highly optimized performance in Lua, you use LuaJIT instead of the default interpreter. If you need highly optimized performance in Python, you use PyPy instead of the default interpreter. As the benchmarks you pointed to show, default Python is sometimes faster than default Lua (yes, if you take into account the number of cores each test uses for each implementation). Similarly, it's likely that optimized Python is sometimes faster than optimized Lua. Interpreters and (JIT) compilers mature, and performance improves. Your stereotypes should as well.

[Linuxdirk]

I literally converted the Lua code to Python, line-by-line. It ran on a single core. The code is right there.

And even if you proved by yourself that even under optimal circumstances the same stuff is slower on Python you still fight for Pythoin to be implemented (what NEVER would happen, and you know that)?

Lua isn't used in most games either

Except most scriptable/moddable games use it.

[prestidigitator]

And even if you proved by yourself that even under optimal circumstances the same stuff is slower on Python you still fight for Pythoin to be implemented (what NEVER would happen, and you know that)?

Um, no?

1. Under optimal circumstances, ONE benchmark was about 20% slower in Python, when someone was trying to make it out to be orders of magnitude slower.
2. Not fighting for it to be implemented, actually. Pointing out flaws in reasoning used to argue that it should never be implemented. There's a difference. As I said:

   Eh. I certainly don't expect it'll happen in the actual game, given the history and attitude of the development team. But someone might be interested in doing it in a fork or in a similar project, so it's still pretty useful info.

Except most scriptable/moddable games use it.

Sure. Python hasn't historically been a very fast language, and games typically have a lot of history to contend with, in libraries, their own code base, developer misconceptions, etc. This is like the argument that most games are written in C, and therefore no game could ever be written in anything else because no other language is good/fast enough. It's a historical argument made by people whose conception of performance is stuck in the past. New languages are being developed. Older languages are maturing and being made faster. Technology moves on, and progress is made. That's a good thing.

Does that mean that you must use Java or Python or whatever for your game? Of course not. But you're conflating choice with necessity, and that's counterproductive in the long run. You folks honestly don't have to make these (false!) arguments and get defensive about them being challenged in order to ENSURE Minetest will never, ever, ever use Python. What exactly are you so worried about?

[ThomasMonroe]

Guys, I didn't want this to be an argument about whether or not Python is better than Lua, else I would have titled the topic differently.
What I wanted was to see how many people would have liked python as a modding language.
And furthermore, this is not a proposal to add it into the Minetest engine. I just wanted to see people's opinions on this, and at the moment, the votes are 14-14.

[Skulls]

Realistically the dev team doesn't have time to monkey with a new scripting language and until you want to pay them a salary the most you can do is beg. That's very very fair and lets take a moment to acknowledge the absolutely fantastic work that they have been doing and are currently doing. The code and game functionality are *very* mature and very well maintained. That's all volunteer work so hats off to the contributors.

A more capable language would help out making rich game content beyond the basic place block, swat mob style games. One aspect of Mine* games is the non-linear game play: you can go over, around, through, whatever you want. But all I've seen have very limited game content beyond building. I'm trying to make a non-linear questing / job system that isn't the regular "kill 10k boars" but is focused on more abstract objectives with small micro actions. Of course this can be done in Lua. But it is soooo much harder than it needs to be.

I want to see Python added but I also realize it is a ton of work to move it into the core. Embedding the Python interpreter itself is not that bad with something like CFFI but that's only the tip of the iceberg. The minetest object has to be ported, security, memory management, testing testing and more testing. Windows, Android, Linux + flavors, BSD, Apple, all that stuff. It has to be done in a fork and matured then romanced into the main codebase.

Lua still sucks, Emacs for life, \r\n is technically the correct line end. ;)

[Byakuren]

I think Python would be more useful than Lua, with even just the standard libraries available, since it has a more useful set of data structures than just tables. I don't know though how sandboxing would work in Python.

I don't think it is really feasible to get Minetest using Python, though. And it wouldn't be my vote for language to switch to anyway.

[Linuxdirk]

A more capable language modding API would help out making rich game content

FTFY

[rnd]

I think Python would be more useful than Lua, with even just the standard libraries available, since it has a more useful set of data structures than just tables.

Can you give example of 'more useful' data structure in python? I looked https://docs.python.org/2/tutorial/datastructures.html and see for example lists (lua has that), Sets (easily implemented using hashtable and operations like union(a,b) =u where for k,v in pairs(a) u[k] = v; for k,v in pairs(b) u[k] = v; too) ...

I find lua more than capable to do pretty much anything short of intensive numeric simulations. Just intensive stuff like pathfinding or other similiar world operations should be written in c++ and included in api. Don't take my word for it, look for example http://torch.ch/

Code: Select all

Torch is a scientific computing framework with wide support for machine learning algorithms that puts GPUs first. It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDA implementation.

Also

A more capable language modding API modders would help out making rich game content

[Byakuren]

I think Python would be more useful than Lua, with even just the standard libraries available, since it has a more useful set of data structures than just tables.

Can you give example of 'more useful' data structure in python? I looked https://docs.python.org/2/tutorial/datastructures.html and see for example lists (lua has that), Sets (easily implemented using hashtable and operations like union(a,b) =u where for k,v in pairs(a) u[k] = v; for k,v in pairs(b) u[k] = v; too) ...

I find lua more than capable to do pretty much anything short of intensive numeric simulations. Just intensive stuff like pathfinding or other similiar world operations should be written in c++ and included in api. Don't take my word for it, look for example http://torch.ch/

Code: Select all

Torch is a scientific computing framework with wide support for machine learning algorithms that puts GPUs first. It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDA implementation.

Also

A more capable language modding API modders would help out making rich game content

torch is fast because the parts that need to be fast are written in C.

[duane]

I think Python would be more useful than Lua, with even just the standard libraries available, since it has a more useful set of data structures than just tables. I don't know though how sandboxing would work in Python.

I have to say that I don't get this at all. I haven't yet looked at a data structure in any other language and thought, "I wish lua had that." Especially when you're talking about python or ruby, the proliferation of data types doesn't add much to the language. With C, at least they're giving you type safety and performance (at the expense of giving me a headache when I have to constantly scan kilometer-long declarations).

The worst problem I've had with using lua tables for everything (even objects) was the overhead from moving array elements, and if you think ahead, that's easy to delay until you've done all the cpu-intensive work.

[sorcerykid]

I've never touched Python myself, never had a need to. When it comes to object-oriented scripting, my forte is Perl and Javascript. Both languages have suited me well for many years (mostly Web-based applications, but also data analysis, and task automation as well).

I personally do not see Python as the next logical step for Minetest, since Lua is already the de facto scripting language for every subgame. I can see no overwhelming benefit to rewriting the script API to support a much smaller code-base. Just my thoughts.

[ThomasMonroe]

Guys, again, I just wanted to see how many people, if we were to hypothetically restart Minetest, would have like to see python instead of Lua.

I stress this point again: THIS IS NOT A PROPOSAL to change the modding language.

[Byakuren]

I think Python would be more useful than Lua, with even just the standard libraries available, since it has a more useful set of data structures than just tables. I don't know though how sandboxing would work in Python.

I have to say that I don't get this at all. I haven't yet looked at a data structure in any other language and thought, "I wish lua had that." Especially when you're talking about python or ruby, the proliferation of data types doesn't add much to the language. With C, at least they're giving you type safety and performance (at the expense of giving me a headache when I have to constantly scan kilometer-long declarations).

The worst problem I've had with using lua tables for everything (even objects) was the overhead from moving array elements, and if you think ahead, that's easy to delay until you've done all the cpu-intensive work.

Stuff from python I would find useful in Lua are sets and their set operations, like union, differencing, etc. I can't think of anything else specifically. (Note I have not used python that much)

[prestidigitator]

Stuff from python I would find useful in Lua are sets and their set operations, like union, differencing, etc. I can't think of anything else specifically. (Note I have not used python that much)

There's lots of other cool stuff in Python that can really help:

• keyword arguments (with dict unpacking)
• full(er) regular expressions with standard syntax
• conditional expressions
• generator expressions and comprehensions
• arguably better/more complete generators and more consistent and comprehensive treatment of iterables
• useful general library functions which solve very common problems (e.g. iteration, dates/times, encoding, formatting, and parsing, math, etc.)
• (probably more, but that's off the top of my head)

There are lots of common problems which you can often solve in Python with very little code (like a single readable line), but that you often have to "reinvent the wheel" for in Lua. Can you do it in Lua? Sure. Should every single modder have to write a bunch of code every time they need to (esp. with little variations on a common problem)? Nah. Python code is often also extremely readable and easy to understand (a common refrain is that "writing Python is like writing pseudo-code," which means it is obviously doing something right). That can be really important in a community of people sharing tons of open-source mods.

Choosing to reduce the difference between languages to a simple matter of data structures or Turing completeness or whatever is too dismissive of real strengths that should not be ignored. (Heck, there's a lot about Go I don't like, but I really appreciate how it centralizes goroutines and channels, and wish more languages made concurrency and communication so elegant and clear.)

Honestly Lua is like Python lite. The one thing Lua does differently and better IMO is something few people use or have even heard about: ephemerons. (Yeah, a little ironically given my statement above, it's a data structure and associated behavior.)

[Linuxdirk]

• full(er) regular expressions with standard syntax

Yes, not having proper Regex leads to unnecessarily complex code and hacky workarounds to re-implement stuff over and over again. But that’s a general Lua downside and not particularly something that is pro Python as modding language.

[sorcerykid]

There are lots of common problems which you can often solve in Python with very little code (like a single readable line), but that you often have to "reinvent the wheel" for in Lua.

I think this same argument could be made about Perl 5, which is one of the reasons it still stands out as one of my favourite scripting languages (the past 30 years I've programmed in countless scripting languages from Applesoft Basic, Visual Basic, QBasic, Hyperscript, OpenScript, awk, ECMAScript, etc).

If it weren't for the memory footprint required by the interpreter, I would absolutely love for Perl 5 to be integrated with Minetest, even just for its powerful string manipulation operators, expressions, and functions (Lua's regex engine can't even come close). But the true beauty of Perl, at least in my view, is that it allows for rapid prototyping without the time spent constantly beautifying code only for a proof of concept. I can't tell you how many times I am forced to write a obfuscated routine in Lua that took only one line in Perl.

[sofar]

You can trivially get lua to do pcre regexes as well as a wide variety of other support, through DSO's. Eg. http://rrthomas.github.io/lrexlib/ (luarocks install lrexlib-pcre2).

Lua is often miscompared (apples to oranges) with other languages and found "lacking", but people forget that essentially Python by itself also can't do anything of significance, and relies heavily on a huge variety of built-in libraries and optional components. Remember that even in python, you already have to `import os` in order to do some really basic stuff. You're just relying on the distribution of Python to provide these bits to you, but you can easily do the same with Lua, so it really isn't what sets the languages apart.

[burli]

Yeah, the difference is, you know Python as a language that comes with tons of libraries. But the interpreter itself can't do much more than the lua interpreter.

If Python would be added to Minetest you had to install all the libs yourself. You would just have the bare interpreter

[prestidigitator]

A little off-topic, but it'd be interesting to see how the cost of crossing the C/Lua boundary with various parameters compares to serializing messages across a pipe (e.g. using CBOR). The latter could be used to build a modding system which would allow the actual mods to be built in several (potentially any) languages. Some scripting platforms like Lua and Python and Java could be done by protecting access to system calls, whereas others (e.g. straight C/C++, Go, etc.) where system calls can be made directly could be run in an OS container of some kind (e.g. Docker, or straight LXC in Linux).

Anyway, fun to ponder and maybe play with prototyping at some point. Kind of the next meta level of building an engine vs. a game. If we aren't at a level of computational performance that'll allow for that sort of thing at this point, we likely will be before long.

If Python would be added to Minetest you had to install all the libs yourself. You would just have the bare interpreter

Probably not. The idea of having "Python" without any libraries at all whatsoever is so useless that it need not even be considered. Again not an apples-to-apples comparison anyway, as all of Lua's standard libraries are included with its embedding in MT. No reason to expect it wouldn't be the same or similar with Python. At the very least you'd want a small selection of core libraries like the "built-in" functions, collections and iterators, math, regexs, etc. Whether you could (or should) strip out things like networking libraries is a reasonable question.