Python as a modding language

[Sergey]

So many references to religion in Lua!

Lua was designed and is being maintained at the Pontifical Catholic University of Rio de Janeiro, which is located in Brazil. Its creators are Roberto Ierusalimschy, Waldemar Celes and Luiz Henrique de Figueiredo.

As atheist all these disturb me a little.

[ThomasMonroe]

why should references to places make you uneasy?

[Skulls]

One thing I'm struggling with is that Lua doesn't seem to be able to handle large programs very well. Sure, it *can* do it, but then so can assembly. But you wouldn't want to do that either. Writing a non trivial mod (aka The Entire Game Part Of The Game) in Lua stuffed in the tiny JIT gives me the shivers. And 99% of your code will run on the server anyway? As mentioned above speed isn't everything. Scalability, maintainability, and expressiveness go a long way.

[Byakuren]

What's wrong with the JIT compiler being tiny? I would say it's a good thing for a programming language's interpreter (for interpreted languages) or runtime (for compiled languages) to be small, as that's less core code you need to worry about being incorrect. It also says nice things about the usability of a language if you are able to implement a lot of it in the language itself.

My main gripe with Lua as the modding language is its dynamic typing, since it seems that a lot of errors are simple ones that could be found statically in a language that supported it. Refactoring becomes a pain because there is no way to see what needs to change after changing some place in the code, other than running the code and waiting for errors. Ideally for me, the modding language would be an expressive, statically typed programming language such as Haskell or similar languages.

However, Lua is not a deal-breaker for me, and I don't see any mods that are actually pushing up against any complexity limits of Lua. Python would not resolve my complaints either as it is still dynamically typed.

[v-rob]

1) Python is slower than Lua, as shown before. Sure, this might be fine for higher end computers, but for slower computers, this is a big problem.

2) This is a lot of work for the devs, who already have enough to work on. It won't happen whether you want it to or not unless you yourself work on it specifically.

3) Two different programming languages together? A) This is confusing and will make it harder for people to work on different mods, and B) some mods are going to be in Lua while others will be in Python, which mean that they will be incompatible. Can you use globals registered in Lua in a Python mod?

4) Python will bloat the size of Minetest.

5) Lua is working just fine for Minetest, so Python is unnecessary.

Logical conclusion: Minetest almost certainly won't use Python as a programming language for the reasons stated above.

[Linuxdirk]

Can you use globals registered in Lua in a Python mod?

With a proper API you can, because the globals are registered in the core, and not in userspace.

Anyways, a Python modding API won’t happen - ever.

[twahm]

Minetest should have python as an alternative language but it doesn't have to leave lua aside, for my biggest reasons why python should be integrated is because of the popularity it has, which drives mod development, data processing and algorithms.

[Linuxdirk]

for my biggest reasons why python should be integrated is because of the popularity it has

Lua is pretty much the standard for game modding/scripting.

https://en.wikipedia.org/wiki/Category: ... ideo_games
https://en.wikipedia.org/wiki/Category: ... me_engines

[rubenwardy]

Python is useful for education and new programmers. I don't see it as more productive as Lua, given that you're discouraged from using third party libs and Lua has an okay object model. See CTF for an example of a large interconnected project in Lua

[Skulls]

Python is useful for education and new programmers. I don't see it as more productive as Lua, given that you're discouraged from using third party libs and Lua has an okay object model. See CTF for an example of a large interconnected project in Lua

I'll check it out, thank you. Python runs Google, OpenStack, and is used in Libre Office, btw :)

[rubenwardy]

Python runs Google, OpenStack, and is used in Libre Office, btw :)

Errr so? I've used python in rather big projects, including those for clients. Only when including external libraries is Python mod mature in terms of Minetest mods. But those can't be allowed for security

[Skulls]

Very true. I'm looking at it from a server side perspective where you would want to know (ish...) what libraries you would be tossing in anyways. My main issue with Lua is the data structures. The client side aspect is frick'n amazing and I'm super excited to see it split up like this but I just see Lua causing more friction than is needed on a large server side project with lots and lots of models.

Was viewtopic.php?f=11&t=6947 the CTF you were mentioning?

[sofar]

Python runs Google, OpenStack, and is used in Libre Office, btw :)

Not very convincing arguments. Google is throwing out python in favor of go, openstack is a performance disaster partially due to python, and libreoffice isn`t exactly known to be responsive or small.

Minetest will never use python for modding, so this whole discussion is kind of moot. There is no way that we'll add 100+ mb just to run python code. Sorry, dont get your hopes up.

[duane]

I won't argue the technical merits of lua vs python, but I haven't been able to tolerate coding in python for years. My original issue with it was the changes made to accommodate unicode, but at this point, even the minor inconvenience of built-in formatting makes me itch.

On the other hand, I've grown to really enjoy lua. I've recoded hundreds of lines of scripts on my server from ruby to lua. They're much easier to follow now, and work better (though that's more due to fixing old problems during the translation). I don't know what you'd call a large project, but I haven't had any trouble maintaining a lua library in the kilo-line range. Luacheck really helps with that, and it works fine on minetest mods.

Now, if I can only force myself to start making C libraries for lua. Ick.

[Skulls]

I've recoded hundreds of lines of scripts on my server from ruby to lua.

In your code how do you handle the mental "is a" (data structure inheritance) or "can do" (interface, poor person's functional inheritance) checks? For example, an apple is a fruit and all fruits can rot.

[duane]

I've recoded hundreds of lines of scripts on my server from ruby to lua.

In your code how do you handle the mental "is a" (data structure inheritance) or "can do" (interface, poor person's functional inheritance) checks? For example, an apple is a fruit and all fruits can rot.

So far, I haven't used inheritance in lua. I'm one of those old curmudgeons who still thinks of objects as another fad. I use tables to contain everything, but I don't always treat them as objects; I just don't like having local functions (or any sort of private methods, generally). However, the lua wiki has some sections on doing it with metatables that makes it look easy enough.

Also, keep in mind that I'm using lua5.3 on my server, which has some small but significant changes from the 5.1 in use on minetest. I haven't had any problems going from one to the other so far, and I seem to recall seeing a library specifically designed to supply a lot of the changes from 5.1 to 5.3.

Apart from copying the minetest dump function, which I'm accustomed to, I've only needed the ports and lfs libraries, neither of which would be useful in minetest. I do all my local gui's as minimal web servers, so ports is important.

[Byakuren]

I've recoded hundreds of lines of scripts on my server from ruby to lua.

In your code how do you handle the mental "is a" (data structure inheritance) or "can do" (interface, poor person's functional inheritance) checks? For example, an apple is a fruit and all fruits can rot.

There is no way to check it statically. I just document in comments or documentation what contracts the inputs and outputs of some function should respect.

[benrob0329]

I've learned to love Lua. Its simple, but flexible. It builds on the basics and does things in a way to keep the language small. It very much can be used standalone, in fact it has a packaging system and a plethora of modules (Luarocks). But its strength is often in how its embedded. Minetest, AwesomeWM, Love2D, Cafu, Amulet, and many more projects use Lua as an embedded scripting language, and many are better off for it.

At the end of the day language X will always be faster than language Y, but raw speed isn't everything. I've seen slow C apps, and fast BASIC ones. More often than not its how you write you code, not what language its in that matters most.

[rubenwardy]

I don't see this being merged into Minetest. This could be done in the similar way the Lua API is structured (C++ embedding the language), but this would be a big undertaking for a fork to make and maintain. I'd prefer some compatibility layer in Lua, which would be slower but would be less breaking.

This could be implemented as a protocol which different things could use - Python, Scratch like stuff, etc. Although you'd lose the speed especially when related to map manipulation. There's a RPi Python project that does this for Minetest, but it reverse engineers the Minecraft Python API.

Was viewtopic.php?f=11&t=6947 the CTF you were mentioning?

No, this: viewtopic.php?f=50&t=12835
Although the link you gave is used in the game.

[Skulls]

Edit: rubenwardy, sweet, thank you. Poking at it now.

The only way I can figure out how to implement some kind of abstraction in Lua is to stuff things in tables, such as "is_fruit = true, is_apple = true" all the way down and up the tree for each data type. This just seems wrong and will add a lot of crazy boilerplate code that needs to be maintained. Everything is a stupid table. The language doesn't seem to be able to scale beyond a script with maybe a dozen types (and those need to be carefully handled with kid gloves).

Lua seems to be fantastic at being the glue to hold things together but for larger game logic I will need to lay the foundational stuff that is baked into other projects. Granted, I'm a complete Lua noob but so far the usual answers (language website, forums, Stackoverflow, even Google) haven't given a decent, maintainable way to write a larger program in Lua. Its all hack some tables and pray. What am I missing?

[rubenwardy]

See https://www.lua.org/pil/16.1.html for the Lua class pattern which allows inheritance and crude polymorphism
Think of tables as one structure to make classes, objects, and namespaces. It's worth noting that Javascript uses a similar design under the hood - they're both prototype languages - and pretty big things have been written in it.

[Skulls]

Well, darn.

From http://www.lua.org/pil/16.4.html:

Lua is not intended for building huge programs, where many programmers are involved for long periods. Quite the opposite, Lua aims at small to medium programs, usually part of a larger system, typically developed by one or a few programmers, or even by non programmers.

[rubenwardy]

You are not going to be making large programs at the scale they mean, and you're unlikely to be working with many programmers continuously. The best mods are closer to independent programs which may use the API of another mod.

[duane]

Well, darn.

From http://www.lua.org/pil/16.4.html:

Lua is not intended for building huge programs, where many programmers are involved for long periods. Quite the opposite, Lua aims at small to medium programs, usually part of a larger system, typically developed by one or a few programmers, or even by non programmers.

You do realize that you already have access to a language that's specifically designed for any scale of project, has full object orientation, and is used extensively in minetest. It's called C++.

Mods are great fun, but the real power is in the game itself, and if you develop something good, you can probably get it put into the code base, benefiting everyone.

[prestidigitator]

wow, i didnt know the diff would be that much

And I think it is more with "normal" functions because in this example most time is spend in the math functions

I randomly picked fannkuch-redux from this benchmark and run it with Lua, LuaJIT and Python3 on an AMD FX6300 hexacore CPU.

The Lua code runs on a single core, the Python code uses all 6 cores

This is the output from normal lua. I add a function to measure the runtime. The last number is the time in seconds

Code: Select all

markus@morpheus:~/dev/lua$ lua fannkuch-redux.lua 12
3968050
Pfannkuchen(12) = 65
1363.03

This is the output of the python version. As you can see on the screenshot it really uses all 6 cores.

Code: Select all

markus@morpheus:~/dev/lua$ python3 fannkuch-redux.py 12
3968050
Pfannkuchen(12) = 65
373.94084000587463

And this is the output of LuaJIT

Code: Select all

markus@morpheus:~/dev/lua$ luajit fannkuch-redux.lua 12
3968050
Pfannkuchen(12) = 65
99.183862

If you divide the Lua time of 1363 by 6 you get the time a multicore version of Lua would need. And now divide the LuaJIT time by 6...

Any questions?

1. You picked a test in which the Lua code is pretty highly optimized (for example, it caches the result of element look-ups, and only tests whether the last array flip needs to be done rather than actually doing it), while from even cursory inspection the Python code very obviously is not.
2. You picked a test where the code does a lot of memory accesses and very little computation, and where the Python implementation uses multiple processes. This is a pretty bad place to use multiprocessing to solve the problem, so again the actual implementation used for Python was a pretty bad one.
3. For some reason you saw fit to include a highly optimized Lua interpreter in the testing, but not a highly optimized Python interpreter (e.g. PyPy).

This is very obviously a terrible way to compare the performance. I have converted the Lua benchmark code pretty much line-for-line into Python:

Code: Select all

import sys


def fannkuch(n):
    sign, maxflips, sum = 1, 0, 0
    p = list(range(n))
    q, s = p[:], p[:]

    while True:
        q0 = p[0]
        if q0 != 0:
            for i in range(1, n):
                q[i] = p[i]
            flips = 1

            while True:
                qq = q[q0]
                if qq == 0:
                    sum += sign * flips
                    if flips > maxflips:
                        maxflips = flips
                    break

                q[q0] = q0
                if q0 >= 3:
                    i, j = 1, q0 - 1
                    while True:
                        q[i], q[j] = q[j], q[i]
                        i += 1
                        j -= 1
                        if i >= j:
                            break
                q0 = qq
                flips += 1

        if sign == 1:
            p[1], p[0], sign = p[0], p[1], -1
        else:
            p[1], p[2], sign = p[2], p[1], 1
            for i in range(2, n):
                sx = s[i]
                if sx != 0:
                    s[i] = sx - 1
                    break
                if i == n - 1:
                    return sum, maxflips

                s[i] = i

                t = p[0]
                for j in range(i + 1):
                    p[j] = p[j + 1]
                p[i + 1] = t


n = int(sys.argv[1])
sum, flips = fannkuch(n)
print(sum)
print('fannkuchen({}) = {}'.format(n, flips))

and compared LuaJIT to PyPy. On my test system I get:

Code: Select all

[root@32bc349d53d7 ~]# time luajit fannkuch-redux.lua 12
3968050
Pfannkuchen(12) = 65

real    0m56.446s
user    0m56.446s
sys     0m0.000s

[root@32bc349d53d7 ~]# time pypy3 fannkuch-redux.py 12
3968050
fannkuchen(12) = 65

real    1m5.676s
user    1m5.549s
sys     0m0.027s

So in an apples-to-apples comparison Python is about 18% slower (66 seconds vs. 56 seconds) for this particular benchmark. Now if you take all the benchmarks on that site where Python is, in fact, comparable or faster even considering the amount of parallelism, the picture starts to look a lot different from how you are attempting to characterize it. ("Any questions?" /s — Honestly I wish people would avoid that kind of disrespectful jibe.)

In addition, none of these benchmarks compare various methods of transitioning between C and the particular scripting language in question, and that is likely to be far more interesting in the Minetest case than shuffling some array elements around like in the benchmark you chose. Comparing similar tasks involving bidirectional calls and data movement between the Lua API and, for example, CFFI for Python would be an interesting exercise. Maybe I'll play with that at some point.