Making modding easier

[voxelproof]

The motif of Minetest vs Minecraft is recurring like some crazy jet boomerang. I have some simple(?) suggestion, which, as I believe, would make Minetest unbeatable in terms of modding at least (thus presumably attracting significant part of Minecraft modding community and making development of new mods quickier and easier).

One important feature frequently added by mods are new types of nodes or other non-cubic forms. By adding an interface for adding new kinds of building blocks, tree shapes and plants and a separate one for defining features of created worlds (similar to what was done in Terasology) were a player could define biomes (by percentage of their surfaces covered by water and different trees and plants, in that number those introduced by a player) it would make ultimate achievement in making modding easy for non-professionals.

The solution used for example in Terasology allows to set steepness and average height of hills and mountains in newly created worlds as well as the maximum depth of oceans and rivers and abundance of caves, water and plants. I don't know the Minetest code (yet) so I cannot predict if it is possible to implement it in MT at all, but if it was, it would leave MC far behind and would boost modding to unimaginable creativity.(Maybe I exagerrate a little bit, but certainly it is worth to think about for a while at least.)

I'm aware that such deep change in the code is possible for version 0.9 rather then any earlier, but it's so obvious to me that I hope some other users will admit it would be a significant and achievable improvement.

[Gael de Sailly]

For my part, the easier modding is already the main reason why I prefer Minetest. But it could always be improved.
It seems you're rather talking about mod/mapgen parameters than modding itself (as I understand it, improving the modding would be improving the Lua API and its functions). It's not the same thing, and for the first there is much more work to be done.

The solution used for example in Terasology allows to set steepness and average height of hills and mountains in newly created worlds as well as the maximum depth of oceans and rivers and abundance of caves, water and plants.

That's perfectly doable now, but it's not user-friendly, that's the least we can say. You have to go to minetest.conf and modify a bunch of obscure parameters composed by 9 numbers (noise parameters). Easily understandable parameters for map generation are something that has been discussed a couple of times, and it seems achievable but really difficult. I coded a lot on map generators, and sometimes I don't even understand the parameters I introduced myself… A solution I see is to create an additional mapgen from scratch that is really made to be easily tweakable with understandable parameters.

[Wuzzy]

I think modding is already reasonable easy. Lua is one of the easiest programming languages I've ever seen.

Rubenwardy wrote the Minetest Modding Book.

We also have a bunch of deveopment tools, but I think there could be more.

The biggest problem of the API atm are:
- Formspec coordinate system is completely messed up and inconsistent and does not make sense
- Lack of documentation. Examples: Return value of many functions have been forgotten; returned fields from *_receive_fields callbacks are not explained
- Giant lua_api.txt handled 100% manually, instead of automatically generating it in a javadoc-like manner

I think the Lua API itself is mostly fine as it is (apart from general formspec weirdness). Focus needs thus to be on documentation and development tools for modders.

Ideas for development tools:
- BiomeDefiner: Because getting the heat and humidity points right is important. Maybe as a sophisticated GeoGebra template?
- MapPreviewer: Given a set of mapgen settings, generates a 2D map preview around (0,0,0). So you can see the results of tweaking mapgen params much faster. Could be combined with BiomeCombiner (not sure). Would also be incredibly useful in Minetest itself
- Some static analysis tools to search the mod code for Minetest-related bugs, deprecated functions, bad coding practices, etc.
- DependencyChecker: Generates a graph with all the dependency relationships of mods, and also looks for errors. Useful for subgames
- NodeDefiner: NodeBoxEditor on steroids. Graphically define a set of simple nodes, spits out Lua source code. More difficult things like on_punch are not touched. Not really important because nodes are usually easy

[Linuxdirk]

- Some static analysis tools to search the mod code for Minetest-related bugs, deprecated functions, bad coding practices, etc.
- DependencyChecker: Generates a graph with all the dependency relationships of mods, and also looks for errors. Useful for subgames

Don't worry. Minetest will throw an exception and the mod will crash the whole server if something is not 100% correct or if there are any missing dependencies.

(One of the biggest design flaws is mods not having separate namespaces and mods being able to crash the server. Fixing both would make modding easier.)

[hajo]

Ideas for development tools:
- DependencyChecker: Generates a graph with all the dependency relationships of mods, ..

- Recipe-checker - look for overlap / duplicate / conflicting recipes.
E.g. I have seen a recipe for 'compressed cobble' (all 9 crafting-spaces filled with cobble)
that cannot be done because there is another recipe for stonebricks that apparently
is used first.

- Map-checker - generate some area around spawn, and check for a list of resources.
E.g. to make sure there is land/wood/coal/iron... not too far away (or to make really difficult maps :)

[voxelproof]

For my part, the easier modding is already the main reason why I prefer Minetest. But it could always be improved

When I posted this thread I wasn't fully aware yet of the number of MT mods released already. What seems now a bigger challenge is maintenance of the compatibility of mods with succesive MT versions and with each other. Obviously some modders abandoned their mods, others rarely do updates, sometimes once uploaded binaries disappear from the sites when nobody downloaded them for a longer period of time. The same goes about texture packs - although there is apparently effort to keep them available, my impression is that as time goes by much of really good work done in the past by users may be lost due to the lack of some kind central repository of user's content.

Maybe the present level of difficulty in modding MT is OK, easier one might cause disorder due to too much content and, consequently, difficulty in finding anything valuable on a pile of trash. Sometimes the better is an enemy of the good ;)

[Wuzzy]

Don't worry. Minetest will throw an exception and the mod will crash the whole server if something is not 100% correct or if there are any missing dependencies.

That's a feature. Minetest shouldn't chug along no matter what, even if the the mod code is utterly broken. Lua is not PHP. xD


I have recently thougt about this topic again.

Here's another idea to make modding easier: Implement core features in the engine. That seems like a no-brainer, but it seems there's a tendency to remove responsibility after responsibility from the engine, forcing mods to re-implement the same core features over and over again.
In general, more key features should go into the engine. Not everything should be outsourced to mods, because this just leads to redundancy. Implementing features in the engine also has the benefit that all subgames benefit from it. The engine can be a great help to standardize things, as well.

Leaving a feature to be implemented by mods makes sense if there is no general solution to a problem which would satisfy most subgames. Like a hunger system, there is a zillion possible ways you can think of different rules how the hunger works.

The second big thing here is the general design of the Lua API itself. The Lua API should be useful enough that you can use it without any wrappers. If you require any “utility mods” to make the API actually useful, then the Lua API has failed. For example, if there's a mod which offers wrapper functions for formspec handling, then that's a sign that the current system is flawed. Instead, the API should be redesigned that you can use the functions directly without much boilerplate and extra-hassle and dependencies. If modders need to write much boilerplate first, the API has failed and mods are redundant again.
The HUD Bars mod is another example for a wrapper mod. While the core Lua API allows you to add more statbars, you can't do so without instantly running into conflicts with other mods. You are forced to hardcode the statbar positions, there are no standardized statbar positions yet. The only way to reliably and portably add more statbars is by using the HUD Bars mod.
Then the player physics override is a big pain in the ass. You can only set the physics override directly. So as soon two mods touch the player jump height, you already have a conflict. Again, the problem is a lack of coordination between mods. So again, you need an utility mod first before you can safely touch the physics override.

The problem in these areas is that multiple mods have access to a resource (e.g. player speed) and each mod is allowed to overwrite it completely, with complete disregard of all the other mods.
A possible solution would be to add functions to enable/disable a certain offset to the player speed.

In future Lua API design, I suggest to consider the following questions:
- Is it really needed in the Lua API, or would it make more sense in the engine (or builtin)?
- If multiple mods use this feature, how would they interact with each other?
- What are potential use cases, and how would these be implemented?
- Does this feature require lots of boilerplate to be usable?

Had these question been considered in the addition of, lets say, player physics overide, it would have been clear from the start that modders will quickly run into problems when using this function more than once.

[Linuxdirk]

Don't worry. Minetest will throw an exception and the mod will crash the whole server if something is not 100% correct or if there are any missing dependencies.

That's a feature. Minetest shouldn't chug along no matter what, even if the the mod code is utterly broken. Lua is not PHP. xD

Under no circumstances 3rd-party code should be able to crash the applications that runs said code. Disabling or unloading a faulty mod is absolutely fine. Crashing is not.

[Wuzzy]

Well. Minetest should indeed not crash (e.g. segfault) when running a mod. That's not what I meant.

But what Minetest should do is stop the server when there's a faulty Lua mod because it has a Lua error. E.g. if there's a syntax error in the Lua code, there's no sensible way to mitigate this (should Minetest guess the correct code?). Or trying to assign a value to some_variable[nil].

[Linuxdirk]

But what Minetest should do is stop the server when there's a faulty Lua mod because it has a Lua error.

I disagree. MT should print what's wrong like it already doe but instead of crashing or shutting down it should disable the faulty mod. But the server should continue running. It's up to the server admin to validate if a mod works as expected.

[rubenwardy]

This can't changed at this stage, as it would introduce lots of bugs. The engine simply isn't designed for this.

My personal game is designed from the start to allow the reloading of the Lua environment on a Lua error. You can't just disable a mod as it populates the Lua environment with globals and registrations, so you need to fully reset the Lua environment and run all init.lua files again. The engine also has to be resistant to state corruption - ie: it should be very hard for Lua to corrupt the engine and put it in a weird state. A benefit to this is you can reload mods on the fly to load any changes.

[Linuxdirk]

You can't just disable a mod as it populates the Lua environment with globals and registrations, ...

See, that's what I'm talking about. Some mods (even some famous ones) pollute the public namespace (the only namespace that exists) with a shit-ton of mod-specific variables. All mods should have their own namespace and should be (re)loadable on the fly as well as being able to disable them. Unfortunately this is simply impossible due to Minetests design.

[Wuzzy]

On the namespace I actually agree. The giant global shared namespace is a pest. You can only avoid problems by religiously using local variables all the time.

Also, all mods are able to overwrite everything at all times, including all of the Minetest Lua API.

The global namespace is a huge bug magnet as well, I've seen many mods which conflict with each other only because they used equal variable names.

I think this unholy mess can only be fixed by extreme measures. Maybe just stop sharing all global variables with other mods altogether, and each mod has (mostly) its own _G (basically what Linuxdirk suggested), completely unaware of the _G in other mods. Only the minetest.* and core.* stuff is automatically shared. Instead, provide a different way for mods to share resources and functions.
Of course, this will break basically all mods with even the most primitive APIs. This is because using global variables is currently the only way to provide functions to other mods.

On the other hand, we will get rid of one of the biggest bug magnets in Minetest modding, and a lot of possible bugs will be prevented, improving overall stability of mods.

[duane]

I think this unholy mess can only be fixed by extreme measures. Maybe just stop sharing all global variables with other mods altogether, and each mod has (mostly) its own _G (basically what Linuxdirk suggested), completely unaware of the _G in other mods. Only the minetest.* and core.* stuff is automatically shared. Instead, provide a different way for mods to share resources and functions.

If you do that, they'll just pollute the core and minetest namespaces. They have legitimate reasons to share data, and doing so invites abuse.

I've never run into any stability problems using pcalls and just letting mods fail, even the mapgen. They were using the memory anyway. Stopping them just keeps them from using more.

Lua has wonderful error handling, readily available. Trying to force more is sort of pointless. If you run bad code, you should expect problems. Mods are just as important to the game as the base code is -- some of them are full-featured mapgens. Use mods that are programmed well if stability is a major concern.

And truthfully, making modding "easier" will tend to make them less stable, as people won't bother learning the system. I think more, better documentation is the only thing minetest really needs.

[voxelproof]

And truthfully, making modding "easier" will tend to make them less stable, as people won't bother learning the system.

To make my opening post more precise, I'd say that it's not about sort of "outsourcing" all modding into some UI integrated with modified C code. But I noticed that great many mods concern only adding new blocks; this could be somehow 'standardized' so to speak. The same is about customizing terrain generation: it would be interesting and inspiring for the players to have some parameters modifiable from the level of adjusting world settings in the main menu. That's all and any 'serious' modding concerning more advanced elements of gameplay involving interactions between different elements of virtual environment should be of course done in a professional way, that is by going down to the level of the code.

I think more, better documentation is the only thing minetest really needs.

I've a feeling that you're terribly right here. I'd like to see somewhere some sort of a 'roadmap' of the source: what part of the code is relevant to which functions, a list of all functions and variables, modules and so on. I don't know, maybe sth like this already exists, but I suppose there's no such concise, systemic summary as for now.

That is, to make a long story short, to be able to change the colour of the skybox without digging through half of the listing.

And to have more clarity as to how the compilation procedure looks like (both in Windows and different Linux distributions); as I see on the forums (especially that about Voxelands) getting a workable binary from the source is a branch of black magic reserved exclusively for adepts of some hermetic order, not for mere mortals ;)