Why aren't mods in C++?

[MrIbby]

So I've been reading through the forum, and here are some of the claims people have made:

• C++ is better suited than Lua for things like mobs.
• Lua is good enough--for adding nodes and items.
• Everything else is slow or inconvenient.

I'm not asking for opinions. I want facts.

Why Lua instead of C++?
Why C++ instead of Lua?
Are there any other plausible options?

Your Answers:

If mods were written in C++, you would have to compile every one of them, increasing the startup time. Also, C++ apps can mess things up more than Lua scripts. [...]

[...] Part of the reason why any higher-level language runs faster is that it's more convenient. I hate working with C because I have to spend much more time thinking about how to store data and free up memory afterward. I have to spend more time testing. Lua gives you easy to use list structures that never write over the rest of your program. Convenience and safety take extra time. [...]

[...] I guess the only one who can really answer your questions is celeron55 since it was his choice. But C++ / Lua seems like a good combination to me. C++ is a popular well known low level language and Lua is a simple, easy to learn high level extension language.

[...] There are tasks for which Lua is the most efficient option, but things like AI mob entities and power/data/item/fluid transmission are definitely not examples. Lua could be very efficient for registering items, nodes, AI mob entities and nodes which conduct transmission of power/data/items/fluids; but it would be very inefficient to use Lua for implementing the functionality of those things, because it would end up being just too slow and very hackish in nature.

[...] Encouraging engine patching might make sense for some things, but I think doing so also has the drawback that the engine then becomes specialized to a specific way of doing things, and with it the limitations of those chosen approaches, with little ability to mod outside the box so to speak. As a core mod, there could be multiple solutions to different problems. [...]

[...] You can already patch the engine if you wanted. It's open source.. if you want to code an AI in C++ that can be enabled from within Lua you can just write it in the current engine codebase.. [...]

[...] there is *always* politics when trying to get a PR accepted: someone doesn't like the way you did it, another person says it breaks compatibility so shouldn't be added, ect. This like you said causes forks which is ok, but that makes the features less accessible to the "masses" and harder to find. [...]

[benrob0329]

If mods were written in C++, you would have to compile every one of them, increasing the startup time. Also, C++ apps can mess things uo more than Lua scripts.

What people are talking about is making APIs in C++ so that Lua scripts don't have to run all the time for things like mobs.

Another option would be a language like Python or TCL, one that ran in a VM would be a performance boost.

[BrandonReese]

If they were in C++ would they also need to be compiled for each platform?

[duane]

Lua is a fast language by design. It benchmarks at about half the speed of C on math intensive work, and that's about as good as you can get from a higher-level language. There's also a significant amount of overhead involved in making function calls to and from the compiled game code, which reduces the speed of mods some.

The C++ version of valleys mapgen, after months of work (and a bit of cheating to speed it up), runs less than three times as fast as the lua version, and that's only counting map generation. There's obviously a lot more processing involved in the game.

Part of the reason why any higher-level language runs faster is that it's more convenient. I hate working with C because I have to spend much more time thinking about how to store data and free up memory afterward. I have to spend more time testing. Lua gives you easy to use list structures that never write over the rest of your program. Convenience and safety take extra time.

If you want another good reason for lua, compare the number of capable C++ developers working on MT to the number of people who can make a mod in lua.

I'm not asking for opinions. I want facts.

As an instructor of mine told me, "There are no facts. There's only opinion and data that may back it up."

[Ferk]

Another option would be a language like Python or TCL, one that ran in a VM would be a performance boost.

LuaJIT is a VM. And I guess most Lua implementations are.
Python would be slower.. and while I personally like TCL, prefix-notation syntax is not popular and would scare many people away.

I'm not asking for opinions. I want facts.

Why Lua instead of C++?
Why C++ instead of Lua?

A "why?" question can't be answered with a fact. Purposes and objectives are only in the eyes of the beholder. Things simply are what they are, they don't really have intrinsic purposes (religion aside). It's entirely your human judgement what you use them for, or whether you use them at all. Of course some choices will make some goals harder to achieve but you can't pinpoint the "best" tool without entering very gray areas with too many circumstantial aspects... like how familiar are you with which language.

I guess the only one who can really answer your questions is celeron55 since it was his choice. But C++ / Lua seems like a good combination to me. C++ is a popular well known low level language and Lua is a simple, easy to learn high level extension language.

[bdjnk]

I'm not asking for opinions. I want facts.

Why Lua instead of C++?
Why C++ instead of Lua?
Are there any other plausible options?

There are a great many plausible options which can be leveraged right now, with no changes to minetest, including C.

[SegFault22]

Important tasks are done in the C++ engine code, such as the item/node registration functionality and entity functionality. To make more advanced mods, such as those which involve transmitting power or information between nodes via cables or making AI-controlled entities, currently we are only able to use the Lua part of the engine, which can cause these functions to run slowly. It would be more efficient to provide the "framework" for the functionality, by implementing new APIs in the C++ part of the engine, which can then be used by Lua mods to add the "stuff" which uses the functionality (register entities that have AI, register nodes that store or produce or use or transmit energy, etc).

A very effective implementation would be the addition of "coremods", which would be compiled C++ code that provides definition of APIs and other functionality to be implemented by the engine when running the server (or the client). They would be dropped into a folder just like Lua mods, so that they can be added or removed as necessary, and without having to re-compile the whole game every time something is changed.
This would require some way to iterate through all files in a coremods directory, similar to the process by which mods are loaded from the mods directory; of course they will be loaded differently than Lua mods because they will be written in C++, but that should not be too difficult to accomplish since C++ is the language in which most of Minetest is written.

That would provide a way to solve the following problems, without requiring their functionality to be added to the engine itself:
- Mobs lagging and being unable to have significantly intelligent AIs
- Energy transmission mods running very slow and requiring a special network hub node or "switching station" in order to function
- Limited functionality of GUIs and node metadata
- Limited functionality of node lighting dynamics, node temperature/pressure/gravity dynamics, fluid node dynamics, etc
- Limited control of node dig time
- Explosion dynamics lagging and being generally hackish in nature
- Impossible to create items/nodes with "dynamic" or "custom" properties such as variable content of different materials or variable quality factor (custom alloys, custom steel making, dynamic mixing of environmental materials like dirt and sand, etc.)
- Limited functionality of plants, especially the kind that grow or spread
including several more which have not been considered yet, or I just forgot.

There are tasks for which Lua is the most efficient option, but things like AI mob entities and power/data/item/fluid transmission are definitely not examples. Lua could be very efficient for registering items, nodes, AI mob entities and nodes which conduct transmission of power/data/items/fluids; but it would be very inefficient to use Lua for implementing the functionality of those things, because it would end up being just too slow and very hackish in nature.

[Ferk]

I think the precompiled coremods idea would have some problems:

- They will not be portable to all platforms Minetest supports (like it was already pointed out in this thread).
- There are not that many mod developers that would use it. A lot of the people making mods are not even programmers. I'd guess most of the people willing to write a C++ mod that implements an AI are already working in the engine code.
- You can already patch the engine if you wanted. It's open source.. if you want to code an AI in C++ that can be enabled from within Lua you can just write it in the current engine codebase.. you'll even have more freedom this way because you'll have the whole engine code exposed for you to change and study instead of having to depend on a limited API. I bet the devs would be happy to accept a PR that provides optional AI behaviors for the entities or many of the other features you mentioned were they nicely implemented (and if they are nicely implemented but still not accepted.. maybe go with that code to one of the forks, the day such significant features were officially rejected in core might be the day a Minetest fork would be able to really set itself apart).
- Creating an API is not easy and it's time consuming. I think Minetest dev resources are already limited.
- We would have 2 different modding APIs for the server.. why would we want to maintain both? one of the APIs is inevitably gonna fall behind and be disregarded.
- Precompiled code could be dangerous for people who just pick whatever binary from the internet and run it.
- Since you can't directly see the code as you can with lua, it won't be as "free as in freedom"-friendly (even though this is not as important since I can see this community really understands and I'd guess those mods will still be FOSS).

On top of that.. I don't think the bottleneck for the performance is in the Lua language. I think the problem is that all the processing is done in the server and most of the lag is network latency. I do not experience any problems with mobs when I play singleplayer or in a local network. I think client-side Lua modding is a more approachable solution.

[JSwigart]

I'm new to the forum so forgive me if I bring up things already answered, but I wander to toss some thoughts in here too.

Just to second the thoughts of others, I think the best modding approach is Lua JIT(just to get the best performing lua parts), but to have built in engine exposed bits for doing computationally expensive operations such as AI and generally anything that requires a per-frame update. As much as possible Lua should be registering infrequent event callbacks or wiring up inputs and outputs in the native systems to do the computational work on. Any per-frame lua updating is going to be slow. Less so with LuaJIT, but if it's anything that is per block or per device is probably going to add up quickly and you'll want to be handled by native code.

As primarily a C++ developer myself, I am very partial to a C++ modding API as a personal preference and for performance, but the issue of compiling across all supported platforms is the primary hurdle to adopting one I think. Unless the engine itself had the tools to compile source files to the platform itself, or there were some sort of web resource to compile all platforms, it's going to be a big burden on modders to use a native modding api.

I think the "core mod" approach is a good middle ground. Someone taking on the burden of creating a core mod is an advanced programmer adding significant functionality that piggybacks off a native engine API for performance reasons and perhaps to utilize other resources(third party libraries etc). Core mods wouldn't be meant for non programmers, so their accessibility to such people would be a non issue. Encouraging engine patching might make sense for some things, but I think doing so also has the drawback that the engine then becomes specialized to a specific way of doing things, and with it the limitations of those chosen approaches, with little ability to mod outside the box so to speak. As a core mod, there could be multiple solutions to different problems. I think the core mod approach is best if the core mods are basically a dynamic library that interfaces with the engine to provide computationally expensive functionality, but that is otherwise itself an extension to the Lua modding API. In other words, if large mod packages were not built as monolithic core mods, but that if mod packages were built as collections of lua scripts that utilize multiple core mods as engine extenders, useful by all modders, not just specific mods.

Code: Select all

-- archer.lua pseudocode
-- quick and dirty ai "system" where a function is an AI "state"
-- and a function is run as a lua coroutine to facilitate asynchronous operations 
-- and easily readible serial execution

pathfind = require "coremod_ai_pathfinding_v0.1"

-- for brevity, assume there is a transition registration system for AI behaviors
function init( entity )
	setState( "state_Idle" )
end

function state_Idle( entity )
	while true do
		if random() % 10 then
			setState( "state_Wander" );
		end
		-- select from a few idle variations
		entity.cycleAnimation("idle"+ random() % 2);
		coroutine.sleep( 10.0 )
	end
end

function state_Wander( entity )
	while true do
		local wanderPos = pathfind.pickReachablePointWithinRadius( { movemode="walk", radius = 1000 } );
		local asyncMove = pathfind.moveEntityTo( entity, wanderPos, { movemode="walk" } );
		-- runs asynchronously by the pathfind module
		 -- returns status whenever async operation fails and allows coroutine to resume
		local moveResult = coroutine.waitfor( asyncMove )
		if moveResult == "failed" then
			-- handle failure
		end
		
		-- wait at each wander position before wandering again
		coroutine.sleep( 10.0 )		
	end
end

function state_Combat( entity )
	local attackRange = 100.0 -- or whatever the bow and arrow range is
	while true do
		entity.enemy = pathfind.findNearestReachableEntity( { team="player" } );
		local enemyDist = distance( entity.position, entity.enemy.position )
		if enemyDist > attackRange then			
			local asyncMove = pathfind.moveEntityTo( entity, { destination=entity.enemy, movemode="run", range=attackRange, needLineOfSight=true } );
			 -- runs asynchronously by the pathfind module
			 -- returns status whenever async operation fails and allows coroutine to resume
			local moveResult = coroutine.waitfor( asyncMove )
			if moveResult == "failed" then
				-- handle failure, pick new target, etc?
			end
		end
		
		-- assumed within attack range
		pathfind.stopMoving();
		-- set up an async
		entity.lookAt( { target=entity.enemy } )
		
		-- play a random attack animation
		local anim = entity.actionAnimation("attack"+ random() % 2);
		coroutine.waitfor( anim ); -- stall script until anim completes
	end
end

function animEvent( entity, animName, eventName, eventArgs )
	if eventName == "fireArrow" then
		entity.fireProjectile( { projectile="wooden_arrow", fromJoint="arrow", target=entity.enemy, offset="center" } )
	endif
end

The point being that the native core modules can do much of the heavy lifting in cooperation with the engine, while exposing easy to use functionality for broad use by lua JIT scripting that still acts as the foundation of the mods themselves.

The expensive stuff ultimately needs to be in native code one way or the other. The core mod idea has the benefit of not burdening the engine itself to adopt approaches and close off alternatives. Given the likely limited resources of the engine team, this would seem to be easier to parallelize the workflow for, with core modders primarily needing lower level access to the engine, and much of the gameplay and AI being built on top of that API through the core mods.

Thoughts?

[benrob0329]

Having a compiler on Linux or *BSD isn't a problem, it would just be a dependency. On Windows however, Clang(++)/LLVM doesn't require MinGW (I think) so that's an option.

Another would be a to have pre compiled mods, that link to MT itself, and get run after the engine starts.

[Byakuren]

obviously we should use a haskell modding API

[benrob0329]

Functional programming is hard to wrap your head around. It might discourage new/young modders.

[domtron vox]

And I didn't realize this was a fairly old thread until I was half way through writing this post. So I'm going to go ahead and post it and hope I don't get tar and feathered.


There are lots of ideas in this area all of which need more research.

For example you could probably get away from the security and, to a lesser extent, platform issues by using a C/C++ JIT compiler(may not be the right term though my google results are what I was thinking of). I've come across micro c/c++ compilers that are intended to be embedded into a program and used during runtime. So you distribute your c++ mod and it will be compiled on the platform in a sand-boxed environment. The modder will still need to make sure it runs well on all platforms but we already have that in Lua to a lesser extent (esspecially if your interacting with the file system).

Another idea you could do right now is create a C++ program that runs alongside minetest and use a mod to bridge the two programs. For example make some sort of C++ program for AI then let it interact with the minetest world through a lua mod (or maybe even the network protocol). This should work well with servers but not so well with singleplayer since you would need to run the extra program.


Quick disclaimer the following are just my thoughts. I have done C++ programming, but certainly do not know what all would go into re/designing the current engine's architecture around Core Mods.

I'm liking the idea of core mods as well. Maybe it could use a C++ JIT compiler.

I think the precompiled coremods idea would have some problems:

- They will not be portable to all platforms Minetest supports (like it was already pointed out in this thread).

This is true but just means the burden is put on modders to make sure it works on multiple platforms. If they are making a Core Mod then it is likely they know what they are doing and know it will need more testing which is the price for performance.

- There are not that many mod developers that would use it. A lot of the people making mods are not even programmers. I'd guess most of the people willing to write a C++ mod that implements an AI are already working in the engine code.

I think that is the point. Most mods would be in lua stil with only a few mods that need heavy processing being made by more experienced mods. That might even be a good learning tool, you work your way up from registering simple nodes to working on more complecated lua mods and then when you are more experienced and start pushing the limmits of what lua can do you switch to making Core Mods. The you go off and make your own engine. :P

- You can already patch the engine if you wanted. It's open source.. if you want to code an AI in C++ that can be enabled from within Lua you can just write it in the current engine codebase.. you'll even have more freedom this way because you'll have the whole engine code exposed for you to change and study instead of having to depend on a limited API. I bet the devs would be happy to accept a PR that provides optional AI behaviors for the entities or many of the other features you mentioned were they nicely implemented (and if they are nicely implemented but still not accepted.. maybe go with that code to one of the forks, the day such significant features were officially rejected in core might be the day a Minetest fork would be able to really set itself apart).

I see a couple problems with this thinking.

First, not everyone will want the same features, but if you add all features that one could want, the engine will be slow.

Second, there is *always* politics when trying to get a PR accepted: someone doesn't like the way you did it, another person says it breaks compatibility so shouldn't be added, ect. This like you said causes forks which is ok, but that makes the features less accessible to the "masses" and harder to find. I think it is best, rather then fragmenting the community with forks that add some C++ feature, let players choose to include Core Mods that does what he wants.

Core Mods also gives modders some autonomy without decreasing the quality of the engine code.

- Creating an API is not easy and it's time consuming. I think Minetest dev resources are already limited.
- We would have 2 different modding APIs for the server.. why would we want to maintain both? one of the APIs is inevitably gonna fall behind and be disregarded.

I think part of the idea is to allow the Core Mods to interface with the engine more closely extending the C++ code directly. You may not need too much of an API to allow them to interface with the engine. And we would want to maintain both so that we can have a better experience and more flexability. If one does fall out of use it would probebly be the Core Mods and we would know that they were a bad idea.

- Precompiled code could be dangerous for people who just pick whatever binary from the internet and run it.
- Since you can't directly see the code as you can with lua, it won't be as "free as in freedom"-friendly (even though this is not as important since I can see this community really understands and I'd guess those mods will still be FOSS).

Both of these could probebly be solved with some sort of sandboxed JIT C++ compiler.

On top of that.. I don't think the bottleneck for the performance is in the Lua language. I think the problem is that all the processing is done in the server and most of the lag is network latency. I do not experience any problems with mobs when I play singleplayer or in a local network. I think client-side Lua modding is a more approachable solution.

With Core Mods you might beable to strip out some of the code from the engine that slows it down. Maybe you could even pull all the legacy code into Core Mods and say "If you don't want this feature broken add in this code mod and it will be just like it was."

I guess in my mind I am thinking that Core Mods could make it so: the engine is treated as a stable core that is only tinkered with occasionally, core mods are treated as performant APIs that add gameplay mechanics but are constantly changing and innovating, and Lua Mods are treated as content creators more or less how they are used now. So that it is really easy to add content, a bit harder to design good core mechanics, and really hard to effect the engine itself.

[theHalfBloodStanger]

I am kinda new around here, and I don't really understand lua past registering nodes and ABMs. Would there be a way to use either language?

[Byakuren]

I am kinda new around here, and I don't really understand lua past registering nodes and ABMs. Would there be a way to use either language?

Yes, but it would require some understanding of Lua to do. You would need to make your other-language code callable from C, and then make your C code callable from Lua, then call the C code from Lua using FFI stuff (I don't know exactly how that works). To do any API calls you will have to pass the Lua state around somehow, because they are only exposed in Lua. On top of this, you will lose a lot of performance because Lua<->C calls are expensive, and there will be additional overhead between your language and C (unless your language is C).

[theHalfBloodStanger]

I am kinda new around here, and I don't really understand lua past registering nodes and ABMs. Would there be a way to use either language?

Yes, but it would require some understanding of Lua to do. You would need to make your other-language code callable from C, and then make your C code callable from Lua, then call the C code from Lua using FFI stuff (I don't know exactly how that works). To do any API calls you will have to pass the Lua state around somehow, because they are only exposed in Lua. On top of this, you will lose a lot of performance because Lua<->C calls are expensive, and there will be additional overhead between your language and C (unless your language is C).

But doesn't it all end up as binary code anyway?

[Byakuren]

I am kinda new around here, and I don't really understand lua past registering nodes and ABMs. Would there be a way to use either language?

Yes, but it would require some understanding of Lua to do. You would need to make your other-language code callable from C, and then make your C code callable from Lua, then call the C code from Lua using FFI stuff (I don't know exactly how that works). To do any API calls you will have to pass the Lua state around somehow, because they are only exposed in Lua. On top of this, you will lose a lot of performance because Lua<->C calls are expensive, and there will be additional overhead between your language and C (unless your language is C).

But doesn't it all end up as binary code anyway?

The Lua gets run by a lua interpreter. This may be the reference interpreter or LuaJIT. The latter does JIT compilation, but that is not really the same thing as translating the whole thing to machine code. In general anyway, it doesn't matter if everything ends up as binary code, because what matters is that it ends up as binary code that runs together.

[theHalfBloodStanger]

OK, then I'd better learn more lua.

[lisacvuk]

Look at how Quake 3 does it. It runs mods in a virtual machine. Security fixed.