celeron55's roadmap

[celeron55]

This is a roadmap created and maintained by myself, which attempts to pick out things that will make Minetest progress forward. To a certain direction, without spinning in random circles. This is some kind of a continuation from http://c55.me/blog/?p=1491 (NOTE: URL not working at the moment).

I created this because nobody else really is in the position of making a global roadmap. These are not the only things that matter, but these are things that definitely do matter.

This roughly follows something that I consider to be the original spirit of Minetest, altough I do have to wiggle my original ideas around quite a bit to make them make sense on what we currently have. I have hard time figuring this out, but this appears to be the best way to make the community work together and the best way to use my time.

It is highly unlikely that I will personally implement more than a minor part of this, so feel free to consider this to be a guide in whatever you do. If you think something should be added or removed from here, throw me a message. I will probably try a few different formats for this and settle with something during the upcoming months.

Goal 1

Modders: Give feedback to the core developers (including me)

• Modders mean those who make subgames or individual mods.
• The engine's purpose is largely to provide a good platform for the content that runs on it, be it subgames or individual mods, and one of the first steps in figuring out what features to add is to hear about what modders would imagine having use for.

Goal 2

Maintain features and art styles for games
After a decade of existing, MT has come far enough that some games want to maintain their function and art style.

Games should be able to maintain their function and art style without players having to change settings.

This means that if something is added or changed in the engine that makes a game work differently or look different by default, the game should be able to change it back to how it was, without having to beg the players to change their settings. This usually means the server, as configured or commanded by the game, sending a setting or option to the client, and the client disabling the new feature or going back to old functionality when so instructed by the server.

Usually the way to do this is using the ObjectRef client control interfaces, like ObjectRef::set_sky(). This puts games first, which is the goal. I recommend not to negotiate generic settings between the server and the client. The hierarchy is hidden in a system like that, and it pushes people into not designing things with care.

My UI ideals

• When starting MT, ideally the user should be faced with a choice between connecting a server, and choosing between locally installed subgames. The "connect to server" option can be just a pseudo-game. (this is what everyone is already really doing, but the menu just doesn't represent it well at all.)
• When picking one, the choice is locked in the next screen. The next screen should contain a back button for going back to the game selection screen.
• The "connect to server" option would have a hardcoded about box explaining it, and the games would have whatever they specify in a file added for this purpose

Focus on these kinds of gameplay

• Voxel-based mechanisms that work well in multiplayer environment
  • This works well together with the moddability and people have been playing with mesecons, pipes and whatnot for a long time already.
• Defending against computer-controlled enemies
  • Defending instead of attacking puts much better use to the aspect of building things... at least until VAEs exist!
  • PvP is generally very hard to balance without making it boring
  • Many types of this require a world that is more fully in an active state than how Minetest currently works.
    • Currently it is assumed that every game wants to maximize world size and player count by taking away from other things, which is not right.
    • It should be possible for a mod or game to pick the tradeoff between world size, active entity count, individual entity performance and the resulting maximum player count.

Implement Voxel Area Entities

• That is, moving chunks of voxels on which you can build on like the static world, and which otherwise work similarly too.
• This is the only logical extension of the modifiable world mechanic. There's no excuse for this not to be implemented at least in the long run.
  • Well, actually, I lied. The other logical extension is making things buildable in finer detail, but that is not what Minetest is about.
• VAEs could have their own Lua environments for API compatibility and performance reasons, or
• a 4th coordinate, w, could be added to all positions. This was deemed as the better option in preliminary discussions.
• These can also be used to make multiple worlds, or partition the main world into superchunks for scaling of a large server. Here's the hype word: MMORPG
• Planning here http://dev.minetest.net/TODO#Voxel_Area_Entities, here http://irc.minetest.net/minetest-dev/20 ... #i_3669359 and here https://irc.minetest.net/minetest-dev/2 ... #i_5835581

Make Lua map generation non-blocking

• The map generator has become very nicely moddable with Lua, and is fast enough in what it does.
• The issue is that the map generator causes the whole server to halt almost all processing when it is running.
• -> Create some way that makes this not be an issue. Eg. something like register_mapgen_thread("mapgen.lua"), which would operate in it's own environment on an API that only exposes LVMs.
• Some other kind of threading model for mods could work instead of this, but they are likely much more complicated to create, and may not give much practical benefit (only downsides due to complication).

Overally fix non-blocking technical issues

• Things that don't prevent playing the game, but make it a lesser experience are often ignored.
• Measure and fix rendering bottlenecks
• Build system could use a clean up. Big or small, doesn't really matter; but don't break things.
  • I would like a /contrib directory for things that someone volunteers to maintain when the core team doesn't want the overhead, and from which things can be moved out if they become widely used.
• There are only hundreds of bugs. Fix (or wontfix) them! https://github.com/minetest/minetest/is ... state=open

Some general guidelines

• Client UI: Keep it generic and customizable so that it works for any game and any server.
• Pull requests: If you test a pull request, always comment on it saying that you did so. The amount of testers is a very important factor in deciding whether to merge or not.
• Maintain fork-friendliness in everything including versioning of things. See http://irc.minetest.net/minetest-dev/20 ... #i_3654822
• If you want to focus on exploration of the infinite world, you have to put way more effort into emergent generation and active systems in the world in order to actually have infinite things in there. With almost any fixed selection of static things, it's incredibly bland. For the king of this stuff, see Dwarf Fortress.
• Don't mind the lesser visuals and take full advantage of them in how they make creating unimagined things easier. There are enough games in this world that focus on visuals by taking away from everything else, or by making everything else harder to do.
• If you make game content, read this: viewtopic.php?p=139470#p139470

Voxel Area Entities

+ Spoiler

This is a thing that people will absolutely love and which provides a huge amount of gameplay value unique to this genre.

There are various ways to make these unnecessarily hard to implement in a fully functional way. This is a way which makes them fairly easy to implement, without any real drawbacks:

Allocate an area of the world to be used for VAEs. For example a 64 node wide slice at the -Z "face" of the world, maybe padded a bit from the edge so that lighting and other things don't freak out. There is completely unused space there as left out by map generation. A check in the regular sending of map data should be put in place to prevent sending stuff from that specially allocated area.

This allows all functions that handle nodes to behave like nothing special was happening; for example furnaces and mesecons will work fine, as far as mods are concerned.

A new ServerActiveObject is created. The server needs a way to keep track of what chunk of nodes belongs to the object. Map sending has to be modified to send the relevant pieces of map from the specially allocated area to clients that see the VAEs.

A new ClientActiveObject is created. The client has to be able to render the allocated piece of the world at the object's position.

They should probably be initially allocated as simply one MapBlock per VAE. It provides enough room to make most things and is very simple, and allocating less than that may not make sense at all.

Falling nodes might be implementable as VAEs; that way they could fall in chunks too in some cases.

[celeron55]

Things that I have left out of consideration for now. I'll see at some point if there is something to point out in these:

• Server administration and hosting

[celeron55]

Old:

Networking: Mostly fine on the lower level, but packet handling, serialization and cross-version compatibility should be made more easily reviewable and more noob-proof.

New:

Networking: The low-level protocol implementation is bloated and not ideal. See extra information in spoiler below. Packet handling, serialization and cross-version compatibility should be made more easily reviewable and more noob-proof.

+ Spoiler

The current resource-hungry and not always optimally working network implementation was not meant to be final. It became to exist when the old implementation was fixed and those fixes went too far in complexity.

Someone should look into it and possibly convert it to Enet. It should not be very difficult to decide whether Enet is the correct choice now that Freeminer has used it for quite long and can be referred to when answers to practical questions are needed.

[celeron55]

I split Goal 1 into 1 and 2 to prioritize subgame descriptions. It has been delayed for way too long already.

[celeron55]

I deleted outdated parts from the first post. Here are notable changes:

• We effectively have releases with multiple subgames now, with the content database accessible from the client.
• I removed every subitem from "Measure and fix rendering bottlenecks". Everything seemed outdated, but the overall topic remains very relevant.
• I guess "Pay some attention to SQLite" is outdated now, if not for any other reason then because large servers have other database backends available. However, it might be worth a look still.
• "Mod security sandboxing (server-side)": It was done and it works. That is an actual achievement for sure.
• "Some kind of built-in profiler" was made. Maybe it's not what I was thinking here, but it exists.
• "Networking: The low-level protocol implementation" was optimized and behaves much better these days.
• I'm going to drop "Multithreading part of the modding API" from this roadmap. It's a complicated subject and I don't want to type that much right now.

Archived old content of the first post:

+ Spoiler

This is a roadmap created and maintained by myself, which attempts to pick out things that will make Minetest progress forward. To a certain direction, without spinning in random circles. This is some kind of a continuation from http://c55.me/blog/?p=1491.

I created this because nobody else really is in the position of making a global roadmap. These are not the only things that matter, but these are things that definitely do matter.

This roughly follows something that I consider to be the original spirit of Minetest, altough I do have to wiggle my original ideas around quite a bit to make them make sense on what we currently have. I have hard time figuring this out, but this appears to be the best way to make the community work together and the best way to use my time.

It is highly unlikely that I will personally implement more than a minor part of this, so feel free to consider this to be a guide in whatever you do. If you think something should be added or removed from here, throw me a message. I will probably try a few different formats for this and settle with something during the upcoming months.

Goal 1

Modders: Give feedback to the core developers (including me)

• Modders mean those who make subgames or individual mods.
• The engine's purpose is largely to provide a good platform for the content that runs on it, be it subgames or individual mods, and one of the first steps in figuring out what features to add is to hear about what modders would imagine having use for.

Get a release out that contains multiple subgames

• This is important as it pushes the best content to the average users.
• The most interesting content is made by individual people, not a committee.
• Subgames need to have descriptions that can be easily viewed in the menu so that new users can understand what the subgames are about.
• Ideal menu:
  • When starting MT, ideally the user should be faced with a choice between connecting a server, and choosing between locally installed subgames. The "connect to server" option can be just a pseudo-game. (this is what everyone is already really doing, but the menu just doesn't represent it well at all.)
  • When picking one, the choice is locked in the next screen. The next screen should contain a back button for going back to the game selection screen.
  • The "connect to server" option would have a hardcoded about box explaining it, and the games would have whatever they specify in a file added for this purpose

Goal 2

Focus on these kinds of gameplay

• Voxel-based mechanisms that work well in multiplayer environment
  • This works well together with the moddability and people have been playing with mesecons, pipes and whatnot for a long time already.
• Defending against computer-controlled enemies
  • Defending instead of attacking puts much better use to the aspect of building things... at least until VAEs exist!
  • PvP is generally very hard to balance without making it boring
  • Many types of this require a world that is more fully in an active state than how Minetest currently works.
    • Currently it is assumed that every game wants to maximize world size and player count by taking away from other things, which is not right.
    • It should be possible for a mod or game to pick the tradeoff between world size, active entity count, individual entity performance and the resulting maximum player count.

Implement Voxel Area Entities

• That is, moving chunks of voxels on which you can build on like the static world, and which otherwise work similarly too.
• This is the only logical extension of the modifiable world mechanic. There's no excuse for this not to be implemented at least in the long run.
  • Well, actually, I lied. The other logical extension is making things buildable in finer detail, but that is not what Minetest is about.
• Planning here http://dev.minetest.net/TODO#Voxel_Area_Entities and here http://irc.minetest.ru/minetest-dev/201 ... #i_3669359

Make Lua map generation non-blocking

• The map generator has become very nicely moddable with Lua, and is fast enough in what it does.
• The issue is that the map generator causes the whole server to halt almost all processing when it is running.
• -> Create some way that makes this not be an issue. Eg. something like register_mapgen_thread("mapgen.lua"), which would operate in it's own environment on an API that only exposes LVMs.
• Some other kind of threading model for mods could work instead of this, but they are likely much more complicated to create, and may not give much practical benefit (only downsides due to complication).

Overally fix non-blocking technical issues

• Things that don't prevent playing the game, but make it a lesser experience are often ignored.
• Measure and fix rendering bottlenecks: viewtopic.php?p=133347#p133347
  • Texture switches (i.e. use generated texture atlases; most important)
  • Data transfer to the GPU (add "Store vertex buffers on GPU" to be a user-visible setting)
  • Implement hiding of nodeboxes that are buried underground at mesh generation (What is the effect on performance? Are there other similar things?)
• Pay some attention to SQLite
  • Determine whether sqlite_synchronous=0 can be used by default, and do so if possible
  • Enable locking so that the database cannot be opened by multiple instances of Minetest
• Build system could use a clean up. Big or small, doesn't really matter; but don't break things.
  • I would like a /contrib directory for things that someone volunteers to maintain when the core team doesn't want the overhead, and from which things can be moved out if they become widely used.
• Mod security sandboxing (server-side) should be set up. It should be complete, but as unobtrusive as possible.
• Some kind of built-in profiler should be made where at least in singleplayer games the client could easily show how much processing time each mod approximately uses in ABMs, entities and global ticks.
• There are only tens of bugs. Fix (or wontfix) them before there are hundreds! https://github.com/minetest/minetest/is ... state=open

Some general guidelines

• Networking: The low-level protocol implementation is bloated and not ideal. See extra information in spoiler below. Packet handling, serialization and cross-version compatibility should be made more easily reviewable and more noob-proof.
• Client UI: Keep it generic and customizable so that it works for any game and any server.
• Pull requests: If you test a pull request, always comment on it saying that you did so. The amount of testers is a very important factor in deciding whether to merge or not.
• Maintain fork-friendliness in everything including versioning of things. See http://irc.minetest.ru/minetest-dev/201 ... #i_3654822
• If you want to focus on exploration of the infinite world, you have to put way more effort into emergent generation and active systems in the world in order to actually have infinite things in there. With almost any fixed selection of static things, it's incredibly bland. For the king of this stuff, see Dwarf Fortress.
• Don't mind the lesser visuals and take full advantage of them in how they make creating unimagined things easier. There are enough games in this world that focus on visuals by taking away from everything else, or by making everything else harder to do.
• If you make game content, read this: viewtopic.php?p=139470#p139470

Voxel Area Entities

+ Spoiler

This is a thing that people will absolutely love and which provides a huge amount of gameplay value unique to this genre.

There are various ways to make these unnecessarily hard to implement in a fully functional way. This is a way which makes them fairly easy to implement, without any real drawbacks:

Allocate an area of the world to be used for VAEs. For example a 64 node wide slice at the -Z "face" of the world, maybe padded a bit from the edge so that lighting and other things don't freak out. There is completely unused space there as left out by map generation. A check in the regular sending of map data should be put in place to prevent sending stuff from that specially allocated area.

This allows all functions that handle nodes to behave like nothing special was happening; for example furnaces and mesecons will work fine, as far as mods are concerned.

A new ServerActiveObject is created. The server needs a way to keep track of what chunk of nodes belongs to the object. Map sending has to be modified to send the relevant pieces of map from the specially allocated area to clients that see the VAEs.

A new ClientActiveObject is created. The client has to be able to render the allocated piece of the world at the object's position.

They should probably be initially allocated as simply one MapBlock per VAE. It provides enough room to make most things and is very simple, and allocating less than that may not make sense at all.

Falling nodes might be implementable as VAEs; that way they could fall in chunks too in some cases.

Multithreading part of the modding API

+ Spoiler

http://irc.minetest.net/minetest-dev/20 ... #i_3662297
http://irc.minetest.net/minetest-dev/20 ... #i_3663675
http://irc.minetest.net/minetest-dev/20 ... #i_3664120

It quite clearly does not seem a good idea to put every mod in their own thread, or even default every mod to have any kind of threading.

Generally more asynchronous APIs probably do make sense.

Mod security sandboxing

+ Spoiler

Yes, there was work on this. Yes, it was partly wasted. Pick it up and make it even better.

Networking

+ Spoiler

The current resource-hungry and not always optimally working network implementation was not meant to be final. It became to exist when the old implementation was fixed and those fixes went too far in complexity.

Someone should look into it and possibly convert it to Enet. It should not be very difficult to decide whether Enet is the correct choice now that Freeminer has used it for quite long and can be referred to when answers to practical questions are needed.

Miscellaneous

+ Spoiler

Begin: 2014-04-26

Goal 3

???

Goal 4

Profit!

[celeron55]

Added to Goal 2:

Maintain features and art styles for games
After a decade of existing, MT has come far enough that some games want to maintain their function and art style.

Games should be able to maintain their function and art style without players having to change settings.

This means that if something is added or changed in the engine that makes a game work differently or look different by default, the game should be able to change it back to how it was, without having to beg the players to change their settings. This usually means the server, as configured or commanded by the game, sending a setting or option to the client, and the client disabling the new feature or going back to old functionality when so instructed by the server.

Usually the way to do this is using the ObjectRef client control interfaces, like ObjectRef::set_sky(). This puts games first, which is the goal. I recommend not to negotiate generic settings between the server and the client. The hierarchy is hidden in a system like that, and it pushes people into not designing things with care.

Also, moved UI stuff to Goal 2 from Goal 1. Nobody has done much to the UI in 5 years, so it must be just fine, right? Right??

[celeron55]

I'll just write this down here so I won't forget it:

These days it seems to me it might be a realistic design goal to have VAEs have their own Lua environments, and use that as one way to multithread Lua stuff. The main world would of course be just one VAE. Let's call it VAE 0 or something.

This of course means anything that wants to be world-global will have to start using a message passing or a thread safe database system. I do believe such can be made quite ergonomic though and it won't be a problem.

Player inventories would have to be transferred between Lua environments when a player moves between VAEs and they would have to be globally accessible also.

If a VAE ship lands on or crashes with VAE 0 (or another VAE), it has to be able to use an API that can map the other VAE into its coordinate space and interact with nodes and entities there. This will have to be a new API. But when stuff within the ship or VAE 0 interact within the VAE, it'll work exactly like before.

---
Maybe the future terminology could be VAE = Space. Space, as in coordinate space. VAE 0 = Space 0. Other name suggestions are of course welcome.

Here's the same text written with that terminology for comparison:

These days it seems to me it might be a realistic design goal to have Spaces have their own Lua environments, and use that as one way to multithread Lua stuff. The main world would of course be just one Space. Let's call it Space 0 or something.

This of course means anything that wants to be world-global will have to start using a message passing or a thread safe database system. I do believe such can be made quite ergonomic though and it won't be a problem.

Player inventories would have to be transferred between Lua environments when a player moves between Spaces and they would have to be globally accessible also.

If a Space ship lands on or crashes with Space 0 (or another Space), it has to be able to use an API that can map the other Space into its coordinate space and interact with nodes and entities there. This will have to be a new API. But when stuff within the ship or Space 0 interact within the Space, it'll work exactly like before.

[celeron55]

Well, I'll write this idea down too:

Entities (objects) in inactive blocks could probably be baked into the mapblock meshes based on the static data and rendered efficiently that way.

EDIT: Also, there could be a way for mods to handle entity staticdata without activating the entities of a mapblock. This could allow entity interaction outside of the active block range without spending too many system resources.