Distributed server?

[nomohakon]

Is it possible to have distributed server in some way? I mean several machines all over the internet, working as one.

[Ferk]

All over the internet? wouldn't this make it very slow? this could potentially multiply the load instead of distributing it, since the workarounds to prevent synchronization problems between machines over the internet would add extra latency, I would guess.

I guess you could set up a distributed mysql database but I'm not sure if it would work as you'd expect.

[steven2612]

I had a similar idea. but nothing coded yet.
there could be several possibilities:

- multiple worlds:
You have multiple server with one world and could go via teleport or gates or similar logic to the other server.
this should be doable with a proxy server without changing too much within the minetest code.
you need a custom authentication and trust system via a central authentification server. and maybe a server to server communication to transport items from one server to another (ender chest, tesseract or similar).

- a real distributed server
one (or multiple) world with threads on different server. so you could split abm's for example and run these on different threads or different servers. or you could define regions where each server thread is responsible for a different region on the map. the problem with this is synchronising the different tasks. but it should be doable with for example the redis backend. mysql would be too slow. redis has some features too which could be used for syncing the tasks (publish/subscribe). for example when opening a chest and this chest is modified by another user/abm the user should get be notified about the changes.

i have already done many researches about both possibilities, both should be doable.

Stefan

[Ferk]

- multiple worlds

This would probably be the easier and guaranteed to work approach, even though it wouldn't really be distributed. But still it'll be a cool way to have a cohesive universe and it'll allow for things like different "dimensions" and so on that people have asked frequently.
Several servers could agree about linking one another by adding teleporters that connect between worlds. Then it would be like being in a single big universe or interconnected worlds.

Imho, it wouldn't matter too much if the inventories of the players were different per-world (it could even be an optional thing, since bringing things from one place to another might not be wanted by the admins.. and maybe the mods from one world are not compatible with the other), but I think the central authentication system is a must because asking for a password somewhere else than the main menu would open the door to phishing and security exploits.

It would be even nicer if we had a public/private-key authentication system like ssh, then the authentication could be seamless and wouldn't require an authentication server or even a password. But I guess this would be too alien for Minetest audience, and probably uncomfortable when playing from a different device if we have to copy over the private key.

[steven2612]

This would probably be the easier and guaranteed to work approach, even though it wouldn't really be distributed.

it's far easier than the real distributed server ;) and it is distributed - but on a higher level ;)

it could be done via 2 methods:
- proxy server:
one or serveral proxy servers which are connected to the normal minetest servers - this would be a minimalistic approach which would need no or minimal changes within the minetest code.
- cross server teleport api:
if it is possible to connect to another server via lua api someone could code a portal mod or similar to teleport to another server. this would be simple too, but currently there is no such api call avaliable and i don't know yet how difficult it would be to implement such a api.

Imho, it wouldn't matter too much if the inventories of the players were different per-world (it could even be an optional thing, since bringing things from one place to another might not be wanted by the admins.. and maybe the mods from one world are not compatible with the other), but I think the central authentication system is a must because asking for a password somewhere else than the main menu would open the door to phishing and security exploits.

yes this could be done by a trust system. so each server owner could trust specific servers and then it would be possible to transfer your inventories to another server. if there is not trust relationship between two servers you have different inventories.

[Ferk]

it is distributed - but on a higher level ;)

I would say it's decentralized, not distributed. As in how the HTTP web as a whole is a decentralized collection of data across servers whereas P2P networks like bittorrent store data that is truly distributed.

- proxy server:
one or serveral proxy servers which are connected to the normal minetest servers - this would be a minimalistic approach which would need no or minimal changes within the minetest code.

I don't understand well that option.
How would the player move from one world to another when using a proxy server? And how do you do it without changes to the minetest client? the only way I can think of would be if you assigned different sectors of the world to each server but that would be too complex and would give limited results. You would actually need the client to disconnect and then reconnect to the world to switch properly, I don't think there's an API for that (which wouldn't be very different from the "cross-server teleport" API), they would have to log again manually, at that point they might as well just switch the server manually.

[steven2612]

- proxy server:
one or serveral proxy servers which are connected to the normal minetest servers - this would be a minimalistic approach which would need no or minimal changes within the minetest code.

I don't understand well that option.
How would the player move from one world to another when using a proxy server? And how do you do it without changes to the minetest client? the only way I can think of would be if you assigned different sectors of the world to each server but that would be too complex and would give limited results. You would actually need the client to disconnect and then reconnect to the world to switch properly, I don't think there's an API for that (which wouldn't be very different from the "cross-server teleport" API), they would have to log again manually, at that point they might as well just switch the server manually.

no thats not so difficult ...

the minetest network protocol is well documentated. it's based on a UDP network protocol. You could now go and build a little proxy server which takes the packets from the client and send it unmodified to the server - also the other way.
In the next step you could expand your server to analyse specific messages - for example a chat message from the server to the client. it could be like this:
"connect to server mynewworld position 200,50,100"
this message is now not delivered to the client but interpreted by the proxy server. it closes the connection to the first server and opens a new connection to the new server "mynewworld". it does the login stuff (which could be stored from the login to the first server) and some initial stuff like teleporting to the new position or similar.
the client would get a world/position update from the proxy server. and now you are connected to a different server.
so the client keeps connected to one server (the proxy server) and the proxy server did transparently the switch to another server.
so client and serverside there are no code changes needed. you only need a mod which delivers the chat message to the client when - for example - you are touching a specific block in the world.

[Ferk]

I thought you couldn't just start sending different data to the client just like that, can you? Wouldn't there be a lot of things breaking? like.. chunks that were cached in the client being remembered whereas the new server has them different. Also since for the client the server didn't change it will be mandatory to manipulate its inventory to check for incompatibilities. What about HUD state? Would the server be able to clean all that up?

Or maybe the client is way dumber than I expected and doesn't really keep much of a saved state. But wouldn't this be a bad thing for the sake of efficiency? Also what if we have in the future a client-side API, how would it cope with something like this?

[rubenwardy]

You'd have to send some delete all blocks packet to the client. Also, the servers would have to be identical in content.

[steven2612]

I thought you couldn't just start sending different data to the client just like that, can you? Wouldn't there be a lot of things breaking? like.. chunks that were cached in the client being remembered whereas the new server has them different. Also since for the client the server didn't change it will be mandatory to manipulate its inventory. Would the proxy be able to clean all that up?

Or maybe the client is way dumber than I expected and doesn't really keep much of a saved state.

as i have seen right now the client is dumb. nearly everything is done serverside (inventory and so on)
but you are right - you have to force a chunk update - so the client views the chunks of the new world and position.
but this could be forced by the proxy server too - maybe to set the player position to a far away position and then back to the right position. this would force the server to deliver a full set of chunks and clears all cached chunks out of the clients memory.

[steven2612]

if you want to share inventorys whith other servers you have to implement some sort of server to server communication too. this would be beyond the proxy. but this could be done via lua modding and - for example - a central (web-)server where player inventorys could be stored. when connecting to a server the mod fetches the inventory from the webserver via curl and when disconnecting it stores the inventory via curl to this webserver.

[Ferk]

And mods that affect the HUD? or player properties (gravity, etc).. I think it will get more complicated the more stuff gets handled by the client (which hopefully will be more in the future, since this reduces server load).

I still think it's better the cross-server teleporting. It requires changes in the client but it'll be cleaner and more efficient (without a proxy in the middle) and reliable (what if the proxy server goes down? the whole thing won't work at all even if the other servers are up).

[steven2612]

And mods that affect the HUD? or player properties (gravity, etc).. I think it will get more complicated the more stuff gets handled by the client (which hopefully will be more in the future, since this reduces server load).

At the moment this is handled by the server mainly. formspecs would be no problem because i don't think there will be a open formspec when doing the teleport.
HUD is another problem - at the moment this is also handled mainly by the server. but if handled by the client this would be no problem too. when connecting to a server, the server sends some initial packets to define the HUD. so you only have to reset the HUD when changing the server - then the new server will define a new HUD.
When you want to synchronize things like "hunger", "livepoints" or similar you have to do this via server to server connection like the inventory problem described before.

i like a dumb client - this gives you more control over cheating possibilities - if everything is done by the server, there are few less possibilities to do cheating.
but there are some things which could be done on client side - yes - mainly visual things - like updating the HUD to view a popup window when looking at a specific block - this would be optimal when done on the client - not on the server. or something like particle systems which are only started when the view range is near enough (torches for example) - this would be produce less lag when done on the client.

but i don't think such things would break the server switch. maybe there is (or should be - i did nor look yet) implemented a reset command in the network protocol - so the client is on a well defined initial state. so the server could start from scratch.

I still think it's better the teleport API. It requires changes in the client but it'll be cleaner and more efficient (without a proxy in the middle) and reliable (what if the proxy server is down? the whole thing won't work even if the other servers are up, that's not very decentralized anymore).

Yes i like this idea too - but it needs a LUA api. i don't know yet how difficult it would be to add such a api call. i've searched the forum and the code but i am not sure if this is trivial.

but i think a central server for login and exchanging things like shared inventory or player stats would be good idea.
maybe even a common player skin ...

[Фред Канниг]

You can try to use for example ZeroNet.

Можно попытаться задействовать например ЗероНет.

This topic echoes the following:
viewtopic.php?f=10&t=23231
viewtopic.php?f=3&t=20821