Creating decentralized worlds with IPFS

[MirceaKitsune]

I have a very interesting idea that I'd like to discuss. It's a pretty radical suggestion, but just as amazing and worth considering in my opinion. It's a proposal I'm planning to poke several game engines about, and Minetest (with other Minecraft like games) would be the first to benefit from such a system.

I would like to start with some background on what this is all about: I've been following the development of the decentralized internet initiative closely for the past few years. This namely includes the IPFS protocol. For those who are not familiar with it: IPFS (Inter Planetary File System) is a file storage system which allows files to be stored in a global decentralized network. Every user who accesses a file also stores and seeds parts of it to other viewers for a limited time (similarly to torrents). Files in IPFS are immutable, meaning that once they're uploaded they cannot be edited nor deleted and are represented by a hash which guarantees their originality. There exist multiple implementations and the library is very flexible... a browser node can even be embedded directly onto a website using JavaScript alone. The goal is to revolutionize the web by allowing entire websites and more to become free from the requirement of a central server... meaning fewer or no storage costs, guaranteed uptime, faster network speeds, and increased safety from censorship.

https://ipfs.io

To offer a simple example, let's compare Youtube to BitTube. The first is a centralized video platform owned by Google, whereas the second is a new video platform which uses IPFS for storage. When you watch a video on Youtube, the video file is downloaded from the Google server closest to your location. When you watch a video on BitTube, some of it might be downloaded from the computer of your next door neighbor, some from an user in a nearby city, and some from another viewer from the other side of the planet... all concurrently so the download speeds add up. While Google has to spend money on drive space and bandwidth, the platform can go down if it experiences server issues, and they can also delete videos they don't like from their site... BitTube has its videos are seeded by viewers who watching them, it's guaranteed to be available as long as you're connected to the internet, and uploads can never be truly deleted once posted (only filtered at HTML / protocol level). It's easy to see why the decentralized web will likely be the biggest revolution in the history of the internet.

Back to Minetest and how this relates: I've been greatly intrigued by the idea of using IPFS in a similar way for game engines... particularly ones that have highly dynamic worlds intended for multiplier environments. A fork of the Minetest engine would be a fantastic candidate for such an experiment. If implemented properly, this would essentially allow online worlds which are hosted by no server yet available 24/7 (granted there are enough peers seeding all of the data): Whenever you connect to a world, you download bits and pieces from various users who have accessed that world and are seeding its files from the IPFS nodes on their machines.

This is how my idea would work like in more technical detail (requires some general understanding of IPFS): Whenever you create a world, it gets registered as a database within IPFS (eg: OrbitDB). Every game is referenced as the ID of that database, you connect to a world by pasting in its hash. The creator of the world is registered as its owner based on their public peer key, and can give other keys various forms of access to join or modify given parts the world. When a player preforms any action that edits the world, changes cause the database to be updated and new files to be referenced where relevant.

There are roughly two categories of storage a Minetest server needs to address: The settings of that server including the properties of players and entities (health, inventory, etc) and the voxels in the world. The first are small configs which are stored in tiny text files, they would be seeded throughout the network easily and players can pin their own settings file for instant access. For the later I'd go with using one file per chunk: A player will only download and seed the file of a chunk if they've accessed that area and loaded it into view, essentially meaning that computers share pieces of the world they've visited with one another in realtime. When changes are made to any chunk (blocks are placed or dug out) the database of the world (eg: OrbitDB) pointing to each chunk is updated to represent the updated file... anyone revisiting the area will then re-download the updated chunk data once the client notices there's a newer file in the database. It must be noted that there would no longer be a server doing any computing, so each client processes the changes to be made and uploads the update... obviously this opens the gate to trolls and cheaters, so there would need to be some form of verification to make sure people aren't adding or deleting whatever they want, this part is a bit more complicated.

I'm putting this idea up in case it inspires anyone to make a fork of the engine and try something out, and even to hear what others think of the idea in general. Just as decentralized storage is going to revolutionize the web, I believe it will revolutionize online gaming in a similar way; Serverless multiplayer worlds are going to bring new possibilities and make things a lot easier, and sandbox environments like those of Minecraft or Second Life would be the first to benefit from such ease of use. As an open-source project with many free thinkers, I can see MineTest having an easier time offering a test case for this possibility.

[Hume2]

Nice idea. I can see several problems those have to be solved:

Is there any mechanism to update mods? Who has the right to update mods?

Everyone connected to the server is something like an owner. All cheats in singleplayer now work in multiplayer too. And there isn't anyone to do anything against it.

What if I receive an old block, I build something here and then I receive the same block from a player who has been a long time offline and he has already built his own house here?

[MirceaKitsune]

Nice idea. I can see several problems those have to be solved:

Is there any mechanism to update mods? Who has the right to update mods?

Everyone connected to the server is something like an owner. All cheats in singleplayer now work in multiplayer too. And there isn't anyone to do anything against it.

What if I receive an old block, I build something here and then I receive the same block from a player who has been a long time offline and he has already built his own house here?

This model would indeed bring up a lot of issues. There is no longer a server to process the mechanics, so the clients of connected players would have to do the job of the server too. To allow other users to edit a world, we must give them write access to its database... but the moment they have that nothing's stopping anyone from using an online tool to just erase the whole thing. There could however be a few solutions to this problem.

The idea I had at first is whether all MineTest clients connected to a world can verify a change and check that it's valid. So if a server has say 10 players, at least 5 of them must agree that an action you've preformed isn't cheating before allowing it to manifest into the database... to fool it more than half of your users would need to be using hacked clients. But what do you do if only one person is online? I don't know how this can be enforced either, since it requires a complex form of database security which OrbitDB (the only viable candidate to my knowledge) doesn't have: If you have an owner key you can edit the database to your liking, it's a black and white thing.

There is however another way. Orbit features a kind of database which allows users to add entries but not remove any. As such, worlds could be stored as change logs rather than a present state description; Instead of saying "the block at position (1, 2, 3) is dirt_with_grass", it would say "at 1:00 AM user Foo dug out the block at position (1, 2, 3), at 2:30 AM user Bar placed the block cobblestone at position (1, 2, 3)" then the client picks the last entry per position when connecting. You can then have blacklists and whitelists of trusted users, so even if a troll messes around you can simply exclude their changes from the world! As an added bonus, any world could also be reverted to any point in the past and you can essentially time travel.

However if this choice is left up to each user, we end up in a situation where everyone sees their own reality; Some players would bump into invisible walls while others would walk through solid surfaces, because each would have a different version of the world loaded. To avoid that problem, we could use a separate database or list describing a version of the world based on valid / invalid users... people can then join what they agree to be the correct version, which would essentially result in forks of the world.

[Hume2]

I can imagine that this might work somehow. If each player had to calculate updates of all blocks those are currently updated, it would lag more. Maybe the players could update only a part of them when there are too many players online.

I thought that playing on a server like this could result in many forks. Is there a way to solve merge conflicts or the forks will diverge forever?

[sorcerykid]

As much as I like the idea in principle, I'm skeptical about whether it would be compatible with the existing framework.

Most games for Minetest, are heavily modded. These mods require a single-threaded Lua interpreter with a persistent environment. I don't see how that would be possible without a centralized server. Add to the fact, the underlying game mechanics are more complex than just "place block" and "dig block". Most player-world and player-player interactions dispatch event messages to Lua for additional processing. So the entire engine (not to mention all games and mods) would need to be rewritten from scratch. At that point, it's no longer Minetest.

I think it would be more realistic and feasible to only take portions of the engine, and decentralize it. For example, offloading map generation and schematic placement would be very easy, by comparison since that is a linear algorithm with mostly predetermined input values.

Just some thoughts to consider.

[MirceaKitsune]

I can imagine that this might work somehow. If each player had to calculate updates of all blocks those are currently updated, it would lag more. Maybe the players could update only a part of them when there are too many players online.

I thought that playing on a server like this could result in many forks. Is there a way to solve merge conflicts or the forks will diverge forever?

Each player should save the updates their own client makes to avoid processing too much data. If another player digs or places a block in your field of view, they could communicate this directly to you (IPFS has a messaging system called PubSub) so you don't have to constantly re-download the chunk file on every change.

I'm not sure what the best way is to calculate the movement of entities such as mobs. You can process them client-side, but if different clients compute the physics differently you can experience some slight drift. PubSub could be used by clients to communicate to one another the average positions of items, making sure they agree on the precise location of each entity.

As much as I like the idea in principle, I'm skeptical about whether it would be compatible with the existing framework.

Most games for Minetest, are heavily modded. These mods require a single-threaded Lua interpreter with a persistent environment. I don't see how that would be possible without a centralized server. Add to the fact, the underlying game mechanics are more complex than just "place block" and "dig block", most player actions (and interactions) dispatch event messages to Lua for additional processing. So essentially the entire engine (not to mention all games and mods) would need to be rewritten from scratch. At that point, it would no longer be Minetest. Just some thoughts.

One of the issues is that clients would need to handle all the Lua processing in lack of a server, including both client and server side Lua. This would be possible in theory, as server Lua could still act in a server-oriented way but be processed by a connected client instead. The main issue here is that it opens the gates to endless cheating and griefing; To avoid that we'd need some complex permission and verification system.

My idea would be for all connected clients to act as a cluster and process the server side collectively; All server actions would be based on distributed processing, multiple machines acting as one by taking all or some parts of a task and checking the results with one another. The more players are online, the greater the load but also the number of machines available to compute server changes in realtime!

[ExeterDad]

Wouldn't mobile clients need to be exempt from their share of the load? I'd think their network connection (if in data mode) and smaller processors would reek havok with the rest of the hive. And if they were to be exempt... they are a larger portion of the multiplayer's these days.

[Hume2]

I think, you didn't get my problem. Imagine the following situation:

Player A lives far away so noone else downloads his blocks. He build a nice house and goes offline. After that, player B goes online and doesn't know anything about player A. He goes to the same blocks where player A lives and builds another house. Now player A goes online and both houses are on the same spot. What now?

Also another problem:

Player A is a cheater, so player B refuses all changes made by player A. However, player C doesn't know that A is a cheater and still accepts changes of both A and B. Player A builds a large wall. Now player B can pass through the wall while C can't. Now C removes some blocks from the wall and gets some material. Player B accepts C's changes but they don't make any sense since the wall was built by A.

I can imagine that this could work like a git repo but there must be a mechanism to solve merge and rebase conflicts. Keep in mind that both versions of the same block could diverge dramatically.

[MirceaKitsune]

I think, you didn't get my problem. Imagine the following situation:

Player A lives far away so noone else downloads his blocks. He build a nice house and goes offline. After that, player B goes online and doesn't know anything about player A. He goes to the same blocks where player A lives and builds another house. Now player A goes online and both houses are on the same spot. What now?

Also another problem:

Player A is a cheater, so player B refuses all changes made by player A. However, player C doesn't know that A is a cheater and still accepts changes of both A and B. Player A builds a large wall. Now player B can pass through the wall while C can't. Now C removes some blocks from the wall and gets some material. Player B accepts C's changes but they don't make any sense since the wall was built by A.

I can imagine that this could work like a git repo but there must be a mechanism to solve merge and rebase conflicts. Keep in mind that both versions of the same block could diverge dramatically.

On the first problem: There would be a common world database pointing to each chunk file, even if it's stored in a decentralized fashion. Its entries are read whenever a player connects or walks to a new area, to make sure they get the latest version of that chunk. I'd suggested OrbitDB for this task, you can read more about it here: https://github.com/orbitdb/orbit-db

The second problem is more complicated. A quick suggestion I had was to use player blacklists / whitelists; You can connect to any world using a "reality filter" that tells you which users changes you accept and which you discard. However each filter would act as its own world, as you could only see and interact with players on the same filter as you... so essentially you'd get a new world whenever you apply a filter. This might be a poor idea in the long run though, a better system should be thought of ultimately.

[sorcerykid]

All server actions would be based on distributed processing, multiple machines acting as one by taking all or some parts of a task and checking the results with one another. The more players are online, the greater the load but also the number of machines available to compute server changes in realtime!

But that's just it, it wouldn't be realtime because all of the message passing for every single user event would be subject to increasing network latency across each additional peer to ensure synchronicity. In multiplayer games demanding responsiveness, like PVP for example, I'm inclined to believe that dispatcing messages across a peer-to-peer network is going to be a far greater bottleneck in contrast to processing messages on a central server.

Distributed computing is much better suited for load balancing of modular systems.Perhaps it could be adapted for CPU intensive tasks that don't entail player interaction (see my comment above about map generation and schematic placement, since those are usually independent of Lua). But I can't see a fully distributed computing model being compatible insofar as Minetest's scripting architecture.

Perhaps a rudimentary proof of concept is in order. I'd be more inclined to change my mind, if there was evidence this is truly feasible :)

[MirceaKitsune]

But that's just it, it wouldn't be realtime because all of the message passing for every single user event would be subject to increasing network latency across each additional peer to ensure synchronicity. In multiplayer games demanding responsiveness, like PVP for example, I'm inclined to believe that dispatcing messages across a peer-to-peer network is going to be a far greater bottleneck in contrast to processing messages on a central server.

Distributed computing is much better suited for load balancing of modular systems.Perhaps it could be adapted for CPU intensive tasks that don't entail player interaction (see my comment above about map generation and schematic placement, since those are usually independent of Lua). But I can't see a fully distributed computing model being compatible insofar as Minetest's scripting architecture.

Perhaps a rudimentary proof of concept is in order. I'd be more inclined to change my mind, if there was evidence this is truly feasible :)

I think the biggest issue is someone actually trying out the concept, which would involve a pretty big development effort. Minetest barely has any shader developers still... finding someone willing to do something so specific would rely on convincing people that the idea is worth it. I'd try it myself but don't know a lot of the code and am extremely busy with my own projects too so no time.

I should be making it more clear however that storing worlds in IPFS and server-less multiplayer would be two different things which can be done without one another. You can keep the current server side, just have it and clients load the world from an IPFS hash or IPNS domain rather than the server's local drive, then have both servers and clients seed the relevant chunks which would greatly increase network speeds. That too would be lovely and also far easier to do!

[Linuxdirk]

Plus: Distributed computing or distributed file systems are entirely in the operating system's domain. Applications should not implement such functionality but the OS itself should where applicable.

[Фред Канниг]

Perhaps it makes sense to think about using BlockChain instead of IPFS as a database.

Возможно имеет смысл подумать о том, чтобы в качестве базы данных вместо IPFS использовать блокчейн.

[bzt]

Hi,

Yes, decentralized network is a good idea, and it can be easily done. It would definitely need a custom implementation to accomodate Minetest gameplay, I don't think any existing high-level library can be tuned enough for that (but I could be wrong about this).

Files in IPFS are immutable, meaning that once they're uploaded they cannot be edited nor deleted

This "feature" makes a filesystem perfectly unusable. Just think about it! It might be good for static webpages, but who wants to store an sqlite db on a CDROM?

Perhaps it makes sense to think about using BlockChain instead of IPFS as a database.

Please don't! Just because BlockChain is a buzzword these days, it does not make it suitable for a database! Actually it is perfectly unusable for any kind of database in the usual sense. BlockChain was developed for one thing, and one thing only: to keep track of changes reliably, not to store a snapshot of data efficiently (just like what Minetest needs to store the actual state of the world with nodes and all). BlockChain data just keeps growing, extremely resource consuming to decode, which means after a while servers would be unable to send it to the clients in a timely manner. It is not a matter of if, it is a matter of when. And if you use pruning on the data to counteract, then you loose the benefits of BlockChain, so then the question is, why to use BlockChain in the first place if you're about to drop the history anyway?

I'd rather suggest to pick a nice and easy to use low-level P2P library instead, and build on that to store and share Minetest data efficiently. That surely can be done. With other words, if we are thinking about torrents, then don't implement how they share data, implement the part how torrent clients share their peers database.

Alternatively a distributed server could use the same algorithm that IRC servers use, not to store the data, but to distribute the server commands. That would require far less bandwidth. There's one problem though, IRC-like protocols are vulnerable to split-brain scenarios. That is hard to workaround, and even if you can decide which server contains the valid data, probably would require server downtime to sync the databases to resolve the split-brain.

Cheers,
bzt

[Фред Канниг]

Hi,

Yes, decentralized network is a good idea, and it can be easily done. It would definitely need a custom implementation to accomodate Minetest gameplay, I don't think any existing high-level library can be tuned enough for that (but I could be wrong about this).

Files in IPFS are immutable, meaning that once they're uploaded they cannot be edited nor deleted

This "feature" makes a filesystem perfectly unusable. Just think about it! It might be good for static webpages, but who wants to store an sqlite db on a CDROM?

Perhaps it makes sense to think about using BlockChain instead of IPFS as a database.

Please don't! Just because BlockChain is a buzzword these days, it does not make it suitable for a database! Actually it is perfectly unusable for any kind of database in the usual sense. BlockChain was developed for one thing, and one thing only: to keep track of changes reliably, not to store a snapshot of data efficiently (just like what Minetest needs to store the actual state of the world with nodes and all). BlockChain data just keeps growing, extremely resource consuming to decode, which means after a while servers would be unable to send it to the clients in a timely manner. It is not a matter of if, it is a matter of when. And if you use pruning on the data to counteract, then you loose the benefits of BlockChain, so then the question is, why to use BlockChain in the first place if you're about to drop the history anyway?

I'd rather suggest to pick a nice and easy to use low-level P2P library instead, and build on that to store and share Minetest data efficiently. That surely can be done. With other words, if we are thinking about torrents, then don't implement how they share data, implement the part how torrent clients share their peers database.

Alternatively a distributed server could use the same algorithm that IRC servers use, not to store the data, but to distribute the server commands. That would require far less bandwidth. There's one problem though, IRC-like protocols are vulnerable to split-brain scenarios. That is hard to workaround, and even if you can decide which server contains the valid data, probably would require server downtime to sync the databases to resolve the split-brain.

Cheers,
bzt

Suggest your decision, what kind of library?

[bzt]

Suggest your decision, what kind of library?

Not sure what you're asking.

As I wrote before, I'd suggest a custom implementation that can accomodate all the specialities of the Minetest data. Sharing data in an incremental way boosts the performance very significantly, but for that you have to know how Minetest updates the data, that's why. (Also a custom implementation can be integrated into the existing engine smoothly without the need of quirks.)

For low-level, I've already suggested the peer exchange part of libtorrent, and/or IRC to distribute server commands.

Other than that, it is pretty simple and straightforward to write a P2P library, I talk from experience. But if you don't feel up to the task, a quick websearch gave me literally hundreds of Open Source and multiplatform implementations. For example here's one that has brainf*ck simple API (I haven't used this lib myself, it was just on the first page of the websearch results and looks simple to use).

And there's libp2p on which IPFS was built upon. You won't need the entire bloated IPFS stack to share data among peers, it's more than enough to use it's low-level library. Libp2p was separated from IPFS and made available individually exactly for that reason. This library is also a very good candidate.

But as I've said, there are hundreds of FOSS implementations, feel free to pick one which you prefer the most, with the API closest to your design.

Cheers,
bzt

[Фред Канниг]

On github, a concept of implementation using the RAFT algorithm is proposed:
https://github.com/aofg/decentralized-mmo

[MirceaKitsune]

On github, a concept of implementation using the RAFT algorithm is proposed
https://github.com/aofg/decentralized-mmo

Very interesting, thanks for sharing!

[Фред Канниг]

This topic echoes the following:
viewtopic.php?f=3&t=13402
viewtopic.php?f=10&t=23231

[Linuxdirk]

This topic echoes the following:

Not even loosely related.

[Фред Канниг]

This topic echoes the following:

Not even loosely related.

In these themes we consider the problem to get rid of dependence on the server.

В этих темах рассматривается задача избавиться от зависимости от сервера.

[stackdump]

Hello,

I'm a programmer interested in building IPFS based chunk server.
My main goal would be to allow syncing between multiple servers, making worlds somewhat decentralized.

Is anyone working on this in the minetest community?

[ywwv]

minetest needs a method of auto-reconnecting to multiple servers. in order to traverse a decentralized map hosted by multiple servers. possibly this could be transparently done server side by a proxy that could seamlessley switch players between server environments. even though it might not be so seamless in practice. its similar to website load-balancing servers. you can awalys buy more bandwidth. but they dont make CPU cores big enough for a certain amount of players . so this is an interesting solution to the problem. the servers could have an internal protocol to "hand off" various chunk simulation to each other

[c56]

instead of just hosting the whole server as p2p application why not host 1 central server that verifys every node addition / deletion and the p2p network purely acts as storage (the map format would also need to have the time encoded and the server would need to have an index of which map blocks are stored by which client ( and before the player leaves his map block would have to be transfered to another player or backup storage on server and at least 3 clients would need to have a storage of every map block as there is no such thing as majority with 2 people) also the server should store sha256 hashes of every map block so in the event of loosing the map block it can brute force what could have been there and compare it with the map seed or it could be used to decide which client has the up to date version of the map block at this point computing power invested could also be used as ingame currency #mesecoin to incentivise people of instead of playing minetest on a real server and opening other apps to fry theyr gpu at the task of hashing map blocks

[Фред Канниг]

Not the last point of the idea of decentralization is to get rid of the authoritarian server as a single point of failure.