Proposal: Peer to Peer filesharing for Minetest Content

Post Reply
User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

For mods we have our ContentDB, forum uploads for small mods, and GitHub releases. These work fine for projects that are mostly code, however they don't work very well for larger projects like large texture packs, worlds, and games with included worlds, which can weigh well into the gigabyte range. I think the sheer size is a big reason we don't have worlds downloadable on ContentDB.

The issue often comes up when wanting to host a large map such as Karsthafen or a server backup that it's simply not obvious where to host it. Most people would rather not have to pay for hosting, but then free hosting can be easily lost such as lost websites like PhotoBucket, MegaUpload in the past. Free hosting can also have malvertising or confusing fake 'download' links. So whether by payment or advertising, commercial file hosters make their money, or else go bust and you lose things to link rot, and hope and pray the original author still has the files and will be active to upload them elsewhere.

I propose that peer-to-peer (P2P) file sharing may be a good solution for these large projects, and that it would enable us to add them to an augmented ContentDB. P2P fits with Minetest's culture of freely redistributable content, is architecturally more resilient, and if there are a good number of seeders it can often be faster than free file hoster's download speeds. Note however that I am not saying we should ever stop serving Minetest content over HTTPS, just that P2P options should be available and would be good for us. There may be difficulties in convincing some people that using P2P is perfectly legal and helpful, but by publishing under free software/free culture licenses the right to redistribute is technically already there, even if it might make some people uncomfortable.

Here is one possible P2P proposal: The Minetest project could host a BitTorrent tracker that provides coordination for that protocol, and possibly a small dedicated seeder just like Internet Archive hosts a tracker and dedicated but low-bandwidth seeder. ContentDB content could be made alternately available via BitTorrent in addition to HTTPS. The official Minetest client may not need BitTorrent capability but to make downloading over BitTorrent seamless and drive more adoption it realistically would, because downloading many mods manually is never fun. BitTorrent is a well-proven protocol and I think it would only take a little bit of effort to set up a tracker and start saving ContentDB bandwidth. Now BitTorrent can tend to attract a lot of leechers, so there should probably be some kind of recognition or reward given to top seeders as well, though maybe some seeders want to remain anonymous. My next proposal can work around the leeching problem somewhat.

Another proposal is to add ContentDB content to the InterPlanetary File System (IPFS). Functionally this is similar to BitTorrent but even more decentralised. ContentDB would publish IPFS hashes, one per version per content item, and clients, again, probably the official Minetest client, would be able to download the ContentDB content from automatically-found peers across the world. An IPNS reference could also point to the latest version of a mod, which can then be checked to see if the mod is up-to-date. Assuming we used the operating system's IPFS daemon, then once a mod is downloaded, it would also be automatically seeded so long as it is kept in the IPFS cache and the IPFS daemon is not shut down. Among other things about IPFS, I am not clear on the policy of IPFS caching and whether all of a person's Minetest mods would expire out of it; IPFS is impressively simple to use while also being a beast to write all the network code I'm sure. I can see people being more hesitant to use IPFS; although it is not very new any more, it it not as well-established as BitTorrent outside a few niche circles. I just thought I had better mention more than one P2P protocol.

To summarise: I think we should make large projects like large texture packs, worlds and games with included worlds to an augmented ContentDB that runs on Peer-to-Peer (P2P) filesharing technology, as well as all the existing ContentDB content and the existing HTTPS delivery. This will save us headache from file hosting websites and make it even easier to get content for Minetest and save us bandwidth. The only drawbacks are: (1) it may make some people uncomfortable or (2) at worst, it hardly saves any bandwidth and resources compared to the centralised ContentDB due to leeching.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Komodo
Member
Posts: 163
Joined: Tue Jan 11, 2022 13:33
GitHub: MeseCraft
In-game: Komodo
Location: God Bless America
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Komodo » Post

Torrent use has been declining ever since streaming media has become the norm. I dont think nontechnical users are familar with torrenting.
Unfortunately, I dont believe it would provide a good reliable user experience. Also, the files being uploaded still need to be seeded and hosted from somewhere. It doesnt really dimish file size requirements, just bandwidth use reduction. Could have a network of torrent hosts seeding content but usually this is done by large institutions or companies to donate to projects like Linux distros from what Ive seen on the net. Also many vpns and edu networks block p2p torrent protocols completely.
🌎 Website | 🌲 MeseCraft Game | 📰 News | 🖌️ ContentDB

c56
Member
Posts: 307
Joined: Wed Apr 21, 2021 03:05
GitHub: tigercoding56
In-game: bm5 or bemo5 also sell_her_on55

Re: Proposal: Peer to Peer filesharing for Minetest Content

by c56 » Post

<irony> i have a idea , instead of adding those things to minetest client , make a new revolutionary CSM api that allows you to implement torrent functionality , with the minor cost of allowing technical players to exploit the game even more which is possible by changing the source files directly so why even bother blocking it </irony>
this is a signature not a place to post messages also if i could change my username i would change it to sell_her_on55

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Komodo wrote:
Thu Sep 29, 2022 00:55
Torrent use has been declining ever since streaming media has become the norm. I dont think nontechnical users are familar with torrenting.
Yes and no. Some people won't pony up for things that are streaming-only or not on their platform. And there's plenty of abandonware and plenty of reasons you should make your games playable offline if you can. For Westerners, parts of Asia and so on, places with reliable internet, sure most people are lax. Also Minetest is heavily skewed towards technical people in my experience.
Komodo wrote:
Thu Sep 29, 2022 00:55
Unfortunately, I dont believe it would provide a good reliable user experience. Also, the files being uploaded still need to be seeded and hosted from somewhere. It doesnt really dimish file size requirements, just bandwidth use reduction.
Yes, obviously someone would have to run a dedicated seeder along with the tracker. Even if it doesn't achieve great speeds, well, it'll probably work out not much slower than some free file hosting sites.
Komodo wrote:
Thu Sep 29, 2022 00:55
Could have a network of torrent hosts seeding content but usually this is done by large institutions or companies to donate to projects like Linux distros from what Ive seen on the net.
Yes it doesn't seem like we have a lot of organisational support. I mean there's not much money in Minetest as most of our devs and creators can attest. But I think if a few people volunteered to always run their BitTorrent clients on their PCs, or a seedbox or two we could get really impressive results. I certainly wouldn't mind running my client when I'm on my PC to help seed.
Komodo wrote:
Thu Sep 29, 2022 00:55
Also many vpns and edu networks block p2p torrent protocols completely.
Some people work around blocks by first VPNing home, assuming VPN protocols aren't blocked. Besides, just because there are draconian measures in place some places, doesn't mean we shouldn't offer it for those who can. And in my proposal I did say we shouldn't get rid of HTTPS, just that larger files can be on P2P.
Last edited by Blockhead on Mon Aug 14, 2023 09:06, edited 2 times in total.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

Astrobe
Member
Posts: 568
Joined: Sun Apr 01, 2018 10:46

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Astrobe » Post

Blockhead wrote:
Thu Sep 29, 2022 02:59
Komodo wrote:
Thu Sep 29, 2022 00:55
Torrent use has been declining ever since streaming media has become the norm. I dont think nontechnical users are familar with torrenting.
Yes and no. Some people won't pony up for things that are streaming-only or not on their platform. And there's plenty of abandonware and plenty of reasons you should make your games playable offline if you can. For Westerners, parts of Asia and so on, places with reliable internet, sure most people are lax. Also Minetest is heavily skewed towards technical people in my experience.
Depends on how it is done. According to WIkipedia, some popular AAA games do use BitTorrent, and the vast majority of their users probably have no idea. Some are known to use Libtorrent under the hood.

I second the proposal not for bandwidth but because ContentDB is a single point of failure. This year I have seen two FOSS projects having trouble because the maintainer of the official site went MIA.

Also, the proposal seems to focus on large mods, but one could consider a way to bundle small or mid-size mods into a single archive. As to which ones, probably the featured mods are good first candidates. Sure, someone at the very least has to run a script to update that bundle, but I would argue that featured mods are generally stable and getting a slightly older version is still better than getting nothing at all.
My game? It's Minefall.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Astrobe wrote:
Mon Aug 14, 2023 08:48
Depends on how it [BitTorrent] is done. According to WIkipedia, some popular AAA games do use BitTorrent, and the vast majority of their users probably have no idea. Some are known to use Libtorrent under the hood.
Huh, didn't realise BitTorrent was so popular for patching MMOs, but it makes sense. It's very unlikely someone's going to successfully generate a hash collision that also delivers a malicious payload unless they're a state actor/megacorp.
Astrobe wrote:
Mon Aug 14, 2023 08:48
I second the proposal not for bandwidth but because ContentDB is a single point of failure. This year I have seen two FOSS projects having trouble because the maintainer of the official site went MIA.
You're right, reliability is important, and 1 ContentDB is not very reliable if it goes down/rubenwardy goes MIA. At least ContentDB's server software is free and it's got an API so it could be mirrored (I wrote about this recently in "Is there a guide to hosting your own ContentDB?). However, the proposal is about more than ContentDB, it's about all the different sites that host "Content", like worlds, since those can't be put on ContentDB.
Astrobe wrote:
Mon Aug 14, 2023 08:48
Also, the proposal seems to focus on large mods, but one could consider a way to bundle small or mid-size mods into a single archive. As to which ones, probably the featured mods are good first candidates. Sure, someone at the very least has to run a script to update that bundle, but I would argue that featured mods are generally stable and getting a slightly older version is still better than getting nothing at all.
Actually my first focus is on worlds, especially worlds with worldmods and games with an included world (with an approach like modgen). Second, on large games, mods and texture packs. Then the rest. Worlds have usually been the least reliable to download long term since there's no provider that is both free/cheap and reliable at the same time. Even with the popular personal cloud storages like Google Drive/Dropbox, there are rate limits and accounts can be deleted.

Large games, mods/packs and texture packs are downloadable from ContentDB but P2P scales better if Minetest got really popular and ContentDB's connection got saturated. What ContentDB is pretty good at is small mods and resolving dependencies. In my estaimation, it's not as hard to host content sized 0-5MB or so, since most people don't browse fast enough for the downloads to queue up.

I do agree that bundles *could* be useful, but even ContentDB doesn't have them. I actually made a request for collections to be added to ContentDB back in May 2022, based on the way that Steam Workshop collections work.

Let's run through a hypothetical CDB-P2P project. The server software would be a fork that makes some changes but tracks mainline. Its core functionality would be to serve magnet or torrent links on the front end instead of direct HTTPS downloads. This is under the assumption of BitTorrent. In case of IPFS, it would just serve ipfs (specific version) or ipns (latest version) URIs.

There would of course have to be a tracker-seeder as well to support the downloads, especially with low availability of seeders for packages at first. I'm not familiar with tracker and seeder software, hopefully there is something decent available without much hassle to set up, e.g. in Debian's package repositories. Again, not an expert, but in case of IPFS a server would have to be made available that can actually serve all of the content over IPFS before it can propagate through the P2P network onto different nodes running IPFS daemons.

For content, CDB-P2P would mirror the mods, modpacks, texture packs, and games from the main ContentDB by keeping track of what's on the main ContentDB using an API key. There would also be a separate worlds section in addition to the mainline ContentDB types. The worlds would be originally sourced, not mirrored.

I would probably say to open the site to registrations but only allow adding worlds - the point is *not* to compete with the main ContentDB for aggregation.

Finally, to add collections to CDB-P2P, that feature would just be pulled from upstream/mainline ContentDB when implemented there. Collections are now implemented! The collection features would of course be part of the fork. Collections would also be mirrored via the API but would available over BitTorrent instead of HTTPS as normal. Some original collections may be provided which include worlds, for example something like "Parkour Maps Pack".
Last edited by Blockhead on Tue Aug 15, 2023 12:23, edited 1 time in total.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Festus1965 » Post

Astrobe wrote:
Mon Aug 14, 2023 08:48
Depends on how it is done. According to WIkipedia, some popular AAA games do use BitTorrent, and the vast majority of their users probably have no idea. Some are known to use Libtorrent under the hood.
Games, ok - what was about Windows itself ?
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Hybrid Dog
Member
Posts: 2828
Joined: Thu Nov 01, 2012 12:46
GitHub: HybridDog

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Hybrid Dog » Post

Instead of a file hosting service, could a free Content Delivery Network service be used to distribute large files?
And would using zsync to update already-downloaded mods, texture packs, etc. have a big impact on bandwidth saving?

‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Hybrid Dog wrote:
Mon Aug 14, 2023 18:57
Instead of a file hosting service, could a free Content Delivery Network service be used to distribute large files?
CDNs are a kind of third way compared to paying for central server hosting for ContentDB or individuals using free/paying for file hosting, which is what I've discussed so far. A CDN could be integrated to deliver the downloads on ContentDB with minimal modification, on a different hostname that points to the CDN, but the main website would probably stay centralised.

But can you tell me more about these "free" CDNs? I'm not familiar with them. I suspect most of them are either not very reliable - and the point of this proposal is reliability - or severely limited and want you to upgrade to go beyond certain data quotas.
Hybrid Dog wrote:
Mon Aug 14, 2023 18:57
And would using zsync to update already-downloaded mods, texture packs, etc. have a big impact on bandwidth saving?
Mods are served inside zip on ContentDB currently. I had to look up zsync, but right on the homepage it says it supports looking inside gzip files, not zips. It's not that .tar.gz isn't easy enough to use on Windows & macOS with a third party bit of software but .zip is integrated right into File Explorer since I think it was Windows Vista, so it remains preferred. You would want either an adaptation of zsync to .zip or make an unpopular move to .tar.gz which is a bit more friction for manual downloaders on Windows & macOS.

The biggest gains to be had would be text-format files inside the archive. If a PNG file, B3D file and so on is changed, there's little to be gained by doing a binary diff on them, as they are formats with compressed data blocks inside of them. Every time the compressed block changes, it tends to change drastically no the difference in size of the compressed block. The big gains would be Lua files, localisation files and OBJ files.

This approach would definitely save bandwidth for ContentDB downloads, but it doesn't really improve reliability except by making the one website scale a bit better - at least, assuming ContentDB has the CPU power to run a zsync daemon to diff all the packages, which I'm sure is more CPU than the current approach.

Funny timing, after my recent comment Rubenwardy went ahead and implemented collections - I'm sure it was in the works before I mentioned it given how big the diff looks, and I don't really want to take any credit. Still interesting though :)
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

Astrobe
Member
Posts: 568
Joined: Sun Apr 01, 2018 10:46

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Astrobe » Post

Blockhead wrote:
Tue Aug 15, 2023 03:20
Funny timing, after my recent comment Rubenwardy went ahead and implemented collections - I'm sure it was in the works before I mentioned it given how big the diff looks, and I don't really want to take any credit. Still interesting though :)
Why did he go for a specific package spec, instead of saying "well, collections are just meta-packages, that is empty packages that just declare required dependencies (even though they do nothing with them), to have the client pull them when the meta-package is installed; this is basically a modpack without including a hardcopy of each mod. All I need to do is to let users tell ContentDB it is a meta-package, so that CDB can generate a nice display from that dependency list"? (few, that was a long question).
My game? It's Minefall.

User avatar
rubenwardy
Moderator
Posts: 6969
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by rubenwardy » Post

Astrobe wrote:
Tue Aug 15, 2023 12:14
Why did he go for a specific package spec, instead of saying "well, collections are just meta-packages, that is empty packages that just declare required dependencies (even though they do nothing with them), to have the client pull them when the meta-package is installed; this is basically a modpack without including a hardcopy of each mod. All I need to do is to let users tell ContentDB it is a meta-package, so that CDB can generate a nice display from that dependency list"? (few, that was a long question).
On the Minetest side, It's not possible for packages to depend on texture packs or games currently, adding collections as packages right now would not work. I also don't want to flood the package list with a load of collections - I'd like to show collections in a different way to packages. This will need to wait for the ContentDB redesign inside the client

On the web side, packages and collections are quite different interfaces. To make it a type of Package, I'd need to refactor a huge amount of code relating to packages. It would be easier to just pretend to the client that collections are packages as part of the API
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
Hybrid Dog
Member
Posts: 2828
Joined: Thu Nov 01, 2012 12:46
GitHub: HybridDog

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Hybrid Dog » Post

Blockhead wrote:
Tue Aug 15, 2023 03:20
But can you tell me more about these "free" CDNs? I'm not familiar with them. I suspect most of them are either not very reliable - and the point of this proposal is reliability - or severely limited and want you to upgrade to go beyond certain data quotas.
I'm also not familiar with them; I just noticed that they exist. There is, for example, Couldflare's CDN, which can be used for websites for free but requires Paid Services if there is a "disproportionate percentage" of large files.

Perhaps there is another option to increase reliability: People who host Minetest servers could voluntarily serve ContentDB downloads, too. The traffic caused by the Minetest server (mapblock sending, etc.) is probably much larger than traffic caused by ContentDB file downloads.

‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪‮
‮‪

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Hybrid Dog wrote:
Sat Aug 19, 2023 09:18
There is, for example, Couldflare's CDN, which can be used for websites for free but requires Paid Services if there is a "disproportionate percentage" of large files.
Sound like ContentDB wouldn't fit into the free tier, given the size of some of the packages, like HD texturepacks and entire games, and possibly just the sheer number of files.
Hybrid Dog wrote:
Sat Aug 19, 2023 09:18
Perhaps there is another option to increase reliability: People who host Minetest servers could voluntarily serve ContentDB downloads, too. The traffic caused by the Minetest server (mapblock sending, etc.) is probably much larger than traffic caused by ContentDB file downloads.
This would fit into the CDB-P2P proposal where server owners would also run a BitTorrent seeder the same as the main seeder that CDB-P2P would run.

This would probably be straightforward to deploy for server owners who have a VPS or homelab. Once the setup is possible to automate for the main seeder, the same or similar script can be run on secondary seeders. The only problem would potentially be bandwidth and storage limits. A lot of VPS providers make those quite expensive in terms of value for money ($/GB) or require an upgraded standard plan where you pay for more CPU, RAM and so on.

However this would mostly benefit singleplayer players, as servers usually directly serve the media content that is used on their server, or refer the Minetest client to a separate HTTP media server. A CDN might be usable for that server to geographically distribute it, but that's a centralised not peer2peer approach. A P2P approach for servers would be connected clients distributing media files to other clients that are connecting, something that a lot of people might be uncomfortable doing or unable to by being at a school, public library etc, so it would be opt-in and probably relatively few people would know about and want to turn it on.

Theoretically one could engineer some kind of cryptocurrency usable on the server where mining is based on serving content to other connecting clients, but that sounds like a lot of work. And this currency would still be relatively hard-to-obtain since new clients or clients with deleted caches much less frequently join servers than returning players.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Festus1965 » Post

That Idea sounds not bad, it would also hardening server fail as same content is available on several independent server.
But : the content chosen should be limited or maybe free to choose by the share file hoster also.

So first make a number of how much content in GB is that.

Then maybe reduce by choosing main parts as engine, and standard/default mods and further by how often installed mods on all servers, like top x mods to spread, as THAT will reduce the load on one server or git and give most change that needed content is available on more server.

Would a git on some admin servers not do the same thing ? As contains automatic update from the source, as I remember my test with gitea where I collected all found mt content.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: Proposal: Peer to Peer filesharing for Minetest Content

by snowyu » Post

Git as backend is enough and simple.

1. More and more Free Git Services: There is an increasing number of free Git hosting services available, such as GitHub, GitLab, and Bitbucket. These platforms provide a convenient infrastructure for hosting and sharing Git repositories, making it easier for Minetest users to contribute and collaborate on content.
2. Distributed Nature of Git itself: Git itself is a distributed version control system, which means that each user has a complete copy of the repository. This decentralized approach is well-suited for peer-to-peer file sharing, as it enables users to share and receive Minetest content directly from one another, without relying on a centralized server.
3. Git Submodules for Referencing Other Git Services: One of the key features of Git is its ability to incorporate submodules, which are essentially references to other Git repositories. This feature can be utilized to extend the modpack's functionality to support not only mods but also subgames, texture packs, and world templates. By using submodules, users can easily include and manage various types of content from different Git repositories within their Minetest setups.

IMO, the modpack should be extended to support the subgame type, texture type and world type. This expansion would allow users to easily customize their Minetest experience by incorporating different subgames, applying various texture packs, and creating or using pre-designed world templates.

But Nobody care about these:

* [Proposal: Consider all LUA scripts on an equal footing](https://github.com/minetest/minetest/issues/13654)
* [[Feature] Use Git Repository to Replace PostgreSQL Database](https://github.com/minetest/contentdb/issues/424)

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

snowyu wrote:
Sun Aug 20, 2023 04:35
2. Distributed Nature of Git itself: Git itself is a distributed version control system, which means that each user has a complete copy of the repository. This decentralized approach is well-suited for peer-to-peer file sharing, as it enables users to share and receive Minetest content directly from one another, without relying on a centralized server.
ChatGPT bends the truth again. Sounds smart, isn't actually that smart. Git may be technically decentralised, in the technical meaning of the term where you don't "check in" to a master server. But really, if there's no mirrors of a repository then it's not really all that decentralised. And SSH and HTTP(S) (which can carry git over the network) are not peer-to-peer protocols. Sending git over those is just the usual client-server model.
snowyu wrote:
Sun Aug 20, 2023 04:35
IMO, the modpack should be extended to support the subgame type, texture type and world type. This expansion would allow users to easily customize their Minetest experience by incorporating different subgames, applying various texture packs, and creating or using pre-designed world templates.
There's such a thing as try to be too accommodating. If you can put games in a modpack and modpacks in a game, and bundle texture packs with a game, the rules become too hard to interpret. In a normal Minetest environment, mods can be enabled to override the current game's copy of that mod/mod with the same name. Now you have a tree or directed graph of override dependencies to work out:

Game A ships with modA, but also with modpack modpackB, which also has a mod called modA. The user then installs modA directly from ContentDB. Finally, the user installs another modpack, modpackD, which has a copy of modA. What happens when the user goes to the mod menu for a new world and enables all the mods? A big mess, that's what.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: Proposal: Peer to Peer filesharing for Minetest Content

by snowyu » Post

1. Git is a distributed version system which means it do not care about discovery client, transport. It's a very basic things. But you can make it with bittorrent(https://github.com/cjb/GitTorrent) , Dat(https://github.com/hackergrrl/hypergit) , IPFS(https://github.com/dhappy/git-remote-ipfs)...

2. No, you misunderstood my point. If the game is a modpack, then the modpack should only contain mods that are closely related to that game, rather than including all mods within the game. This allows for maximum mod reuse. The issues you mentioned are related to mod name conflicts and the inability to specify mod versions in mod.conf, both of which have been addressed in my pull request.

Other_Cody
Member
Posts: 49
Joined: Wed May 08, 2019 23:25
In-game: Other_Cody

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Other_Cody » Post

I think my question and this peer to peer filesharing post are similar
so this post my be suited for this forum discussion.

Maybe the Generic Inoffensive Flags mod,
the skinsdb, and the contentdb could be made into a downloader / uploader and you could post into those from the minetest client with or without your forum.minetest.net account.

Than players can post or even link to there minetest forum account, if they wish to, to the minetest client and do forum posts from the minetest client.

Though I do not yet know how to link the minetest client to, optionally, sign in as a forum.minetest.net user.

Though keeping the minetest server accounts separate from the forum account may also help security, as many passwords and user names will not be linked into one account than.

Is there any safe floss client mod yet made that does that?

Maybe even world date can be exchanged, optionally, if users wish to do that.

The minetest client also has a matrix chat
https://content.minetest.net/packages/j ... trix_chat/
mod
if that could be built on to make a file sharing mod.

https://content.minetest.net/packages/bell07/skinsdb/

https://content.minetest.net/packages/A ... ric_flags/

Even though I use Trisquel Gnu/Linux from

https://trisquel.info/

I think https://guix.gnu.org/

is working on a guix package program
to help with securely sharing packages.

That may also help share minetest games and mods.
For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Other_Cody wrote:
Sun Aug 27, 2023 19:56
I think my question and this peer to peer filesharing post are similar
so this post my be suited for this forum discussion.
Sorry but not really. Your ideas are related to working out how to fit the wider Minetest community services like chat, SkinsDB and ContentDB into the Minetest client itself. That is meant to help people engage with the community and not have to switch applications so much. In contrast, my idea is a relatively narrow focus on how to distribute downloads of Content in a way that doesn't have the centralisation problems that a single host like ContentDB has, which is an issue that is kind of separated from the "front end" or user view.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

Astrobe
Member
Posts: 568
Joined: Sun Apr 01, 2018 10:46

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Astrobe » Post

snowyu wrote:
Sun Aug 27, 2023 14:26
1. Git is a distributed version system which means it do not care about discovery client, transport. It's a very basic things. But you can make it with bittorrent(https://github.com/cjb/GitTorrent) , Dat(https://github.com/hackergrrl/hypergit) , IPFS(https://github.com/dhappy/git-remote-ipfs)...
The counter-argument is in your claim: why would Git want BitTorrent if it can replace it? Because it cannot.

Git could fit the bill as a distributed file sharing system, but it is a hijacking similar to using it as a back up system. It is both overkill and not entirely fit to the task, unlike true distributed file sharing systems, like Bittorent.

For instance, if you however go for Git, naively cloning repositories would lead to increased transfer times; you'd rather make a shallow copy. But then you are both still transferring some Git bookkeeping stuff (wasted bandwidth, wasted disk space) and still lose some functionality (the version history) - although, granted, very few users will miss it. Therefore, it is a slightly bad deal from the get go.

The issue here is that Git would solve problems you don't have, while introducing problems you wouldn't have without it. Use the right tool for the right job.
My game? It's Minefall.

User avatar
Tuxilio
Member
Posts: 140
Joined: Mon Nov 28, 2022 12:18
In-game: gamer777 tuxilio

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Tuxilio » Post

It would be a good idea to have P2P filesharing in Minetest, but it needs to be built into Minetest itself so that the user does not have to do anything.

Other_Cody
Member
Posts: 49
Joined: Wed May 08, 2019 23:25
In-game: Other_Cody

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Other_Cody » Post

guix.gnu.org

has

"
Copyright © 2012, 2013, 2014 Ludovic Courtès <ludo@gnu.org>
Copyright © 2019 Mathieu Othacehe <m.othacehe@gmail.com>

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.

* MAYBE Add a substituter that uses the GNUnet DHT or [[http://libswift.org][libswift]]

Would be neat if binaries could be pushed to and pulled from the GNUnet DHT or
rather libswift (since DHTs aren’t suited for large payloads). Guix users
would sign their binaries, and define which binaries they trust.
"

in it's TODO file in the git repository of https://git.savannah.gnu.org/git/guix

So maybe something like these or guix publish could help distribute minetest.

Though a torrent or git repository may also work.

Like
Emigrate from Github
viewtopic.php?f=3&p=428472#p428472

could be made into something like

git.minetest.net

or have a

torrent.minetest.net

to host minetest torrent files of at least the minetest engine, minetest_game, and irrlichtmt.
For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life.

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: Proposal: Peer to Peer filesharing for Minetest Content

by snowyu » Post

Astrobe wrote:
Mon Aug 28, 2023 07:39
The counter-argument is in your claim: why would Git want BitTorrent if it can replace it? Because it cannot.

Git could fit the bill as a distributed file sharing system, but it is a hijacking similar to using it as a back up system. It is both overkill and not entirely fit to the task, unlike true distributed file sharing systems, like Bittorent.

For instance, if you however go for Git, naively cloning repositories would lead to increased transfer times; you'd rather make a shallow copy. But then you are both still transferring some Git bookkeeping stuff (wasted bandwidth, wasted disk space) and still lose some functionality (the version history) - although, granted, very few users will miss it. Therefore, it is a slightly bad deal from the get go.

The issue here is that Git would solve problems you don't have, while introducing problems you wouldn't have without it. Use the right tool for the right job.
First of all, the best technology does not necessarily mean the most appropriate technology to use. I didn't say that Git can replace BitTorrent or Dat; what I meant is that Git lacks discovery etc capabilities.

Secondly, Git can preserve version history, which means different dependency versions of a repository can be found in the repository itself. And it can also perform shallow copies. Is it a waste of bandwidth? Once the code is retrieved from Git, does it still require network communication? Does it waste more than the current communication protocol used in Minetest? As for disk space, with the current storage units being in terabytes, is it really a significant waste compared to the space occupied by games nowadays?

Lastly, I believe that Git is currently the most cost-effective free space storage service available. It is also the most practical and freely mirrorable storage service for open-source projects.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Blockhead » Post

Other_Cody wrote:
Sat Sep 02, 2023 21:47
Would be neat if binaries could be pushed to and pulled from the GNUnet DHT or
rather libswift (since DHTs aren’t suited for large payloads). Guix users
would sign their binaries, and define which binaries they trust.
These are some decent suggestions for integrating networking into a/the Minetest client when connecting to CDB-P2P as was proposed. Something in C++ might be preferable just due to the codebase. Digital signature is something we hadn't thought about before, as HTTPS was considered "enough", but it definitely wouldn't hurt.
snowyu wrote:
Sun Sep 03, 2023 03:32
First of all, the best technology does not necessarily mean the most appropriate technology to use. I didn't say that Git can replace BitTorrent or Dat; what I meant is that Git lacks discovery etc capabilities.

Secondly, Git can preserve version history, which means different dependency versions of a repository can be found in the repository itself. And it can also perform shallow copies. Is it a waste of bandwidth? Once the code is retrieved from Git, does it still require network communication? Does it waste more than the current communication protocol used in Minetest? As for disk space, with the current storage units being in terabytes, is it really a significant waste compared to the space occupied by games nowadays?

Lastly, I believe that Git is currently the most cost-effective free space storage service available. It is also the most practical and freely mirrorable storage service for open-source projects.
You actually make a pretty good case that git with BitTorrent for transport could work, especially since you mention shallow copies. That would also make mods updatable from the command line. But it would also require git, which is something present ContentDB doesn't require but does integrate with. I didn't see recent activity from the GitTorrent project you previously linked, and there are a number of open issues. Would it still work with recent git clients and servers?
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

Astrobe
Member
Posts: 568
Joined: Sun Apr 01, 2018 10:46

Re: Proposal: Peer to Peer filesharing for Minetest Content

by Astrobe » Post

snowyu wrote:
Sun Sep 03, 2023 03:32
Secondly, Git can preserve version history, which means different dependency versions of a repository can be found in the repository itself. And it can also perform shallow copies. Is it a waste of bandwidth?
In absolute terms, yes, as it transfers metadata related to Git.

I don't see the "shallow copy from tag" (to get older versions) as a major advantage over just letting archives of older versions live in a torrent. In particular, users usually do want the latest version; they only want to go back to older versions if the author really screwed up.
Once the code is retrieved from Git, does it still require network communication?
It does not, as any other file transfer solution.
Does it waste more than the current communication protocol used in Minetest?
Difficult to say. If we are talking about installs/updates from CDB, it probably uses raw HTTP/HTTPS to transfer a compressed archive, which is probably in absolute terms better than Git using the same transport (assuming it does compress the data as well).
As for disk space, with the current storage units being in terabytes, is it really a significant waste compared to the space occupied by games nowadays?
Well, I've heard this argument for ages. But these past years it was proven wrong. For instance, the company I work for had to reduce the size of the storage embedded in the product because the price of this storage more than doubled.

Perhaps the few people who run MT servers from a Raspi can tell you more about this. The Raspi uses an SD card as storage. Consumer grade SD cards are relatively cheap, but they are notoriously unreliable and slow. Reliable and fast SDs are a lot more expensive.

It's always a good thing to keep an eye on resource usage, and make compromises when you get a clearly good trade-off.
Lastly, I believe that Git is currently the most cost-effective free space storage service available.
Nitpicking but Git is not a service. Gitlab, GitHub, etc. are.

That means in particular that your are in the same situation than when using Dropbox, Onedrive, Mediafire, whatever file hosting solution that has existed or still exists. You depend on a company that may vanish or terminate your account at any time, and is subject to other interference as well, such as the ridiculous DMCA law in the US.

Sure, you can mirror your repo somewhere else, but you still have the problem of "updating links" where ever you posted its URI and/or inform people that it has moved.
My game? It's Minefall.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests