Public Remote Media Server Project

[lag01]

As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...

[sofar]

As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...

It could, or, you can deploy the media server on a CDN, and automatically do this on the server side. Not that it matters much at this point, since the throughput needed to host the PRMS is trivial, and the client download rate is the limiting factor.

[sofar]

Quick stats on the server:

last full week: 11gb total, 3k unique vsits, 1.5 million files requested.

Please keep suggesting mods, especially if they have moved.

[texmex]

I suggest you add the bgm mod since it bandwidth heavy!

https://gitlab.com/timnxx20/bgm

[sofar]

I suggest you add the bgm mod since it bandwidth heavy!

https://gitlab.com/timnxx20/bgm

The included tracks are taken from the album "Resolute" by "Blue Dot Sessions" and are released under a CC BY-NC 4.0 license.

Sorry, I won't add this, especially since it's both large and NC licensed. Servers that force players to download these media files should have to bear the burden themselves to serve these to their players.

[sofar]

I've updated the topic to include a little blurb that states that mods should be freely distributable, as well as a quick statement that we can refuse mods for various reasons. Most of the time it's as simple as "this mod already contains all these assets" or "this is an obsolete mod" but sometimes it can be content or license related. For instance, I'll likely not add things like nudity, etc.

[BuckarooBanzay]

Can you add the following mods/packs:

• display_modpack viewtopic.php?f=11&t=19365
• Botic font for display_modpack: https://github.com/pyrollo/font_botic

Thanks for your service :)

[sofar]

Can you add the following mods/packs:

• display_modpack viewtopic.php?f=11&t=19365

Contains NC material. Maybe replace it with a font that's not NC?

[*] Botic font for display_modpack: https://github.com/pyrollo/font_botic[/list]
)

The font is SIL, so I've added that (it's also tiny, really).

[rubenwardy]

Hi, please may you add capturetheflag and ctf_pvp_engine?

https://github.com/MT-CTF/capturetheflag
https://github.com/MT-CTF/ctf_pvp_engine

Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)

[sofar]

Hi, please may you add capturetheflag and ctf_pvp_engine?

https://github.com/MT-CTF/capturetheflag
https://github.com/MT-CTF/ctf_pvp_engine

Done, added.

Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)

That's a bug in the client? What client version are you seeing this in?

The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".

Somehow there are clients out there that (1) completely barf the remote media protocol and then (2) ask for the plain file instead of the sha1 from the media server.

I've seen this on the server side, but I don't get this on my own clients and I've never been able to reproduce it, because it is specific to some clients somehow. I've wanted to figure out why, but all I can see from the protocol implementation is that my PRMS side is correct and it had those files properly to begin with.

FYI, the PRMS has these resources and will share them by sha1sum properly:

Code: Select all

$ find . -name wool_pink.png | xargs sha1sum
c431049287c68a56bf08f933ccb3b59e0bf3c9ee  ./MineClone2/mods/ITEMS/mcl_wool/textures/wool_pink.png
96d5fa1509960522cf20bbef370e31775d5deed4  ./minetest_game/mods/wool/textures/wool_pink.png
c71c9c8a2178c7945e42656e312b69324ac33732  ./capturetheflag/mods/shooter/textures/wool_pink.png
96d5fa1509960522cf20bbef370e31775d5deed4  ./capturetheflag/mods/wool/textures/wool_pink.png
$ find . -name default_tool_bronzesword.png | xargs sha1sum
476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./minetest_game/mods/default/textures/default_tool_bronzesword.png
476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./capturetheflag/mods/default/textures/default_tool_bronzesword.png

...

$ curl http://minetestmedia.foo-projects.org/c431049287c68a56bf08f933ccb3b59e0bf3c9ee -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   203  100   203    0     0  50750      0 --:--:-- --:--:-- --:--:-- 50750
$ file c431049287c68a56bf08f933ccb3b59e0bf3c9ee 
c431049287c68a56bf08f933ccb3b59e0bf3c9ee: PNG image data, 16 x 16, 4-bit colormap, non-interlaced
$ sha1sum c431049287c68a56bf08f933ccb3b59e0bf3c9ee 
c431049287c68a56bf08f933ccb3b59e0bf3c9ee  c431049287c68a56bf08f933ccb3b59e0bf3c9ee

So, from the PRMS side, everything is OK. Hence my suspicion that there are bugged/old clients, somehow. I know that pre-0.4.14 is probably broken, maybe even 0.4.15.

[Lejo]

That's a bug in the client? What client version are you seeing this in?

The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".

And what about only providing the raw file and not the sha1sum?
I‘m using this way and it worked for my client.

[sofar]

And what about only providing the raw file and not the sha1sum?
I‘m using this way and it worked for my client.

That would break if mod A and mod B both wanted to provide different versions of the same file name. Which happens, a lot.

The above method is broken by design. The PRMS has many files that are duplicate filenames but a different sha1sum. Only this way will clients be sure to get the actual content they need, and not something randomly picked.

[sofar]

[EDIT] this experiment is cancelled now [/EDIT]


I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

It's possible that newer clients might see this texture somehow, but I can't figure out from the code why. If you see this texture appear in 0.4.17 versions of minetest, please send me a screenshot and the server address so I can try to reproduce the issue.

[Linuxdirk]

I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.

[sofar]

I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.

These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.

So it's not as black and white as "a double wrong". These clients allow themselves to be passed malicious content by a MITM, effectively.

Anyway, it's an experiment, and I'm going to make sure this isn't a bigger nuisance or causes other problems. If you spot issues, please let me know ASAP.

[sofar]

Well, after compiling 0.4.13 and attempting to see if the placeholder image actually shows up in-game, I've stopped the experiment. The 0.4.13 MT client code does download the placeholder, but then properly checks the checksum and does not use the placeholder image in-game, and so it's more efficient for the PRMS to return a 404 instead of a placeholder image.

I'm not evil enough to generate sha1 collisions on the fly :).

[Linuxdirk]

These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.

Now it makes absolute sense to me.

[sofar]

The following mods were *removed* due to no longer being available for updates:

git://github.com/MT-Modder/hydroponics
git://github.com/bell07/minetest-skinsdb
git://github.com/tenplus1/mob_horse
git://github.com/tenplus1/mobs_animal
git://github.com/tenplus1/mobs_monster
git://github.com/tenplus1/mobs_npc
git://github.com/tenplus1/farming.git

If people want these still hosted at the PRMS, send me the proper git locations, and I can add them back.

[Thomas-S]

As far as the tenplus1 mods are concerned, this should be the new git locations:

https://notabug.org/TenPlus1/mob_horse.git
https://notabug.org/TenPlus1/mobs_animal.git
https://notabug.org/TenPlus1/mobs_monster.git
https://notabug.org/TenPlus1/mobs_npc.git
https://notabug.org/TenPlus1/farming.git

[sofar]

As far as the tenplus1 mods are concerned, this should be the new git locations:

https://notabug.org/TenPlus1/mob_horse.git
https://notabug.org/TenPlus1/mobs_animal.git
https://notabug.org/TenPlus1/mobs_monster.git
https://notabug.org/TenPlus1/mobs_npc.git
https://notabug.org/TenPlus1/farming.git

WHY does notabug not allow/provide git:// support? WHY?

sigh, added. With grumbling.

[Walker]

I have added this service to my five servers

add server running under KNOPPIX ( Debian ) with minetest-server 0.4.17.1
and the minetest-game 0.4.17.1 or the mineclone2 0.45.1 subgame
minetest can be found here: https://repo.or.cz/MineClone/MineClone2.git

the list of used 3rd person mods by servers:

Spoiler

--- German-Mineclone2-Night-server ( Port 30000 )
gameid = MineClone2
backend = sqlite3
creative_mode = false
enable_gemage = true
player_backend = files
server_announce = true
load_mod_areas = true
load_mod_protector = false
load_mod_sethome = true
load_mod_invisible = true
load_mod_invhack = true
load_mod_xban2 = true

--- German-MineClone2-server ( Port 30001 )
gameid = MineClone2
backend = sqlite3
creative_mode = false
enable_gemage = true
player_backend = files
server_announce = true
load_mod_areas = true
load_mod_protector = false
load_mod_sethome = true
load_mod_invisible = true
load_mod_invhack = true
load_mod_xban2 = true

--- German-Vanilla-server ( Port 30002 )
gameid = minetest
backend = sqlite3
creative_mode = false
enable_gemage = true
player_backend = files
server_announce = true
load_mod_areas = true
load_mod_protector = true
load_mod_sethome = true
load_mod_gravelsieve = true
load_mod_bonemeal = true
load_mod_throwing = true
load_mod_3d_armor = true
load_mod_farming = true
load_mod_shields = true
load_mod_hazmat_suit = true
load_mod_mtg_i18n = true
load_mod_hud = true
load_mod_mobs = true
load_mod_technic_armor = true
load_mod_3d_armor_stand = true
load_mod_hunger = true
load_mod_anvil = false
load_mod_inventorybags = true
load_mod_intllib = true
load_mod_mobs_mc = true
load_mod_wieldview = true
load_mod_mobs_monster = true
load_mod_mobs_animal = true
load_mod_invisible = true
load_mod_invhack = true
load_mod_unified_inventory = true
load_mod_unified_inventory_plus = true
load_mod_xban2 = true

--- German-Underground-server ( Port 30003 )
gameid = minetest
backend = sqlite3
creative_mode = false
enable_gemage = true
player_backend = files
server_announce = true
load_mod_areas = true
load_mod_protector = true
load_mod_sethome = true
load_mod_gravelsieve = true
load_mod_bonemeal = true
load_mod_throwing = true
load_mod_3d_armor = true
load_mod_farming = true
load_mod_shields = true
load_mod_hazmat_suit = true
load_mod_mtg_i18n = true
load_mod_hud = true
load_mod_mobs = true
load_mod_technic_armor = true
load_mod_3d_armor_stand = true
load_mod_hunger = true
load_mod_anvil = false
load_mod_inventorybags = true
load_mod_intllib = true
load_mod_mobs_mc = true
load_mod_wieldview = true
load_mod_mobs_monster = true
load_mod_mobs_animal = true
load_mod_invisible = true
load_mod_invhack = true
load_mod_unified_inventory = true
load_mod_unified_inventory_plus = true
load_mod_xban2 = true

--- German-Trash-server ( Port 30004 )
gameid = minetest
backend = sqlite3
creative_mode = false
enable_gemage = true
player_backend = files
server_announce = true
load_mod_areas = true
load_mod_protector = true
load_mod_sethome = true
load_mod_gravelsieve = true
load_mod_bonemeal = true
load_mod_throwing = true
load_mod_3d_armor = true
load_mod_farming = true
load_mod_shields = true
load_mod_hazmat_suit = true
load_mod_mtg_i18n = true
load_mod_hud = true
load_mod_mobs = true
load_mod_technic_armor = true
load_mod_3d_armor_stand = true
load_mod_hunger = true
load_mod_anvil = false
load_mod_inventorybags = false
load_mod_intllib = true
load_mod_mobs_mc = true
load_mod_wieldview = true
load_mod_mobs_monster = true
load_mod_mobs_animal = true
load_mod_invisible = true
load_mod_invhack = true
load_mod_unified_inventory = true
load_mod_unified_inventory_plus = true
load_mod_xban2 = true

P.S. sorry for my bad english, i am from german ^^

[sofar]

load_mod_areas = true

I can't do anything with this data.

Please list only new mods that were not added before, and give me the git URL for them. If you say "load_mod_areas" then I have no idea what git URL that is, or whether you use a fork of it at a different URL or not.

For reference, a git url starts with "git://"

[sofar]

Update:

You can get a list of currently included mods from this URL: http://media.foo-projects.org/remotes. You can use this to check if your favorite mods are already included or not. Or send corrections and updates.

[jas]

Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)

Thank you for your consideration

[sofar]

Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)

Added both. They'll be included in a few hours as usual.