Public Remote Media Server Project

User avatar
lag01
Member
Posts: 293
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag

Re: Public Remote Media Server Project

by lag01 » Post

As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

lag01 wrote:As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...
It could, or, you can deploy the media server on a CDN, and automatically do this on the server side. Not that it matters much at this point, since the throughput needed to host the PRMS is trivial, and the client download rate is the limiting factor.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Quick stats on the server:

last full week: 11gb total, 3k unique vsits, 1.5 million files requested.

Please keep suggesting mods, especially if they have moved.

User avatar
texmex
Member
Posts: 1751
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex

Re: Public Remote Media Server Project

by texmex » Post

I suggest you add the bgm mod since it bandwidth heavy!

https://gitlab.com/timnxx20/bgm
Mods | Support Mesehub: bc1qluuests9rxmlnvpjrhsnyjg9ucwy6z3r0y3srw

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

texmex wrote:I suggest you add the bgm mod since it bandwidth heavy!

https://gitlab.com/timnxx20/bgm
The included tracks are taken from the album "Resolute" by "Blue Dot Sessions" and are released under a CC BY-NC 4.0 license.
Sorry, I won't add this, especially since it's both large and NC licensed. Servers that force players to download these media files should have to bear the burden themselves to serve these to their players.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

I've updated the topic to include a little blurb that states that mods should be freely distributable, as well as a quick statement that we can refuse mods for various reasons. Most of the time it's as simple as "this mod already contains all these assets" or "this is an obsolete mod" but sometimes it can be content or license related. For instance, I'll likely not add things like nudity, etc.

User avatar
BuckarooBanzay
Member
Posts: 367
Joined: Tue Apr 24, 2018 05:58
GitHub: BuckarooBanzay
IRC: BuckarooBanzai
In-game: BuckarooBanzai

Re: Public Remote Media Server Project

by BuckarooBanzay » Post

Can you add the following mods/packs: Thanks for your service :)

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

BuckarooBanzay wrote:Can you add the following mods/packs:
Contains NC material. Maybe replace it with a font that's not NC?
BuckarooBanzay wrote:[*] Botic font for display_modpack: https://github.com/pyrollo/font_botic[/list]
)
The font is SIL, so I've added that (it's also tiny, really).

User avatar
rubenwardy
Moderator
Posts: 6094
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: United Kingdom
Contact:

Re: Public Remote Media Server Project

by rubenwardy » Post

Hi, please may you add capturetheflag and ctf_pvp_engine?

https://github.com/MT-CTF/capturetheflag
https://github.com/MT-CTF/ctf_pvp_engine

Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

rubenwardy wrote:Hi, please may you add capturetheflag and ctf_pvp_engine?

https://github.com/MT-CTF/capturetheflag
https://github.com/MT-CTF/ctf_pvp_engine
Done, added.
rubenwardy wrote: Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)
That's a bug in the client? What client version are you seeing this in?

The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".

Somehow there are clients out there that (1) completely barf the remote media protocol and then (2) ask for the plain file instead of the sha1 from the media server.

I've seen this on the server side, but I don't get this on my own clients and I've never been able to reproduce it, because it is specific to some clients somehow. I've wanted to figure out why, but all I can see from the protocol implementation is that my PRMS side is correct and it had those files properly to begin with.

FYI, the PRMS has these resources and will share them by sha1sum properly:

Code: Select all

$ find . -name wool_pink.png | xargs sha1sum
c431049287c68a56bf08f933ccb3b59e0bf3c9ee  ./MineClone2/mods/ITEMS/mcl_wool/textures/wool_pink.png
96d5fa1509960522cf20bbef370e31775d5deed4  ./minetest_game/mods/wool/textures/wool_pink.png
c71c9c8a2178c7945e42656e312b69324ac33732  ./capturetheflag/mods/shooter/textures/wool_pink.png
96d5fa1509960522cf20bbef370e31775d5deed4  ./capturetheflag/mods/wool/textures/wool_pink.png
$ find . -name default_tool_bronzesword.png | xargs sha1sum
476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./minetest_game/mods/default/textures/default_tool_bronzesword.png
476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./capturetheflag/mods/default/textures/default_tool_bronzesword.png

...

$ curl http://minetestmedia.foo-projects.org/c431049287c68a56bf08f933ccb3b59e0bf3c9ee -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   203  100   203    0     0  50750      0 --:--:-- --:--:-- --:--:-- 50750
$ file c431049287c68a56bf08f933ccb3b59e0bf3c9ee 
c431049287c68a56bf08f933ccb3b59e0bf3c9ee: PNG image data, 16 x 16, 4-bit colormap, non-interlaced
$ sha1sum c431049287c68a56bf08f933ccb3b59e0bf3c9ee 
c431049287c68a56bf08f933ccb3b59e0bf3c9ee  c431049287c68a56bf08f933ccb3b59e0bf3c9ee

So, from the PRMS side, everything is OK. Hence my suspicion that there are bugged/old clients, somehow. I know that pre-0.4.14 is probably broken, maybe even 0.4.15.

User avatar
Lejo
Member
Posts: 670
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: Public Remote Media Server Project

by Lejo » Post

sofar wrote: That's a bug in the client? What client version are you seeing this in?

The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".
And what about only providing the raw file and not the sha1sum?
I‘m using this way and it worked for my client.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Lejo wrote: And what about only providing the raw file and not the sha1sum?
I‘m using this way and it worked for my client.
That would break if mod A and mod B both wanted to provide different versions of the same file name. Which happens, a lot.

The above method is broken by design. The PRMS has many files that are duplicate filenames but a different sha1sum. Only this way will clients be sure to get the actual content they need, and not something randomly picked.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

[EDIT] this experiment is cancelled now [/EDIT]


I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

It's possible that newer clients might see this texture somehow, but I can't figure out from the code why. If you see this texture appear in 0.4.17 versions of minetest, please send me a screenshot and the server address so I can try to reproduce the issue.

User avatar
Linuxdirk
Member
Posts: 2507
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Public Remote Media Server Project

by Linuxdirk » Post

sofar wrote:I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".
So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Linuxdirk wrote:
sofar wrote:I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".
So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.
These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.

So it's not as black and white as "a double wrong". These clients allow themselves to be passed malicious content by a MITM, effectively.

Anyway, it's an experiment, and I'm going to make sure this isn't a bigger nuisance or causes other problems. If you spot issues, please let me know ASAP.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Well, after compiling 0.4.13 and attempting to see if the placeholder image actually shows up in-game, I've stopped the experiment. The 0.4.13 MT client code does download the placeholder, but then properly checks the checksum and does not use the placeholder image in-game, and so it's more efficient for the PRMS to return a 404 instead of a placeholder image.

I'm not evil enough to generate sha1 collisions on the fly :).

User avatar
Linuxdirk
Member
Posts: 2507
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Public Remote Media Server Project

by Linuxdirk » Post

sofar wrote:These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.
Now it makes absolute sense to me.

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

The following mods were *removed* due to no longer being available for updates:

git://github.com/MT-Modder/hydroponics
git://github.com/bell07/minetest-skinsdb
git://github.com/tenplus1/mob_horse
git://github.com/tenplus1/mobs_animal
git://github.com/tenplus1/mobs_monster
git://github.com/tenplus1/mobs_npc
git://github.com/tenplus1/farming.git

If people want these still hosted at the PRMS, send me the proper git locations, and I can add them back.

Thomas-S
Member
Posts: 110
Joined: Sun Sep 06, 2015 12:23
GitHub: Thomas--S
IRC: Thomas-S
In-game: Thomas-S
Location: Bavaria, Germany

Re: Public Remote Media Server Project

by Thomas-S » Post


sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

WHY does notabug not allow/provide git:// support? WHY?

sigh, added. With grumbling.

User avatar
Walker
Member
Posts: 1078
Joined: Tue Oct 03, 2017 09:22
In-game: Walker

Re: Public Remote Media Server Project

by Walker » Post

I have added this service to my five servers

add server running under KNOPPIX ( Debian ) with minetest-server 0.4.17.1
and the minetest-game 0.4.17.1 or the mineclone2 0.45.1 subgame
minetest can be found here: https://repo.or.cz/MineClone/MineClone2.git

the list of used 3rd person mods by servers:
+ Spoiler
P.S. sorry for my bad english, i am from german ^^

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Walker wrote: load_mod_areas = true
I can't do anything with this data.

Please list only new mods that were not added before, and give me the git URL for them. If you say "load_mod_areas" then I have no idea what git URL that is, or whether you use a fork of it at a different URL or not.

For reference, a git url starts with "git://"

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

Update:

You can get a list of currently included mods from this URL: http://media.foo-projects.org/remotes. You can use this to check if your favorite mods are already included or not. Or send corrections and updates.

User avatar
jas
Member
Posts: 510
Joined: Mon Jul 24, 2017 18:15
GitHub: jastevenson303
IRC: jas_
In-game: jas
Contact:

Re: Public Remote Media Server Project

by jas » Post

Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)

Thank you for your consideration
2009-10-7 02:04:55: ACTION[Server]: jastest activates

sofar
Developer
Posts: 2128
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Post

jas wrote:Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)
Added both. They'll be included in a few hours as usual.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest