Public Remote Media Server Project

User avatar
lag01
Member
 
Posts: 293
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag

Re: Public Remote Media Server Project

by lag01 » Wed Sep 19, 2018 04:45

As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...
 

sofar
Developer
 
Posts: 2066
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Wed Sep 19, 2018 16:31

lag01 wrote:As my server works from WiFi now, i added this service to configuration. I wonder if media can be fetched from multiple server simultaneously by client...


It could, or, you can deploy the media server on a CDN, and automatically do this on the server side. Not that it matters much at this point, since the throughput needed to host the PRMS is trivial, and the client download rate is the limiting factor.
 

sofar
Developer
 
Posts: 2066
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Sun Dec 30, 2018 07:39

Quick stats on the server:

last full week: 11gb total, 3k unique vsits, 1.5 million files requested.

Please keep suggesting mods, especially if they have moved.
 

User avatar
texmex
Member
 
Posts: 1579
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex
 

sofar
Developer
 
Posts: 2066
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Sun Dec 30, 2018 19:41

texmex wrote:I suggest you add the bgm mod since it bandwidth heavy!

https://gitlab.com/timnxx20/bgm


The included tracks are taken from the album "Resolute" by "Blue Dot Sessions" and are released under a CC BY-NC 4.0 license.


Sorry, I won't add this, especially since it's both large and NC licensed. Servers that force players to download these media files should have to bear the burden themselves to serve these to their players.
 

sofar
Developer
 
Posts: 2066
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Sun Dec 30, 2018 19:47

I've updated the topic to include a little blurb that states that mods should be freely distributable, as well as a quick statement that we can refuse mods for various reasons. Most of the time it's as simple as "this mod already contains all these assets" or "this is an obsolete mod" but sometimes it can be content or license related. For instance, I'll likely not add things like nudity, etc.
 

BuckarooBanzay
Member
 
Posts: 293
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai
 

sofar
Developer
 
Posts: 2066
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Public Remote Media Server Project

by sofar » Sun Dec 30, 2018 22:22

BuckarooBanzay wrote:Can you add the following mods/packs:


Contains NC material. Maybe replace it with a font that's not NC?

BuckarooBanzay wrote:
  • Botic font for display_modpack: https://github.com/pyrollo/font_botic
  • )


    The font is SIL, so I've added that (it's also tiny, really).
     

    User avatar
    rubenwardy
    Moderator
     
    Posts: 5835
    Joined: Tue Jun 12, 2012 18:11
    Location: United Kingdom
    GitHub: rubenwardy
    IRC: rubenwardy
    In-game: rubenwardy

    Re: Public Remote Media Server Project

    by rubenwardy » Mon Dec 31, 2018 17:02

    Hi, please may you add capturetheflag and ctf_pvp_engine?

    https://github.com/MT-CTF/capturetheflag
    https://github.com/MT-CTF/ctf_pvp_engine

    Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

    2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
    2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Tue Jan 01, 2019 04:09

    rubenwardy wrote:Hi, please may you add capturetheflag and ctf_pvp_engine?

    https://github.com/MT-CTF/capturetheflag
    https://github.com/MT-CTF/ctf_pvp_engine


    Done, added.

    rubenwardy wrote:Also, any idea why normal MTG resources aren't being picked up? I get a lot of messages like:

    2018-12-31 17:00:06: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/d ... esword.png not found (HTTP response code said error) (response code 404)
    2018-12-31 17:00:27: ERROR[CurlFetch]: http://minetestmedia.foo-projects.org/wool_pink.png not found (HTTP response code said error) (response code 404)


    That's a bug in the client? What client version are you seeing this in?

    The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".

    Somehow there are clients out there that (1) completely barf the remote media protocol and then (2) ask for the plain file instead of the sha1 from the media server.

    I've seen this on the server side, but I don't get this on my own clients and I've never been able to reproduce it, because it is specific to some clients somehow. I've wanted to figure out why, but all I can see from the protocol implementation is that my PRMS side is correct and it had those files properly to begin with.

    FYI, the PRMS has these resources and will share them by sha1sum properly:

    Code: Select all
    $ find . -name wool_pink.png | xargs sha1sum
    c431049287c68a56bf08f933ccb3b59e0bf3c9ee  ./MineClone2/mods/ITEMS/mcl_wool/textures/wool_pink.png
    96d5fa1509960522cf20bbef370e31775d5deed4  ./minetest_game/mods/wool/textures/wool_pink.png
    c71c9c8a2178c7945e42656e312b69324ac33732  ./capturetheflag/mods/shooter/textures/wool_pink.png
    96d5fa1509960522cf20bbef370e31775d5deed4  ./capturetheflag/mods/wool/textures/wool_pink.png
    $ find . -name default_tool_bronzesword.png | xargs sha1sum
    476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./minetest_game/mods/default/textures/default_tool_bronzesword.png
    476d7d4497c2d5f7f42a3aceefa6511f934e5f26  ./capturetheflag/mods/default/textures/default_tool_bronzesword.png

    ...

    $ curl http://minetestmedia.foo-projects.org/c431049287c68a56bf08f933ccb3b59e0bf3c9ee -O
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   203  100   203    0     0  50750      0 --:--:-- --:--:-- --:--:-- 50750
    $ file c431049287c68a56bf08f933ccb3b59e0bf3c9ee
    c431049287c68a56bf08f933ccb3b59e0bf3c9ee: PNG image data, 16 x 16, 4-bit colormap, non-interlaced
    $ sha1sum c431049287c68a56bf08f933ccb3b59e0bf3c9ee
    c431049287c68a56bf08f933ccb3b59e0bf3c9ee  c431049287c68a56bf08f933ccb3b59e0bf3c9ee



    So, from the PRMS side, everything is OK. Hence my suspicion that there are bugged/old clients, somehow. I know that pre-0.4.14 is probably broken, maybe even 0.4.15.
     

    User avatar
    Lejo
    Member
     
    Posts: 634
    Joined: Mon Oct 19, 2015 16:32
    GitHub: Lejo1
    In-game: Lejo

    Re: Public Remote Media Server Project

    by Lejo » Wed Jan 02, 2019 08:55

    sofar wrote:That's a bug in the client? What client version are you seeing this in?

    The client should never ask for "wool_ping.png" but instead ask for "<prms_url>/<sha1ofresource>". But wait, it should ONLY ask for those URL's if the PRMS even says that "I have that sha1sum".

    And what about only providing the raw file and not the sha1sum?
    I‘m using this way and it worked for my client.
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Wed Jan 02, 2019 18:42

    Lejo wrote:And what about only providing the raw file and not the sha1sum?
    I‘m using this way and it worked for my client.


    That would break if mod A and mod B both wanted to provide different versions of the same file name. Which happens, a lot.

    The above method is broken by design. The PRMS has many files that are duplicate filenames but a different sha1sum. Only this way will clients be sure to get the actual content they need, and not something randomly picked.
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Wed Jan 02, 2019 20:16

    [EDIT] this experiment is cancelled now [/EDIT]


    I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

    It's possible that newer clients might see this texture somehow, but I can't figure out from the code why. If you see this texture appear in 0.4.17 versions of minetest, please send me a screenshot and the server address so I can try to reproduce the issue.
     

    User avatar
    Linuxdirk
    Member
     
    Posts: 2177
    Joined: Wed Sep 17, 2014 11:21
    Location: Germany
    In-game: Linuxdirk

    Re: Public Remote Media Server Project

    by Linuxdirk » Thu Jan 03, 2019 09:28

    sofar wrote:I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

    So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Thu Jan 03, 2019 18:10

    Linuxdirk wrote:
    sofar wrote:I'm running an experiment on the PRMS - the PRMS will be sending a custom error image to clients that ask for the wrong files and do not honor the remote media protocol. The image contains the text "game client is out of date. Get it without ads for free: minetest.net".

    So you intercept and modify the payload of your "customers"? The idea behind this is actually pretty great and hopefully will help in fighting malicious clients but two wrongs don't make a right.


    These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.

    So it's not as black and white as "a double wrong". These clients allow themselves to be passed malicious content by a MITM, effectively.

    Anyway, it's an experiment, and I'm going to make sure this isn't a bigger nuisance or causes other problems. If you spot issues, please let me know ASAP.
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Thu Jan 03, 2019 20:01

    Well, after compiling 0.4.13 and attempting to see if the placeholder image actually shows up in-game, I've stopped the experiment. The 0.4.13 MT client code does download the placeholder, but then properly checks the checksum and does not use the placeholder image in-game, and so it's more efficient for the PRMS to return a 404 instead of a placeholder image.

    I'm not evil enough to generate sha1 collisions on the fly :).
     

    User avatar
    Linuxdirk
    Member
     
    Posts: 2177
    Joined: Wed Sep 17, 2014 11:21
    Location: Germany
    In-game: Linuxdirk

    Re: Public Remote Media Server Project

    by Linuxdirk » Fri Jan 04, 2019 07:00

    sofar wrote:These clients are not compatible with the remote media protocol (largely, 0.4.11 and 0.4.13) and they should have not been built with curl support. They ask for resources by file name which means that if your remote media server has 25 copies of default_grass.png, the client can't ask for a specific one anyway. This is why the remote media protocol works by content sha1sums, and it should only ask for the sha1sums that the MT server told it to get. Note that throughout all this, the client can just discard the placeholder texture that the PRMS sends if a client does this, because the MT server told it that it needed sha1sum X, and the PRMS sends it the placeholder texture which does not have sha1sum X, but some other value.

    Now it makes absolute sense to me.
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar
     

    Thomas-S
    Member
     
    Posts: 109
    Joined: Sun Sep 06, 2015 12:23
    Location: Bavaria, Germany
    GitHub: Thomas--S
    IRC: Thomas-S
    In-game: Thomas-S
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar
     

    User avatar
    Walker
    Member
     
    Posts: 380
    Joined: Tue Oct 03, 2017 09:22
    In-game: Walker

    Re: Public Remote Media Server Project

    by Walker » Wed Feb 20, 2019 15:56

    I have added this service to my five servers

    add server running under KNOPPIX ( Debian ) with minetest-server 0.4.17.1
    and the minetest-game 0.4.17.1 or the mineclone2 0.45.1 subgame
    minetest can be found here: https://repo.or.cz/MineClone/MineClone2.git

    the list of used 3rd person mods by servers:

    + Spoiler


    P.S. sorry for my bad english, i am from german ^^
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Wed Feb 20, 2019 17:59

    Walker wrote:load_mod_areas = true


    I can't do anything with this data.

    Please list only new mods that were not added before, and give me the git URL for them. If you say "load_mod_areas" then I have no idea what git URL that is, or whether you use a fork of it at a different URL or not.

    For reference, a git url starts with "git://"
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Wed Feb 20, 2019 18:26

    Update:

    You can get a list of currently included mods from this URL: http://media.foo-projects.org/remotes. You can use this to check if your favorite mods are already included or not. Or send corrections and updates.
     

    User avatar
    jas
    Member
     
    Posts: 452
    Joined: Mon Jul 24, 2017 18:15
    GitHub: jastevenson303
    IRC: jas_
    In-game: jas

    Re: Public Remote Media Server Project

    by jas » Wed Feb 27, 2019 23:53

    Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

    Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)

    Thank you for your consideration
    2019-10-14 09:12:51: ACTION[Server]: jas activates
     

    sofar
    Developer
     
    Posts: 2066
    Joined: Fri Jan 16, 2015 07:31
    GitHub: sofar
    IRC: sofar
    In-game: sofar

    Re: Public Remote Media Server Project

    by sofar » Thu Feb 28, 2019 01:35

    jas wrote:Hello. Thank you for the list; if you're interested I'm using this for git://github.com/jastevenson303/Glitchtest.git --

    Particularly FaceDeer's dfcaverns modpack @ git://github.com/facedeer/dfcaverns.git (LICENSE.txt)


    Added both. They'll be included in a few hours as usual.
     

    PreviousNext

    Return to Minetest-related projects



    Who is online

    Users browsing this forum: No registered users and 1 guest