Simple Static Remote Media Server

[Lejo]

A static remote media server done simple.

Why static?

For all other remote media services you need a paid root server.
But you can everywhere get a simple static webserver where you just need to upload the medias.
You don't need to install complex programs you just generate the index file.
Sadly github pages don't work as they return 405 Not Allowed if you send post data.
There are enough other ones.

Installation

Download at least the make_index_mth.py
This download also contains > 64000 medias from minetest servers.

Just copy your media cache file after you joined the servers you want to have the medias from.
Next you execute the make_index_mth.py file python make_index_mth.py
Last you just copy the new media file which now includes the index.mth to your webserver
Then you only need to set in your serverconfig the link to the media_server http://yourdomain.com/media/

Download

Download
Source
License of the code only: MIT

I prefer every server to do it's own media server to keep index.mth size low and 100% of the media available there.

[TalkLounge]

An it's even faster

Average
sofar(IPv6): 173 ms
sofar(IPv4): 235 ms
Lejo(IPv4): 135,5 ms

[LMD]

Going to advertise my own Java implementation (https://github.com/appgurueu/minetest-m ... va-edition).
I don't think this is a working media server, judging by the code (https://github.com/Lejo1/minetest-media ... dex_mth.py)

The actual media server spec is a bit like the following :
1. Minetest client requests media corresponding to MD5 hashes (checksums), got nothing to do with filenames, using POST
2. Server checks which hashes it has and servers them

I have no idea what you are doing there, especially with "hex_decode". Have you tested to make sure that it's actually working ?

> And it's even faster
Nothing but some claim, only justified by a few pings... I think that it's hardly possible to be faster than my Java or sofar's Go implementation...

[Lejo]

I don't think this is a working media server, judging by the code (https://github.com/Lejo1/minetest-media ... dex_mth.py)

Sure it works don't judge things you haven't tested!
It generates a static index.mth file containing the signature, version and the hashes.

The actual media server spec is a bit like the following :
1. Minetest client requests media corresponding to MD5 hashes (checksums), got nothing to do with filenames, using POST
2. Server checks which hashes it has and servers them

It's sha1 but yes it works this way. It just doens't matter if the server sends some more sha1 hashes...
Just read this.

I have no idea what you are doing there, especially with "hex_decode". Have you tested to make sure that it's actually working ?

The medias in your cache are hex encoded sha1 because the sha1 are just binary bytes you can't use for a filename.

> And it's even faster
Nothing but some claim, only justified by a few pings... I think that it's hardly possible to be faster than my Java or sofar's Go implementation...

I agree that Ping doesn't says a lot.
Your system needs to calculate what to send, my doesn't so it's a bit faster.
Well, we all know that we are talking about ms so this doesn't matter.

[LMD]

I'm not an expert. I've tested my server and made sure it actually delivers the media.
Whether your server is capable of delivering media to a new (MT 5) Minetest client, I don't know.

I guess we need an expert, so I will leave sofar a reply in his project.

> Your system needs to calculate what to send
Of course it does ! There is no other way, AFAIK. Probably old protocols. That way the client can know which files your server can serve. So there won't be that much failed requests. It likely saves you LOTS of time.
Also, the calculations I do are extremely efficient. Binary searches and such. Number of hashes * binary lookup. That's not alot. My implementation does this in, huh, no idea, but extremely fast. And this only needs to be done a few times (when index.mth is requested).

As said, not a Minetest dev. Probably sofar knows more about advantages or disadvantages of both approaches.

[LMD]

Ah, now I know why I have been mislead. I thought you were just giving file names. You aren't. Your file names aren't the hashes.

But this means that the given source code is not complete. There is no easy way of adding media; it needs additional shell scripts and stuff.

My impl is also easy to use (as easy as running a JAR with arguments) and automatically retrieves media which does not need to be in any specific format (normal Minetest subgame/mod folders will all work).

[sofar]

Sure it works don't judge things you haven't tested!
It generates a static index.mth file containing the signature, version and the hashes.

He can read code. He can see that you're using `filename.decode("hex")` which does not generate sha1sums. So clearly, it is an incorrect implementation.

Your implementation is not functional. The only reason MT clients don't complain is because the client implementation has proper fallbacks. You don't understand the protocol and are not properly testing it yourself. Hint: watch the web logs, and hexdump your index.mth.

This is far worse than sfan5's shell script, which actually does this correctly. Your code is non-functional.

[Linuxdirk]

I love how one implementation is not functional, and one implementation is in a dying language.

You people should support existing projects instead of fiddling around with this crap :D

[Lejo]

So first of all I tested it very often especially now again as you made me unsure if it really works. I can still be wrong but I just don't see it right now. Please correct me.
First thing I now check again. Is the media name is the sha1 hash in hex of the file, right? Check this
What I do is to just decode the filename.

> Your system needs to calculate what to send
Of course it does ! There is no other way, AFAIK. Probably old protocols. That way the client can know which files your server can serve. So there won't be that much failed requests. It likely saves you LOTS of time.
Also, the calculations I do are extremely efficient. Binary searches and such. Number of hashes * binary lookup. That's not alot. My implementation does this in, huh, no idea, but extremely fast. And this only needs to be done a few times (when index.mth is requested).

Yes it's your way you do it. But the index.mth file which tells the client which files exist can also be static. It just contains all files the server has. About bandwith... later.
I know calculating is no reason to do this, it's much more what I saided before:

For all other remote media services you need a paid root server.
But you can everywhere get a simple static webserver where you just need to upload the medias.
You don't need to install complex programs you just generate the index file.

Ah, now I know why I have been mislead. I thought you were just giving file names. You aren't. Your file names aren't the hashes.

Sure my files names are sha1 hashes as hex!
The index.mth tells the client which files exist.

But this means that the given source code is not complete. There is no easy way of adding media; it needs additional shell scripts and stuff.

Sure it is! Just add your media and execute python make_index_mth.py it will generate the index.mth file.

My impl is also easy to use (as easy as running a JAR with arguments) and automatically retrieves media which does not need to be in any specific format (normal Minetest subgame/mod folders will all work).

For my you not even need to install a program just copy your media cache(containing your server medias) and execute make_index_mth.py

He can read code. He can see that you're using `filename.decode("hex")` which does not generate sha1sums. So clearly, it is an incorrect implementation.

Sure he can. The file names are just hex encoded sha1 hashes. See again

Your implementation is not functional. The only reason MT clients don't complain is because the client implementation has proper fallbacks. You don't understand the protocol and are not properly testing it yourself. Hint: watch the web logs, and hexdump your index.mth.

I checked my whole verbose stream! I extra added logging to my client and it never calls any error.
I also saw on my webserver in the access.log that the hashes got requested.
Don't want to go in detail here.

He is sending each MT client that connects an 875KB file. Every, single, client, Every time they miss 1 media file. My implementation would only send ~30 bytes or so if only 1 file is needed from the server.

That's true and it's also the reason why I thought about alternatives which can also get hosted for free.
Just like php but php is ahhhhhhh please not.
There is no free host where you can run your own programm for free.(I haven't found one. Tell me if you know one please :D)
I would be better if you have it per server so the index.mth only sends needed files.
But come on it's still a question of less then a seconds to download it.

I ... don't understand why people are making inadequate solutions if a feature rich, open source and efficient implementation exist. A static index.mth is a ridiculous way to waste internet bandwidth and make MT clients do more work for no reason. Especially for a public service.

You people should support existing projects instead of fiddling around with this crap :D

As you maybe remember I multiple times asked you to add my whole project as it contains outdated files or skins. Which have no git link. Thats the reason why I added one remote media for alllll media the client download: just copy the cache.
In addition to this I multiple times used all my allowed bandwith of my server so I wanted a server for the whole media.

This is far worse than sfan5's shell script, which actually does this correctly. Your code is non-functional.

I didn't find it? I would have used it for my server if I knew about it.

[TalkLounge]

I made a speed test

+ Spoiler

Ping data
sofar(IPv6): 173 ms, located in USA
sofar(IPv4): 235 ms, located in USA
Lejo(IPv4): 135 ms, located in USA
Wildes Land 2: 47 ms, located in Germany(Where I am living)(My minetest server)

Test info
Server tested with: Wildes Land 2
MT Version tested with: 0.4.17

WL2 has a whole media size of 31.1 MB

Lejo's media server
Lejo fully support my server. All media were downloaded from his gitlab.

sofar's media server
sofar just support a few mods from my server. At least 10.7 MB were downloaded from my minetest server and not from the media server, because these files are music files added to my own not published mod.

comparison
The comparison shows, that both are equal fast, but Lejo's media server send all data in the same time with a lower ping. sofar's media server only send 2/3 data with a higher ping, the remaining media was sent by my minetest server, with a very little ping.

Result
I think, if sofar add all media of my server, his server would be slower than Lejo's.

Conclusion
I will use Lejo's media server now.

[sofar]

I made a speed test

Conclusion
I will use Lejo's media server now.[/spoiler]

This conclusion is garbage because you did not account for the amount of raw bytes transferred(rx/tx) to each client.

Lejo1's implementation forces each client to download 875Kb of data from the remote media server, while much, much less is needed.

A proper comparison would also account for speed of connection. Ping values are almost meaningless and skew the viewer into thinking something is "slow" because it's comparing apples to transistors.

[LMD]

> I think, if sofar add all media of my server, his server would be slower than Lejo's.
This is (obviously) not true. I would rather say the exact opposite is true; the more media a media server (that's its purpose) can serve, the faster the process.

[TalkLounge]

This is (obviously) not true. I would rather say the exact opposite is true; the more media a media server (that's its purpose) can serve, the faster the process.

No, because my minetest server has a ping from 47 ms and sofar's media server a ping from 173 ms. So the media needs longer to travel to me. But that's 100% a theory.

This conclusion is garbage because you did not account for the amount of raw bytes transferred(rx/tx) to each client.

How to do this? You can make your own comparison, if you want to. You can use my minetest server if you want to. Just tell me, when I should switch from Lejo's media server to yours.

Lejo1's implementation forces each client to download 875Kb of data from the remote media server, while much, much less is needed.

Lejo's media server fully support my minetest server. Yeah, the client don't need the file names from the other servers, which are implemented in the nearly 1 MB file. But it's only 1 MB per connection. In comparison to my minetest media from 31.1 MB... That's nothing.

A proper comparison would also account for speed of connection.

That's how quick I was connected to my minetest server. My internet bandwidth is almost the same.

[Lejo]

The goal of the server is NOT to be faster!
The goal is to be simple and free to setup, like my way.
Maybe it’s a few ms slower but that’s not much.

[sofar]

The goal of the server is NOT to be faster!
The goal is to be simple and free to setup, like my way.
Maybe it’s a few ms slower but that’s not much.

You may as well have not even made the python script, and remove the index.mth file (or put the 6-byte header in it). It would work just the same, since it contains garbage data.

[Festus1965]

The goal of the server is NOT to be faster!
The goal is to be simple and free to setup, like my way.
Maybe it’s a few ms slower but that’s not much.

I still read the posts here in forum 1 or 2 times a day ... and get headage - as of still nobody seam to understand to main goal for ALL gamer.

For a server, where only or most server local (under 20 ms) gamer are, a own simple media server is worth, as time and sure bandwidth are available.

But as like the only one server-master cheating ping, only one media-server is also cheating as it depends on the location of gamer if it is better or not.

Like I wrote at gitlab (I deleted my acc) similar to the discussion of client could do own ping, if was denied as of too often, like a ping flood, nobody thought about just to let this ping once a week and store the result in the favorites file, and update it when time of needed as new ip ?

So I see very strong connection to your project here also, lag for all is only be fight down if all come together:
- server-list in 5 continents or client does ping itself so that gamer take more local server or understand why others may lag,
- servers will not move, but can support better (short, faster ?) with own media-server or link to nearest of them but
- or better, also media servers will be spread over the continents or more, and let client decide onto ping to them (however) where to get it fastest (ping and/or throughput).

I still cant believe that your really think Internet is everywhere the same and so diff in time and load - as it is maybe for electric power (as NOT, see 220 V Europe and 110 V in USA). You sound like kids, telling milk comes from super-marked, not knowing anymore about the cow is the source even where they are.

so how to test for real advantage of this?
* local home server, local home user - and diff between external and/against internal media server
* external server, but local home media-server, where also use client - against outside media-server
?
as always short also mean fast, as local internet connections SURE are bigger and easier to use than world wirde connections as low ping and limited bandwidth.

[sofar]

But as like the only one server-master cheating ping, only one media-server is also cheating as it depends on the location of gamer if it is better or not.

Ping is almost entirely irrelevant for the remote media protocol. Most of the time is spent downloading the media, and the performance of that does not correlate with ping values, at least not for media servers which are hosted in reasonably modern hosting sites.

The limiting factor is bandwidth, not ping. Most of the time this is limited by the clients' available bandwidth.

[Festus1965]

Ping is almost entirely irrelevant for the remote media protocol.

I know that - but you have to write down, what that mean to media-servers, position (spreading) and connection so make them a win !!! (for most gamer)

[Lejo]

Ah I finally found sfan5's script
It does the same just that it gets the sha1-binary using openssl dgst -binary -sha1
And I do it using decoding the filename which is the hex encoded sha1 hash.

typically rendered as a hexadecimal number

You may as well have not even made the python script, and remove the index.mth file (or put the 6-byte header in it). It would work just the same, since it contains garbage data.

Could you please maybe tell me whats wrong instead of telling it's garbage...
If I use an empty header(MTHS\x00\x01) then the client gives an info otherwise not:

Code: Select all

INFO[Main]: Client: Failed to remote-fetch 4570 files. Requesting them the usual way.

[sofar]

Could you please maybe tell me whats wrong instead of telling it's garbage...

Ah I finally found sfan5's script
It does the same just that it gets the sha1-binary using openssl dgst -binary -sha1

[Lejo]

@sofar So last time I try to explain you how it works. I'm not sure if you don't understand or if you don't want to understand...
The filename of an media file is the sha1 hash as hex. I just decode the filename to get the sha1 as it is a lot faster then generating the hash.
Proof:

Code: Select all

user@pc:~/minetest-media$ openssl dgst -sha1 media/0000d50cfbb0f91790b80abccdd0a670e6146d8b 
SHA1(media/0000d50cfbb0f91790b80abccdd0a670e6146d8b)= 0000d50cfbb0f91790b80abccdd0a670e6146d8b

I will stop this discussion here as I don't understand what's wrong in your opinion(except the bandwith thing)...

[LMD]

> it is a lot faster then generating the hash.
Maybe, but as already said before, you don't provide a proper way to add stuff.
Also, current implementations only generate hashes on startup (which is a pretty fast thing btw).

> what's wrong in your opinion
Your media server is static. That's the problem. It's some "outdated" (at least IMO) method. You send all the data, which is often not required. Calculating what to send is also extremely fast.
Furthermore, (not sure whether sofar would agree with me on this) you are using GitLab / GitHub Pages. This likely makes the media sending process slower I think. Such services are sending unnecessary headers. Imagine sending thousands of 16x16 pixel art images. The images are likely only 256 bytes ore less if indexed colors are used. Your headers however can be larger than the actual media you are sending.

My implementation takes advantage of implementing this based only on server sockets (pretty much "from scratch"). This allows me to greatly reduce the overhead. I only do the basics of request analysis. Responses are sent quite immediately with little overhead.

Furthermore, it also provides a neat way of providing arguments (such as port) and scans the media on initialization, which makes adding easier.

TL;DR : I consider my application better - easy to use, made from scratch, little overhead, and blazing fast. But probably I only do so because it's mine ;)

[Lejo]

> it is a lot faster then generating the hash.
Maybe, but as already said before, you don't provide a proper way to add stuff.
Also, current implementations only generate hashes on startup (which is a pretty fast thing btw).

Next and last time:
You can just run the make_index_mth.py script and it will generate the index.mth for the files also the new added!

> what's wrong in your opinion
Your media server is static. That's the problem. It's some "outdated" (at least IMO) method. You send all the data, which is often not required. Calculating what to send is also extremely fast.
Furthermore, (not sure whether sofar would agree with me on this) you are using GitLab / GitHub Pages. This likely makes the media sending process slower I think. Such services are sending unnecessary headers. Imagine sending thousands of 16x16 pixel art images. The images are likely only 256 bytes ore less if indexed colors are used. Your headers however can be larger than the actual media you are sending.

Sure calculation is fast as I already noticed multiple times it's easier and free to set up a server as you don't need any root server. About headers: Maybe, but thats just an assumption. That's different between host you can choose any.

Furthermore, it also provides a neat way of providing arguments (such as port) and scans the media on initialization, which makes adding easier.

Media scanning is not even needed in my way you can just copy your cache which contains all medias of the server you joined.

[sofar]

@sofar So last time I try to explain you how it works. I'm not sure if you don't understand or if you don't want to understand...

I literally gave you the answer, using your own words. The answer to your problem is right there, yet you fail to see it.

I've implemented a correct implementation of the remote media protocol. You have not. Maybe you should read my implementation, or even LMD's implementation, which is also correct, but I understand it 100%. And your implementation isn't.

Hint: this line is (very) wrong: https://github.com/Lejo1/minetest-media ... _mth.py#L8

[Lejo]

I literally gave you the answer, using your own words. The answer to your problem is right there, yet you fail to see it.

I've implemented a correct implementation of the remote media protocol. You have not. Maybe you should read my implementation, or even LMD's implementation, which is also correct, but I understand it 100%. And your implementation isn't.

Hint: this line is (very) wrong: https://github.com/Lejo1/minetest-media ... _mth.py#L8

Do you read what I write you?
The Filename of the medias in the cache are the sha1 hashes in hex-format
So I just decode the filename and I will get the sha1 binary. I don't need to generate it new as sfan5 does it.

Btw. I tested it: Sfan5's script and my script generated the exactly same index.mth file(testing with one file.)
I won't bring any proofs here just because they are already posted and I don't wont to waste more work here.