[Tool] Minetest Media Server Java Edition Early Alpha

[LMD]

Minetest Media Server Java Edition
Also called MT Media Server JE in short.
An implementation of a Minetest Media Server written in Java.
Serves both index.mth and the actual media files.

Requirements
Requires Java `1.8` or higher. Probably also works with `1.6`.
Tested using Ubuntu 16.04 with Minetest `5.0.1`, and Java `11` on IntelliJ `2019.2`.
Below instructions are all for Linux/Ubuntu.

Building
Run `cd src` and `javac Main.java`. I however recommend using IntelliJ for building. Builds are already available in `out`. Directory is IntelliJ project.

Running
Run `cd out/production` and `java Main <port> <path_to_media>` or `java -jar MinetestMediaServerJavaEdition.jar`

Arguments :
* `<port>` - Integer port number, for example `8000`, note that you will need root permission for port numbers below `1024`
* `<path_to_media>` - Path to retrieve media from, for example `/home/user/.minetest/games/mineclone2`, can be given multiple times to retrieve media from multiple directories

Example use :
0. Change directory to the one you extracted it to, for example `cd /home/user/Downloads/MinetestMediaServerJavaEdition`
1. `cd out/artifacts/MinetestMediaServerJavaEdition_jar`
2. `java -jar MinetestMediaServerJavaEdition.jar 8000 /home/user/.minetest/games/mineclone2 /home/user/.minetest/games/nodecore`

Will retrieve media from MineClone2 and NodeCore. Server will run on port 8000.
Accordingly you would have to set your `remote_media` setting in `minetest.conf` to `http://<your_ip_or_url>:<port>/`.
Using this example, if you have an url `mt.media.server`, it would look like this : `http://mt.media.server:8000/`

Output (for this example) if successfull (note that the number of media files can of cours be different) :

Code: Select all

Retrieving media from /home/lars/.minetest/games/mineclone2
Retrieving media from /home/lars/.minetest/games/nodecore
Finished retrieving media - 1840 media files found
Finished starting server - now running on port 8000

Example configuration is already default run configuration if you open this in IntelliJ.

Licensing
Licensed under AGPLv3. Roughly summarized (not legally binding) : This software comes without any warranty.
If you modify it and redistribute or run it (on your server), you have to keep the license.

Please visit https://github.com/appgurueu/minetest-m ... va-edition for the GitHub project.

[sofar]

I find the choice of Java for something this simple just unwise. You don't make a hot air balloon with a steam engine. Note that the implementation I wrote more than a year ago has been proven to be stable and highly efficient, and it is the choice for at least a dozen Minetest servers already.

You don't need to run a media server if you host a Minetest server. Visit this thread on how to use the community media server:

viewtopic.php?f=14&t=18951&start=25

Instead of writing his own implementation, LMD should have studied `mtmediasrv` and started a discussion on how to improve it. So say the knights who say NIH.

[LMD]

I find the choice of Java for something this simple just unwise.

It's my choice. I already gave the reasons on the Unofficial Discord : Java has plenty of builtin libraries. Java is pre-installed on most computers. Java is fast, Java is secure. Java is stable and platform-independent. Java is a common programming language. You may find tons of reasons against using Java, but I can do the same for Go. I like Java, and this is why I will continue using it.

You don't make a hot air balloon with a steam engine.

I do. You used Go but you require the user to install a FCGI implementation. I wrote all server stuff by hand only using ServerSocket and my code isn't that much longer.

Note that the implementation I wrote more than a year ago has been proven to be stable and highly efficient, and it is the choice for at least a dozen Minetest servers already.

I respect your implementation. I still see some room for improvement. Probably you didn't notice, but my implementation also serves the media files - it's not exactly the same. This also was a learning project for me - I just wanted to code a server using nothing but "a steam engine". The title says "Early Alpha", and this implies that you can certainly not expect it to be "proven stable". Concerning efficiency, I already saw some problems in your implementation, although I wouldn't claim that mine is faster - I have not benchmarked both implementations yet.

You don't need to run a media server if you host a Minetest server. Visit this thread on how to use the community media server: viewtopic.php?f=14&t=18951&start=25

I already visited the thread. I am not running a server right now. But probably not anybody wants to use your service. Some may want to host their own (for example to make adding files easier), which I consider easier using my standalone Java application.

Instead of writing his own implementation, LMD should have studied `mtmediasrv` and started a discussion on how to improve it.

At first : Why do you talk about me in the third person, while using the second person ("You") in the rest of your reply ? This feels strangely distant, to be honest.
I studied it (there was no other way to get the specification), but as mentioned above, wanted to implement my own, using a different approach - not requiring any FCGI server or using Go as language, but instead platform-independent Java without any libs. Repeating myself, this project had educational purpose as well. It should explicitly be mentioned that I consider `mtmediasrv` a very helpful project.

So say the knights who say NIH.

I get neither abbreviation nor reference as english language learner.
EDIT : Ok, haha, very funny.

[twoelk]

A use case I could see would be a closed environment such as a school, which might need full control of the served media as parents probably expect from them.

[sofar]

Probably you didn't notice, but my implementation also serves the media files

I did notice.

Your implementation also doesn't allow for HTTPS and HTTP2 pipelining, because you're effectively implementing the entire protocol stack yourself.

You can probably make a web server do http2 in Java, but why would you if nginx or apache can just do this for you? Oh sorry, I already know your argument against that.

but you require the user to install a FCGI implementation

No, I don't. You just need to run the binary. For running there is no requirement at all to install some other thing. It just needs a vanilla nginx/apache and the published binary builds for Linux/x86_64.

[sofar]

A use case I could see would be a closed environment such as a school, which might need full control of the served media as parents probably expect from them.

This can be done with both implementations. There is no restriction in either implementation that would prevent anyone from running their own instance of either mtmediasrv or this Java implementation.

[LMD]

> Your implementation also doesn't allow for HTTPS
Yes. A disadvantage, but not too critical - MT Media Servers serve publicly available files and no secrets. MT itself doesn't encrypt it's transmissions AFAIK.

> because you're effectively implementing the entire protocol stack yourself
...and I'm proud of that. This increases both educational value and performance. As bonus I do not require an additional file server.

> It just needs a vanilla nginx/apache
This is a requirement. Apache needs to be installed.

[rubenwardy]

Directly exposing such a thing without nginx/Apache is very bad practice, and will not perform well. There's a reason why basically all production software is placed behind a reverse-proxy and not exposed directly - because reverse-proxies are fast, allow caching, reduce attack surface areas, and reduce the complexity of the programs.

> Your implementation also doesn't allow for HTTPS
Yes. A disadvantage, but not too critical.

HTTPS increases performance by allowing HTTP/2.

This increases both educational value and performance.

Educational value yes, but not performance

Java is fast, Java is secure. Java is stable

[sofar]

because you're effectively implementing the entire protocol stack yourself

...and I'm proud of that. This increases both educational value and performance. As bonus I do not require an additional file server.

This is something I see inexperienced software developers do, and it's a terrible attitude, and potentially very very costly to you and your career in the long run.

First, by placing all your expertise in a single platform, you are going to be in really big trouble if that platform declines or becomes obsolete. And of all platforms, Java may appear alive and well, but this is largely dependent on an older industry still backing it, and more and more those older companies are going to be folding and make place for newer solutions.

Second, by not diversifying your developer skills you are excluding knowledge of what makes Java bad. You are showering in the Java goodness but you are unaware of the things that make Java unacceptable for various tasks. Yes, it's great you can implement a HTTP file server, but you shouldn't. Yes, it's great you can use JIT, but tasks like this shouldn't use JIT languages at all and be precompiled with hardening options enabled in the compiler or using RUST if you don't like Golang. You effectively have skill blindness - you don't know what's bad because all you can see is things that look attractive to you. If you only have a hammer, everything looks like a nail.

Third, Java security and performance are a complete hype. Unless you know exactly what you are doing, no language is safe, and every language can be exploited. Making inefficient software is trivial - it's really difficult to make things efficient. This goes for Java too. From what I can see from your code, you aren't doing things too smartly either, for instance, you are literally loading all the media into RAM and serving them from memory. This may work great if you are hosting a minetest_game server only (a few MB tops), but on the PRMS this would eat a few gigabytes of RAM (https://github.com/appgurueu/minetest-m ... r.java#L95). So much for using resources wisely? Surely your external internet connection is slower than your drive, and you would be better off serving the static content from disk and not caching it.

Another major problem is that you're avoiding learning about making interoperable components. Try building software in parts that are separated out in different processes and have different privilege levels. It is a huge gain to your knowledge skill if you understand well when to create a specific socket protocol, use pipes or message passing like DBus and understand their design and why they are useful (and also why despite the generic hate for them they are still used widely in every OS).

Try, in the future, to build your software using parts that exist and are proven. Build with safeguards in place. Confine your software to the parts that need solving, and don't re-implement parts that work well. Only if you find that those parts are deficient, replace them with other implementations or fix them, and definitely don't re-implement them from scratch or you will repeat all the mistakes that already was learned from.

I'd be really cautious to let this run facing a public internet. Especially since, as pointed out, you have absolutely no shielding in place from shenanigans.

[LMD]

> This is something I see inexperienced software developers do
Do no expect me to be experienced. I am not even 18 yrs old.

> Something like : Don't focus on Java
Yes. But what else to use ? C is too old, and C++ is it's successor. I'm not sure what to use. For scripting languages, there's already a variety (Python, JS, Lua) I know quite good and use, but focusing on performance, I choose Java.

> Something like : You are storing all the files in RAM
Yes. This is very easy to change, however. I decided to store them in RAM as it won't become that much; most games are uncompressed smaller than 50 Mb, but if you wish to, I can change this.

But thanks for taking the time to write a proper advice :D this is what an apprentice dev needs.

[TumeniNodes]

> This is something I see inexperienced software developers do
Do no expect me to be experienced. I am not even 18 yrs old.

So, why not approach those with far more experience and knowledge, and also when they offer you advice, follow it.?
Rather than continuously want to go against their offers of giving you advice, information, etc based on their experience?

Every attempt they make to offer you advice/guidance, you counter them and ignore their experience. It becomes frustrating to watch, I am sure even more so to be involved in.
There are already a few instances of this within these forums.

If you are learning, and desire to learn from those who can teach... then calm down, and learn.
You will learn more, faster, and grow by understanding mistakes before they are even made.
While mistakes are part of learning, sometimes the mistakes are so simple, it is best to learn them from those who have made them, than making them yourself, just for the sake of it.

In your last sentence above, you seem to understand. But a better place often to find such things out is on either
#minetest or
#minetest-hub

Adding my own experiences here, I do not think you will find people more willing to help and with a good amount of patience, than from those above who are taking the time to offer their advice.

[LMD]

> So, why not approach those with far more experience and knowledge, and also when they offer you advice, follow it.?

If you want to make me follow your advice, you have to convince me. I won't blindly follow advice based on "experience" I can't even estimate. Is sofar more experienced than rubenwardy ? I do not know.

> Every attempt they make to offer you advice/guidance, you counter them and ignore their experience. It becomes frustrating to watch, I am sure even more so to be involved in.

1. How can you ever say which experience I ignore ? I do not ignore anything, I take everything into account.
2. Countering something is nothing "bad" - I only try to start a discussion. They have the right to criticize me, and agreed, they probably may have more experience, but I as well have the right to criticize their criticism, and so on... this is what a forum is for : discussion. Discussion requires counterarguments.

> In your last sentence above, you seem to understand.

Glad that you think I understand it.

> Adding my own experiences here, I do not think you will find people more willing to help and with a good amount of patience, than from those above who are taking the time to offer their advice.

And I'm taking the time to fuel the discussion :D

[_E7]

So i actually tried holding back replying here but oh well... I'll try to cut you some slack for being young but let's face it your reasoning is flawed to the bone.

Instead of deconstructing your posts point by point here is some well meaning advice: Stop repeating Java's marketing slogans. They are just that, marketing slogans. The cross platform abilities are really nothing special once you put things into perspective, the performance is only good when you compare it inside it's own niche of languages, the resource usage is downright horrible and labeling something secure just because it by design doesn't allow for a certain class of bugs is silly.

Just be honest: You are familiar with Java and you like it. There is really absolutely nothing wrong with that. Besides building a case how you choose Java based on efficiency and resource use is not going to work anyways. To me Java is actually a KO criteria for (server and to a lesser extend desktop) software as long as there is ANY other option.

I also think it's somewhat funny how you evaluate languages. C is not an option because it's old? Actually C would be a decent choice here considering the size of the task and how efficiency seems to be a major objective but OK, it's old... Of course there is the whole memory management thing which makes it easy to shoot yourself in the foot but that's a different topic. Also C++ is not a successor to C. It's a superset with standardized OOP features. Guess what? Processors have no concept of OOP so OOP means overhead. Again, if efficiency is the objective... And that's just the tip of the iceberg. Really, there is a lot more things to consider than just C being old. To be clear i am not saying that you should use C but more that you should get a bit more background knowledge before making bold claims. I am sure there would be more options on your list then.

Anyways, don't let all that distract you to much. Write your Java implementation. Why not? If people think it's useful they will use it and if they don't, well... I don't really have a horse in this race anyways. If i'd find myself in a situation where i'd need a media server i'd probably use neither project and it won't be all that much because of the languages they are coded in ;)

[LMD]

> Just be honest: You are familiar with Java and you like it.
True. I could also use Kotlin, Scala or Groovy, but they all use the JVM.

In fact, you are "wrecking" all languages here. Any suggestions on what to use ?

But certainly a good evaluation on languages ! I'm trying to seek and alternative to Java with it's own proper VM...

[_E7]

Well, i think if you want performance you should go for something that compiles to native code and forget about VMs. Beyond that the choice is pretty much yours. It's basically about what features and want and what kind of trade offs you are willing to make.

Since you seem to focus on portability without having users compile stuff you probably want static builds but i guess any semi serious language that compiles to native code could do that. Just the amount of work required might differ.

Next big thing would be if you want the compiler to take care of the memory management. Leaving it to the compiler might be convenient but again introduces overhead. Doing it yourself gives you a maximum of control and ability to optimize but you have to be careful not to introduce (sometimes quite serious and hard to debug) bugs.

I'd say everything else is either secondary or simply personal preference.

PERSONALLY i would go for C but then it's a language i am familiar with and which i happen to like. Dejavu, eh? In my opinion it's a very simple (learning curve way less steep than C++) and clear language. Performance wise it's probably as good as it gets short of writing assembler (assuming you don't tell it to do stupid things) and it doesn't get in your way but that's also one of the major drawbacks. You are pretty much on your own with everything and there are practically no safety nets. It's quite good at teaching you to write tidy code and paying attention to detail though because if you don't you will certainly suffer ;)

[sofar]

I like where this discussion is going.

Honestly, I give you credit for ignoring me and doing it anyway. There is really no better way to learn sometimes than to do something even though people told you it was unwise, and then making up your mind afterwards. And maybe, especially in this case, we're not talking about a very complex system that would be unmanageable - heck, you can probably implement it in a shell script anyway.

I started programming in BASIC and immediately grasped for assembly and C, but it took me years to actually get anywhere with it, At the time Python and Java didn't exist, so I was stuck with Pascal / Modula-2 until I got better hardware. Nowadays, the situation is the opposite - too many languages are available, and they have rich ecosystems.

That's when the rule no longer is: "the best language for the task".

Instead, it becomes more "what would get the best adoption in this ecosystem" IMHO.

If you like Golang, great! But don't come near scientific data study community members. You'll want R instead.

Like Big Data and mapreduce, and Python? Ouch, better learn Java fast :)

Program in C++? Don't write Linux kernel code.

Know Perl and want to write GUI's? Better shape up on C++.

This is why I am disliking Java so much right now: the ecosystem is great, but the community around it doesn't have any track record of being great in microservices, it's entirely absent in it, even.

But please, do continue to attempt to prove me wrong in the future :)

[Sires]

Fast? Java
Secure? Java
Stable? Java
Hotel? Trivago


sorry, someone had to do it