[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

by sofar » Post

A Minetest Remote Media server implementation (as fcgi server).

This program is useful to distribute minetest media (textures, models, sounds) to minetest clients for multiplayer server owners that wish to have their media hosted on a `remote media server` URL. Doing this as a separate download removes some of the download bandwidth from the actual game server and offloads it to a different HTTP or HTTPS server. This will work for all clients that have cURL support enabled.


Requirements
  • nginx or apache, with fcgi
  • webserver must be able to access file sockets in /run/mtmediasrv
  • systemd for controlling the service startup
  • (optional) go to build the service(there are binary builds available for Linux)
This program is intended to run as fcgi process and handle POST requests for the `/index.mth` URI. It listens on a local unix domain socket, and needs to read the media files in the media folder to create sha1 hashes. It creates a hash list of files it has available and keeps this in memory.

At startup, the program can optionally scan mods and subgames to find and copy or hardlink (the default) all the assets into the webroot. The hardlink method is better for space, but may fail if the media is not on the same filesystem as the webroot.

When a client connects, the client POSTS their list of known sha1 hashes of files they need.

The program verifies for each needed hash that the server has the hash listed, and returns the needed hashes only, and only hashes that it has the files for.

The actual file content is not served by this program, for this you need to have your web server serve that content as static files.

Last, it needs to be noted that minetest clients older than, and including, 0.4.13 are NOT supported by this service. These will fail to properly POST the needed hash data to the service and will fail to load any remote media. Please tell your users to upgrade to a newer MT version.


Building

not recommended, there are binary releases available!

Run `make` in the folder to create the binary `mtmediasrv`.

Please note the currently hardcoded values in the binary and example configuration files.


Installation

Several example config and service units are provided as an example on how to deploy this service.

Once you have configured your web server properly so that it serves up the static content files by hash, point the mtmediasrv to the same folder and it will talk to minetest clients that request media automatically. If the content changes, you need to restart the program.

You should not have a file called `index.mth` in the media folder, although this will not break anything, it will probably be confusing.

opy and edit the `mtmediasrv.yaml` file and point it at the proper webroot, socket path, and mediapath entries. Place it in `/etc/` or `~/.config/`.

logging

The program logs all clients and writes out the following data to the log output (journal/syslog):
  • remote address
  • user agent
  • hashes needed/hashes given
  • content-length of sent bytes
Example output:

Code: Select all

mtmediasrv: 5.4.3.2:34567 'Minetest/0.4.15 (Linux/4.10.1 x86_64)' 64/64 1286

Github Project: https://github.com/sofar/mtmediasrv
License: AGPL-3.0
Download: (binaries!) https://github.com/minetest-tools/mtmed ... x86_64.zip
Download: (source!) https://github.com/sofar/mtmediasrv/archive/master.zip

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Post

TODO: currently this still requires web servers to provide a file of hashed-named-files in the media folder. I intend to make this easier and make the daemon create those files for the hoster automatically so you can just point the daemon at your mod folders.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Post

DONE: the program is now entirely self-sufficient. You will not need any shell script or external tools to collect and serve remote media. Just edit and put the config file in place and point it at your webroot and asset folders.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Post

Changed the topic to be a bit more specific - "Minetest Remote Media Server"

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Post

I will post binary builds so this is even easier to deploy. I've added 32bit and 64bit binaries for Linux on the `releases` page.

User avatar
texmex
Member
Posts: 1753
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: tacotexmex

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by texmex » Post

Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Post

texmex wrote:Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?
It should always be beneficial, even for smaller servers, because the HTTP downloads are async in the client, which means that while the main server thread is running (working to send already logged in players game data stuff) and sending blocks, the client can download media over HTTP at the same time, and HTTP downloads are very small and efficient.

Of course, a second benefit is that the remote media server can run on a different server, thus offloading bandwidth to a completely different IP address.

User avatar
texmex
Member
Posts: 1753
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: tacotexmex

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by texmex » Post

Cool, thanks!

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by rubenwardy » Post

I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often

Code: Select all

go get https://github.com/spf13/viper
package https:/github.com/spf13/viper: "https://" not allowed in import path



go get github.com/spf13/viper
package github.com/spf13/viper: cannot download, $GOPATH not set. For more details see: go help gopath


export GOPATH=/usr/lib/go
export PATH=$PATH:$GOPATH/bin


make install
go install: no install location for directory /home/minetest/mtmediasrv outside GOPATH
	For more details see: go help gopath
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

rubenwardy wrote:I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often
Did you check (1) the binary release zip file that has prebuilt binaries, and (2) has the systemd service unit for this program in it?

I could make it even more comfortable, of course, by completing the `make install` command that is available in the Makefile. Sure.

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:38, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:I putted my datas to 217.160.13.85/remote_media
But how can i check if it is working?
If you ask this question, you haven't understood how this program works. Please read the requirements and installation section of the readme for instructions and pointers.

https://github.com/minetest-tools/mtmed ... stallation

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:38, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:I don't realy understand how to start it.
But if I start it it only says:

Code: Select all

2017/11/23 20:11:27 Scaning mediapath: /var/www/html/remote_media
2017/11/23 20:11:27 link /var/www/html/remote_media/3d_armor_boots_admin.png /var/www/media/858b868cdb6b830b936b4bbc3a8556dfebe5453b: file exists
Does i have to set anything to minetest.conf?
Don't put the files in the webroot. mtmediasrv manages the webroot contents entirely for you, and you shouldn't put any files there at all. As a matter of fact, you should empty it out before you start (entirely, including subfolders).

Make sure to set `mediapath` to the place where your mods and subgames are, for instance:

Code: Select all

mediapath:
 - /usr/share/minetest
 - /usr/local/share/minetest
 - /home/lejo/.minetest/mods
 - /home/lejo/.minetest/games
Second, you need to setup apache/nginx to have it connect to it's socket properly.

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:39, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:I cleared the webroot and setted the mediapath.
I started it but it still tried to acess my old mediapth.
But I setted a new one.
Maybe show me your yaml config file entirely, so I can take a look.

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:39, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:Sorry, I forgot to copy it to /etc , but i still get this Error:

Code: Select all

2017/11/26 11:06:27 mtmediasrv: net.Listen: listen unix /run/mtmediasrv/sock: bind: no such file or directory
I've made a small modification to the example service file that should prevent this issue and create the missing folder for you (if you use the systemd service file as recommended). I've also updated the readme.md to reflect those chantes. You may want to look at the service file in github one more time:

https://github.com/minetest-tools/mtmed ... stallation
Lejo wrote:Does I have to create the file or does I have to install viper and then run Makefile?

Does I have to set in minetest.conf sth. like this?

Code: Select all

remote_media = 217.160.13.85/
You do not need to install viper assuming you've dowloaded the binary build for Linux/x86_64.

You should set the `remote_media` like this:

remote_media = http://217.160.13.85/

(assuming you're not hosting over https)

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:38, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:I got this Error:

Code: Select all

Nov 27 17:08:40 localhost systemd[1]: [/etc/systemd/system/mtmediasrv.service:8] Executable path is not absolute, ignoring: +/usr/bin/mkdir -p /run/mtmediasrv
Nov 27 17:08:40 localhost systemd[1]: [/etc/systemd/system/mtmediasrv.service:9] Executable path is not absolute, ignoring: +/usr/bin/chown www /run/mtmediasrv
You must be running an ancient version of systemd. What output do you get from this command?

Code: Select all

systemctl --version

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:38, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo wrote:I got this:

Code: Select all

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN
So, in that case, you'll need to manually, after booting, create /var/mtmediasrv and chown it to www.

User avatar
Lejo
Member
Posts: 718
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by Lejo » Post

Last edited by Lejo on Tue Nov 28, 2017 18:35, edited 1 time in total.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Post

Lejo, can you please start an issue on github, this forum thread shouldn't be about debugging your installation problems.


Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests