[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

by sofar » Wed Apr 19, 2017 07:47

A Minetest Remote Media server implementation (as fcgi server).

This program is useful to distribute minetest media (textures, models, sounds) to minetest clients for multiplayer server owners that wish to have their media hosted on a `remote media server` URL. Doing this as a separate download removes some of the download bandwidth from the actual game server and offloads it to a different HTTP server. This will work for clients that have cURL support enabled.


Requirements

  • nginx or apache, with fcgi
  • webserver must be able to access file sockets in /run/mtmediasrv
  • systemd for controlling the service startup
  • (optional) go to build the service(there are binary builds available for Linux)

This program is intended to run as fcgi process and handle POST requests for the `/index.mth` URI. It listens on a local unix domain socket, and needs to read the media files in the media folder to create sha1 hashes. It creates a hash list of files it has available and keeps this in memory.

At startup, the program can optionally scan mods and subgames to find and copy or hardlink (the default) all the assets into the webroot. The hardlink method is better for space, but may fail if the media is not on the same filesystem as the webroot.

When a client connects, the client POSTS their list of known sha1 hashes of files they need.

The program verifies for each needed hash that the server has the hash listed, and returns the needed hashes only, and only hashes that it has the files for.

The actual file content is not served by this program, for this you need to have your web server serve that content as static files.

Last, it needs to be noted that minetest clients older than, and including, 0.4.13 are NOT supported by this service. These will fail to properly POST the needed hash data to the service and will fail to load any remote media. Please tell your users to upgrade to a newer MT version.


Building

not recommended, there are binary releases available!

Run `make` in the folder to create the binary `mtschemsrv`.

Please note the currently hardcoded values in the binary and example configuration files.


Installation

Several example config and service units are provided as an example on how to deploy this service.

Once you have configured your web server properly so that it serves up the static content files by hash, point the mtmediasrv to the same folder and it will talk to minetest clients that request media automatically. If the content changes, you need to restart the program.

You should not have a file called `index.mth` in the media folder, although this will not break anything, it will probably be confusing.

opy and edit the `mtmediasrv.yaml` file and point it at the proper webroot, socket path, and mediapath entries. Place it in `/etc/` or `~/.config/`.

logging

The program logs all clients and writes out the following data to the log output (journal/syslog):

  • remote address
  • user agent
  • hashes needed/hashes given
  • content-length of sent bytes

Example output:

Code: Select all
mtmediasrv: 5.4.3.2:34567 'Minetest/0.4.15 (Linux/4.10.1 x86_64)' 64/64 1286



Github Project: https://github.com/sofar/mtmediasrv
License: AGPL-3.0
Download: (binaries!) https://github.com/sofar/mtmediasrv/releases/tag/v2
Download: (source!) https://github.com/sofar/mtmediasrv/archive/master.zip
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Wed Apr 19, 2017 07:48

TODO: currently this still requires web servers to provide a file of hashed-named-files in the media folder. I intend to make this easier and make the daemon create those files for the hoster automatically so you can just point the daemon at your mod folders.
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Wed Apr 19, 2017 18:31

DONE: the program is now entirely self-sufficient. You will not need any shell script or external tools to collect and serve remote media. Just edit and put the config file in place and point it at your webroot and asset folders.
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu Apr 27, 2017 05:39

Changed the topic to be a bit more specific - "Minetest Remote Media Server"
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu May 11, 2017 04:59

I will post binary builds so this is even easier to deploy. I've added 32bit and 64bit binaries for Linux on the `releases` page.
 

User avatar
texmex
Member
 
Posts: 696
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by texmex » Thu May 11, 2017 08:41

Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu May 11, 2017 17:19

texmex wrote:Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?


It should always be beneficial, even for smaller servers, because the HTTP downloads are async in the client, which means that while the main server thread is running (working to send already logged in players game data stuff) and sending blocks, the client can download media over HTTP at the same time, and HTTP downloads are very small and efficient.

Of course, a second benefit is that the remote media server can run on a different server, thus offloading bandwidth to a completely different IP address.
 

User avatar
texmex
Member
 
Posts: 696
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex
 

User avatar
rubenwardy
Moderator
 
Posts: 5244
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by rubenwardy » Sun Sep 10, 2017 16:58

I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often

Code: Select all
go get https://github.com/spf13/viper
package https:/github.com/spf13/viper: "https://" not allowed in import path



go get github.com/spf13/viper
package github.com/spf13/viper: cannot download, $GOPATH not set. For more details see: go help gopath


export GOPATH=/usr/lib/go
export PATH=$PATH:$GOPATH/bin


make install
go install: no install location for directory /home/minetest/mtmediasrv outside GOPATH
   For more details see: go help gopath
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Mon Sep 11, 2017 05:23

rubenwardy wrote:I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often


Did you check (1) the binary release zip file that has prebuilt binaries, and (2) has the systemd service unit for this program in it?

I could make it even more comfortable, of course, by completing the `make install` command that is available in the Makefile. Sure.
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Wed Nov 22, 2017 23:31

Lejo wrote:I putted my datas to 217.160.13.85/remote_media
But how can i check if it is working?


If you ask this question, you haven't understood how this program works. Please read the requirements and installation section of the readme for instructions and pointers.

https://github.com/minetest-tools/mtmediasrv#installation
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Fri Nov 24, 2017 01:06

Lejo wrote:I don't realy understand how to start it.
But if I start it it only says:
Code: Select all
2017/11/23 20:11:27 Scaning mediapath: /var/www/html/remote_media
2017/11/23 20:11:27 link /var/www/html/remote_media/3d_armor_boots_admin.png /var/www/media/858b868cdb6b830b936b4bbc3a8556dfebe5453b: file exists

Does i have to set anything to minetest.conf?


Don't put the files in the webroot. mtmediasrv manages the webroot contents entirely for you, and you shouldn't put any files there at all. As a matter of fact, you should empty it out before you start (entirely, including subfolders).

Make sure to set `mediapath` to the place where your mods and subgames are, for instance:

Code: Select all
mediapath:
 - /usr/share/minetest
 - /usr/local/share/minetest
 - /home/lejo/.minetest/mods
 - /home/lejo/.minetest/games


Second, you need to setup apache/nginx to have it connect to it's socket properly.
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Sun Nov 26, 2017 07:21

Lejo wrote:I cleared the webroot and setted the mediapath.
I started it but it still tried to acess my old mediapth.
But I setted a new one.


Maybe show me your yaml config file entirely, so I can take a look.
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Sun Nov 26, 2017 20:46

Lejo wrote:Sorry, I forgot to copy it to /etc , but i still get this Error:
Code: Select all
2017/11/26 11:06:27 mtmediasrv: net.Listen: listen unix /run/mtmediasrv/sock: bind: no such file or directory


I've made a small modification to the example service file that should prevent this issue and create the missing folder for you (if you use the systemd service file as recommended). I've also updated the readme.md to reflect those chantes. You may want to look at the service file in github one more time:

https://github.com/minetest-tools/mtmediasrv#installation

Lejo wrote:Does I have to create the file or does I have to install viper and then run Makefile?

Does I have to set in minetest.conf sth. like this?
Code: Select all
remote_media = 217.160.13.85/


You do not need to install viper assuming you've dowloaded the binary build for Linux/x86_64.

You should set the `remote_media` like this:

remote_media = http://217.160.13.85/

(assuming you're not hosting over https)
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Mon Nov 27, 2017 19:02

Lejo wrote:I got this Error:
Code: Select all
Nov 27 17:08:40 localhost systemd[1]: [/etc/systemd/system/mtmediasrv.service:8] Executable path is not absolute, ignoring: +/usr/bin/mkdir -p /run/mtmediasrv
Nov 27 17:08:40 localhost systemd[1]: [/etc/systemd/system/mtmediasrv.service:9] Executable path is not absolute, ignoring: +/usr/bin/chown www /run/mtmediasrv


You must be running an ancient version of systemd. What output do you get from this command?

Code: Select all
systemctl --version
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Tue Nov 28, 2017 06:19

Lejo wrote:I got this:
Code: Select all
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN


So, in that case, you'll need to manually, after booting, create /var/mtmediasrv and chown it to www.
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 1600
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Tue Nov 28, 2017 18:12

Lejo, can you please start an issue on github, this forum thread shouldn't be about debugging your installation problems.
 

User avatar
Lejo
Member
 
Posts: 285
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

Next

Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 1 guest