[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

[TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Server

by sofar » Wed Apr 19, 2017 07:47

A Minetest Remote Media server implementation (as fcgi server).

This program is useful to distribute minetest media (textures, models, sounds) to minetest clients for multiplayer server owners that wish to have their media hosted on a `remote media server` URL. Doing this as a separate download removes some of the download bandwidth from the actual game server and offloads it to a different HTTP server. This will work for clients that have cURL support enabled.


Requirements

  • nginx or apache, with fcgi
  • webserver must be able to access file sockets in /run/mtmediasrv
  • systemd for controlling the service startup
  • (optional) go to build the service(there are binary builds available for Linux)

This program is intended to run as fcgi process and handle POST requests for the `/index.mth` URI. It listens on a local unix domain socket, and needs to read the media files in the media folder to create sha1 hashes. It creates a hash list of files it has available and keeps this in memory.

At startup, the program can optionally scan mods and subgames to find and copy or hardlink (the default) all the assets into the webroot. The hardlink method is better for space, but may fail if the media is not on the same filesystem as the webroot.

When a client connects, the client POSTS their list of known sha1 hashes of files they need.

The program verifies for each needed hash that the server has the hash listed, and returns the needed hashes only, and only hashes that it has the files for.

The actual file content is not served by this program, for this you need to have your web server serve that content as static files.


Building

not recommended, there are binary releases available!

Run `make` in the folder to create the binary `mtschemsrv`.

Please note the currently hardcoded values in the binary and example configuration files.


Installation

Several example config and service units are provided as an example on how to deploy this service.

Once you have configured your web server properly so that it serves up the static content files by hash, point the mtmediasrv to the same folder and it will talk to minetest clients that request media automatically. If the content changes, you need to restart the program.

You should not have a file called `index.mth` in the media folder, although this will not break anything, it will probably be confusing.

opy and edit the `mtmediasrv.yaml` file and point it at the proper webroot, socket path, and mediapath entries. Place it in `/etc/` or `~/.config/`.

logging

The program logs all clients and writes out the following data to the log output (journal/syslog):

  • remote address
  • user agent
  • hashes needed/hashes given
  • content-length of sent bytes

Example output:

Code: Select all
mtmediasrv: 5.4.3.2:34567 'Minetest/0.4.15 (Linux/4.10.1 x86_64)' 64/64 1286



Github Project: https://github.com/sofar/mtmediasrv
License: AGPL-3.0
Download: (binaries!) https://github.com/sofar/mtmediasrv/releases/tag/v1
Download: (source!) https://github.com/sofar/mtmediasrv/archive/master.zip
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Wed Apr 19, 2017 07:48

TODO: currently this still requires web servers to provide a file of hashed-named-files in the media folder. I intend to make this easier and make the daemon create those files for the hoster automatically so you can just point the daemon at your mod folders.
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest fcgi Media Ser

by sofar » Wed Apr 19, 2017 18:31

DONE: the program is now entirely self-sufficient. You will not need any shell script or external tools to collect and serve remote media. Just edit and put the config file in place and point it at your webroot and asset folders.
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu Apr 27, 2017 05:39

Changed the topic to be a bit more specific - "Minetest Remote Media Server"
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu May 11, 2017 04:59

I will post binary builds so this is even easier to deploy. I've added 32bit and 64bit binaries for Linux on the `releases` page.
 

User avatar
texmex
Member
 
Posts: 589
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by texmex » Thu May 11, 2017 08:41

Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?
Taking a break from Minetest.
Mods Commons, Sling, hbSprint, Floodables, item_drop⑂, Telemosaic⑂, Herbs⑂.
Texturepacks Good Morning Craft
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - A Minetest Remot Media Se

by sofar » Thu May 11, 2017 17:19

texmex wrote:Hi sofar! This sounds great for large server admins. Is there a gain to be had from this from small servers to or is it not worth the hassle?


It should always be beneficial, even for smaller servers, because the HTTP downloads are async in the client, which means that while the main server thread is running (working to send already logged in players game data stuff) and sending blocks, the client can download media over HTTP at the same time, and HTTP downloads are very small and efficient.

Of course, a second benefit is that the remote media server can run on a different server, thus offloading bandwidth to a completely different IP address.
 

User avatar
texmex
Member
 
Posts: 589
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: texmex
 

User avatar
rubenwardy
Developer
 
Posts: 4718
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by rubenwardy » Sun Sep 10, 2017 16:58

I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often

Code: Select all
go get https://github.com/spf13/viper
package https:/github.com/spf13/viper: "https://" not allowed in import path



go get github.com/spf13/viper
package github.com/spf13/viper: cannot download, $GOPATH not set. For more details see: go help gopath


export GOPATH=/usr/lib/go
export PATH=$PATH:$GOPATH/bin


make install
go install: no install location for directory /home/minetest/mtmediasrv outside GOPATH
   For more details see: go help gopath
 

sofar
Developer
 
Posts: 1073
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [TOOL]mtmediasrv[mtmediasrv] - Minetest Remote Media Ser

by sofar » Mon Sep 11, 2017 05:23

rubenwardy wrote:I highly suggest providing instructions on how to install go-lang, and how to create a systemd service (I've done it before, but I still had to look up the directory for it). I'm just going to use sfan5's script for now, as that is considerably simpler and I don't update the server's mods very often


Did you check (1) the binary release zip file that has prebuilt binaries, and (2) has the systemd service unit for this program in it?

I could make it even more comfortable, of course, by completing the `make install` command that is available in the Makefile. Sure.
 


Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 2 guests