[Mod] DataMiner v2.2 (Player Analytics Toolbox) [dataminer]

Post Reply
User avatar
Posts: 1451
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

[Mod] DataMiner v2.2 (Player Analytics Toolbox) [dataminer]

by sorcerykid » Post


DataMiner Mod v1.2
dataminer (by sorcerykid)

DataMiner is an analytical tool for Minetest server operators and administrators, providing comprehensive player and server statistics within a graphical user-interface, in addition to an API for custom-tailored log analysis and reporting within command-line Lua scripts. DataMiner is intended to be compatible with all versions of Minetest 0.4.14+, and has been tested under CentOS/Red Hat Linux.



Download Archive (.zip)
Download Archive (.tar.gz)


Auth Redux Mod

ActiveFormspecs Mod

Polygraph Mod

Source Code License:

The MIT License (MIT)

Chat Command Usage:

There are two ways to generate reports, both in-game and from the command line. This section will show you how to use the chat command from within Minetest. After logging into your server as an administrator, press F5 and enter "/statmon" into chat. A screen like the one below will appear, showing a chart of player activity alongside a variety of player and server statistics for the past 24-hours.


The drop-down menu in the upper-left allows you to chart alternate datasets:
  • Total Players
    Total number of unique players that joined per hour
  • Total Player Sessions
    Total number of player sessions per hour
  • Total Login Failures
    Total number of failed logins per hour
  • Total Login Attempts
    Total number of attempted logins per hour
    Maximum Connected Clients
    Maximum number of connected clients per hour
    Minimum Connected Clients
    Minimum number of connected clients per hour
All charts are color-coded as a visual aid. In addition the y-axis is appropriately scaled to permit direct comparisons between related datasets (for example, the intervals for Max vs Min Connected Clients will be the same).

At the bottom of the screen is a summary of statistics for the entire 24-hour period. Most are derived from those above, with a few additions:
  • Maximum Player Lifetime
    Maximum session duration amongst all players per day
  • Average Player Lifetime
    Average session duration amongst all players per day
  • Total New Players
    Total number of new players that joined per day
  • Overall Server Uptime
    Aggregate uptime percentage of the server per day
You can navigate between days using the arrow buttons at the top of the screen. The double-arrows will jump forward or backward in 7-day increments. The selected dataset will remain the same, allowing for at-a-glance comparisons across a range of dates.

Command Line Usage:

The command-line interface offers a convenient alternative to the chat command if you only have access to a terminal, but not the Minetest client. From the shell, change to the tools subdirectory. Then, issue the following command:

Code: Select all

cd dataminer/tools
lua statmon.lua days=1 ~/.minetest/worlds/world/auth.dbx
Be sure to enter the correct path to the auth.dbx file of your world. The "days" parameter is also required. It specifies the number of days in the past to start reviewing. Hence, "0" is today, "1" is yesterday, and so forth. After the command is issued, DataMiner will index the database transaction log and cache the authentication database into memory (this may take a few seconds, but is a one-time operation).
  • Image
The I, J, K, and M keys allow you to navigate between days and to scroll through the list of players. By holding down the Shift key, you can jump in larger increments. The Player Details panel displays statistics for each player obtained from the authentication database including Initial Login, Latest Login, Total Sessions, and Total Lifetime.

When you are finished, the Q key will return you to the shell. You can also keep the statmon.lua script running in the background. The journal is opened in read-only mode, so it will not interfere with the Minetest server

Technical Information:

Mod and game developers, as well as server operators, can interface programmatically with DataMiner, allowing for a any number of applications. This section describes the API in detail.

The JournalIndex object contains all of the logic for indexing and analyzing the database transaction log (using an algorithm that I'd devised from an earlier incarnation of the same mod from the JT2 server).

The following function will instantiate a new JournalIndex object:
  • catalog = JournalIndex( path, name )
The path and name parameters specify the journal file to index and analyze, and are both required. Generally speaking, the file should be named "auth.dbx", but this is not mandatory.

Next, you will need to index the journal, using the process( ) method. This is a one-time operation. It could take a few seconds depending on the size of the file (I tested over 16 months worth of data from the JT2 server, and it clocked on average about 9 seconds).
  • catalog.prepare( )
Lastly, the journal can be analyzed by passing a date-offset to the analyze( ) method.
  • results = catalog.analyze( days )
The results table will contain the following fields (if the transaction log is incomplete for this day, then values will be 0):
  • total_players
  • total_players_new
  • total_sessions
  • total_failures
  • total_attempts
  • server_uptime
  • max_clients
  • max_clients
  • max_player_lifetime
  • avg_lifetime
  • player_stats
  • hourly_stats
The hourly_stats sub-table is an array of 24 elements, each corresponding to an hour:
  • sessions
  • players
  • failures
  • attempts
You can call the analyze( ) function again, to obtain another set of data for a different day. Since the journal is indexed in memory, this should typically be a very fast operation.

User avatar
Posts: 6278
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: United Kingdom

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by rubenwardy » Post

This is very impressive, well done! This is possibly one of the best formspecs I've ever seen

Posts: 548
Joined: Tue Jun 16, 2015 16:23

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by amadin » Post

When I had a server with up to 40 players online, he started lagging, and I could not figure it out because of the hard drive or the processor. Special Linux programs did not give an exact answer. I would like at least your mod could give such an answer, but I see that it does not.

New member
Posts: 4
Joined: Sun Dec 03, 2017 13:06

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by Tricobalt » Post

Polygraph bitbucket repo appears to be private, I can't clone it and when visiting the page it redirects me to a login page.

Posts: 4131
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by Sokomine » Post

Thanks for sharing this mod! I'm sure it can be very useful on a lot of servers. It just isn't very useful in singleplayer so there's usually less feedback in the thread. Which is a pity because there's certainly a lot of work that went into these mods. Hope other server owners will install and use them to their advantage.
A list of my mods can be found here.

Post Reply

Who is online

Users browsing this forum: Bastrabun and 1 guest