[Mod] DataMiner v2.2 (Player Analytics Toolbox) [dataminer]

User avatar
sorcerykid
Member
 
Posts: 852
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

[Mod] DataMiner v2.2 (Player Analytics Toolbox) [dataminer]

by sorcerykid » Mon Aug 06, 2018 03:54

Image

DataMiner Mod v1.2
dataminer (by sorcerykid)

DataMiner is an analytical tool for Minetest server operators and administrators, providing comprehensive player and server statistics within a graphical user-interface, in addition to an API for custom-tailored log analysis and reporting within command-line Lua scripts. DataMiner is intended to be compatible with all versions of Minetest 0.4.14+, and has been tested under CentOS/Red Hat Linux.

Repository:

https://bitbucket.org/sorcerykid/dataminer

Download Archive (.zip)
Download Archive (.tar.gz)

Dependencies:

Auth Redux Mod
https://bitbucket.org/sorcerykid/auth_rx

ActiveFormspecs Mod
https://bitbucket.org/sorcerykid/formspecs

Polygraph Mod
https://bitbucket.org/sorcerykid/polygraph

Source Code License:

The MIT License (MIT)

Chat Command Usage:

There are two ways to generate reports, both in-game and from the command line. This section will show you how to use the chat command from within Minetest. After logging into your server as an administrator, press F5 and enter "/statmon" into chat. A screen like the one below will appear, showing a chart of player activity alongside a variety of player and server statistics for the past 24-hours.

Image

The drop-down menu in the upper-left allows you to chart alternate datasets:

  • Total Players
    Total number of unique players that joined per hour
  • Total Player Sessions
    Total number of player sessions per hour
  • Total Login Failures
    Total number of failed logins per hour
  • Total Login Attempts
    Total number of attempted logins per hour
    Maximum Connected Clients
    Maximum number of connected clients per hour
    Minimum Connected Clients
    Minimum number of connected clients per hour
All charts are color-coded as a visual aid. In addition the y-axis is appropriately scaled to permit direct comparisons between related datasets (for example, the intervals for Max vs Min Connected Clients will be the same).

At the bottom of the screen is a summary of statistics for the entire 24-hour period. Most are derived from those above, with a few additions:

  • Maximum Player Lifetime
    Maximum session duration amongst all players per day
  • Average Player Lifetime
    Average session duration amongst all players per day
  • Total New Players
    Total number of new players that joined per day
  • Overall Server Uptime
    Aggregate uptime percentage of the server per day
You can navigate between days using the arrow buttons at the top of the screen. The double-arrows will jump forward or backward in 7-day increments. The selected dataset will remain the same, allowing for at-a-glance comparisons across a range of dates.

Command Line Usage:

The command-line interface offers a convenient alternative to the chat command if you only have access to a terminal, but not the Minetest client. From the shell, change to the tools subdirectory. Then, issue the following command:

Code: Select all
cd dataminer/tools
lua statmon.lua days=1 ~/.minetest/worlds/world/auth.dbx

Be sure to enter the correct path to the auth.dbx file of your world. The "days" parameter is also required. It specifies the number of days in the past to start reviewing. Hence, "0" is today, "1" is yesterday, and so forth. After the command is issued, DataMiner will index the database transaction log and cache the authentication database into memory (this may take a few seconds, but is a one-time operation).

    Image

The I, J, K, and M keys allow you to navigate between days and to scroll through the list of players. By holding down the Shift key, you can jump in larger increments. The Player Details panel displays statistics for each player obtained from the authentication database including Initial Login, Latest Login, Total Sessions, and Total Lifetime.

When you are finished, the Q key will return you to the shell. You can also keep the statmon.lua script running in the background. The journal is opened in read-only mode, so it will not interfere with the Minetest server

Technical Information:

Mod and game developers, as well as server operators, can interface programmatically with DataMiner, allowing for a any number of applications. This section describes the API in detail.

The JournalIndex object contains all of the logic for indexing and analyzing the database transaction log (using an algorithm that I'd devised from an earlier incarnation of the same mod from the JT2 server).

The following function will instantiate a new JournalIndex object:

    catalog = JournalIndex( path, name )
The path and name parameters specify the journal file to index and analyze, and are both required. Generally speaking, the file should be named "auth.dbx", but this is not mandatory.

Next, you will need to index the journal, using the process( ) method. This is a one-time operation. It could take a few seconds depending on the size of the file (I tested over 16 months worth of data from the JT2 server, and it clocked on average about 9 seconds).

    catalog.prepare( )
Lastly, the journal can be analyzed by passing a date-offset to the analyze( ) method.

    results = catalog.analyze( days )
The results table will contain the following fields (if the transaction log is incomplete for this day, then values will be 0):

  • total_players
  • total_players_new
  • total_sessions
  • total_failures
  • total_attempts
  • server_uptime
  • max_clients
  • max_clients
  • max_player_lifetime
  • avg_lifetime
  • player_stats
  • hourly_stats
The hourly_stats sub-table is an array of 24 elements, each corresponding to an hour:

  • sessions
  • players
  • failures
  • attempts
You can call the analyze( ) function again, to obtain another set of data for a different day. Since the journal is indexed in memory, this should typically be a very fast operation.
 

User avatar
rubenwardy
Moderator
 
Posts: 5492
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by rubenwardy » Mon Aug 06, 2018 09:43

This is very impressive, well done! This is possibly one of the best formspecs I've ever seen
 

amadin
Member
 
Posts: 544
Joined: Tue Jun 16, 2015 16:23

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by amadin » Mon Aug 06, 2018 11:21

When I had a server with up to 40 players online, he started lagging, and I could not figure it out because of the hard drive or the processor. Special Linux programs did not give an exact answer. I would like at least your mod could give such an answer, but I see that it does not.
 

Tricobalt
New member
 
Posts: 4
Joined: Sun Dec 03, 2017 13:06

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by Tricobalt » Mon Aug 06, 2018 13:38

Polygraph bitbucket repo appears to be private, I can't clone it and when visiting the page it redirects me to a login page.
 

Sokomine
Member
 
Posts: 3616
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: [Mod] DataMiner v2.2 (Player Analytics Toolbox) [datamin

by Sokomine » Sun Aug 12, 2018 10:25

Thanks for sharing this mod! I'm sure it can be very useful on a lot of servers. It just isn't very useful in singleplayer so there's usually less feedback in the thread. Which is a pity because there's certainly a lot of work that went into these mods. Hope other server owners will install and use them to their advantage.
A list of my mods can be found here.
 


Return to WIP Mods



Who is online

Users browsing this forum: Bing [Bot] and 5 guests