[Mod] DataMiner v2.2 (Player Analytics Toolbox) [dataminer]

[sorcerykid]

DataMiner Mod v1.2
dataminer (by sorcerykid)

DataMiner is an analytical tool for Minetest server operators and administrators, providing comprehensive player and server statistics within a graphical user-interface, in addition to an API for custom-tailored log analysis and reporting within command-line Lua scripts. DataMiner is intended to be compatible with all versions of Minetest 0.4.14+, and has been tested under CentOS/Red Hat Linux.

Repository:

https://bitbucket.org/sorcerykid/dataminer

Download Archive (.zip)
Download Archive (.tar.gz)

Dependencies:

Auth Redux Mod
https://bitbucket.org/sorcerykid/auth_rx

ActiveFormspecs Mod
https://bitbucket.org/sorcerykid/formspecs

Polygraph Mod
https://bitbucket.org/sorcerykid/polygraph

Source Code License:

The MIT License (MIT)

Chat Command Usage:

There are two ways to generate reports, both in-game and from the command line. This section will show you how to use the chat command from within Minetest. After logging into your server as an administrator, press F5 and enter "/statmon" into chat. A screen like the one below will appear, showing a chart of player activity alongside a variety of player and server statistics for the past 24-hours.



The drop-down menu in the upper-left allows you to chart alternate datasets:

• Total Players
  Total number of unique players that joined per hour
• Total Player Sessions
  Total number of player sessions per hour
• Total Login Failures
  Total number of failed logins per hour
• Total Login Attempts
  Total number of attempted logins per hour
  Maximum Connected Clients
  Maximum number of connected clients per hour
  Minimum Connected Clients
  Minimum number of connected clients per hour

All charts are color-coded as a visual aid. In addition the y-axis is appropriately scaled to permit direct comparisons between related datasets (for example, the intervals for Max vs Min Connected Clients will be the same).

At the bottom of the screen is a summary of statistics for the entire 24-hour period. Most are derived from those above, with a few additions:

• Maximum Player Lifetime
  Maximum session duration amongst all players per day
• Average Player Lifetime
  Average session duration amongst all players per day
• Total New Players
  Total number of new players that joined per day
• Overall Server Uptime
  Aggregate uptime percentage of the server per day

You can navigate between days using the arrow buttons at the top of the screen. The double-arrows will jump forward or backward in 7-day increments. The selected dataset will remain the same, allowing for at-a-glance comparisons across a range of dates.

Command Line Usage:

The command-line interface offers a convenient alternative to the chat command if you only have access to a terminal, but not the Minetest client. From the shell, change to the tools subdirectory. Then, issue the following command:

Code: Select all

cd dataminer/tools
lua statmon.lua days=1 ~/.minetest/worlds/world/auth.dbx

Be sure to enter the correct path to the auth.dbx file of your world. The "days" parameter is also required. It specifies the number of days in the past to start reviewing. Hence, "0" is today, "1" is yesterday, and so forth. After the command is issued, DataMiner will index the database transaction log and cache the authentication database into memory (this may take a few seconds, but is a one-time operation).

• 

The I, J, K, and M keys allow you to navigate between days and to scroll through the list of players. By holding down the Shift key, you can jump in larger increments. The Player Details panel displays statistics for each player obtained from the authentication database including Initial Login, Latest Login, Total Sessions, and Total Lifetime.

When you are finished, the Q key will return you to the shell. You can also keep the statmon.lua script running in the background. The journal is opened in read-only mode, so it will not interfere with the Minetest server

Technical Information:

Mod and game developers, as well as server operators, can interface programmatically with DataMiner, allowing for a any number of applications. This section describes the API in detail.

The JournalIndex object contains all of the logic for indexing and analyzing the database transaction log (using an algorithm that I'd devised from an earlier incarnation of the same mod from the JT2 server).

The following function will instantiate a new JournalIndex object:

• catalog = JournalIndex( path, name )

The path and name parameters specify the journal file to index and analyze, and are both required. Generally speaking, the file should be named "auth.dbx", but this is not mandatory.

Next, you will need to index the journal, using the process( ) method. This is a one-time operation. It could take a few seconds depending on the size of the file (I tested over 16 months worth of data from the JT2 server, and it clocked on average about 9 seconds).

• catalog.prepare( )

Lastly, the journal can be analyzed by passing a date-offset to the analyze( ) method.

• results = catalog.analyze( days )

The results table will contain the following fields (if the transaction log is incomplete for this day, then values will be 0):

• total_players
• total_players_new
• total_sessions
• total_failures
• total_attempts
• server_uptime
• max_clients
• max_clients
• max_player_lifetime
• avg_lifetime
• player_stats
• hourly_stats

The hourly_stats sub-table is an array of 24 elements, each corresponding to an hour:

• sessions
• players
• failures
• attempts

You can call the analyze( ) function again, to obtain another set of data for a different day. Since the journal is indexed in memory, this should typically be a very fast operation.

[rubenwardy]

This is very impressive, well done! This is possibly one of the best formspecs I've ever seen

[amadin]

When I had a server with up to 40 players online, he started lagging, and I could not figure it out because of the hard drive or the processor. Special Linux programs did not give an exact answer. I would like at least your mod could give such an answer, but I see that it does not.

[Tricobalt]

Polygraph bitbucket repo appears to be private, I can't clone it and when visiting the page it redirects me to a login page.

[Sokomine]

Thanks for sharing this mod! I'm sure it can be very useful on a lot of servers. It just isn't very useful in singleplayer so there's usually less feedback in the thread. Which is a pity because there's certainly a lot of work that went into these mods. Hope other server owners will install and use them to their advantage.

[AmyMoriyama]

oops wrong mod.. this reply can be deleted.

[MCL]

2022-08-20 21:00:31: ACTION[Main]: Server: Shutting down
2022-08-20 21:00:32: ERROR[Main]: ModError: Failed to load and run script from /home/minetest/minetest-5.5.0/bin/../mods/dataminer/init.lua:
2022-08-20 21:00:32: ERROR[Main]: ...inetest/minetest-5.5.0/bin/../mods/dataminer/catalog.lua:18: Cannot open journal file for reading.
2022-08-20 21:00:32: ERROR[Main]: stack traceback:
2022-08-20 21:00:32: ERROR[Main]: [C]: in function 'error'
2022-08-20 21:00:32: ERROR[Main]: ...inetest/minetest-5.5.0/bin/../mods/dataminer/catalog.lua:18: in function 'JournalIndex'
2022-08-20 21:00:32: ERROR[Main]: ...e/minetest/minetest-5.5.0/bin/../mods/dataminer/init.lua:418: in main chunk

After creating the auth.dbx file manually:

Loading ActiveFormspecs Mod
2022-08-20 21:07:17: ACTION[Main]: Server: Shutting down
2022-08-20 21:07:18: ERROR[Main]: ModError: Failed to load and run script from /home/minetest/minetest-5.5.0/bin/../mods/dataminer/init.lua:
2022-08-20 21:07:18: ERROR[Main]: ...inetest/minetest-5.5.0/bin/../mods/dataminer/catalog.lua:71: attempt to perform arithmetic on local 'optime' (a nil value)
2022-08-20 21:07:18: ERROR[Main]: stack traceback:
2022-08-20 21:07:18: ERROR[Main]: ...inetest/minetest-5.5.0/bin/../mods/dataminer/catalog.lua:71: in function 'prepare'
2022-08-20 21:07:18: ERROR[Main]: ...e/minetest/minetest-5.5.0/bin/../mods/dataminer/init.lua:419: in main chunk

[sorcerykid]

Thanks for the heads up. It appears the first error is caused by the auth.dbx file not being found or having the wrong permissions.

What version of Auth Redux is installed? Have you tried running the command statmon.lua tool?

[MCL]

Thanks for the heads up. It appears the first error is caused by the auth.dbx file not being found or having the wrong permissions.

auth.dbx does exist, I created it manually and that's probably the issue, should I change te perms w/ chmod?

What version of Auth Redux is installed?

The one from the master branch

Have you tried running the command statmon.lua tool?

No, but I will
EDIT: Just did, and

Code: Select all

lua: statmon.lua:56: The 'days' parameter is invalid or missing.
stack traceback:
        [C]: in function 'error'
        statmon.lua:56: in main chunk
        [C]: in ?

[MCL]

Auth Redux doesn't seem to be working for me either...

Code: Select all

2022-08-21 18:22:42: WARNING[Main]: Mod auth_rx at /home/minetest/minetest-5.5.0/bin/../mods/auth_rx:
2022-08-21 18:22:42: WARNING[Main]:     description.txt is deprecated, please use mod.conf instead.
2022-08-21 18:22:42: ACTION[Main]: Server: Shutting down
2022-08-21 18:22:42: ERROR[Main]: ModError: Failed to load and run script from /home/minetest/minetest-5.5.0/bin/../mods/auth_rx/init.lua:
2022-08-21 18:22:42: ERROR[Main]: ...e/minetest/minetest-5.5.0/bin/../mods/auth_rx/filter.lua:488: The specified ruleset file does not exist.
2022-08-21 18:22:42: ERROR[Main]: stack traceback:
2022-08-21 18:22:42: ERROR[Main]:       [C]: in function 'error'
2022-08-21 18:22:42: ERROR[Main]:       ...e/minetest/minetest-5.5.0/bin/../mods/auth_rx/filter.lua:488: in function 'refresh'
2022-08-21 18:22:42: ERROR[Main]:       ...e/minetest/minetest-5.5.0/bin/../mods/auth_rx/filter.lua:607: in function 'AuthFilter'
2022-08-21 18:22:42: ERROR[Main]:       /home/minetest/minetest-5.5.0/bin/../mods/auth_rx/init.lua:21: in main chunk

[mcaygerhard]

all the errors were addressed in my minenux version.. https://codeberg.org/minenux/minetest-mod-auth_rx

it automatically init the necessary things.. if you installed manually for any OS, but the tools of course will only work on linux

[sorcerykid]

Thanks for bringing both of these issues to my attention.

The first error ("statmon.lua:56: The 'days' parameter is invalid or missing.") is due to not providing the number of days as a paramter to the script. Please refer to section "Command Line Usage" in the first post, as it explains how to correctly invoke the statmon.lua script.

The second error ("filter.lua:488: The specified ruleset file does not exist.") is caused by a missing ruleset file. To remedy, simply create an empty file in the world directory called "greenlist.mt". This should have been automatically created, however, assuming that you ran the install script provided:

Re: [Mod] Auth Redux v2.1 (Authentication Handler) [auth_rx]

Admittedly, this should have been better documented. So I'll add it to the wiki on the Github repo.