[Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx]

[rubenwardy]

It's a custom designed database format. You may gain speed but you lose reliability and the use of standard tools. Plus the engine will soon support other databases like postgresql which will be faster

Also worth noting that percentages aren't a good way to measure performance anyway - 1ms is twice as fast as 2ms but the difference is negligible.

You should also make sure you're using the latest version of the builtin auth, there were optimisations to do with string concatenation. It was also be good to note the number of records, and maybe even show a complexity graph (time vs number of records). This would be cool to show how each scales

[sorcerykid]

Version 2.13 Released

Last week I completed work on a full-scale Database Management Console for Auth Redux. Minetest server operators now have a means to directly examine and manipulate the authentication database in-game from a simple but highly-flexible graphical user interface.



Important Note: This feature requires version 2.3 or higher of the ActiveFormspecs Mod be installed.

To open the database management console, simply type "/auth" into chat (requires the "server privilege"). A screen similar to the one shown above will appear with the following important elements:

• A. The number of matching records appears here. As queries are performed, this number will change. If no matching records are found, then "No Records Selected" will be shown.
  
  B. The "Reverse Sort" option toggles whether to sort the dataset in descending order rather than the default ascending order. This option takes effect after clicking the "Sort" button.
  
  C. This table displays the currently matching records as a dataset. The cursor is used to select individual records within the dataset for use with the "Delete" and "Sort" buttons. If the cursor is in the header row, then all records in the dataset are selected.
  
  D. The user-defined columns and their corresponding formulas are listed in this table. By double-clicking on any column definition, the formula will be copied into the field below for editing.
  
  E. The left and right arrows change the order of columns within the results table. For convenience, the horizontal scroll position of the results table remains static as columns are adjusted.
  
  G. The "Add" button inserts the formula from the adjecent input field into the list of column definitions. Any valid MARS expression is allowed, but the result must evaluate to a string datatype.
  
  H. This dropdown menu identifies the active database column. It is intended for use with the "Set" and "Sort" buttons. It also allows for copying values of cells within the results table.
  
  I. Double-clicking a cell within the results table copies its value into this field for editing. Any valid MARS expression is allowed, but the resulting datatype must conform to the database schema.
  
  J. The "Set" button modifies the active database column of the selected records, given the expression provided. The "Del" button deletes the selected records, so use with extreme caution.
  
  K. The "Sort" button re-orders the dataset by the active database column. Sorting operations are progressive, so that multiple columns can be sorted to create a hierarchical arrangement of data.
  
  L. The recent query history appears in this table as a list of selectors. The highlighted selector corresponds to the dataset shown in the results table.
  
  M. An 'if' or 'unless' conditional expression can be entered into this field for selecting records within the current dataset.
  
  N. The "Clear" button will expunge all selectors from the query history, restoring the default selector (all records from the database).
  
  O. The "Query" button initiates another query, provided a valid conditional expression is entered into the field above.

Queries are always performed against the last matching records, so drill-down data analysis can be accomplished through the use of cascading selectors. For example, you can select all users that created an account prior to 2018:

• Code: Select all

  if $oldlogin is /?-?-2017</d

You can then further refine your query by selecting only those users that signed on in the past week:

• Code: Select all

  if $newlogin gt -1w

The resulting query would look like this:

• 

You can review the results of earlier queries by clicking a different selector in the history list. The datasets are cached, so there is no performance penalty in doing so. However, if you initiate a new query, then it will be inserted at that point in the history list. You can always click the "Clear" button to expunge the history list and start from scratch.

There are endless possibilities for exploring your database through the use of simple queries:

• if $newlogin lt -90d
  Select users that haven't joined in the past 90 days
  
  if $total_sessions eq 0
  Select users that have never successfully logged in
  
  if $lifetime lt 5m
  Select users that have played for less than 5 minutes
  
  if size($assigned_privs) eq 0
  Select users that have not been granted any privileges
  
  if 'basic_privs' in $assigned_privs
  Select trusted users (such as moderators and administrators)
  
  unless $username is /*,*/
  Select users with all numeric, uppercase, and symbolic names
  
  if $username is /*=*=*/
  Select players with multiple symbols in their name
  
  if len($username) lt 3
  Select users with an extremely short name

Since columns can only output strings, I've provided some additional string conversion and formatting functions for use in column formulas:

• cal(a,b)
  Returns moment a as a string given a multi-character format specified by b
  • Y - a year in the format '18'
  • YY - a year in the format '2018'
  • M - a month in the format '02'
  • MM - a month in the format 'Feb'
  • D - a day in the format '31'
  • DD - a weekday in the format 'Tue'
  • h - an hour in the format '8'
  • m - a minute in the format '35'
  • s - a second in the format '15'
• when(a,b)
  Returns interval a as a string according to the scale specified by b
  • y - in years
  • w - in weeks
  • d - in days
  • h - in hours
  • m - in minutes
  • s - in seconds
• join(a,b)
  Returns series a as a string concatenated by delimeter b

I spent a lot of time optimizing the MARS interpreter to be as fast and efficient as possible during queries. The lexer is executed only during the preprocessing stage and the operand parsers are called directly by-reference . Constant values (including all literals) are cached, so they only need to be evaluated once. Functions incorporate "smart-caching" so that if only literals are passed as arguments, then the constant-values will propagate upwards. Similar "smart-caching" is also supported for array literals and interpolated strings.

For example, the following query on a dataset of 522,000 records takes only about 1.0 seconds:

• Code: Select all

  if $username in ('Nemo','sorcerykid','publicworks')

  

In this case the left-hand operand cannot be cached, but the right-hand operand is cached and therefore only needs to be evaluated once since all elements of the series are constants. Generally speaking, functions with variable arguments tend to be the most expensive in terms of performance. However, by properly structuring your queries, heavier operations can be limited to a much smaller dataset.

For example, all three of these selectors produce the same results, yet the second one is twice as fast the first even though it is requires pattern matching against multiple numeric fields. And the third one is four times as fast.

• Code: Select all

  if date($oldlogin) gte 01-01-2018 -- takes 4.7 seconds with 522,000 records
  
  if $oldlogin is /?-?-2018>/d -- takes 2.1 seconds with 522,000 records)
  
  if $oldlogin gte at("2018-01-01T00:00:00Z") -- takes 1.0 seconds with 522,000 records

For datasets of under 50,000 records, the speed difference will be negligible. Therefore, avoiding function calls is only a concern if you are working with extremely large datasets.

[twoelk]

duh......

ok by now I really hope you ad an help button for ingame help soon ;-)
and may the tooltips be extensiv

[sorcerykid]

Auth Redux Lite!

Auth Redux Lite is available for server operators that want a barebones, no-frills authentication handler for their server. The entire mod consists of a single 12.8 kB file (for comparison, I believe the sauth mod is about 11 kB).

• https://bitbucket.org/snippets/sorceryk ... of-auth_rx

This fork uses the same database API and architecture as Auth Redux, so migration is completely seamless. It is also fully compatible with DataMiner. If you are already using Auth Redux, simply backup your existing auth_rx installation (as a tar.gz) and then save the above snippet into the auth_rx directory as an "init.lua" file. The original license terms apply, of course.

However, if you are migrating from the builtin auth handler of Minetest 0.4, then there are a couple additional steps:

1. Create an "auth_rx" subdirectory under mods.
2. Download the Auth Redux Database Import Script snippet and save as "convert.awk"
3. Download the Auth Redux Lite snippet and save as "init.lua".
4. Run the command below to convert the database (with your correct world path).

   Code: Select all

   awk -f convert.awk -v mode=install ~/.minetest/worlds/world/auth.txt

I'd also recommend moving or simply renaming the "auth.txt" file in your world directory to "auth.txt.bak" for safekeeping. That's it!

[sorcerykid]

How To: Adding support for 'on_auth_fail' callback

To get the most benefit from Auth Redux, I strongly recommend that you compile your server with support for the new on_auth_fail callback. This callback was introduced in Minetest 5.0. If you are running an earlier version, then the source code will need to be modified.

Most of the applicable changes are documented in PR #7073. However, since Minetest 0.4.x also includes a legacy authentication mechanism, additional changes are required. To automate this process, I've provided patch files for all stable versions 0.4.14 and later.

Patch for 'on_auth_fail' callback (Minetest 0.4.x)

Simply download the appropriate patch into a temporary location, and issue the following git command within your local repository. Then rebuild the minetestserver as usual.

Code: Select all

git apply <patch-file-path>

To test whether the new callback is working, add the ruleset below to the greenlist.mt file. Now, login to your server with an incorrect password. Then reconnect, and you should get the error message, "ip_attempts=1, ip_failures=1".

Code: Select all

try "ip_attempts=$ip_attempts, ip_failures=$ip_failures"
when $ip_failures gt 0 fail
pass now

If it doesn't work, then double-check that the minetestserver executable was installed into the correct location; i.e. /usr/local/bin.

[sorcerykid]

Version 2.14 Released

Custom column headings are now available in the Database Management Console. Previously, only the raw formulas were shown, but now you can include a more descriptive name as well:

Here I'm performing a query for all users that have at least two symbols in their name. The results are, well, interesting to say the least!



The default column headings and formulas are used in this example, but they can easily be changed using the following format:

• <column_header>=<column_formula>.

By convention, I'm using camel-case, since it tends to be easier to read. But any alphanumeric characters are accepted.

• Username
  $username
• OldLogin
  $oldlogin->cal('D-MM-YY')
• NewLogin
  $newlogin->cal('D-MM-YY')
• Lifetime
  $lifetime->when('h')
• TotalSessions
  $total_sessions->str()
• TotalAttempts
  $total_attempts->str()
• TotalFailures
  $total_failures->str()
• AssignedPrivs
  $assigned_privs->join(',')

On a separate note, I implemented some more optimizations to the authentication handler, which should benefit high-traffic servers.

Registered on_prejoinplayer() callback
First and foremost, I addressed one of the longstanding bottlenecks -- validating case-insensitive duplicate names on new player joins. This process required searching the entire database for matching lowercase or uppercase names, sometimes costing up to 0.5 seconds of CPU time. This was unacceptable in my view, as a handful of new players joining in rapid succession could potentially cripple an underpowered server. Now a separate cache of case-insensitive name variants is maintained in memory, and the difference in performance is remarkable:

• 305.51 milliseconds average (before)
• 0.0126 milliseconds average (now)

Registered get_auth() callback

A second area of concern is the get_auth( ) function, which is one of the most frequently invoked callbacks in Minetest. Nearly every player action (including chat messages) are deferred to get_auth( ). While the overhead is small, I still feel it's a waste of processor cycles to be regenerating the same static data repeatedly. Therefore, I now cache the return values of the get_auth( ) function for users that are logged into the server to further improve performance.

• 0.013 milliseconds average (before)
• 0.002 milliseconds average (now)

[GoldFireUn]

Wow sorcerykid! You have done a LOT of extremely useful work. Those GUIs are really good, too, how do you get them to go like that? I hate coding new formspecs, it seems I constantly toggle to the lua API.txt every other line of code I write . . .

[sorcerykid]

duh......

ok by now I really hope you ad an help button for ingame help soon ;-)
and may the tooltips be extensiv

Excellent suggestions as always, twoelk. Documentation is high on my to-do list. I'd been planning to convert all the relevant posts in this forum topic to markdown for distribution with the source code (still a WIP). I hadn't considered tooltips or in-game help yet. But. I'll see what I can do along that front. Anything to make it easier for the end -user :)

Wow sorcerykid! You have done a LOT of extremely useful work. Those GUIs are really good, too, how do you get them to go like that? I hate coding new formspecs, it seems I constantly toggle to the lua API.txt every other line of code I write . . .

Heya Goldfire! Thanks for the words of encouragement. It's really nice to see you. Hope all is well!

Rest assured, designing formspecs is not my favourite pasttime :P I suppose it's a product of excessive perfectionism combined with my love of graphic design lol. I typically create a mockup of each layout using the drawing primitives in MS Word. With that as a visual guide, I build a static formspec string by hand one element at a time. After it looks half-way decent, i break the string apart and insert variables and other control structures where appropriate.

The screencap above, probably took an entire day of tweaking to get right, because there was so much content to pack into a limited screen space. I wanted the GUI to be at least half-way intelligible, without appearing too cluttered :D

[sorcerykid]

Hi all, I've finished converting all of the documentation for Auth Redux to markdown. It is now published on GitLab!



I've organized the contents into 9 sections, so it should be much easier to navigate than via the (very long) forum post.

1. Basic Database Import
2. Auth Redux vs SQLite3
3. Advanced Database Import
4. Working with Rulesets
5. Command-Line Analytics Script
6. Database Management Console
7. Additional Callback Support
8. Migrating to Auth Redux Lite
9. Technical Information

These docs are also mirrored to the GitHub Wiki, so they can be cloned to your local machine for quick reference.

Clone URL: https://github.com/sorcerykid/auth_rx.wiki.git

[sorcerykid]

Version 2.15 Released

A new version of Auth Redux is ready for download. Here is a complete change log:

• added raw database methods to auth handler API
• extended last-login chat command with elapsed time
• implemented chat command for session history
• house-cleaning of builtin auth handler routines

First and foremost, I extended the "/last-login" chat command so that it shows both the elapsed number of days in addition to the timestamp. If no parameter is supplied, it will default to the current player.

• /last-login sorcerykid
  Last login was 2018-09-25T13:42:15Z (21 days ago)

I also added a "/login-history" chat command. It shows the initial login timestamp, the total number of sessions, and the cumulative time spent playing. If no parameter is supplied, it will default to the current player.

• /login-history sorcerykid
  Initial login was 2016-12-26T04:12:10 (28 total sessions, 49.5 hours spent playing)

I've also extended the authentication handler API so that mods can now retrieve the raw database record for any given player. The get_auth_raw( ) method returns a table with the following fields:

• password - hashed account password
• oldlogin - timestamp of the original login
• newlogin - timestamp of the recent login
• lifetime - cumulative time spent playing
• total_sessions - total count of successful logins
• total_attempts - total count of login requests
• total_failures - total count of unsuccessful logins
• assigned_privs - list of privileges

Since the table is a copy, it may be safely modified. The inverse set_auth_raw( ) method will record the changes back to the database. Bear in mind, however, that this latter method bypasses the transaction log. So use it with caution!

• db_record = minetest.get_auth_raw( player_name )
  
  minetest.get_auth_raw( player_name, db_record )

If you are planning to release mods that use these methods, then be sure to specify auth_rx as a dependency. Also include a sanity check for both methods at the head of your Lua script.

I also did some much needed internal housekeeping of the builtin auth handler. In particular I culled the auth_pass( ) function which has multiple levels of redundant function calls and function pointers. When invoking minetest.set_player_privs( ) or minetest.set_password( ), this is what used to happen:

• minetest.set_player_privs( ) =
  auth_pass("set_player_privileges") =
  function( ... ) ->
  core.get_auth_handler( ) ->
  core.registered_auth_handler( ) - >
  auth_handler[ name ]( ... ) ->
  set_player_privileges( )

This is how it works now:

• minetest.set_player_privs( ) =
  set_player_privileges( )

I was able to strip out 3 function calls and 2 function pointers which were extraneous. I will later fine-tune the get_auth_handler( ) function as well. But at least this change should address some of the still lingering overhead

[Linuxdirk]

Over-engineering at it's best, but I must admit I love all the features! :)

[Lejo]

Delete Me

[sorcerykid]

Delete Me

Hi Lejo, I'm not sure what you are asking?

Anyway, on a side note, I'm working on some improvements to the journal architecture. Currently when the server starts, the transaction log is read in its entirety to determine if a rollback of the database is necessary. For very busy servers, this can increase the server startup time as the transaction log grows (it doesn't introduce any overhead while players are online, of course). I will be caching the journal file offset in the header of the database so this step is no longer necessary. In addition, a unique hash will be stored in the database header to validate the journal for consistency with the corresponding database at server startup.

I expect all of this should be seamless from the perspective of server operators. A one-time shell script will be provided to convert the dbx and the db files accordingly. I'll post further updates once I'm closer to having a fully working prototype.

[Lejo]

I just deleted the whole journal because it results in a three minute server start.
The Lite version is sadly not working.
It would be great if there were a version without any extra stuff.
Only need the mod to fix the auth bug of minetest.

[sorcerykid]

What problem do you have with the lite version? Are you getting an error at startup? If you can give me more specifics I can look into a fix.

In the meantime, it is safe to remove the journal if it gets too big (as long as the server is not running). I will try to have a beta of the new journal architecture available by the end of the week, as that is intended to address your problem. I'll focus on updating Auth Redux Lite first, so that you can have it working on your server.

[Lejo]

I get this error:

Code: Select all

Attempt to call "auth_db.select_record" (a nil value)

Could you please also please make set_password return true like this commit to make sure that you know if your password change was successful.

[rubenwardy]

Please may you redo your benchmarks against the dev branch of sauth? It turns out that sauth didn't have an index on the username. sauth has also added caching and partitioning

[lilo]

Hi, I tried the MOD now:) Unfortunately, I get the following message when I start:

Code: Select all

Failed to load and run script from C:\minetest-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx\init.lua:
...0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/filter.lua:488: The specified ruleset file does not exist.
stack traceback:
[C]: in function 'error'
...0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/filter.lua:488: in function 'refresh'
...0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/filter.lua:607: in function 'AuthFilter'
...t-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx\init.lua:21: in main chunk

What is to do?

I created an new world and an dummy auth.txt and follow your instruction.
I got an auth.db but no auth.dbx.

Greets

[sorcerykid]

Greeting! There are three files required for Auth Redux to run. They must be located in your world directory

• auth.db (master database)
  stores all account login information for your players; avoid editing this file directly
  
  auth.dbx (transaction log)
  used for rollback in case of server crash; can safely be deleted when server is offline
  
  greenlist.mt (ruleset definition)
  allows for automatic filtering of login requests given a list of pass and fail conditions

By running the included "tools/convert.awk" script in install mode, all three files will be created for you.

Code: Select all

awk -f convert.awk -v mode=install <path_to_original_auth_file>

If for some reason you cannot run AWK scripts on your system, then the files can be created manually as follows:

• auth.db should contain auth_rx/2.1 @0
  
  auth.dbx contains a single empty line
  
  greenlist.mt should contain pass now

Once the required files are in your world directory, then you can start your server as you normally do. You can check if Auth Redux is running by typing the /db chat command. Hope this helps!

[lilo]

Hi,

I created these files of myself, but error won't go away :/

greets

[sorcerykid]

I'm investigating to see if this is an OS-related issue. I tested with a fresh install of Minetest under Windows 10, and haven't been able to reproduce the error. I suspect the breakage may be due to file paths being mangled somehow.

Could you double check to make sure there is a file named "greenlist.mt" in your world directory (the same location as map.sqlite)? What is the full path of your world directory?

[ShadMOrdre]

sorcerykid,

I just installed auth_rx and dataminer yesterday. After resolving the auth_rx issue, which was to use the awk script with the install vs convert mode, the issue then became dataminer.

I downloaded and installed auth_rx and dataminer yesterday from github, after our other conversation :) I had previously already installed the latest versions of formspecs and polygraph.

Apparently, dataminer, has an issue with the first line of auth.dbx being blank. After removing the blank line at the beginning of auth.dbx, eveything worked fine. I'd like to see more data in the default install, but I haven't played with it enough yet to say what I'd actually like to see available. I will keep you informed on this, if you'd like.

I must admit, while being well documented, for some users, this might be too complex a process to install and use.

Otherwise, great job here. Can't wait to see the other project...hint hint....

Shad

[lilo]

Hi,

i also use Windows 10 :)

This is my CMD Command:

Code: Select all

C:\tmp\minetest-0.4.17.1-win64 Servertest\mods\auth_rx\tools>"C:\Program Files (x86)\GnuWin32\bin\awk.exe" -f convert.awk -v mode=convert "C:\tmp\minetest-0.4.17.1-win64 Servertest\worlds\Servertest\auth.txt"
Converting C:\tmp\minetest-0.4.17.1-win64 Servertest\worlds\Servertest\auth.txt...
Done! 0 of 0 total records were imported to auth.db (0 records skipped).

I get only auth.db.

Greets

[ShadMOrdre]

lilo,

Use:

Code: Select all

awk -f convert.awk -v mode=install ~/.minetest/worlds/world/auth.txt

This will create the required files.

[lilo]

Hi,

thx, this command create all 3 files :) But i get this error message, wenn i start my world:

Code: Select all

2019-02-19 21:45:09: [Main]: Automatically selecting world at [C:\tmp\minetest-0.4.17.1-win64 Servertest\bin\..\worlds\Servertest]
2019-02-19 21:45:35: WARNING[Main]: BanManager: creating C:\tmp\minetest-0.4.17.1-win64 Servertest\bin\..\worlds\Servertest\ipban.txt
2019-02-19 21:45:36: ACTION[Main]: Reading authentication data from disk...
2019-02-19 21:45:36: ERROR[Main]: Invalid header in authentication database.
2019-02-19 21:45:36: ERROR[Main]: ModError: Failed to load and run script from C:\tmp\minetest-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx\init.lua:
2019-02-19 21:45:36: ERROR[Main]: ...est-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/db.lua:197: Fatal exception in AuthDatabase:reload( ), aborting.
2019-02-19 21:45:36: ERROR[Main]: stack traceback:
2019-02-19 21:45:36: ERROR[Main]: 	[C]: in function 'error'
2019-02-19 21:45:36: ERROR[Main]: 	...est-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/db.lua:197: in function 'db_reload'
2019-02-19 21:45:36: ERROR[Main]: 	...est-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx/db.lua:271: in function 'connect'
2019-02-19 21:45:36: ERROR[Main]: 	...t-0.4.17.1-win64 Servertest\bin\..\mods\auth_rx\init.lua:142: in main chunk

In my auth.dbx is "auth_rx/2.1 @0" only one entry, so i think it's correct.

greets