[Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx]

User avatar
rubenwardy
Moderator
 
Posts: 5592
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by rubenwardy » Sun Aug 19, 2018 15:04

It's a custom designed database format. You may gain speed but you lose reliability and the use of standard tools. Plus the engine will soon support other databases like postgresql which will be faster

Also worth noting that percentages aren't a good way to measure performance anyway - 1ms is twice as fast as 2ms but the difference is negligible.

You should also make sure you're using the latest version of the builtin auth, there were optimisations to do with string concatenation. It was also be good to note the number of records, and maybe even show a complexity graph (time vs number of records). This would be cool to show how each scales
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Mon Aug 20, 2018 23:44

Version 2.13 Released

Last week I completed work on a full-scale Database Management Console for Auth Redux. Minetest server operators now have a means to directly examine and manipulate the authentication database in-game from a simple but highly-flexible graphical user interface.

Image

Important Note: This feature requires version 2.3 or higher of the ActiveFormspecs Mod be installed.

To open the database management console, simply type "/auth" into chat (requires the "server privilege"). A screen similar to the one shown above will appear with the following important elements:

    A. The number of matching records appears here. As queries are performed, this number will change. If no matching records are found, then "No Records Selected" will be shown.

    B. The "Reverse Sort" option toggles whether to sort the dataset in descending order rather than the default ascending order. This option takes effect after clicking the "Sort" button.

    C. This table displays the currently matching records as a dataset. The cursor is used to select individual records within the dataset for use with the "Delete" and "Sort" buttons. If the cursor is in the header row, then all records in the dataset are selected.

    D. The user-defined columns and their corresponding formulas are listed in this table. By double-clicking on any column definition, the formula will be copied into the field below for editing.

    E. The left and right arrows change the order of columns within the results table. For convenience, the horizontal scroll position of the results table remains static as columns are adjusted.

    G. The "Add" button inserts the formula from the adjecent input field into the list of column definitions. Any valid MARS expression is allowed, but the result must evaluate to a string datatype.

    H. This dropdown menu identifies the active database column. It is intended for use with the "Set" and "Sort" buttons. It also allows for copying values of cells within the results table.

    I. Double-clicking a cell within the results table copies its value into this field for editing. Any valid MARS expression is allowed, but the resulting datatype must conform to the database schema.

    J. The "Set" button modifies the active database column of the selected records, given the expression provided. The "Del" button deletes the selected records, so use with extreme caution.

    K. The "Sort" button re-orders the dataset by the active database column. Sorting operations are progressive, so that multiple columns can be sorted to create a hierarchical arrangement of data.

    L. The recent query history appears in this table as a list of selectors. The highlighted selector corresponds to the dataset shown in the results table.

    M. An 'if' or 'unless' conditional expression can be entered into this field for selecting records within the current dataset.

    N. The "Clear" button will expunge all selectors from the query history, restoring the default selector (all records from the database).

    O. The "Query" button initiates another query, provided a valid conditional expression is entered into the field above.
Queries are always performed against the last matching records, so drill-down data analysis can be accomplished through the use of cascading selectors. For example, you can select all users that created an account prior to 2018:

You can then further refine your query by selecting only those users that signed on in the past week:

The resulting query would look like this:

    Image
You can review the results of earlier queries by clicking a different selector in the history list. The datasets are cached, so there is no performance penalty in doing so. However, if you initiate a new query, then it will be inserted at that point in the history list. You can always click the "Clear" button to expunge the history list and start from scratch.

There are endless possibilities for exploring your database through the use of simple queries:

    if $newlogin lt -90d
    Select users that haven't joined in the past 90 days

    if $total_sessions eq 0
    Select users that have never successfully logged in

    if $lifetime lt 5m
    Select users that have played for less than 5 minutes

    if size($assigned_privs) eq 0
    Select users that have not been granted any privileges

    if 'basic_privs' in $assigned_privs
    Select trusted users (such as moderators and administrators)

    unless $username is /*,*/
    Select users with all numeric, uppercase, and symbolic names

    if $username is /*=*=*/
    Select players with multiple symbols in their name

    if len($username) lt 3
    Select users with an extremely short name
Since columns can only output strings, I've provided some additional string conversion and formatting functions for use in column formulas:

  • cal(a,b)
    Returns moment a as a string given a multi-character format specified by b
    • Y - a year in the format '18'
    • YY - a year in the format '2018'
    • M - a month in the format '02'
    • MM - a month in the format 'Feb'
    • D - a day in the format '31'
    • DD - a weekday in the format 'Tue'
    • h - an hour in the format '8'
    • m - a minute in the format '35'
    • s - a second in the format '15'
  • when(a,b)
    Returns interval a as a string according to the scale specified by b
    • y - in years
    • w - in weeks
    • d - in days
    • h - in hours
    • m - in minutes
    • s - in seconds
  • join(a,b)
    Returns series a as a string concatenated by delimeter b
I spent a lot of time optimizing the MARS interpreter to be as fast and efficient as possible during queries. The lexer is executed only during the preprocessing stage and the operand parsers are called directly by-reference . Constant values (including all literals) are cached, so they only need to be evaluated once. Functions incorporate "smart-caching" so that if only literals are passed as arguments, then the constant-values will propagate upwards. Similar "smart-caching" is also supported for array literals and interpolated strings.

For example, the following query on a dataset of 522,000 records takes only about 1.0 seconds:

    Code: Select all
    if $username in ('Nemo','sorcerykid','publicworks')


    Image
In this case the left-hand operand cannot be cached, but the right-hand operand is cached and therefore only needs to be evaluated once since all elements of the series are constants. Generally speaking, functions with variable arguments tend to be the most expensive in terms of performance. However, by properly structuring your queries, heavier operations can be limited to a much smaller dataset.

For example, all three of these selectors produce the same results, yet the second one is twice as fast the first even though it is requires pattern matching against multiple numeric fields. And the third one is four times as fast.

    Code: Select all
    if date($oldlogin) gte 01-01-2018 -- takes 4.7 seconds with 522,000 records

    if $oldlogin is /?-?-2018>/d -- takes 2.1 seconds with 522,000 records)

    if $oldlogin gte at("2018-01-01T00:00:00Z") -- takes 1.0 seconds with 522,000 records
For datasets of under 50,000 records, the speed difference will be negligible. Therefore, avoiding function calls is only a concern if you are working with extremely large datasets.
 

twoelk
Member
 
Posts: 1259
Joined: Fri Apr 19, 2013 16:19
Location: northern Germany
GitHub: twoelk
IRC: twoelk
In-game: twoelk

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by twoelk » Tue Aug 21, 2018 14:20

duh......

ok by now I really hope you ad an help button for ingame help soon ;-)
and may the tooltips be extensiv
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Wed Aug 22, 2018 19:08

Auth Redux Lite!

Auth Redux Lite is available for server operators that want a barebones, no-frills authentication handler for their server. The entire mod consists of a single 12.8 kB file (for comparison, I believe the sauth mod is about 11 kB).

This fork uses the same database API and architecture as Auth Redux, so migration is completely seamless. It is also fully compatible with DataMiner. If you are already using Auth Redux, simply backup your existing auth_rx installation (as a tar.gz) and then save the above snippet into the auth_rx directory as an "init.lua" file. The original license terms apply, of course.

However, if you are migrating from the builtin auth handler of Minetest 0.4, then there are a couple additional steps:

  1. Create an "auth_rx" subdirectory under mods.
  2. Download the Auth Redux Database Import Script snippet and save as "convert.awk"
  3. Download the Auth Redux Lite snippet and save as "init.lua".
  4. Run the command below to convert the database (with your correct world path).
    Code: Select all
    awk -f convert.awk -v mode=install ~/.minetest/worlds/world/auth.txt
I'd also recommend moving or simply renaming the "auth.txt" file in your world directory to "auth.txt.bak" for safekeeping. That's it!
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Sat Aug 25, 2018 20:59

How To: Adding support for 'on_auth_fail' callback

To get the most benefit from Auth Redux, I strongly recommend that you compile your server with support for the new on_auth_fail callback. This callback was introduced in Minetest 5.0. If you are running an earlier version, then the source code will need to be modified.

Most of the applicable changes are documented in PR #7073. However, since Minetest 0.4.x also includes a legacy authentication mechanism, additional changes are required. To automate this process, I've provided patch files for all stable versions 0.4.14 and later.

Patch for 'on_auth_fail' callback (Minetest 0.4.x)

Simply download the appropriate patch into a temporary location, and issue the following git command within your local repository. Then rebuild the minetestserver as usual.

Code: Select all
git apply <patch-file-path>

To test whether the new callback is working, add the ruleset below to the greenlist.mt file. Now, login to your server with an incorrect password. Then reconnect, and you should get the error message, "ip_attempts=1, ip_failures=1".

Code: Select all
try "ip_attempts=$ip_attempts, ip_failures=$ip_failures"
when $ip_failures gt 0 fail
pass now

If it doesn't work, then double-check that the minetestserver executable was installed into the correct location; i.e. /usr/local/bin.
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Mon Aug 27, 2018 23:44

Version 2.14 Released

Custom column headings are now available in the Database Management Console. Previously, only the raw formulas were shown, but now you can include a more descriptive name as well:

Here I'm performing a query for all users that have at least two symbols in their name. The results are, well, interesting to say the least!

Image

The default column headings and formulas are used in this example, but they can easily be changed using the following format:

    <column_header>=<column_formula>.
By convention, I'm using camel-case, since it tends to be easier to read. But any alphanumeric characters are accepted.

  • Username
    $username
  • OldLogin
    $oldlogin->cal('D-MM-YY')
  • NewLogin
    $newlogin->cal('D-MM-YY')
  • Lifetime
    $lifetime->when('h')
  • TotalSessions
    $total_sessions->str()
  • TotalAttempts
    $total_attempts->str()
  • TotalFailures
    $total_failures->str()
  • AssignedPrivs
    $assigned_privs->join(',')
On a separate note, I implemented some more optimizations to the authentication handler, which should benefit high-traffic servers.

Registered on_prejoinplayer() callback
First and foremost, I addressed one of the longstanding bottlenecks -- validating case-insensitive duplicate names on new player joins. This process required searching the entire database for matching lowercase or uppercase names, sometimes costing up to 0.5 seconds of CPU time. This was unacceptable in my view, as a handful of new players joining in rapid succession could potentially cripple an underpowered server. Now a separate cache of case-insensitive name variants is maintained in memory, and the difference in performance is remarkable:

  • 305.51 milliseconds average (before)
  • 0.0126 milliseconds average (now)
Registered get_auth() callback

A second area of concern is the get_auth( ) function, which is one of the most frequently invoked callbacks in Minetest. Nearly every player action (including chat messages) are deferred to get_auth( ). While the overhead is small, I still feel it's a waste of processor cycles to be regenerating the same static data repeatedly. Therefore, I now cache the return values of the get_auth( ) function for users that are logged into the server to further improve performance.

  • 0.013 milliseconds average (before)
  • 0.002 milliseconds average (now)
 

User avatar
GoldFireUn
Member
 
Posts: 98
Joined: Sun Aug 21, 2016 13:30
Location: Wisconsin, USA
GitHub: BluebirdGreycoat
In-game: GoldFireUn

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by GoldFireUn » Tue Aug 28, 2018 03:10

Wow sorcerykid! You have done a LOT of extremely useful work. Those GUIs are really good, too, how do you get them to go like that? I hate coding new formspecs, it seems I constantly toggle to the lua API.txt every other line of code I write . . .
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Fri Aug 31, 2018 16:52

twoelk wrote:duh......

ok by now I really hope you ad an help button for ingame help soon ;-)
and may the tooltips be extensiv


Excellent suggestions as always, twoelk. Documentation is high on my to-do list. I'd been planning to convert all the relevant posts in this forum topic to markdown for distribution with the source code (still a WIP). I hadn't considered tooltips or in-game help yet. But. I'll see what I can do along that front. Anything to make it easier for the end -user :)

GoldFireUn wrote:Wow sorcerykid! You have done a LOT of extremely useful work. Those GUIs are really good, too, how do you get them to go like that? I hate coding new formspecs, it seems I constantly toggle to the lua API.txt every other line of code I write . . .


Heya Goldfire! Thanks for the words of encouragement. It's really nice to see you. Hope all is well!

Rest assured, designing formspecs is not my favourite pasttime :P I suppose it's a product of excessive perfectionism combined with my love of graphic design lol. I typically create a mockup of each layout using the drawing primitives in MS Word. With that as a visual guide, I build a static formspec string by hand one element at a time. After it looks half-way decent, i break the string apart and insert variables and other control structures where appropriate.

The screencap above, probably took an entire day of tweaking to get right, because there was so much content to pack into a limited screen space. I wanted the GUI to be at least half-way intelligible, without appearing too cluttered :D
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.12 (Authentication Handler) [auth_rx

by sorcerykid » Mon Sep 03, 2018 22:39

Hi all, I've finished converting all of the documentation for Auth Redux to markdown. It is now published on GitLab!

Image

I've organized the contents into 9 sections, so it should be much easier to navigate than via the (very long) forum post.

  1. Basic Database Import
  2. Auth Redux vs SQLite3
  3. Advanced Database Import
  4. Working with Rulesets
  5. Command-Line Analytics Script
  6. Database Management Console
  7. Additional Callback Support
  8. Migrating to Auth Redux Lite
  9. Technical Information
These docs are also mirrored to the GitHub Wiki, so they can be cloned to your local machine for quick reference.

Clone URL: https://github.com/sorcerykid/auth_rx.wiki.git
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by sorcerykid » Tue Oct 16, 2018 17:38

Version 2.15 Released

A new version of Auth Redux is ready for download. Here is a complete change log:

  • added raw database methods to auth handler API
  • extended last-login chat command with elapsed time
  • implemented chat command for session history
  • house-cleaning of builtin auth handler routines
First and foremost, I extended the "/last-login" chat command so that it shows both the elapsed number of days in addition to the timestamp. If no parameter is supplied, it will default to the current player.

    /last-login sorcerykid
    Last login was 2018-09-25T13:42:15Z (21 days ago)
I also added a "/login-history" chat command. It shows the initial login timestamp, the total number of sessions, and the cumulative time spent playing. If no parameter is supplied, it will default to the current player.

    /login-history sorcerykid
    Initial login was 2016-12-26T04:12:10 (28 total sessions, 49.5 hours spent playing)

I've also extended the authentication handler API so that mods can now retrieve the raw database record for any given player. The get_auth_raw( ) method returns a table with the following fields:

  • password - hashed account password
  • oldlogin - timestamp of the original login
  • newlogin - timestamp of the recent login
  • lifetime - cumulative time spent playing
  • total_sessions - total count of successful logins
  • total_attempts - total count of login requests
  • total_failures - total count of unsuccessful logins
  • assigned_privs - list of privileges
Since the table is a copy, it may be safely modified. The inverse set_auth_raw( ) method will record the changes back to the database. Bear in mind, however, that this latter method bypasses the transaction log. So use it with caution!

    db_record = minetest.get_auth_raw( player_name )

    minetest.get_auth_raw( player_name, db_record )
If you are planning to release mods that use these methods, then be sure to specify auth_rx as a dependency. Also include a sanity check for both methods at the head of your Lua script.

I also did some much needed internal housekeeping of the builtin auth handler. In particular I culled the auth_pass( ) function which has multiple levels of redundant function calls and function pointers. When invoking minetest.set_player_privs( ) or minetest.set_password( ), this is what used to happen:

    minetest.set_player_privs( ) =
    auth_pass("set_player_privileges") =
    function( ... ) ->
    core.get_auth_handler( ) ->
    core.registered_auth_handler( ) - >
    auth_handler[ name ]( ... ) ->
    set_player_privileges( )

This is how it works now:

    minetest.set_player_privs( ) =
    set_player_privileges( )

I was able to strip out 3 function calls and 2 function pointers which were extraneous. I will later fine-tune the get_auth_handler( ) function as well. But at least this change should address some of the still lingering overhead
 

User avatar
Linuxdirk
Member
 
Posts: 1717
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk
 

User avatar
Lejo
Member
 
Posts: 474
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by sorcerykid » Sat Nov 10, 2018 19:02

Lejo wrote:Delete Me


Hi Lejo, I'm not sure what you are asking?

Anyway, on a side note, I'm working on some improvements to the journal architecture. Currently when the server starts, the transaction log is read in its entirety to determine if a rollback of the database is necessary. For very busy servers, this can increase the server startup time as the transaction log grows (it doesn't introduce any overhead while players are online, of course). I will be caching the journal file offset in the header of the database so this step is no longer necessary. In addition, a unique hash will be stored in the database header to validate the journal for consistency with the corresponding database at server startup.

I expect all of this should be seamless from the perspective of server operators. A one-time shell script will be provided to convert the dbx and the db files accordingly. I'll post further updates once I'm closer to having a fully working prototype.
 

User avatar
Lejo
Member
 
Posts: 474
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by Lejo » Sun Nov 11, 2018 10:33

I just deleted the whole journal because it results in a three minute server start.
The Lite version is sadly not working.
It would be great if there were a version without any extra stuff.
Only need the mod to fix the auth bug of minetest.
 

User avatar
sorcerykid
Member
 
Posts: 913
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by sorcerykid » Sun Nov 11, 2018 15:44

What problem do you have with the lite version? Are you getting an error at startup? If you can give me more specifics I can look into a fix.

In the meantime, it is safe to remove the journal if it gets too big (as long as the server is not running). I will try to have a beta of the new journal architecture available by the end of the week, as that is intended to address your problem. I'll focus on updating Auth Redux Lite first, so that you can have it working on your server.
 

User avatar
Lejo
Member
 
Posts: 474
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by Lejo » Tue Nov 13, 2018 14:47

I get this error:
Code: Select all
Attempt to call "auth_db.select_record" (a nil value)

Could you please also please make set_password return true like this commit to make sure that you know if your password change was successful.
 

User avatar
rubenwardy
Moderator
 
Posts: 5592
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: [Mod] Auth Redux v2.14 (Authentication Handler) [auth_rx

by rubenwardy » Sat Jan 05, 2019 22:43

Please may you redo your benchmarks against the dev branch of sauth? It turns out that sauth didn't have an index on the username. sauth has also added caching and partitioning
 

Previous

Return to WIP Mods



Who is online

Users browsing this forum: No registered users and 2 guests