[Mod] ActiveFormspecs [formspecs]

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

[Mod] ActiveFormspecs [formspecs]

by sorcerykid » Fri Jan 05, 2018 21:43

Image

ActiveFormspecs Mod v2.4
formspecs (by sorcerykid)

ActiveFormspecs is a self-contained API that provides secure session tracking, session-based state tables, and localized event handling of formspecs for individual mods as well as entire subgames in Minetest. It evolved out of a recurring need for secure "node formspecs" on my server, but without the burden of "reinventing the wheel" with every new project. ActiveFormspecs adheres to the best practices described in sofar's excellent Securing Formspec Code topic.

Since I've had many requests for source code from the just_test_tribute subgame, I finally decided to release ActiveFormspecs with instructions and code examples so that other mod authors can start making use of this framework as well. It's easy to install and use, and arguably a more robust alternative to the builtin formspecs API.

ActiveFormspecs is intended to be compatible with all versions of Minetest 0.4.14+. It has been in continuous use on my server since December 2016 with only minor revisions. So it should prove secure and stable enough for any production environment, as long as you follow the instructions below.

Repository:

https://bitbucket.org/sorcerykid/formspecs

Download Archive (.zip)
Download Archive (.tar.gz)

Dependencies:

None.

Revision History:


Source Code License:

The MIT License (MIT)

Overview:

ActiveFormspecs is a framework that abstracts the builtin formspec API of Minetest. It is intended to address a number of known security issues related to formspecs:

    Secure Session Tracking
    Formspec names have been deprecated as they can be easily forged. Now each formspec session is assigned a unique session ID. Due to the persistent nature of the Minetest client-server protocol (unlike HTTP, for example), all session tracking is performed server-side. Negotiation and validation with the client is entirely unnecessary. Thus, integrity of the session ID is always guaranteed.

    Session-Based State Table
    Since the session ID token is retained throughout the lifetime of the formspec, it is therefore possible to update a formspec dynamically (e.g. in response to an event) with contextual data spanning multiple instances. This data is stored server-side via a session-based state table and it can even be initialized from within the formspec string itself using a new "hidden" element.

    Localized Event Handling
    The minetest.register_on_player_receive_fields( ) method has also been deprecated. Instead, each formspec is assigned its own callback function at runtime, which allows for completely localized event handling. This callback function is invoked after any event associated with the formspec (hence the moniker "ActiveFormspecs"). Both the meta table and form fields are passed as arguments.
The project is a WIP and will be undergoing continuous development based upon your suggestions as well as my personal needs. Version 3.0 is already underway, and I am planning to introduce substantial improvements to the core functionality. New features and bug-fixes will be announced here as they become available. During significant milestones, I will include a roadmap so as to gauge your feedback about long-term goals. I will make every effort to ensure backward compatibility, when possible.

Installation:

  1. Unzip the archive into the mods directory of your subgame
  2. Rename the formspecs-master directory to "formspecs"
  3. Add "formspecs" to the depends.txt file of mods using the ActiveFormspecs API.
Usage Instructions:

An interactive form session monitor can be accessed in-game via the /fs chat command (requires "server" privilege). A realtime summary of form sessions is displayed along with navigation buttons and related statistics.

    Image
The summary is sorted chronologically and divided into four columns per player

  • player - the name of the player viewing the formspec
  • origin - the mod (or node, if attached) that created the formspec
  • idletime - the elapsed time since a formspec event or signal
  • lifetime - the elapsed time since the formspec was first opened
A detached formspec can be opened using the minetest.create_form( ) method:

    form_name = minetest.create_form( meta, player_name, formspec, on_close )
    • meta - an optional session-based state-table (can be nil)
    • player_name - a valid player name
    • formspec - a standard formspec string
    • on_close - a callback function to be invoked after an event or signal
    • form_name - the hashed session token (guaranteed to be unique)
The formspec can subsequently be re-opened or closed using the following methods:

    minetest.update_form( player_name, formspec )
    • player_name - a valid player name
    • formspec - a standard formspec string
    minetest.destroy_form( player_name )
    • player_name - a valid player name
The associated on_close( ) callback function will be automatically invoked with three arguments during a formspec event or signal, including closure. The return value is currently ignored, but in the future it may allow for additional behavior.

    on_close( meta, player, fields )
    • meta - the session-based state table
    • player - the player object
    • fields - the table of submitted fields
In the event of abnormal formspec termination, the callback function will receive a signal indicating the specific condition that led to the closure. One of the following values will be stored in the "quit" field:

  • minetest.FORMSPEC_SIGEXIT (player clicked exit button or pressed esc key)
  • minetest.FORMSPEC_SIGQUIT (player logged off)
  • minetest.FORMSPEC_SIGKILL (player was killed)
  • minetest.FORMSPEC_SIGTERM (server is shutting down)
  • minetest.FORMSPEC_SIGPROC (programmatic closure)
  • minetest.FORMSPEC_SIGTIME (timeout reached)
Formspecs are also supported within a node definition. If an on_open( ) method is defined, then it will be invoked whenever the player right-clicks the node. The return value must be a standard formspec string, or nil to abort.

    nodedef.on_open( pos, player, fields )
    • pos - the position of the node
    • player - the player object
The on_close( ) method is invoked during any formspec event or signal. If defined, it will receive three arguments. Currently, the return value is ignored, but in the future it may allow for additional behavior.

    nodedef.on_close( pos, player, fields )
    • pos - the position of the node (or your own state table)
    • player - the player object
    • fields - the table of submitted fields
The "hidden" formspec element of allows for a slightly different workflow, by presetting the state table from the formspec string itself. The key-value pairs are used only for initialization and will never be transmitted to the client. An optional data type can be specified within the element as follows:

    hidden[name;value;string]
    hidden[name;value;boolean]
    hidden[name;value;number]
    hidden[name;value]
If the data type is not specified, then no type-conversion will occur. Here are some examples:

Code: Select all
hidden[pos_x;-240;number]"
hidden[wool_color;darkgreen;string]"
hidden[is_dead;true;boolean]"

Through the use of form timers, it possible to re-open a formspec at regular intervals, or even to close the formspec after a specified period. There is no need for complex chains of minetest.after( ) or additional globalstep registrations. A form timer persists for the duration of a form session.

    timer = minetest.get_form_timer( player_name, form_name )
    Returns a form timer object associated with the form session of a given player.
Three methods are exposed by the form timer object, providing similar behavior to node timers.

    timer.start( timeout )
    Starts a timer with the given timeout in seconds.

    timer.stop( )
    Cancels a running timer.

    timer.get_state( )
    Returns a table with information about the running timer including

    • elapsed - the number of seconds since the timer started
    • remain - the number of seconds until the timer expires
    • overrun - the number of milliseconds overrun for this period (only valid within an on_close callback)
    • counter - the number of periods that have accrued
The associated on_close( ) callback function will be notified of timer expiration via a SIG_TIME signal.

To better understand this methodology, here are some working code examples. Let's say that we want to register an "uptime" chat command that displays the current server uptime to any player, with an option to automatically refresh the formspec each second.

Code: Select all
minetest.register_chatcommand( "uptime", {
        description = "View the uptime of the server interactively",
        func = function( player_name, param )
                local is_refresh = true

                local get_formspec = function( )
                        local uptime = minetest.get_server_uptime( )

                        local formspec = "size[4,2]"
                                .. default.gui_bg_img
                                .. string.format( "label[0.5,0.5;%s %d secs]",
                                        minetest.colorize( "#FFFF00", "Server Uptime:" ), uptime
                                )
                                .. "checkbox[0.5,1;is_refresh;Auto Refresh;" .. tostring( is_refresh ) .. "]"
                        return formspec
                end
                local on_close = function( meta, player, fields )
                        if fields.quit == minetest.FORMSPEC_SIGTIME then
                                minetest.update_form( player_name, get_formspec( ) )

                        elseif fields.is_refresh then
                                is_refresh = fields.is_refresh == "true"
                                if is_refresh == true then
                                        minetest.get_form_timer( player_name ).start( 1 )
                                else
                                        minetest.get_form_timer( player_name ).stop( )
                                end
                        end
                end

                minetest.create_form( nil, player_name, get_formspec( ), on_close )
                minetest.get_form_timer( player_name ).start( 1 )
        end
} )


Of course, we could implement similar functionality without the need for a chat command. Perhaps we want to display the server uptime only when a privileged player clicks on a Nyan Cat while limiting the total number of refreshes to ten.

Code: Select all
minetest.register_privilege( "uptime", "View the uptime of the server interactively" )

local get_nyancat_formspec = function( meta )
        local uptime = minetest.get_server_uptime( )
        local formspec = "size[4,3]"
                .. default.gui_bg_img
                .. string.format( "label[0.5,0.5;%s %0.1f %s]",
                        minetest.colorize( "#FFFF00", "Server Uptime:" ),
                        meta.is_minutes == true and uptime / 60 or uptime,
                        meta.is_minutes == true and "mins" or "secs"
                )
                .. "checkbox[0.5,1;is_minutes;Show Minutes;" .. tostring( meta.is_minutes ) .. "]"
                .. "button[0.5,2;2.5,1;update;Refresh]"
                .. "hidden[view_count;1;number]"
                .. "hidden[view_limit;10;number]"               -- limit the number of refreshes!
        return formspec
end

minetest.override_item( "nyancat:nyancat", {
        description = "System Monitor",

        on_open = function( meta, player )
                local player_name = player:get_player_name( )

                if meta.is_minutes == nil then meta.is_minutes = true end

                if minetest.check_player_privs( player, "uptime" ) then
                        return get_nyancat_formspec( meta )
                else
                        minetest.chat_send_player( player_name, "Your privileges are insufficient." )
                end
        end,
        on_close = function( meta, player, fields )
                local player_name = player:get_player_name( )

                if not minetest.check_player_privs( player, "uptime" ) then return end

                if fields.update then
                        if meta.view_count == meta.view_limit then
                                minetest.destroy_form( player_name )
                                minetest.chat_send_player( player_name, "You've exceeded the refresh limit." )
                        else
                                meta.view_count = meta.view_count + 1
                                minetest.update_form( player_name, get_nyancat_formspec( meta ) )
                        end

                elseif fields.is_minutes then
                        meta.is_minutes = fields.is_minutes == "true"
                        minetest.update_form( player_name, get_nyancat_formspec( meta ) )
                end
        end
} )
Last edited by sorcerykid on Tue Feb 27, 2018 21:40, edited 6 times in total.
 

sofar
Developer
 
Posts: 1653
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Mod] ActiveFormspecs [formspecs]

by sofar » Sat Jan 06, 2018 07:22

I'm sure that many people would appreciate some actual revision control, a mod.conf at minimum (to avoid the whole directory naming problem) and perhaps a description.txt, but, since it's not something easily enabled, I'm sure that it will give people significant installation problems. The fact that this has to be inserted into `defaults` dependencies make it more powerful, but some subgames may not even have a `default` mod, and it will still remain fragile because a mod named `abacus` will load before `default` if it doesn't depend on it. So, I understand why you did this, but it will make adoption difficult and have hard to find problems.

I don't like the fact that the formnames are really predictable. You might as well send the same formspec name every time. But this code never even checks them either, and I think it really should, after all any discrepancy in the formname is a sign of tampering. It's a little bit like allowing a burglar into your house just because he had a key, and ignoring the ski mask on his head and crowbar in his hand :)

Introducing a new formspec keyword 'hidden' is an easy route for abuse, since it will get sent verbatim to the client, and is therefore completely not hidden at all. Since an attacker now knows some of the values in the meta storage, the attacker can be even more effective at exploiting any vulnerabilities in the callback handler.

FWIW layering some new API on top of the node formspecs isn't a bad idea, but I think it's unlikely that it will prevent any attack vector at all since there the design of node formspecs don't allow for any effective defense against them. Providing an extended API for those might make people think that they get some security, while it's actually not the case. I would currently advocate for not using node formspecs at all if you care about security - if you don't mind much you won't need a new API either, but maybe that's just me. If you are going to try and improve things, you'll need to come up with a lot of checks to perform to prevent abuse.

Sorry if this sounds very negative, there's definitely nothing really bad about this code, and it'll help with the code flow of formspecs for sure, I just can't not comment on the things that I would like to see improved with it.
 

User avatar
rnd
Member
 
Posts: 200
Joined: Sun Dec 28, 2014 12:24
IRC: ac_minetest
In-game: rnd

Re: [Mod] ActiveFormspecs [formspecs]

by rnd » Sat Jan 06, 2018 09:11

1. hidden[...] is good idea, basically adds server side form data instead of adding it in form names,
maybe name "data[...]" or 'formdata[...]" would be better description? But yes, if complete form string is sent to
Code: Select all
minetest.show_formspec( player:get_player_name( ), form.name, formspec )

client, then not so much hidden :D.. would be if you stripped it from formstring before sending it to client..
All form data is then actually saved in minetest.forms[ player:get_player_name( ) ].

2. Misleading descriptions 'Secure Session Tracking'
Attacker can still trigger form.proc since there are no checks for formname, everything is happily accepted without question. You never really do that 'context checks' properly ( attacker could fake form request but would not do any 'close' ). Only 'secure' thing is attacker can't 'easily' forge form.meta form data (here we assume that you fixed 'hidden' in 1.), but that needs to be done separately by 'modder' inside form.proc, is NOT integrated in this mod.
While this is good for abuses attacker still has some 'door left open'.

Also attacker can monitor how many forms were opened so far by simply observing form name
Code: Select all
minetest.form_id = minetest.form_id + 1
   form.name = player:get_player_name( ) .. ":" .. string.format( "%#06x", minetest.form_id )

giving him some insight whats going on on server.

Code: Select all
minetest.register_on_player_receive_fields( function( player, formname, fields )
   local form = minetest.forms[ player:get_player_name( ) ]

   if form then
      -- perhaps we should merge meta into fields table?
      form.proc( form.meta, player, fields )

      if fields.quit then
         minetest.forms[ player:get_player_name( ) ] = nil
      end
   end
end )


3. From 'security' view sofar's 'secure forms' version is adequate, this not so much.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Sat Jan 06, 2018 10:08

sofar wrote:Sorry if this sounds very negative, there's definitely nothing really bad about this code, and it'll help with the code flow of formspecs for sure, I just can't not comment on the things that I would like to see improved with it.


Thanks for your feedback. I'll try my best to respond to your concerns and questions.

The dependency in default is not a requirement, it is only a recommendation since some users might be compelled to update the default mod. It not, then ActiveFormspecs can be installed in a typical fashion. I will, of course, include description.txt and mod.conf files as suggested.

Form names are never used for flow control, so their predictability cannot be exploited. I could check them, nonetheless, but then my assertion in the post above that form names are deprecated and no longer used would be false. Any attempt to tamper with the name would be futile since ActiveFormspecs can only execute a callback procedure that is assigned to that user during an existing formspec session, which is maintained exclusively server-side in a key-value store. I can look into validating form names, if that is something that gives server operators more peace of mind. But like I said, formspec names need to be abandoned altogether because they offer no security.

A proper warning about the hidden formspec element will be added to the post above, so that mod athors are aware that such data is exposed to the client. Keep in mind, however, that in the next version of ActiveFormspecs, I am planning to altogether strip such elements from the string just prior to it being sent to the client.

To be clear, "node formspecs" are not supported. In fact, the very impetus for this framework was to overcome the inherent security flaws of node formspecs with a superior API. On the JT2 sever, for example, node formspecs were completely removed from the subgame and replaced with ActiveFormspecs-compliant callbacks -- this includes the default unlocked chests, bookshelves, furnaces, vessels shelves, etc. When maikerumine transferred ownership of the server to me, formspec security was my first goal. It's no coincidence that version 1.0 of ActiveFormspecs is dated December 15, 2016, exactly one week prior the relaunch of JT2.

What ActiveFormspecs does provide is an extension to the node definition for specialized callbacks. This is accomplished behind the scenes by hooking into the minetest.register_node( ) method. In reality, it is no different than a mod author defining their own on_rightclick( ) callback which subsequently invokes minetest.create_form( ) with an embedded callback function. I am merely offering mod authors a shorthand for this existing workflow, one that I think helps to better compartmentalize such repetitive logic.

Since this is the first public release, improvements can and should be expected. My hope is to fix any bugs as they are brought to my attention. So I'm all ears if something needs to be fixed :)
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Sat Jan 06, 2018 10:55

rnd wrote:1. hidden[...] is good idea, basically adds server side form data instead of adding it in form names,
maybe name "data[...]" or 'formdata[...]" would be better description? But yes, if complete form string is sent to


Thanks for the input. Indeed, we are thinking the same in this respect. See my reply above. I am planning to strip the hidden elements.

2. Misleading descriptions 'Secure Session Tracking'
Attacker can still trigger form.proc since there are no checks for formname, everything is happily accepted without question. You never really do that 'context checks' properly ( attacker could fake form request but would not do any 'close' ). Only 'secure' thing is attacker can't 'easily' forge form.meta form data (here we assume that you fixed 'hidden' in 1.), but that needs to be done separately by 'modder' inside form.proc, is NOT integrated in this mod.
While this is good for abuses attacker still has some 'door left open'.


This is not entirely true. It is impossible to "fake a form request" since session tracking in ActiveFormspecs is never contingent on any tainted (user-supplied) data. The best a would-be attacker can do is submit a form for an existing formspec session, which is the expected logic anyway. Once a new form is requested, the previous session is automatically voided and cannot be reused even by that same player. Hence session tracking is secure. Form names, in stark contrast, are not secure.

Also attacker can monitor how many forms were opened so far by simply observing form name


I don't regard this as a vulnerability, since monitoring the number of formspec sessions doesn't offer any advantage. I can readily view a server's uptime and max_lag by typing /status, without a hacked client. Then, I can obtain the login id of the server operator with the /admin command. Then, I can visit servers.minetest.net to review a complete list of installed mods on that sever as well as the engine version and subgame name. In the case of some prominent servers, I can even even download the full source code from GitHub! Dare I say, there is already an overwhelming amount of information for would-be attackers to gain keen insight about the internal operations of a server. :P
 

sofar
Developer
 
Posts: 1653
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Mod] ActiveFormspecs [formspecs]

by sofar » Sat Jan 06, 2018 18:14

Form names, in stark contrast, are not secure.


Having them, and checking them, is more secure than not having them, since it makes it more difficult for an attacker.

Having predictable form names, as you implemented it, and then checking them, does nothing for complexity. The way `fsc` implements it, the formnames are unpredictable and with enough entropy that they are incredibly hard to predict. This does increase the attack complexity, and therefore, should be a desirable practice.

It's for a good reason that "Attack Complexity" is part of CVSS scoring. (https://www.first.org/cvss/user-guide#2 ... Complexity)

It's maybe over the top what fsc does, but, omitting a cheap check is in my opinion a mistake.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Fri Jan 12, 2018 17:09

cVersion 2.1 Released

A new version of ActiveFormspecs is ready for download. Here is a complete change log:

  • Various code refactoring and better comments
  • Introduced password hashing of form names
  • Improved sanity checks during form submission
  • Fully reworked parsing of hidden elements
  • Ensured hidden elements are always stripped
  • Gave hidden elements default-state behavior
  • Localized old node registration functions
  • Included player object within form table
  • added signal handling on formspec termination
  • added support for callbacks in node overrides
In addition, I tied up a few loose ends by including a description.txt and a mod.conf with the distribution.

Below are some important release notes regarding functional changes that could impact existing users:

    Password hashing of form names

    Form names are now hashed via Minetest's builtin password function using a random seed. While it's not Fort Knox style security, it should provide a satisfactory degree of intrusion detection and prevention. The likelihood of predicting any form name from the client-side using this new method is effectively nil.

    Callback support in node overrides

    Attached formspecs were previously supported only during node registration. I've included a hook into minetest.override_item( ), so both methods are now possible.

    Code: Select all
    minetest.override_item( "default:bookshelf", {
            on_open = function( pos, player )
                    return new_bookshelf_formspec
            end,
            on_close = function( ) end,
    } )


    As usual, you can define both on_open( ) and on_close( ) callbacks for use with a node.

    Formspec termination signals

    Callbacks are now notified of formspec termination via signals. Whenever a signal is received, you can perform cleanup and exit gracefully. One of the following values will be stored in the "quit" field:

    • minetest.FORMSPEC_SIGEXIT (player clicked exit button or pressed esc key)
    • minetest.FORMSPEC_SIGQUIT (player logged off)
    • minetest.FORMSPEC_SIGKILL (player was killed)
    • minetest.FORMSPEC_SIGTERM (server is shutting down)
    • minetest.FORMSPEC_SIGCALL (closure requested)
    • minetest.FORMSPEC_SIGTIME (timeout reached)
    The normal termination signal is minetest.FORMSPEC_SIGEXIT, which equates to true for backward compatibility. All other signals indicate an abnormal condition.

    Data types of hidden elements

    My original implementation for hidden elements was subpar at best. Now the parser is better equipped to handle a variety of data types including booleans, numbers, and strings. To specify a data type, simply add a third attribute within the element as follows:

      hidden[name;value;string]
      hidden[name;value;boolean]
      hidden[name;value;number]
      hidden[name;value]
    If a data type is not specified, then no type-conversion will occur. Here are some examples:

    Code: Select all
    hidden[pos_x;-240;number]"
    hidden[wool_color;darkgreen;string]"
    hidden[is_dead;true;boolean]"


    Hidden elements serve only as default, initial values for the state table (and thus are never transmitted to the client). This can be useful in situations where your formspec strings are generated beforehand. To override any of the presets, just include the corresponding key-value pair within the state table that is passed to minetest.create_form( )

All of the changes documented above have been rigorously tested on both my test server and my production server over the past week. There are no known bugs, but if you encounter a problem, then please notify me. Thanks!
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Sat Jan 20, 2018 05:53

cVersion 2.2 Released

A new version of ActiveFormspecs is available for download. Here is the change log:

  • Introduced player-name arguments for all API calls
  • Added signal for programmatic formspec closure
  • Ensured callbacks are notified of session resets
  • Renamed some local variables to improve clarity
I also included working examples with full source code, all of which can be found in samples.lua.

This release isn't nearly as extensive as Version 2.1, but there are still a couple notable changes worth mentioning.

  1. I discovered that calls to minetest.create_form( ) would clobber existing sessions without warning, thereby disobeying the signal handling framework devised in the previous version. This is no longer an issue, as on_close( ) callbacks will now be notified via a SIGPROC signal in the event of such a termination.

  2. All the API functions of ActiveFormspecs now accept player names as arguments. Player objects will continue to be supported up until Version 3.0, at which point they will be deprecated. The phasing out of player objects has been on my roadmap for awhile, since it is consistent with the conventions of the Minetest API.

These latest updates have been tested on my server. But if you experience any problems, please feel free to report them here.
 

sofar
Developer
 
Posts: 1653
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Mod] ActiveFormspecs [formspecs]

by sofar » Sat Jan 20, 2018 06:30

You really need to learn how to publish code in git, svn or whatever you're comfortable in.

That way I can see your individual changes. Now I can only see the `diff` between your last version and your new version. It's almost impossible to review this code without re-reading the entire mod over from top to bottom. With git, you can see, when done properly, each individual change set with, when done properly, a coherent explanation as the commit message.

Just dumping the final tarbals on a http server was out of fashion in 1980.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Thu Jan 25, 2018 02:07

Version 3.0a Unreleased

The past few weeks I've been working on a highly experimental branch of ActiveFormspecs.

https://bitbucket.org/sorcerykid/formsp ... ch/dev-3.0

Version 3.0 is a radical departure from previous releases. It incorporates numerous compatibility-breaking changes including a completely object-oriented API, independent timers, benchmarking hooks, and a realtime session monitor. In contrast to the linear, top-down approach of traditional formspecs, the newly structured framework affords a greater degree of flexibility to mod authors. In fact, one of my long-term goals is to extend ActiveFormspecs with cooperative multitasking support for embedded applications.

So there are indeed some big plans in the works! But for starters, here's a quick run-down of the key features in this alpha-version:

1. Form Session Objects

With ActiveFormspecs 3.0, form sessions are now entirely self-contained objects with their own methods and properties. For this reason, the functions minetest,create_form( ) and minetest.update_form( ) and minetest.destroy_form( ) have been deprecated. Instead, the following constructor function will return a reference to a new FormSession object:

    fs = FormSession( meta, player_name, on_open, on_close )
Properties specific to the form session can be obtained directly from the following fields:

  • fs.id - the session id
  • fs.meta - the state table
  • fs.player - the player object
  • fs.name - the hashed session token
Methods for updating and destroying and validating the form session are also available:

    fs.destroy( )
    Invokes the on_close( ) callback, closes the formspec, and ends the form session

    fs.update( )
    Invokes the on_open( ) callback and re-opens the formspec

    fs.get_idletime( )
    Returns the elapsed time since the callbacks were triggered

    fs.get_lifetime( )
    Returns the elapsed time since the form session was started

    fs.is_active( )
    Returns true if the form session is still active, otherwise false
The destroy( ) and update( ) methods have no effect unless the form session is still active. In addition, these methods should never be called from within an on_open( ) callback, since the form session is in a volatile state.

2. Callbacks Revisited

Attached formspecs now have three additional callbacks at their disposal, each of which is optional. Used together, however, they can provide a much cleaner and simpler workflow.

The can_open( ) callback is triggered when a player right-clicks the node. A return value of true is required to open the formspec and start the form session. Otherwise no further callbacks will be triggered.

    can_open = nodedef.can_open( pos, player )
    • pos - the position of the node
    • player - a player object
The before_open( ) callback is triggered once, immediately before the first on_open( ) callback, allowing the state table to be initialized. By default, the state table will contain only the fields pos and node.

    meta = nodedef.before_open( pos, node, player )
    • pos - the position of the node
    • node - the node
    • player - the player object
The after_open( ) callback is triggered immediately after the formspec is opened and the form session is started. The state table will therefore already be populated with values from the hidden fields. This is an ideal place to start a timer (see below).

    nodedef.after_open( fs )
    • fs - the form session object

The on_close( ) callback is still triggered during a formspec event or signal. However, the parameters have changed:

    nodedef.on_close( fs, fields )
    • fs - the form session object
    • fields - the table of submitted fields
This difference also applies to the on_close( ) callback of detached formspecs.

3. Working with Timers

Timers are managed natively within ActiveFormspecs 3.0. There is no need for complex chains of minetest.after( ) or additional globalstep registrations. Similar to node timers, every form session has its own independent timer that persists for the duration of that same form session. This makes it possible to update a formspec at regular intervals, or even to destroy the formspec after a specified period.

    fs.start_timer( timeout , periods)
    Starts a timer with the given timeout in seconds, repeating for an optional number of periods

    fs.stop_timer( )
    Cancels a running timer.

    fs.reset_timer( )
    Restarts a running timer.

    fs.get_timer_state( )
    Returns a table with information about the running timer including
    • elapsed - the number of seconds since the timer started
    • remain - the number of seconds until the timer expires
    • overrun - the number of milliseconds overrun for this period (only valid within an on_close callback)
    • counter - the number of periods that have accrued
The associated on_close( ) callback will be notified of timer expiration via a SIG_TIME signal. The callback can then stop or restart the timer, as well as perform other necessary tasks.

4. Benchmarking Basics

Formspecs have the potential to consume disproportionate amounts of CPU time. Clocking the creation and event handling of individual form sessions is nearly impossible with the builtin mod_profiler. However, ActiveFormspecs 3.0 provides benchmarking hooks specifically for this purpose. These can be enabled in minetest.conf:

Code: Select all
enable_formspecs_benchmarking = true

Alternatively, you can issue a /set chat command while in game:

Code: Select all
/set -n enable_formspecs_benchmarking = true

To clock the execution of a single formspec session, simply call the get_proctime( ) method.

    fs.get_proctime( )
    Returns a table with total CPU times for each respective callback
    • on_open - the cumulative execution time in seconds for the on_open( ) callback
    • on_close - the cumulative execution time in seconds for the on_close( ) callback

Eventually, I plan to integrate the Polygraph mod with these benchmarking hooks so that a variety of performance metrics can be logged and charted on the fly.

5. Monitoring Made Easy

Perhaps the most noticeable feature of ActiveFormspecs 3.0 is the interactive form session monitor. The /fs chat command, displays a realtime summary of form sessions along with navigation buttons and related statistics.

Image

The summary is sorted chronologically and divided into seven columns per player

  • player - the name of the player viewing the formspec
  • origin - the mod (or node, if attached) that created the formspec
  • counter - the number of updates to the formspec
  • timeout - the timeout in seconds of the form session timer
  • proctime- the total CPU time in seconds for the formspec callbacks
  • idletime - the elapsed time since a formspec event or signal
  • lifetime - the elapsed time of the form session
Clicking the 'x' button beside any player will immediately terminate their form session, so use with caution.

6. Closing Thoughts

I will publish change logs here with links to the repository as work continues on this project. This is an alpha version and is made available for testing purposes only. So until the full release, all of features documented above are susceptible to change without notice. I make no guarantees about stability. Of course, I am always open to suggestions and feedback going forward. Thanks!
 

Sokomine
Member
 
Posts: 3616
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: [Mod] ActiveFormspecs [formspecs]

by Sokomine » Sun Jan 28, 2018 01:45

You have created many nice formspecs and shown screenshots of them. And this is an api for formspecs. I wonder if you could create something that could help create stories/adventures in the game. Something along the lines of the old point&click adventures where you would click on a mob/npc, "read" what it has to say, and select your answer to the mob's question from usually 1-3 options. The trouble with that is that the text would be provided by whoever writes the story; the formspec would have to deal with it and present the text in a readable way that doesn't look too terrible.
A list of my mods can be found here.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Mon Jan 29, 2018 23:39

Formatting of user-input is one of the biggest of obstacles of almost any UI design. But with formspecs, it's extraordinarily difficult since there is little to no control over any aspect of text layout (i.e. kerning, line-height, margins, alignment, etc. are all at the discretion of the engine) The inventory grid further adds to the complexity, and effectively undermines any chance of portability.

What you've described is a perfectly reasonable design goal. But I'm not sure how it could be implemented without resorting to some type of kludge. I don't think that Minetest's formspecs are fit enough (yet) for applications that require visual consistency. Perhaps an approximation of formatting could be achieved by hacking the core.wrap_text( ) routine:

https://github.com/minetest/minetest/bl ... s.lua#L311
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Tue Jan 30, 2018 07:41

Version 2.3 Released

A new version of ActiveFormspecs is available for download. Here is the change log:

  • Corrected erroneous value of formspec exit signal
  • Removed two experimental form session methods
  • Included timestamp and origin within form table
  • Added form session validation on destroy and update
  • Introduced form timers with start and stop methods
  • Created routine to notify callbacks of timeout
  • Added support for internal statistical tracking
  • Added chat command to view form session summary

I backported several features from the dev-3.0 branch. Of course, this meant removing object-oriented constructs for consistency with the original framework while maintaining as much functionality as possible.

  1. Form timers are made available through the minetest.get_form_timer( ) function:

      timer = minetest.get_form_timer( player )
    Two methods are exposed by the timer object. Although similar to node timers, take note of the dot syntax.

      timer.start( timeout )
      Starts a timer with the given timeout in seconds.

      timer.stop( )
      Cancels a running timer.
    The associated on_close( ) callback will be notified of timer expiration via a SIG_TIME signal.

  2. The interactive form session monitor can be accessed via the /fs chat command (requires "server" privilege).

    Image

    A realtime summary of form sessions is displayed along with navigation buttons and related statistics.

    • player - the name of the player viewing the formspec
    • origin - the mod (or node, if attached) that created the formspec
    • idletime - the elapsed time since a formspec event or signal
    • lifetime - the elapsed since the formspec was first opened
    To close the formspec of any player, just click the corresponding 'x' button (use with discretion, of course).
 

Sokomine
Member
 
Posts: 3616
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: [Mod] ActiveFormspecs [formspecs]

by Sokomine » Wed Jan 31, 2018 01:43

sorcerykid wrote:What you've described is a perfectly reasonable design goal. But I'm not sure how it could be implemented without resorting to some type of kludge. I don't think that Minetest's formspecs are fit enough (yet) for applications that require visual consistency. Perhaps an approximation of formatting could be achieved by hacking the core.wrap_text( ) routine:

Your screenshots of formspecs look so good that I was hoping you'd found a way around this old problem. Seems it was "just" hard work.
core.wrap_text alone might not be sufficient. There are diffrent font sizes to consider (also horizontally spaced) and letters of diffrent width. Not that other programs havn't solved that yet...we just don't have them available in MT :-(.
A list of my mods can be found here.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Thu Feb 01, 2018 03:24

Thanks, I'm afraid it's just some good old fashioned trial and error combined with a helpful dose of perfectionism :D

When I was younger I did read a lot about software usability (particularly GUIs). In fairness, that combined with my love of graphic design has undoubtedly offered a lot of inspiration as well.

I'm hoping that we will some "upgrades" to formspecs in the coming months, but progress appears to be very slow on that front. So I'm not holding my breath. I think we'll have to make do with what we've got for the time being :-/
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Fri Feb 02, 2018 17:13

I've now included code samples that are Version 3.0a compatible. They can both be found in the samples.lua file.

  • Sample 1 shows how to create an attached formspec within a node override. It features a session-based timer, signal handling for auto-refresh and auto-close, state table initialization from hidden formspec elements, after_open( ) and before_open( ) callbacks, event processing of two checkboxes, and basic access control.

  • Sample 2 shows how to open a formspec from a chat command. It features a session based state table, signal handling for auto refresh, in-line state table initialization, and event processing of a single checkbox.

If you have any technical questions or need clarification about the code samples, feel free to ask.

https://bitbucket.org/sorcerykid/formsp ... ew-default
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Mon Feb 19, 2018 23:26

Version 2.4 Released

A new version of ActiveFormspecs is available for download. Here is the change log:

  • Various code refactoring and better comments
  • Full rewrite of timer queue for higher precision
  • Added method to retrieve state of form timers
To improve the accuracy and the precision of form timers, I implemented a high-resolution monotonic clock with graceful 32-bit overflow. Fractional timer intervals are now supported. I rigorously benchmarked the timer queue under various conditions to ensure optimal performance (achieving a consistent baseline overhead of 0.003698 to 0.004306 seconds in a five minute period with no active timers).

I also added a third method to the timer object, which is backported from the dev-3.0 branch.

    timer.get_state( )
    Returns a table with information about the running timer including

    • elapsed - the number of seconds since the timer started
    • remain - the number of seconds until the timer expires
    • overrun - the number of milliseconds overrun for this period
    • counter - the number of periods that have accrued

All of these values are calculated with realtime accuracy and millisecond precision. However, it is important to note that overrun is valid only within an on_close( ) callback, otherwise it will be zero.
 

sofar
Developer
 
Posts: 1653
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Mod] ActiveFormspecs [formspecs]

by sofar » Tue Feb 20, 2018 01:10

This already exists:

https://github.com/minetest-mods/timer

But, from the link I can't even find any mentions of a `timer` class or even a `get_state` method.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Tue Feb 20, 2018 01:43

There are no dependencies on other mods, so I'm not sure what that link is referring to.

The first post contains usage instructions for ActiveFormspecs v2.4.
 

sofar
Developer
 
Posts: 1653
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Mod] ActiveFormspecs [formspecs]

by sofar » Tue Feb 20, 2018 03:51

Your forum post mentions a timer object.

timer.get_state( )


However, in none of the brances of the formspec mod can I find any mention of this timer object. Is this a fictious object, or did you implement this?
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Tue Feb 27, 2018 21:55

Quick announcement! The code samples in the first post are now up to date. All changes are reflected in the samples.lua file as well.

https://bitbucket.org/sorcerykid/formsp ... amples.lua

I also expanded the documentation, and added two missing functions. If there is anything else I overlooked, feel free to point it out.
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Thu Mar 01, 2018 00:06

I am planning to make a couple breaking changes to the 3.0 branch this weekend. The most important is the constructor function:

    fs = FormSession( meta, player_name, formspec, on_close )
The third parameter will be replaced by a formspec string. This is intended to simplify the internal logic of the constructor, but also to allow more flexibility in the passing of formspec strings without requiring an intermediary callback. I explored several different workflows, but I believe this is the most feasible and efficient

The rationale is that a form session object is intended to represent an active form session. The on_open callback, however, must be invoked prior to the completion of the constructor itself, which fundamentally violates this model.

For attached formspecs, the on_open callback in the node definition will be renamed to get_formspec to better reflect its purpose:

    nodedef.get_formspec( meta, player )
Of course, I welcome feedback. If there is a specific use case in which a callback is preferable, I'll gladly reconsider. Thanks!
 

User avatar
sorcerykid
Member
 
Posts: 771
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: [Mod] ActiveFormspecs [formspecs]

by sorcerykid » Sun Aug 12, 2018 16:48

Hi, all! Since I've been having problems with the BitBucket service lately, I've mirrored this repo to Notabug and MeseHub. Eventually I plan to migrate to one of these two hosts. I just haven't decided which one yet

The latest release can also be downloaded from the Minetest ContentDB:

https://content.minetest.net/packages/s ... formspecs/

I believe that Notabug syncs git repos automatically, so the latest commits should appear there besides Bitbucket. I'll be sure to add these links to the first post as well.
 


Return to WIP Mods



Who is online

Users browsing this forum: Krock and 14 guests