[Mod] Pluggable Helpers (auto-dependency API) [plugins]

Post Reply
User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

[Mod] Pluggable Helpers (auto-dependency API) [plugins]

by sorcerykid » Post

Pluggable Helpers v1.0
plugins (by sorcerykid)

Today I am proud to announce the initial beta release of Pluggable Helpers with the goal of encouraging the cooperative development of helper functions in Minetest mods and games. Pluggable Helpers automates the entire process from downloading to installation, including management of dependencies, within a distributed architecture backed by the BitBucket Cloud for secure version control.

I developed this mod in order to address several of the concerns that were raised here and here.

Image

Repository:

https://bitbucket.org/sorcerykid/plugins

Download Archive (.zip)
Download Archive (.tar.gz)

Dependencies:

None.

Source Code License:

The MIT License

Installation:
  1. Unzip the archive into the mods directory of your game.
  2. Rename the markup-master directory to "plugins".
  3. Add "plugins" as a dependency to any mods using the API.
Overview

Modularity is extremely important when it comes to maintaining large scale code-bases such as games and mods in Minetest. Helper classes and methods and even libraries serve this purpose. But so often they are re-implemented over-and-over again since nobody wants to rely on external dependencies in their mods and games. Eventually some helper functions may be integrated into the engine, but even that is often a lengthy review process with its own share of obstacles and pitfalls.

This is where Pluggable Helpers comes into the picture.
  • "The core philosophy of Pluggable Helpers is to empower the community to create an evolving game-development API through the use of a jointly maintained public repository of helper classes, methods, and libraries that can be downloaded and installed on-the-fly with no intervention required by the end-user."
Although the Pluggable Helpers mod has no dependencies in and of itself, it does need to perform occasional HTTP requests. Therefore, it must be added to the list of secure_http_mods in minetest.conf.

The remainder of this document pertains to mod and game developers only. End-users should never need to be concerned with these implementation details. The Pluggable Helpers mod is literally plug and play.

Getting Started

Let's begin with a simple helper definition. We'll define a new "math.square" helper method, that returns the square of any given number:

Code: Select all

author = "sorcerykid"
version = "1.0"
license = "MIT"
depends = { }
imports = { }

return function ( v )
	return v ^ 2
end
After some basic unit-testing, I then login to the Management Console at plugins.mytuner.net:30005 with my Content DB username and submit the helper for review into the Bitbucket VCS. Assuming the helper is approved, it will be converted to a helper definition and published to the the Remote Repository.

Image

Now, on my local machine, I can easily incorporate this helper function into my mod as follows:

Code: Select all

plugins.require( "math.square" )
print( math.square( 2 ) )
That's literally all there is to it! The helper will be downloaded via HTTP from the remote repository at plugins.mytuner.net and installed within my local repository at server startup. If there were any dependencies, all of them would be downloaded and installed as well.

On subsequent server startups, of course, the helper will be loaded directly from my local repository, so there is no requirement to even be connected to the Internet. The entire process is very efficient, almost imperceptible from an end-user perspective.

Supported Helper Types

There are three types of helpers available: methods, classes, and libraries. Each is intended for a specific purpose:
  • Helper Methods
    A helper method extends one of the builtin global libraries, whether it be "minetest", "string", "math", "table", etc. with an additional custom method. The earlier example of "math.square" would be considered a helper method since it extends the global math library. Once a helper method is required within any mod, then it is made available to all other mods. If another mod requires the same helper method, then it will not be loaded twice.

    A helper method must be named in snake_case notation. Only lowercase letters, underscores, and numerals are permitted.

    Helper Classes
    A helper class is a class constructor that returns a new instance of an object with its own methods and properties. It is usually stateful in natture, and may even be derived from a base class. Once a helper class is required within a mod, then will be available to all other mods just as with helper methods.

    A helper class must be named in CamelCase notation. Only letters and numerals are permitted.

    This is an example of a helper class definition called "StaticSet", for easily working with immutable membership sets.

    Code: Select all

    author = "sorcerykid"
    version = "1.0"
    license = "MIT"
    depends = { }
    imports = { }
    return function ( list )
            local data = { }
            local self = { length = 0 }
    
            for i, v in ipairs( list ) do
                    data[ v ] = true
            end
    
            self.length = #list
            self.exists = function ( elem )
                    return data[ elem ] ~= nil
            end
    end
    
    Now, adding a membership set to a mod is as simple as a few lines of code.

    Code: Select all

    plugins.require( "StaticSet" )
    
    local ranks = StaticSet { "basic", "staff", "admin", "owner" }
    
    print( ranks.exists( "basic" ) )
    
    Helper Libraries
    Helper libraries are a special type of persistent helper class that is instantiated at server startup. They can serve a dual purpose of either being a barebones library for consolidating related helper methods or else a fully-fledged module that encapsulates various public and private methods and properties. Unlike helper methods and classes, libraries are always included rather than required.

    A helper library must be named in CamelCase notation. Only letters and numerals are permitted.

    This is an example helper library called "DebugLog" that provides several methods for posting messages to the Minetest debug log, including statistics of how many warnings were posted.

    Code: Select all

    author = "sorcerykid"
    version = "1.0"
    license = "MIT"
    depends = { }
    imports = { }
    prototype = ""
    
    return function ( this )
    	local warning_count = 0
    
    	this.post_warning = function ( message )
    		minetest.log( "warning", message )
    		warning_count = warning_count + 1
    	end
    	this.post_action = function ( message )
    		minetest.log( "action", message )
    	end
    	this.get_warning_count = function ( )
    		return warning_count
    	end
    end
    
    Placing this code into a mod and running Minetest will print "Total warnings: 2" to the console:

    Code: Select all

    local logger = plugins.include( "DebugLog" )
    
    logger.post_warning( "You made a boo boo!" )
    logger.post_action( "We're doing something now" ) 
    logger.post_warning( "Oops, I did it again!" )
    
    print( "Total warnings: " .. logger.get_warning_count( ) )
    
In all three cases above, if the helper definition is not found within the local repository, then it will be downloaded and installed to the local repository automatically. However, if it is no longer required or included by any mod nor listed within the dependency tree of another helper, then it will be uninstalled from the local repository automaticallu. No modification to depends.txt or mod.conf is necessary.

Conflict Resolution of Helper IDs

One of the primary reasons that I developed Pluggable Helpers was to resolve the very obvious potential for code duplication whenever the Minetest API is expanded. A good example of this situation was the introduction of vector.dot and vector.cross in Minetest 5.0.

Up until recently, any mods that needed these helper functions had no choice but to define them manually. Of course they are now both redundant. Thankfully, Pluggable Helpers provides for conflict resolution in this situation.

At startup, each required helper ID is checked against the global environment. If the method or class is already defined by the builtin Lua or Minetest API, then that function will be used. Otherwise, the corresponding helper will be downloaded from the remote repository. This provides a seamless means to backport newer API functions for legecy versions of Minetest.

Let's say that I wrote a fancy new mod that depends on minetest.encode_base64, yet I just realized that Minetest 0.4.14 does not support that API function even though that's among my target userbase. I could simply write a Lua variant of the Base64 encoder and submit it as a helper method. Now it takes just one line of code to ensure my script will run properly regardless of the Minetest version.

Code: Select all

plugins.require( "minetest.encode_base64' )

print( minetest.encode_base64( "Wow! This will even work in Minetest 0.4.14!" ) )
Extending the Helper Environment

For security reasons, helper functions are always executed within a restrictive environment. The Pluggable Helpers sandbox includes the builtin Lua functions shown below in addition to all of the core Lua and Minetest libraries (such as math, string, vector, io, etc.), and any dependencies.
  • next
  • pairs
  • ipairs
  • assert
  • print
  • error
  • dofile
  • loadfile
  • loadstring
  • getmetatable
  • setmetatable
  • pcall
  • rawequal
  • rawget
  • rawset
  • select
  • tonumber
  • tostring
  • type
  • unpack
  • dump
It is possible to further customize this sandbox by using the following advanced API function:
  • plugins.register_class( name, ref )
    Adds the given library or class constructor to the global helper environment, so that it is accessible to all helper functions. The "ref" parameter must be a table reference in the case of libraries or a function reference in the case of class constructors. The "name" parameter is the name or an alias by which to make the library or class constructor available within the helper environment. It need not be the original name, but it must adhere to the naming conventions previously described.
You should take great care to only add what is essential, rather than polluting the global helper environment.

Using Helpers in Mods

The following API functions should only ever be called at startup, otherwise they will raise an exception. Hence, they are best placed at the head of your script.
  • plugins.include( id )
    Returns a reference to the helper library with the given ID. If the library is not found within the local repository, then it will be asynchronously downloaded and installed.

    Note that helper libraries are purposefully not added to the global environment. You should therefore store the result into a local variable. There is no penalty for including the same helper library in multiple Lua scripts. In fact, that is the correct way to use this API function.

    Code: Select all

    -- Correct usage in init.lua:
    local mylib = plugins.include( "MyGreatLibrary" )
    
    -- Incorrect usage in init.lua
    mylib = plugins.include( "MyGreatLibrary" )
    
    plugins.require( id )
    Returns a reference to the helper method or helper class with the given ID. If the class or method is not found within the local repository, then it will be asynchronously downloaded and installed.

    As mentioned previously, helper methods and helper classes are both available globally. It is entirely acceptable to ignore the return value of this API function and use the helper class or method globally. But you can localize the reference for efficiency. Both of the following examples are the same, but the latter takes advantage of the shorthand.

    Code: Select all

    plugins.include( "math.square" )
    local square = math.square
    print( square( 2 ) )
    
    local square = plugins.include( "math.square" )
    print( square( 2 ) )
    
Helper Version Tags

All helpers must adhere to the SemVer standard, with the first numeral designating a major release and the second numeral, a minor release. This makes it possible to automatically distinguish between version changes.

To require or include a specific version of a helper, simply suffix the ID with a slash and the version tag. For example if only version 1.0 of "math.square" is installed in the local repository and your mod requires version 1.1, then the latest version will be downloaded and installed from the remote repository. For this reason, helpers must always be backward compatible to avoid breakage, as only a single helper can be installed at one time.

For convenience, both of the API functions above allow for splitting the ID and version into a two element array. So all three of these examples are acceptable:

Code: Select all

plugins.require( "math.square/1.1" )
plugins.require "math.square/1.1"
plugins.require { "math.square", "1.1" }
plugins.require( { "math.square", "1.1" } )
Submission Guidelines

One of the central tenets of Pluggable Helpers, is to streamline the review process so that additional functionality can be made available to games and mods within a reasonable timeframe. Obviously, I want to provide developers the greatest degree of creative freedom, but without unduly sacrificing the security and integrity of games and mods that rely on this shared code-base.

All helper methods, classes, and libraries will therefore be vetted for compliance with several core guidelines prior to being published. The goal is to limit any chance of an untrusted helper going rogue.
  • Helpers must be useful
    A helper function must have some degree of utility, and it must do exactly what its name implies without unanticipated side-effects. For example, a helper called "math.add" that subtracts two numbers and prints the result will be rejected.
  • Helpers must be generic
    Generally speaking a helper function should be as generic as possible, unless it is intended to perform a common and repetitive task that would be best consolidated into a function. For example, a helper called "math.square" that squares a given number may be generic enough for inclusion. But, a helper called "math.multiply" that returns the product of 5 x 4 would be rejected.
  • Helpers must be secure
    It should go without saying that all helper functions must respect the security and integrity of the target installation environment. Any attempt to defeat the Pluggable Helpers sandbox or to override the builtin security model of the Minetest engine (client or server) or to abuse the Pluggable Helpers infrastructure will be grounds for rejection.
  • Helpers must be open-source
    In order to build an effective community-focused API for Minetest, it is essential that every helper function be objectively compatible with the principles of FOSS. Any attempt to obfuscate or encrypt or copy-protect source code or to impose a copyright clause that inhibits free distribution and adaptation will be grounds for rejection
  • Helpers must be concise
    The goal of a helper method or class is to perform a small task and to do it in the most efficient and effective way possible. Embedding an entire game into a single function is obviously impractical. Note that this limitation does not apply to helper libraries, which may be as sophisticated as the target application demands (see above)
There are obviously exceptions to every rule, and each helper will be considered on a case by case basis. More often than not, as long as there is a reasonable effort to adhere to these guidelines, then approval is assured.

Also keep in mind that no single coding style is imposed on submissions (other than the aforementioned naming conventions). But, please try to ensure that your source code is easily readable so thatit can be jointly maintained by the community.

The Lua Code Guidelines for Minetest are a good starting point. While not enforced in this project, they are strongly encouraged.

Unit-Testing of Helpers

Prior to submitting any helper for review, it is important to unit-test it first. This can be accomplished by creating a helper definition and manually installing it into the local repository. The "source" subdirectory within the plugins mod directory is the default location for the local repository, but this may be changed to a symbolic link if needed.

A helper definition must be a plain-text file with read and write permissions by the Minetest process. It is named exactly as the helper ID and consists of a Lua snippet in the following format:

Code: Select all

author = "testuser"
version = "1.2"
license = "GPLv3"
depends = { "math.min", "math.max" }
imports = { "math.min", "math.max" }

return function ( v, v_min, v_max )
	return min( v_max, max( v_min, v ) )
end
Once the helper is ready for unit-testing, I would save its definition file to ./minetest_game/mods/plugins/source/math.clamp

Important: Be certain to require or include any test helpers in at least one mod (or else list them as dependencies) before launching Minetest, otherwise they will deleted from "source" subdirectory during startup.

Layout of a Helper Submission

A helper submission is a plain-text file describing the proposed helper. It is the precursor to a helper definition which is the Lua snippet that the Pluggable Helpers Mod downloads and installs from the remote repository.

A valid helper submission consists of 5 fields and an anonymous function at the end of the file.
  • Author - This is author's registered username on the Content DB
  • License - This is a valid FOSS license as listed on the Content DB
  • Version - This is the SemVer tag corresponding to the current release
  • Depends - This is a list of helper methods, helper classes, or helper libraries that must be installed prior to this helper being required or included (optional)
  • Imports - This is a list of helper methods, helper classes, or builtin library methods that are to be imported into the helper's environment when it is required or included (optional)
  • Prototype - This is a special sixth field that is needed only in the case of helper libraries. It allows for specifying a base class for the constructor function to extend at startup.
This is an example skeleton of a helper submission:

Code: Select all

Author:
sorcerykid

License:
MIT

Version:
1.0

Depends:

Imports:

function ( )
end
Later this week I will announce the launch of the Pluggable Helpers Minetest server, which provides a centralized interface for anyone with a trusted ContentDB account to submit new helpers for publication. So stay tuned!

Project Goals
In the short-term, I would really like to bring on-board a team of trusted community members to assist with the review of new submissions. Obviously, Rubenwardy, as the maintainer of the ContentDB itself, and the participation of other core developers and active contributors would be most welcome and encouraged for this project to be successful.

Eventually, it may be worth migrating the remote repository and the submission server to a subdomain on minetest.net, at least if it becomes popular enough. That would not only make it more official but also ensure that it truly is a community endeavor. Of course this need not happen for the interim. I am more than happy to maintain the repository on a volunteer basis in much the same way as sofar hosts the Public Remote Media Server Project independent of Minetest.

Right now I am also working on bringing a Wiki online, so that all approved helper functions can be easily documented. It will be located at http://plugins.mytuner.net/wiki/ when ready.

User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by LMD » Post

I have to ask though:

Why not just go with mods? I wondered whether I should implement dependency management in modlib, and came to the conclusion that I should just leave it as-is, having the engine handle mod(ule) dependencies.
My stuff: Projects - Mods - Website

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by sorcerykid » Post

LMD wrote:Why not just go with mods? I wondered whether I should implement dependency management in modlib, and came to the conclusion that I should just leave it as-is, having the engine handle mod(ule) dependencies.
The engine doesn't manage dependencies. it only checks to ensure that mods of a specific name exist and are loaded in the correct order at runtime. The end-user is responsible for manually downloading, installing, and uninstalling all mods of the required version and fork from the Content DB or various git repositories such as GitHub, GitLab, MeseHub, Bitbucket, or Notabug.

Moreover, mods have their own share of shortcomings. They are overkill for small-scale helper functions. They can freely pollute the global namespace. They are not true "modules" in the standard Lua sense. And there is no means of version consistency checking.

In fact, Minetest makes no distinction between packages and modules (i.e. libraries). Everything is a package, even when it is intended to be a library. Yet in most popular scripting languages whether it be Perl, Python, or even Lua itself these are two very different concepts. As a result developers are forced to use arcane statements like local mylibrary = dofile(minetest.get_modpath("mymod").."/mylibrary.lua") in order to work around this limitation.

MoNTE48
Member
Posts: 323
Joined: Sat Apr 06, 2013 11:58
GitHub: MoNTE48
In-game: MoNTE48
Location: Internet

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by MoNTE48 » Post

I have to admit it's cool! It had to happen earlier and was simply obliged to become official now!

I will definitely use this and try to make some contribution when it is available.
Some MultiCraft APIs may be useful for many mods.

Edit.
A small question.
Using this and creating a small builtin patch, is it possible to add support for the latest APIs in 0.4.*, as well as duplicate all the new in-dev APIs, so that I can be sure that my game for 5.2 will start and work in 5.1, 5.0, 0.4?
It is better to have an API with a `""` for the Sky API than an error while loading or `nil` during some action!

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by Linuxdirk » Post

Formspec on point as always and functionality over-the-top as always :) I absolutely love your mods and wish I had a real use case for them.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by TumeniNodes » Post

sorcerykid
It's so nice to see you back (I know you weren't gone long) and just going full throttle : )
Plenty of really good output
A Wonderful World

MoNTE48
Member
Posts: 323
Joined: Sat Apr 06, 2013 11:58
GitHub: MoNTE48
In-game: MoNTE48
Location: Internet

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by MoNTE48 » Post

Only in the morning I saw this mod. Immediately faced with the problem described.
I needed transliteration of the text. Which (of course) requires a library. After some searching (I really did not want to make this table) I found it.
https://github.com/MultiCraft-Community ... lugify.lua
I am sure it will be useful to others.
Last edited by MoNTE48 on Sun Sep 06, 2020 11:41, edited 1 time in total.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by sorcerykid » Post

Greetings! I'm happy to report that my issue Add support for PUT and DELETE methods in HTTPRequest was just resolved with the merging of PR #9909 yesterday!

I've been waiting patiently for this feature to be added so I could finish work on the VCS backend component. The Management Console server should be ready to go online very soon, and we will have a fully working distributed architecture. I'm very excited about moving this project forward again, because it opens up so many new possibilities for cooperative development.

Here's a quick walk-through of the basic functionality. Most features are tested and working, including back-end integration with the BitBucket VCS. All that remains is to develop the Web-based account verification system, using ContentDB bearer tokens.

Image
Pluggable Helpers Management Console
https://vimeo.com/453830542

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by sorcerykid » Post

MoNTE48 wrote:
Tue Mar 17, 2020 15:19
I have to admit it's cool! It had to happen earlier and was simply obliged to become official now!

I will definitely use this and try to make some contribution when it is available.
Some MultiCraft APIs may be useful for many mods.

A small question.
Using this and creating a small builtin patch, is it possible to add support for the latest APIs in 0.4.*, as well as duplicate all the new in-dev APIs, so that I can be sure that my game for 5.2 will start and work in 5.1, 5.0, 0.4?
It is better to have an API with a `""` for the Sky API than an error while loading or `nil` during some action!
I'm thrilled to have your support, and I'd like to work closely with you on incorporating this API into your servers. Any contributions from the MultiCraft project would be more than welcomed. Since you are already a trusted community member, I can give you "reviewer" permissions in the Management Console. That way you can approve and deploy any pending submissions.

As for back-porting new features from Minetest 5.x, that is one of the intended applications of Pluggable Helpers. When I began work on this project, I envisioned helper functions serving as a compatibility layer for legacy engine versions. Some parts of the API are impossible to extend (like methods of LuaEntitySAOs), but anything in the global namespace is fair game for emulation.
Linuxdirk wrote:
Tue Mar 17, 2020 17:47
Formspec on point as always and functionality over-the-top as always :) I absolutely love your mods and wish I had a real use case for them.
Thank you so much. I always welcome your input. I hope it proves useful for modders that wish to contribute to the project!
TumeniNodes wrote:
Tue Mar 17, 2020 19:28
sorcerykid
It's so nice to see you back (I know you weren't gone long) and just going full throttle : )
Plenty of really good output
Absolutely! It's great to be back in action after my extended hiatus. The past several months I've been heavily focused on server maintenance. But you know I can't stay away from Minetest for too long. It is kind of my obsession after all. And I'm glad to be part of such an awesome community, yourself included :)

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: [Mod] Pluggable Helpers (auto-dependency API) [plugins]

by sorcerykid » Post

After doing some more research, I learned that ObjectRefs rely on metatables. Therefore, PlayerSAOs and LuaEntitySAOs could be extended after all (earlier I claimed it wasn't possible to overwrite the methods of objects).

Re: Is it possible to overwrite ObjectRef:punch?

I also found out that at server startup the media isn't processed until after all mods have finished loading. So in theory, helper libraries could include an optional payload, in addition to just source code -- whether that be Base64 encoded textures, sounds, or models -- all of which could be downloaded and installed on the fly.

This certainly opens the doors to many other possible applications for Pluggable Helpers. I'm very excited!

Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests