RocketLib Toolkit (Lua-based SQLite3 map reader)

Post Reply
User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

RocketLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

Image

RocketLib Toolkit v1.2

RocketLib Toolkit is a purely Lua-driven map reader for SQLite3 databases with an extensive API for use by command line tools. Mods can also make use of the API for reading offline copies of the map.sqlite database directly in game so long as your mod is listed as a "trusted mod" in minetest.conf (however, it may induce lag).

Just to to showcase how easy it is to get started examining your map database with RocketLib Toolkit, it takes only 16 lines of Lua code to search for all mapblocks that have dropped items:

Code: Select all

require( "maplib" )

local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )

for index, block in map_db.iterate( ) do
        local count = 0
        for i, v in ipairs( block.object_list ) do
                if v.name == "__builtin:item" then
                        count = count + 1
                end
        end
        if count > 0 then
                print( string.format( "%d dropped items in mapblock (%s)",
                        count, pos_to_string( decode_pos( index ) ) ) )
        end
end
Here's are the results for a world in which I've littered several random items.

Image

I released this library back in April, just before the forum crash. Unfortunately since the original topic was lost, I didn't get around to reposting it. So I'll do my best to resurrect as much information as possible.

Repository:

https://bitbucket.org/sorcerykid/rocketlib

Download Archive (.zip)
Download Archive (.tar.gz)

Source Code License:

The MIT License

Installation Instructions:

RocketLib Toolkit depends on the Lua modules lsqlite3 and zblip, which can be installed using Luarocks.

Luarocks itself can be obtained from https://github.com/luarocks/luarocks/wiki/Download

Getting Started

The map reader library provides a fully object-oriented API, so it is straightfoward to examine mapblocks and their contents without having to worry about the underlying database architecture.

The available class constructors are as follows:

BlobReader( input )
Provides an interface to serially parse a BLOB with a variety of known datatypes.
  • input is the raw map block data obtained from the "data" field of the "blocks" table (required)
The BlobReader class defines the following public methods:
  • BlobReader::read_u8( )
    Read an 8-bit unsigned integer and advance the pointer by one byte.

    BlobReader::read_u16( )
    Read a 16-bit unsigned integer and advance the pointer by two bytes.

    BlobReader::read_u32( )
    Read a 32-bit unsigned integer and advance the pointer by four bytes.

    BlobReader::read_s16( )
    Read a 16-bit signed integer and advance the pointer by two bytes.

    BlobReader::read_s32( )
    Read a 32-bit signed integer and advance the pointer by four bytes.

    BlobReader::read_f1000( )
    Read a floating point and advance the pointer by four bytes.

    BlobReader::read_v3f1000( )
    Read a 3-dimensional floating point array and advance the pointer by 12 bytes.

    BlobReader::read_zlip( )
    Slurp a zlib compressed data stream and advance the pointer accordingly.

    BlobReader::read_string( len )
    Read a non-terminated text string of len bytes and then advance the pointer accordingly to len. If len is not provided, slurp a multiline terminated text string and advance the pointer accordingly.
MapArea( pos1, pos2 )
Delineates a mapblock area to be examined, while also providing various area calculation methods.
  • pos1 is lowest boundary mapblock position to iterate
  • pos2 is the highest boundary mapblock position to iterate
The MapArea class provides the following public methods:
  • MapArea::get_min_pos( )
    Return the lowest boundary mapblock position of the area as a table {x,y,z}

    MapArea::get_max_pos( )
    Return the highest boundary mapblock position of the area as a table {x,y,z}

    MapArea::get_volume( )
    Calculate the volume of the area in cubic mapblocks and return the result

    MapArea::has_index( idx )
    Returns true if the specified mapblock index is contained within the area

    MapArea::has_pos( pos )
    Returns true if the specified mapblock position is contained within the area
MapBlock( blob, is_preview, get_checksum )
Parses the mapblock data and calculates the associated checksum. For efficiency, the nodemeta map and the node list are not parsed automatically, but they can be obtained using the corresponding methods.
  • blob is the raw mapblock data obtained from "data" field of the "blocks" table (required).
  • is_preview is a boolean indicating whether to parse the BLOB.
  • get_checksum is the checksum function to calculate the checksum and length of the BLOB.
The MapBlock class defines the following public methods:
  • MapBlock::get_node_list( )
    Parses the raw node list of the mapblock and returns a node_list table.

    The node_list table is an ordered array of exactly 4096 elements, corresponding to the 16x16x16 matrix of nodes comprising the mapblock. The position of a node can be obtained from its index using the decode_node_pos( ) helper function. Each entry of the node_list table contains a subtable with three fields: id, param1, and param2.

    Note that the id refers to the content ID which varies between mapblocks. You must cross-reference the content ID to determine the actual registered node name.

    MapBlock::get_nodemeta_map( )
    Parses the raw nodemeta map and returns a nodemata_map table.

    The nodemeta_map table is an associative array of node indexes mapped to node metadata. Each entry of the nodemeta_map table is a subtable containing the following fields:
    • fields is a subtable containing the user-defined metadata for the node, as ordinary key-value pairs.
    • is_private is a boolean specifying whether the metadata of the node is private
    • inventory is a subtable containing the inventory of the node as an array of tables, with two fields for each inventory slot: item_name and item_count
The MapBlock class defines the following public read-only properties:
  • MapBlock::version
    The version of the mapblock.

    MapBlock::flags
    The flags of the mapblock.

    MapBlock::content_width
    The size of the content_ids in bytes. This is either 1 or 2, depending on the version.

    MapBlock::params_width
    The size of param1 and param2 in bytes. This is always 2.

    MapBlock::object_list
    An array of objects stored in the mapblock. Each entry contains a subtable with seven fields: type, pos, version, name, staticdata, hp, velocity, and yaw.

    MapBlock::nodename_map
    An associative array of registered node names indexed by content IDs.

    MapBlock::timestamp
    The timetamp when the mapblock was last modified by the engine. Note that this value is not necessarily a reliable means to determine if a mapblock was changed or not. For that you should perform a checksum comparison.
MapDatabase( path, is_preview, is_summary )
Opens an existing map.sqlite database from disk and prepares the necessary SQL statements.
  • path is the path to the sqlite3 map database to be opened in read-only mode (required)
  • is_preview is a boolean indicating whether mapblocks are to be parsed by default
  • is_summary is a boolean indicating whether checksums apply to all mapblocks by default
The MapDatabase class defines the following public methods:
  • MapDatabase::enable_preview( )
    Enable parsing of mapblocks by default

    MapDatabase::disable_preview( )
    Disable parsing of mapblocks by default, only calculate checksum and length

    MapDatabase::enable_summary( )
    Enable cumulative checksum calculations by default

    MapDatabase::disable_summary( )
    Disable cumulative checksum calculations by default

    MapDatabase::change_algorithm( algorithm )
    Switches to a different checksum algorithm, either 'adler32' or 'crc32'.

    MapDatabase::create_cache( use_memory, on_step )
    Create a cache database storing cross-references of mapblock position hashes, thereby speeding up successive queries. If use_memory is true, the cache database will be memory resident. Otherwise a file named "map.sqlite-cache" will be created in the same directory as the map database. The optional on_step hook can be used to update a progress bar for lengthy operations.

    MapDatabase::get_length( )
    Returns the total number of mapblocks. If the cache is available, then it will be used.

    MapDatabase::get_area_length( area )
    Returns the total number of mapblocks inside the given area. The cache is required for this operation.

    MapDatabase::iterate( on_step )
    Returns an iterator function, for looping over all existing mapblocks. The optional on_step hook can be used to update a progress bar for length operations

    MapDatabase::iterate_area( area, on_step )
    Returns an iterator function, for looping over all existing mapblocks inside the given area. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is required for this operation.

    MapDatabase::select( on_step )
    Returns an array of all hashed positions for all mapblocks. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is not used for this operation (but I will consider making it optional)

    MapDatabase::select_area( area, on_step )
    Returns an array of hashed positions for all mapblocks inside the given area. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is required for this operation.

    MapDatabase::has_index( index )
    Returns a boolean indicating whether a mapblock exists with the given index.

    MapDatabase::get_mapblock( index )
    Returns the mapblock with the given index as a MapBlock object.

    MapDatabase::get_mapblock_raw( index )
    Returns the raw mapblock data as a BLOB without calculating the checksum and length.

    MapDatabase::close( index )
    Closes the map database (but it doesn't close the cache database, which is a known bug).
Several helper functions are also available for debugging and conversion purposes.
  • maplib.decode_pos( index )
    Converts the given mapblock index to a mapblock position.

    maplib.encode_pos( pos )
    Converts the given mapblock position to a mapblock index.

    maplib.decode_node_pos( node_index, index )
    Converts the given node index within the given mapblock to a node position in world coordinates. If the mapblock index parameter is not provided, then the result will be relative to a mapblock at {0,0,0}. Note: For consistency with Lua conventions, node indexes are always 1-based even though mapblock indexes are 0-based.

    maplib.encode_node_pos( pos )
    Converts the given node position into a both a node index and a mapblock index.

    maplib.hash_node_pos( [/i]node_pos[/i] )
    Returns the equivalent of minetest.hash_node_position( ).

    maplib.unhash_node_pos( node_hash )
    Returns the equivalent of minetest.get_position_from_hash( ).

    maplib.pos_to_string( pos )
    Returns a string representing the given position as "(x,y,z)".

    maplib.dump( buffer )
    Returns a string representing a memory dump of the given buffer.
For simplicity, you may wish to localize these functions at the head of your script. Also note that you can use these helper functions without loading the entire RocketLib API:

Code: Select all

local maplib = require "helpers"     -- just the minimal API

print( maplib.pos_to_string( { x = 0, y = 10, z = 0 } ) )
This can be useful in post-processing scripts, when you need to perform conversions but without directly accessing the map database.

Code Samples

This script will print the size of the mapblock at (0,-1,0) in bytes.

Code: Select all

require( "maplib" )

local pos = { x = 0, y = -1, z = 0 }
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", true, true )

local block = map_db.get_mapblock( encode_pos( pos ) )

if block then
        print( string.format( "Size of mapblock at (%s) = %d bytes", pos_to_string( pos ), block.length ) )
else
        print( "Mapblock not found!" )
end
This script will print the position and the crc32 checksum of every mapblock.

Code: Select all

require( "maplib" )

local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", true, true )

for index, block in map_db.iterate( ) do
	print( string.format( "(%s) %d", pos_to_string( decode_pos( index ) ), block.checksum ) )
end
This script will print a list of all nodes found in the map with their usage counts.

Code: Select all

local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
local counts = { }

for index, block in map_db.iterate( status.on_step ) do
        local node_list = block.get_node_list( )
        local nodename_map = block.nodename_map

        for _, node in ipairs( node_list ) do
                local name = nodename_map[ node.id ]
                if counts[ name ] then
                        counts[ name ] = counts[ name ] + 1
                else
                        counts[ name ] = 1
                end
        end
end

for k, v in pairs( counts ) do
	print( string.format( "%-8d %s", v, k ) )
end
This script will print the number of mapblocks that contain at least one nyancat.

Code: Select all

require( "maplib" )

local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
local count = 0

for index, block in map_db.iterate( ) do
        for k, v in pairs( block.nodename_map ) do
                if v == "nyancat:nyancat" then
                        count = count + 1
                end
        end
end

print( count )
This script will print the number of mapblocks within the range (-100,-100,-100) to (100,-10,100).

Code: Select all

require( "maplib" )

local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )

map_db.create_cache( true )

local area = MapArea( { x = -100, y = -100, z = -100 }, { x = 100, y = -10, z = 100 } )
local count = map_db.get_area_length( area )

print( count )
Last edited by sorcerykid on Sun Oct 04, 2020 22:13, edited 3 times in total.

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: MapLib Toolkit (Lua-based SQLite3 map reader)

by Sokomine » Post

Very intresting. I've always wondered how many map blocks on a server are actually *used* by players in order to build something. Map blocks where only ressources where gathered and torches placed are of no intrest. Chests, doors, signs etc. are signs of civilization. I wonder how many mapblocks on a real server are thus used.
A list of my mods can be found here.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: MapLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

Version 1.1 Released

MapLib Toolkit now has a new name: RocketLib Toolkit! To reflect this change, the repository location has moved to

https://bitbucket.org/sorcerykid/rocketlib/src/master/

Here is a complete change log:
  • Included README.txt file with usage instructions
  • Added database method to read raw mapblock data
  • Added private string-matching helper function
  • Allowed for switching crc32 to adler32 checksum
  • Localized math and string library functions
  • Renamed nodemeta_list to reflect being a map
  • Appended parentheses around position strings
  • Switched node_list array to one-based indexing
  • Fixed parsing of itemstack names with digits
  • Included empty itemstacks in inventory list
  • Added helper function for encoding node positions
  • Added helper function for decoding node positions
  • Updated all references from former library name
  • Deprecated index_to_pos helper function
Although I've taken great care to avoid breaking compatibility, there are two changes that might require updating your scripts:

First and foremost, I renamed the get_nodemeta_list( ) method to get_nodemeta_map( ) to reflect the fact that the returned dataset is actually an associative array rather than an ordered array. This is for consistency with the rest of the API. I apologize for the confusion. I'm not sure how that ever got past my original code reviews.

Secondly, I removed support for the get_checksum parameter that was only used by the get_mapblock() method. It didn't make sense, given that the API never exposed the zlib checksum functions anyway. Now I've implemented checksum selection as its own method of the MapDatabase class: change_algorithm( algorithm )

In other news, a couple new helper functions have been introduced to aid with conversion between node indexes and node positions. The decode_node_pos() function in particular is essential for outputing the node's position in world coordinates, as this example illustrates:

Code: Select all

local path = "/root/.minetest/worlds/world/map.sqlite"
local map_db = MapDatabase( path, false )

local idx = maplib.encode_pos( { x = 0, y = 0, z = 0 } )
local block = map_db.get_mapblock( idx )

if block then
        local nodemeta_map = block.get_nodemeta_map( )
        local nodename_map = block.nodename_map
        local node_list = block.get_node_list( )

        for i, m in pairs( nodemeta_map ) do
                local name = nodename_map[ node_list[ i ].id ]
                print( "Found node '" .. name .. "' with metadata at " ..
                        maplib.pos_to_string( maplib.decode_node_pos( i, idx ) ) )
        end
end
It's also worth noting that I decided to deprecate the index_to_string() helper function because it was bound to lead to confusion with these new node-related functions. However, it's still easy to perform the same conversion as follows:

Code: Select all

index_to_string( index )  -->  pos_to_string( decode_pos( index ) )
Last but not least, I opted for 1-based indexing of the node_list array, which is consistent with Lua conventions. This means that the keys of the nodemeta_map table will directly correspond with the node indexes rather than being offset by one.

talamh
Member
Posts: 156
Joined: Sun Nov 12, 2017 18:24

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by talamh » Post

Looking forward using this as it be very useful for testing the output of a few little tools I am developing. Just curious if you fell into the trap trying to calculate a mapblocks coordinates independently of each other using bitwise shifting. I wasted more time than I care to admit on that one! I spotted in one of Lags python scripts that he made the same mistake, bitwise shifting works like a charm until x or y goes negative.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

talamh wrote:Looking forward using this as it be very useful for testing the output of a few little tools I am developing. Just curious if you fell into the trap trying to calculate a mapblocks coordinates independently of each other using bitwise shifting. I wasted more time than I care to admit on that one! I spotted in one of Lags python scripts that he made the same mistake, bitwise shifting works like a charm until x or y goes negative.
Cool! It's definitely become an invaluable tool for server administration tasks. Originally I had tried messing around with Andrej's Map Unexplore scripts. But to be honest, I'm not at all familiar with Python, so I couldn't wrap my head around the code. And given that there was effectively no API, just a lot of hardcoded routines, I found the scripts to be far too inflexible for my needs :0

As for the mapblock positions, those are calculated exactly per the official specifications. No shortcuts taken. I recall that ShadowNinja tried that trick once, and it broke the engine. Suffice it to say, celeron55 was not pleased.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

Version 1.2 Released

A new version of RocketLib Toolkit is ready for download. Here is a complete change log:
  • Fixed botched calculation in node position encoder
  • Included code samples and edited documentation
  • Added helper function for hashing node positions
  • Added helper function for unhashing node positions
  • Organized all helper functions into separate file
  • Localized results of conditional pattern matches
All of the helper functions have now been moved into their own module, "helpers.lua", so so they can be used independently of the main RocketLib API:

Code: Select all

local maplib = require "helpers"     -- just the minimal API

print( maplib.pos_to_string( { x = 0, y = 10, z = 0 } ) )
This can be useful in post-processing scripts, when you need to perform conversions but without directly accessing the map database. For added simplicity, you may wish to localize the functions at the head of your script.

I also ported the minetest.hash_node_position() and minetest.get_position_from_hash() helper functions from the Minetest API, which allow for using node positions as table keys.
  • maplib.hash_node_pos( node_pos )
    Returns the equivalent of minetest.hash_node_position( ).

    maplib.unhash_node_pos( node_hash )
    Returns the equivalent of minetest.get_position_from_hash( ).
Keep in mind, that node hashes are NOT interchangeable with node indexes. A node's hash is derived from it's (x,y,z) position in world coordinates, whereas the node's index is relative to its (x,y,z) position within a mapblock.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

Version 1.3 Released

A new version of RocketLib Toolkit is ready for download. Here is a complete change log:
  • Fixed parsing of private node metadata fields
  • Moved checksum setting into class constructor
  • Exposed helper functions via main module API
  • Implemented working NodeMetaRef helper class
  • Returned node total in metadata parser method
  • Improved parsing of inventories with named lists
  • Updated sample scripts and README.txt
To make working with helper functions more convenient, they are now exposed via the main "maplib.lua" module. So there is no need to require "helpers" simultaneously.

Code: Select all

local maplib = require "maplib"
local map_db = MapDatabase( "/home/minetest/worlds/world/map.sqlite", false )

for index, block in map_db.iterate( ) do
        local count = 0
        for i, v in ipairs( block.object_list ) do
                if v.name == "__builtin:item" then
                        count = count + 1
                end
        end
        if count > 0 then
                print( string.format( "%d dropped items in mapblock (%s)",
                        count, maplib.pos_to_string( maplib.decode_pos( index ) )
                ) )
        end
end
I also completed work on a NodeMetaRef helper class, so it should be a viable alternative to direct table lookups.

NodeMetaRef( block)
Provides a simple yet efficient interface for working with node metadata, while emulating aspects of Minetest's API.
  • block is the mapblock to examine
The NodeMetaRef class provides the following public methods:
  • NodeMetaRef::contains( idx, key )
    Returns true if the given metadata field exists at the given node index, or false otherwise.

    NodeMetaRef::get_raw( idx, key )
    Returns the raw value of the given metadata field at the given node index.

    NodeMetaRef::get_string( idx, key )
    Returns the string value of the given metadata field at the given node index.

    NodeMetaRef::get_float( idx, key )
    Returns the floating point value of the given metadata field at the given node index.

    NodeMetaRef::get_int( idx, key )
    Returns the integer value of the given metadata field at the given node index.

    NodeMetaRef::to_table( idx )
    Returns a deserialized table of the "fields" and "inventory" entries at the given node index.

    NodeMetaRef::is_private( idx, key )
    Returns true if the given metadata field is set to private at the given node index, or false otherwise.
For all intents and purposes, the following two snippets of code are roughly equivalent:

Code: Select all

local meta = NodeMetaRef( block )
local value = meta.get_string( 5, "infotext" )

local value = block.get_nodemeta_map( )[ 5 ].fields.infotext
The first option is more efficient for multiple lookups, since the node metadata is deserialized and cached internally. Whereas, the second option is fine for just a single lookup.

The inventory parser has also been improved to account for multiple lists in addition to item wear and item metadata. So any code in which you are indexing the "inventory" table directly will need to index the subtable for the corresponding list instead.

Code: Select all

local item_name = meta.inventory[ slot_idx ].item_name

local item_name = meta.inventory[ slot_idx ][ list_name ].name
The following key-value pairs are available for each item: name, count, wear, metadata. Notice that "item_name" and "item_count" have been deprecated. If a slot is empty, then the table will be empty.

jyamaha
New member
Posts: 2
Joined: Sat Jun 03, 2023 18:55

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by jyamaha » Post

Apologies in advance for digging up an old thread. It looks like support for this utility may not exist given the 3 year hiatus. However, hopefully there are experts that can help out with an issue I'm having.

First, let me be helpful and document how to get the RocketLib toolkit working for MacOS since the existing docs do not cover this case:

1) Install luarocks from command line
a) brew update
b) brew install luarocks

2) Use luarocks to install dependencies
a) luarocks install lsqlite3
b) luarocks install lsqlite3complete
c) luarocks install lua-zlib

3) Download RocketLib package and decompress these

Now the samples will run as long as properly edited for your database location, etc.

The issue I'm having is when I run samples/finditems.lua I get an error.

The error is this:

Code: Select all

samples % lua ./finditems.lua 
Creating cache...
Examining database...
lua: ./maplib.lua:229: Invalid params_width, aborting!
stack traceback:
	[C]: in function 'error'
	./maplib.lua:229: in function 'MapBlock'
	(...tail calls...)
	./maplib.lua:550: in for iterator 'for iterator'
	./finditems.lua:149: in main chunk
	[C]: in ?
I'm stuck. Let me know if I need to supply more verbose error reports - this is the default level.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: RocketLib Toolkit (Lua-based SQLite3 map reader)

by sorcerykid » Post

Thanks for bringing this issue to my attention. Rest assured, I still support these command-line scripts and routinely use them for server administration.

The error you encountered is likely due to a recent change in map database format. Minetest 5.5 introduced zstd compression, so any tools that parse mapblocks will need to be updated. I'll try to put this on my priority list.

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests