RocketLib Toolkit v1.2
RocketLib Toolkit is a purely Lua-driven map reader for SQLite3 databases with an extensive API for use by command line tools. Mods can also make use of the API for reading offline copies of the map.sqlite database directly in game so long as your mod is listed as a "trusted mod" in minetest.conf (however, it may induce lag).
Just to to showcase how easy it is to get started examining your map database with RocketLib Toolkit, it takes only 16 lines of Lua code to search for all mapblocks that have dropped items:
Code: Select all
require( "maplib" )
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
for index, block in map_db.iterate( ) do
local count = 0
for i, v in ipairs( block.object_list ) do
if v.name == "__builtin:item" then
count = count + 1
end
end
if count > 0 then
print( string.format( "%d dropped items in mapblock (%s)",
count, pos_to_string( decode_pos( index ) ) ) )
end
end
I released this library back in April, just before the forum crash. Unfortunately since the original topic was lost, I didn't get around to reposting it. So I'll do my best to resurrect as much information as possible.
Repository:
https://bitbucket.org/sorcerykid/rocketlib
Download Archive (.zip)
Download Archive (.tar.gz)
Source Code License:
The MIT License
Installation Instructions:
RocketLib Toolkit depends on the Lua modules lsqlite3 and zblip, which can be installed using Luarocks.
Luarocks itself can be obtained from https://github.com/luarocks/luarocks/wiki/Download
Getting Started
The map reader library provides a fully object-oriented API, so it is straightfoward to examine mapblocks and their contents without having to worry about the underlying database architecture.
The available class constructors are as follows:
BlobReader( input )
Provides an interface to serially parse a BLOB with a variety of known datatypes.
- input is the raw map block data obtained from the "data" field of the "blocks" table (required)
- BlobReader::read_u8( )
Read an 8-bit unsigned integer and advance the pointer by one byte.
BlobReader::read_u16( )
Read a 16-bit unsigned integer and advance the pointer by two bytes.
BlobReader::read_u32( )
Read a 32-bit unsigned integer and advance the pointer by four bytes.
BlobReader::read_s16( )
Read a 16-bit signed integer and advance the pointer by two bytes.
BlobReader::read_s32( )
Read a 32-bit signed integer and advance the pointer by four bytes.
BlobReader::read_f1000( )
Read a floating point and advance the pointer by four bytes.
BlobReader::read_v3f1000( )
Read a 3-dimensional floating point array and advance the pointer by 12 bytes.
BlobReader::read_zlip( )
Slurp a zlib compressed data stream and advance the pointer accordingly.
BlobReader::read_string( len )
Read a non-terminated text string of len bytes and then advance the pointer accordingly to len. If len is not provided, slurp a multiline terminated text string and advance the pointer accordingly.
Delineates a mapblock area to be examined, while also providing various area calculation methods.
- pos1 is lowest boundary mapblock position to iterate
- pos2 is the highest boundary mapblock position to iterate
- MapArea::get_min_pos( )
Return the lowest boundary mapblock position of the area as a table {x,y,z}
MapArea::get_max_pos( )
Return the highest boundary mapblock position of the area as a table {x,y,z}
MapArea::get_volume( )
Calculate the volume of the area in cubic mapblocks and return the result
MapArea::has_index( idx )
Returns true if the specified mapblock index is contained within the area
MapArea::has_pos( pos )
Returns true if the specified mapblock position is contained within the area
Parses the mapblock data and calculates the associated checksum. For efficiency, the nodemeta map and the node list are not parsed automatically, but they can be obtained using the corresponding methods.
- blob is the raw mapblock data obtained from "data" field of the "blocks" table (required).
- is_preview is a boolean indicating whether to parse the BLOB.
- get_checksum is the checksum function to calculate the checksum and length of the BLOB.
- MapBlock::get_node_list( )
Parses the raw node list of the mapblock and returns a node_list table.
The node_list table is an ordered array of exactly 4096 elements, corresponding to the 16x16x16 matrix of nodes comprising the mapblock. The position of a node can be obtained from its index using the decode_node_pos( ) helper function. Each entry of the node_list table contains a subtable with three fields: id, param1, and param2.
Note that the id refers to the content ID which varies between mapblocks. You must cross-reference the content ID to determine the actual registered node name.
MapBlock::get_nodemeta_map( )
Parses the raw nodemeta map and returns a nodemata_map table.
The nodemeta_map table is an associative array of node indexes mapped to node metadata. Each entry of the nodemeta_map table is a subtable containing the following fields:
- fields is a subtable containing the user-defined metadata for the node, as ordinary key-value pairs.
- is_private is a boolean specifying whether the metadata of the node is private
- inventory is a subtable containing the inventory of the node as an array of tables, with two fields for each inventory slot: item_name and item_count
- MapBlock::version
The version of the mapblock.
MapBlock::flags
The flags of the mapblock.
MapBlock::content_width
The size of the content_ids in bytes. This is either 1 or 2, depending on the version.
MapBlock::params_width
The size of param1 and param2 in bytes. This is always 2.
MapBlock::object_list
An array of objects stored in the mapblock. Each entry contains a subtable with seven fields: type, pos, version, name, staticdata, hp, velocity, and yaw.
MapBlock::nodename_map
An associative array of registered node names indexed by content IDs.
MapBlock::timestamp
The timetamp when the mapblock was last modified by the engine. Note that this value is not necessarily a reliable means to determine if a mapblock was changed or not. For that you should perform a checksum comparison.
Opens an existing map.sqlite database from disk and prepares the necessary SQL statements.
- path is the path to the sqlite3 map database to be opened in read-only mode (required)
- is_preview is a boolean indicating whether mapblocks are to be parsed by default
- is_summary is a boolean indicating whether checksums apply to all mapblocks by default
- MapDatabase::enable_preview( )
Enable parsing of mapblocks by default
MapDatabase::disable_preview( )
Disable parsing of mapblocks by default, only calculate checksum and length
MapDatabase::enable_summary( )
Enable cumulative checksum calculations by default
MapDatabase::disable_summary( )
Disable cumulative checksum calculations by default
MapDatabase::change_algorithm( algorithm )
Switches to a different checksum algorithm, either 'adler32' or 'crc32'.
MapDatabase::create_cache( use_memory, on_step )
Create a cache database storing cross-references of mapblock position hashes, thereby speeding up successive queries. If use_memory is true, the cache database will be memory resident. Otherwise a file named "map.sqlite-cache" will be created in the same directory as the map database. The optional on_step hook can be used to update a progress bar for lengthy operations.
MapDatabase::get_length( )
Returns the total number of mapblocks. If the cache is available, then it will be used.
MapDatabase::get_area_length( area )
Returns the total number of mapblocks inside the given area. The cache is required for this operation.
MapDatabase::iterate( on_step )
Returns an iterator function, for looping over all existing mapblocks. The optional on_step hook can be used to update a progress bar for length operations
MapDatabase::iterate_area( area, on_step )
Returns an iterator function, for looping over all existing mapblocks inside the given area. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is required for this operation.
MapDatabase::select( on_step )
Returns an array of all hashed positions for all mapblocks. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is not used for this operation (but I will consider making it optional)
MapDatabase::select_area( area, on_step )
Returns an array of hashed positions for all mapblocks inside the given area. The optional on_step hook can be used to update a progress bar for lengthy operations. The cache is required for this operation.
MapDatabase::has_index( index )
Returns a boolean indicating whether a mapblock exists with the given index.
MapDatabase::get_mapblock( index )
Returns the mapblock with the given index as a MapBlock object.
MapDatabase::get_mapblock_raw( index )
Returns the raw mapblock data as a BLOB without calculating the checksum and length.
MapDatabase::close( index )
Closes the map database (but it doesn't close the cache database, which is a known bug).
- maplib.decode_pos( index )
Converts the given mapblock index to a mapblock position.
maplib.encode_pos( pos )
Converts the given mapblock position to a mapblock index.
maplib.decode_node_pos( node_index, index )
Converts the given node index within the given mapblock to a node position in world coordinates. If the mapblock index parameter is not provided, then the result will be relative to a mapblock at {0,0,0}. Note: For consistency with Lua conventions, node indexes are always 1-based even though mapblock indexes are 0-based.
maplib.encode_node_pos( pos )
Converts the given node position into a both a node index and a mapblock index.
maplib.hash_node_pos( [/i]node_pos[/i] )
Returns the equivalent of minetest.hash_node_position( ).
maplib.unhash_node_pos( node_hash )
Returns the equivalent of minetest.get_position_from_hash( ).
maplib.pos_to_string( pos )
Returns a string representing the given position as "(x,y,z)".
maplib.dump( buffer )
Returns a string representing a memory dump of the given buffer.
Code: Select all
local maplib = require "helpers" -- just the minimal API
print( maplib.pos_to_string( { x = 0, y = 10, z = 0 } ) )
Code Samples
This script will print the size of the mapblock at (0,-1,0) in bytes.
Code: Select all
require( "maplib" )
local pos = { x = 0, y = -1, z = 0 }
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", true, true )
local block = map_db.get_mapblock( encode_pos( pos ) )
if block then
print( string.format( "Size of mapblock at (%s) = %d bytes", pos_to_string( pos ), block.length ) )
else
print( "Mapblock not found!" )
end
Code: Select all
require( "maplib" )
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", true, true )
for index, block in map_db.iterate( ) do
print( string.format( "(%s) %d", pos_to_string( decode_pos( index ) ), block.checksum ) )
end
Code: Select all
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
local counts = { }
for index, block in map_db.iterate( status.on_step ) do
local node_list = block.get_node_list( )
local nodename_map = block.nodename_map
for _, node in ipairs( node_list ) do
local name = nodename_map[ node.id ]
if counts[ name ] then
counts[ name ] = counts[ name ] + 1
else
counts[ name ] = 1
end
end
end
for k, v in pairs( counts ) do
print( string.format( "%-8d %s", v, k ) )
end
Code: Select all
require( "maplib" )
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
local count = 0
for index, block in map_db.iterate( ) do
for k, v in pairs( block.nodename_map ) do
if v == "nyancat:nyancat" then
count = count + 1
end
end
end
print( count )
Code: Select all
require( "maplib" )
local map_db = MapDatabase( "/home/minetest/.minetest/worlds/world/map.sqlite", false )
map_db.create_cache( true )
local area = MapArea( { x = -100, y = -100, z = -100 }, { x = 100, y = -10, z = 100 } )
local count = map_db.get_area_length( area )
print( count )