numpy-izing minetestmapper

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

numpy-izing minetestmapper

by spillz » Post

Updated Mar 14, 14

What is it? an improved version of the python minetestmapper:

1. overhauled command line arguments
2. control the drawn region (--region, --minheight, and --maxheight options)
3. draw fog to have node further from the view point fade (--fog to control strength, --fogcolor to choose color)
4. draw cross-sections as well as top down view (--facing)
5. draw nodes that are larger than a single pixel (--pixelspernode option)
6. faster: 8 to 10 times faster than the original minetestmapper.py (but still slower than the c++ code)

The script is here: https://github.com/spillz/minetest/blob ... r-numpy.py
(repo is here: https://github.com/spillz/minetest)

You must have a recent python and numpy (see www.scipy.org) installed.

Performance

As a test I tried out the original and numpy versions of minetest mapper the on the Feb 25 posting of Stampy's world on Ubuntu laptop:

minetestmapper.py
Time 1m23s
Memory Usage: 52MB

minetestmapper-numpy.py
Time 10s
Memory Usage: 13MB

Examples

Examples using Morn76's Stampy's world:

Cross section (5 pixels per node)

Code: Select all

python minetestmapper-numpy.py --facing east --bgcolor lightblue --region 48 128 -384 -128 --drawscale --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye.png
Image

Top view (3 pixels per node)

Image

Command line arguments

Code: Select all

./minetestmapper-numpy.py -h

Code: Select all

usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--ugcolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--drawunderground-standalone]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--pixelspernode PPN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog (default grey)
  --ugcolor COLOR       set the color for underground areas (default purple)
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas overlaid on the map
  --drawunderground-standalone
                        draw underground areas as a standalone map
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes, default = -2000 2000 -2000 2000)
  --maxheight YMAX      don't draw above height YMAX (default = 500)
  --minheight YMIN      don't draw below height YMIN (defualt = -500)
  --pixelspernode PPN   number of pixels per node (default = 1)
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)
TODO:
1. Isometric view
2. GUI
3. Water transparency
4. Requests???
Last edited by spillz on Sat Apr 19, 2014 04:39, edited 1 time in total.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

Updated, numpy-ized the drawing code too, so that now the stampyworld render takes about 10s.

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

by paramat » Post

Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.

Code: Select all

sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16
A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.
Last edited by paramat on Wed Mar 05, 2014 12:16, edited 1 time in total.

User avatar
rubenwardy
Moderator
Posts: 6978
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

by rubenwardy » Post

Awesome, almost three times as fast.
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

by Krock » Post

Again a mapper? Please add LevelDB support. Thanks.
spillz wrote:(repo is here: https://github.com/spillz/minetest)
Ah, here we've got some kind of link destroyer --^

Good work anyway!
Last edited by Krock on Wed Mar 05, 2014 13:00, edited 1 time in total.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

Krock wrote:Again a mapper? Please add LevelDB support. Thanks.
How is it different from the SQL database? I.e. is the layout identical? I don't know anything about LevelDB but so long as it has a python binding it shouldn't be too hard to add support for.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

rubenwardy wrote:Awesome, almost three times as fast.
Should be more like 8 to 10 times with latest version but YMMV

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

paramat wrote:Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.

Code: Select all

sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16
Maybe an overflow issue in the hex lookups (not my code), I will take a look.
A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.
By default? I don't have strong feeling either way, but it should be made easy for anyone to permanently set their preference without the need to set command line args every time.

sfan5
Moderator
Posts: 4095
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

by sfan5 » Post

Krock wrote:
spillz wrote:(repo is here: https://github.com/spillz/minetest)
Ah, here we've got some kind of link destroyer --^

Good work anyway!
Just remove the ) from the URL
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

@paramat: the range bug might be a 64 bit issue. No such problem on my 32 bit laptop, but it is present on 64 bit desktop. Can you confirm? (Which is a pain because I do most of my spare time coding on the 32 bit laptop)

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

by paramat » Post

My system:
Processor Intel® Core™2 CPU T7400 @ 2.16GHz × 2
OS type 32-bit

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

That's not the problem then. Hmm...

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

Latest version cleans up the code a bit (no more globals), changes the command line syntax plus adds new options

Code: Select all

./minetestmapper-numpy.py --help

usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN
Next steps... will try to eke slightly better performance out of it, fix rendering glitches, look into levelDB, and think about mapping caverns and cross-sections
Last edited by spillz on Thu Mar 06, 2014 04:42, edited 1 time in total.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

@paramat: can you try again with latest version using --region switch? thx
Last edited by spillz on Thu Mar 06, 2014 14:23, edited 1 time in total.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

Experimenting with cross-sections. Odd looking... (vertical axis label should be y instead of z)

[img=Cross sectional map image]http://s8.postimg.org/m8vxe91yd/mapx3.png[/img]

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

And here's karsthafen, around the city:

[img=cross section of karsthafen]http://s2.postimg.org/4tpsan0nd/mapkarst.png[/img]

4aiman
Member
Posts: 1208
Joined: Mon Jul 30, 2012 05:47

by 4aiman » Post

Always wanted to ask... Is it possible lo use lua VoxelManipulator to read terrain?
Maybe then it would be possible to make different kind of maps w/o hardcoding it in C?

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

by Krock » Post

@spillz
Image
Or is that link an archivement, when reaching 666 posts?
Since there are 5 posts of you in a row, it should be possible to edit one.

@4aiman
I think the voxelManipulator is for mods only and the direct reading with sqlite3 in the map should be the fastest way.
Last edited by Krock on Sun Mar 09, 2014 07:50, edited 1 time in total.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

sfan5
Moderator
Posts: 4095
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

by sfan5 » Post

Krock wrote:direct reading with sqlite3 in the map should be the fastest way.
You will not be able to support all upcoming map backends when you read directly from the map db.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

by Krock » Post

sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.
You will not be able to support all upcoming map backends when you read directly from the map db.
How much backends are planned yet?
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

sfan5
Moderator
Posts: 4095
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

by sfan5 » Post

Krock wrote:
sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.
You will not be able to support all upcoming map backends when you read directly from the map db.
How much backends are planned yet?
Planned, none (apart maybe adding sophia db.
But everyone could add a backend (open source yey!), I e.g. added a backend which used lua to do thing (you could use lua to connect to mysql for example).
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

by paramat » Post

spillz wrote:@paramat: can you try again with latest version using --region switch? thx
Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

paramat wrote:
spillz wrote:@paramat: can you try again with latest version using --region switch? thx
Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.
Cross-sec still buggy so not committed yet. Will try to fix it soon.

@krock: I will try to contain myself (maybe my last thread post was unnecessary, but I have always thought it legitimate to post updates as replies because the edit button doesn't show up as new activity and is easily missed). But also feel free not to read this thread ;)
Last edited by spillz on Sun Mar 09, 2014 20:18, edited 1 time in total.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

I pushed latest changes to github, with new command line options "--facing", "--fog" and "--fogcolor"

Code: Select all

python minetestmapper-numpy.py --help

Code: Select all

usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--drawscale]
                               [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)

Code: Select all

python minetestmapper-numpy.py --region 0 660 300 600 --fogcolor lightblue --bgcolor lightblue --minheight 0 --maxheight 700 --drawplayers  --draworigin --facing south --fog 0.3 --drawscale karsthafen map.png
Sample output (karthafen at z position 600, facing south):

[img=cross section of karsthafen]http://i.imgur.com/NtJS7jO.png[/img]

Still has a few bugs and thinking about other features. e.g. gui, isometric view, drawing caves and transparent glass/water

PS: You need to install numpy for this to work (www.scipy.org)
Last edited by spillz on Thu Mar 13, 2014 22:58, edited 1 time in total.

spillz
Member
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Post

(Last unsolicited post from me for a while, krock, promise)

A few bug fixes and here's one more option --pixelspernode

Code: Select all

python minetestmapper-numpy.py --drawscale --facing east --bgcolor lightblue --region 48 128 -384 -128 --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye2.png
Each node takes the specified number of pixels but the edge shading operates at the 1 pixel level:

Stampy's world. Cross Section (5 ppn)... grr... bug with "sky" looking like it is in front of the nodes:

Image

Stampy's world. Topo view (3 ppn):

Image
Last edited by spillz on Fri Mar 14, 2014 04:12, edited 1 time in total.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 2 guests