numpy-izing minetestmapper

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

numpy-izing minetestmapper

by spillz » Wed Mar 05, 2014 03:06

Updated Mar 14, 14

What is it? an improved version of the python minetestmapper:

1. overhauled command line arguments
2. control the drawn region (--region, --minheight, and --maxheight options)
3. draw fog to have node further from the view point fade (--fog to control strength, --fogcolor to choose color)
4. draw cross-sections as well as top down view (--facing)
5. draw nodes that are larger than a single pixel (--pixelspernode option)
6. faster: 8 to 10 times faster than the original minetestmapper.py (but still slower than the c++ code)

The script is here: https://github.com/spillz/minetest/blob/master/util/minetestmapper-numpy.py
(repo is here: https://github.com/spillz/minetest)

You must have a recent python and numpy (see www.scipy.org) installed.

Performance

As a test I tried out the original and numpy versions of minetest mapper the on the Feb 25 posting of Stampy's world on Ubuntu laptop:

minetestmapper.py
Time 1m23s
Memory Usage: 52MB

minetestmapper-numpy.py
Time 10s
Memory Usage: 13MB

Examples

Examples using Morn76's Stampy's world:

Cross section (5 pixels per node)

Code: Select all
python minetestmapper-numpy.py --facing east --bgcolor lightblue --region 48 128 -384 -128 --drawscale --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye.png


Image

Top view (3 pixels per node)

Image

Command line arguments
Code: Select all
./minetestmapper-numpy.py -h


Code: Select all
usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--ugcolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--drawunderground-standalone]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--pixelspernode PPN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog (default grey)
  --ugcolor COLOR       set the color for underground areas (default purple)
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas overlaid on the map
  --drawunderground-standalone
                        draw underground areas as a standalone map
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes, default = -2000 2000 -2000 2000)
  --maxheight YMAX      don't draw above height YMAX (default = 500)
  --minheight YMIN      don't draw below height YMIN (defualt = -500)
  --pixelspernode PPN   number of pixels per node (default = 1)
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)


TODO:
1. Isometric view
2. GUI
3. Water transparency
4. Requests???
Last edited by spillz on Sat Apr 19, 2014 04:39, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 04:41

Updated, numpy-ized the drawing code too, so that now the stampyworld render takes about 10s.
 

User avatar
paramat
Developer
 
Posts: 3376
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

by paramat » Wed Mar 05, 2014 12:08

Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.
Code: Select all
sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16


A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.
Last edited by paramat on Wed Mar 05, 2014 12:16, edited 1 time in total.
 

User avatar
rubenwardy
Moderator
 
Posts: 5707
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
 

User avatar
Krock
Developer
 
Posts: 4252
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

by Krock » Wed Mar 05, 2014 12:58

Again a mapper? Please add LevelDB support. Thanks.
spillz wrote:(repo is here: https://github.com/spillz/minetest)

Ah, here we've got some kind of link destroyer --^

Good work anyway!
Last edited by Krock on Wed Mar 05, 2014 13:00, edited 1 time in total.
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 14:52

Krock wrote:Again a mapper? Please add LevelDB support. Thanks.


How is it different from the SQL database? I.e. is the layout identical? I don't know anything about LevelDB but so long as it has a python binding it shouldn't be too hard to add support for.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 14:53

rubenwardy wrote:Awesome, almost three times as fast.


Should be more like 8 to 10 times with latest version but YMMV
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 15:15

paramat wrote:Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.
Code: Select all
sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16



Maybe an overflow issue in the hex lookups (not my code), I will take a look.

A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.


By default? I don't have strong feeling either way, but it should be made easy for anyone to permanently set their preference without the need to set command line args every time.
 

User avatar
sfan5
Moderator
 
Posts: 3786
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 05, 2014 15:30

Krock wrote:
spillz wrote:(repo is here: https://github.com/spillz/minetest)

Ah, here we've got some kind of link destroyer --^

Good work anyway!

Just remove the ) from the URL
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 17:12

@paramat: the range bug might be a 64 bit issue. No such problem on my 32 bit laptop, but it is present on 64 bit desktop. Can you confirm? (Which is a pain because I do most of my spare time coding on the 32 bit laptop)
 

User avatar
paramat
Developer
 
Posts: 3376
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

by paramat » Wed Mar 05, 2014 20:03

My system:
Processor Intel® Core™2 CPU T7400 @ 2.16GHz × 2
OS type 32-bit
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 20:37

That's not the problem then. Hmm...
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 06, 2014 04:31

Latest version cleans up the code a bit (no more globals), changes the command line syntax plus adds new options

Code: Select all
./minetestmapper-numpy.py --help

usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN


Next steps... will try to eke slightly better performance out of it, fix rendering glitches, look into levelDB, and think about mapping caverns and cross-sections
Last edited by spillz on Thu Mar 06, 2014 04:42, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 06, 2014 14:14

@paramat: can you try again with latest version using --region switch? thx
Last edited by spillz on Thu Mar 06, 2014 14:23, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 06:46

Experimenting with cross-sections. Odd looking... (vertical axis label should be y instead of z)

[img=Cross sectional map image]http://s8.postimg.org/m8vxe91yd/mapx3.png[/img]
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 07:05

And here's karsthafen, around the city:

[img=cross section of karsthafen]http://s2.postimg.org/4tpsan0nd/mapkarst.png[/img]
 

4aiman
Member
 
Posts: 1208
Joined: Mon Jul 30, 2012 05:47

by 4aiman » Sun Mar 09, 2014 07:36

Always wanted to ask... Is it possible lo use lua VoxelManipulator to read terrain?
Maybe then it would be possible to make different kind of maps w/o hardcoding it in C?
 

User avatar
Krock
Developer
 
Posts: 4252
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

by Krock » Sun Mar 09, 2014 07:48

@spillz
Image
Or is that link an archivement, when reaching 666 posts?
Since there are 5 posts of you in a row, it should be possible to edit one.

@4aiman
I think the voxelManipulator is for mods only and the direct reading with sqlite3 in the map should be the fastest way.
Last edited by Krock on Sun Mar 09, 2014 07:50, edited 1 time in total.
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
sfan5
Moderator
 
Posts: 3786
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Sun Mar 09, 2014 08:39

Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
Krock
Developer
 
Posts: 4252
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

by Krock » Sun Mar 09, 2014 09:24

sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.

How much backends are planned yet?
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
sfan5
Moderator
 
Posts: 3786
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Sun Mar 09, 2014 10:28

Krock wrote:
sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.

How much backends are planned yet?

Planned, none (apart maybe adding sophia db.
But everyone could add a backend (open source yey!), I e.g. added a backend which used lua to do thing (you could use lua to connect to mysql for example).
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
paramat
Developer
 
Posts: 3376
Joined: Sun Oct 28, 2012 00:05
Location: UK
GitHub: paramat
IRC: paramat

by paramat » Sun Mar 09, 2014 17:21

spillz wrote:@paramat: can you try again with latest version using --region switch? thx

Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 19:27

paramat wrote:
spillz wrote:@paramat: can you try again with latest version using --region switch? thx

Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.


Cross-sec still buggy so not committed yet. Will try to fix it soon.

@krock: I will try to contain myself (maybe my last thread post was unnecessary, but I have always thought it legitimate to post updates as replies because the edit button doesn't show up as new activity and is easily missed). But also feel free not to read this thread ;)
Last edited by spillz on Sun Mar 09, 2014 20:18, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 13, 2014 22:36

I pushed latest changes to github, with new command line options "--facing", "--fog" and "--fogcolor"

Code: Select all
python minetestmapper-numpy.py --help


Code: Select all
usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--drawscale]
                               [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)



Code: Select all
python minetestmapper-numpy.py --region 0 660 300 600 --fogcolor lightblue --bgcolor lightblue --minheight 0 --maxheight 700 --drawplayers  --draworigin --facing south --fog 0.3 --drawscale karsthafen map.png


Sample output (karthafen at z position 600, facing south):

[img=cross section of karsthafen]http://i.imgur.com/NtJS7jO.png[/img]

Still has a few bugs and thinking about other features. e.g. gui, isometric view, drawing caves and transparent glass/water

PS: You need to install numpy for this to work (www.scipy.org)
Last edited by spillz on Thu Mar 13, 2014 22:58, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Fri Mar 14, 2014 03:54

(Last unsolicited post from me for a while, krock, promise)

A few bug fixes and here's one more option --pixelspernode

Code: Select all
python minetestmapper-numpy.py --drawscale --facing east --bgcolor lightblue --region 48 128 -384 -128 --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye2.png


Each node takes the specified number of pixels but the edge shading operates at the 1 pixel level:

Stampy's world. Cross Section (5 ppn)... grr... bug with "sky" looking like it is in front of the nodes:

Image

Stampy's world. Topo view (3 ppn):

Image
Last edited by spillz on Fri Mar 14, 2014 04:12, edited 1 time in total.
 

Next

Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 1 guest