Minetestmapper rewritten into C++

[Sokomine]

With the help of ShadowNinja and VanessaE, I modified a local copy of ShadowNinjas branch of the minetestmapper with the intention of getting a smaller file size where the buildings are easier to see. Of course that map looks far less good than the "normal" one - that is intended. My goal is to get a more abstract, much smaller map that helps players locate roads and towns.

Modifications done:
1. The colors where changed so that grass, dirt and papyrus got the grass-color. Sand, desert_sand, desert_stone, cactus and dry shrub all counts as desert_sand. Stone and gravel retain their color. All other content - possibly player-generated - is drawn in red.
2. The trees turned out to be annoying. Thus, leaves, apples, trees and torches are ignored/treated as air by the mapper.
3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

The result can be seen here: 1.7 MB map of Redcrabs server

It would be more like a real map if the streets where visible as such.

[twoelk]

now that looks really interesting!

I have been playing with the mapper for some while now, hoping not to have to change the code but have since come to the conclusion I can not avoid it to get the results I want.

But if somebody else is allready working on it ;-)

For one I wanted to get rid of the default color table that the program keeps defaulting to. For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest. If I delete everything else in the color.txt it takes the builtin values for grass,stone,etc. Setting them another color, such as white, blocks everything underneath.

So my sollution was either to define an ignore value such as is used for air or make the program use the values described in the external color.txt file only. (as far as I can see there is no ignore value, it is simply hardcoded to ignore air. With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.

[Sokomine]

For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest.

You can invert the exclusion of leaves/apples/etc and let it exclude anything that is not default:rail. That's pretty easy. I did it for a test with torches. Right now, all these changes are done to the source code and cannot be passed as parameters. But then, those are special maps anyway, and compiling is fast. Much faster than the actual map generation.

With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

There has to be an internal distinction between air and vacuum. Try to find out the node names!

The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.

Oh yes. That sounds very reasonable. All those odd sky plattforms which are not visible from the ground but still block up view have puzzled me a lot. And there have been several players wondering how to get to the space ships visible on the map of Redcrabs.

[A while later]After experimenting a bit with it, drawing for example one layer at a time (nodes at height 0-15 only, or 16-32), part of it turned out to be intresting. Plotting the areas below ground does not help much as that would require inverting what is drawn - instead of normal nodes, air would suddenly become particulary intresting. Else it's all more or less stone.

I'll attach the colors.txt file and a list of the tiny modifications to ShadowNinjas branch that might help to experiment a bit with it.

[Krock]

*BUMP*

It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?
Thanks in advance.

[sfan5]

When positive x and/or positive z coords are entered into the geometry option, the cpp minetestmapper generates white space from 0,0 to the specified positive coord and then it renders the area image.

This would be a cool feature indeed and usefull for many things. Slicing at a given hight would not only allow something like hight=30 to exclude platforms above that vallue but hight=-130 would give me a layout of all my tunnels at that level (and show me all caves as well of course)

Ok, another feature that would be nice (or perhaps it's a code line that can be tweaked as needed) is to specify a height or altitude to ignore things (kind of like how minetestmapper ignores the clouds).

3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?

I have implemented the features that were requested
Here's my fork: https://github.com/sfan5/minetest-mapper-cpp

[spillz]

Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)

[sfan5]

Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)

That would require me to include many Minetest headers too..

Random news: I added LevelDB support, but some optiomisations to the code wouldn't hurt.

[sfan5]

I just made a better (automatically generateable) colors.txt for my fork.
Even though the colors.txt was automatically generated using a (pretty primitive) algoritm they still look nice.
Here's a comparison:

[LazyJ]

I've been trying sfan5's fork of MinetestMapper C++.

The reuslts, so far, have been mixed but encouraging.



--backend sqlite3
I tried generating updates to the overview maps that I post in our Minetest.net Forums', LinuxGaming.us server thread. Our server is using the default, sqlite3 backend.

The four, 1,000x1,000 block, 'section' maps take about 2 minutes each when I have used Mireq's or, at present, VanessaE's MinetestMapper C++ fork. sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

The 'Main Section' overview map is 3,300x2,400. It usually takes about 5 minutes to render. When I tried to render it with sfan5's fork it took over 3 hours.



--backend leveldb
At first I kept getting 'Unknown map backend: leveldb' error messages.

The included README.rst file instructs to compile the minetestmapper by using:

Code: Select all

cmake .

make

To get the '--backend leveldb' to work I had to recompile minetestapper with:

Code: Select all

cmake . -DENABLE_LEVELDB=1

make

A 1,000x1,000, overview map, of a leveldb backend world, took about 5 minutes.

A 2,000x2,000, overview map, of the same leveldb backend world, took about 16 minutes.


Hopefully some of this info will help improve future updates to the minestmapper.

Thanks to Mireq, Sean884, Sokomine, VanessaE and sfan5 for progressing this vital Minetest tool.

[sfan5]

sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

Could you try reverting https://github.com/sfan5/minetest-mappe ... 807659e056 and then trying again. my mapper should not be slower than mireq's or VanessaE's.

Edit: I have found out why mireq's/VanessaE's mapper is faster:

(It does not generate the full map)

[Enke]

Pretty.....

*drools

[Argos]

I added a few patches on top of sfan5's version:

• Fixed the computation of block numbers to be included in the map (off-by-one)
• Failure to open output file is now caught (instead of causing SEGV)
• Fixed --min-y and --max-y (was broken)
• Added option --verbose which:
  • reports world and map statistics and coordinate limits
  • reports database access statistics
• Added option --forcegeometry, which creates a map of the requested geometry, even if the world is smaller
  (useful if the map is larger than minetestmapper will handle: generate the map in sections, with --forcegeometry, and glue those together)
• Fixed a memory leak in the sqlite database code (which was also a minor performance issue)
• Improved database performance:
  • by default, only blocks that are needed, are requested from the database now.
    leveldb should benefit greatly (I expect)
    as should sqlite when generating partial maps
  • bulk reading of one world row at a time is still an option for sqlite.
    I suspect it might be faster in some cases, depending on the map and on which part of it is being rendered.

Code on github

Miscellaneous info:

I tested my changes with an sqlite database on a system equipped with an SSD.
I'd like to hear about the performance effects when using an HDD.

I haven't gotten around to testing using a leveldb world yet. (feel free to beat me to it :-)

I noticed the code contains two licenses (LGPL 2.1 & BSD-2), without being clear about their respective statuses. That's something to be sorted out / clarified...

[sfan5]

I added a few patches on top of sfan5's version:[...]

I have merged some of them for now, I'll some more later.
Thanks for your contribution

[Amaz]

sfan5, I've been trying to use your auto generate colors, how do I use the avgcolors.py? When I run it like python avgcolors.py, it outputs this: Usage: avgcolor.py <input> So I'm presuming that there is something that I need to add after the avgcolors.py? I'm using Ubuntu. And where should the path $(find /minetest/dir -type f -name $2) in the command file lead to?
Thanks!

[sfan5]

I've clarified the instructions a bit.

[Argos]

I have merged some of them for now, I'll some more later.
Thanks for your contribution

Thanks.

Please let me know if there is anything I can contribute to getting them merged swiftly and smoothly.

[JPRuehmann]

Hello
Are there future plans to add redis backend support?
Thanks,
JPR

[sfan5]

Are there future plans to add redis backend support?

I already wanted to it yesterday, but here is it:
https://github.com/minetest/minetestmap ... be61c84336

[JPRuehmann]

Works
Thanks for the quick Fix.
JPR

[JPRuehmann]

Hello
I have a problem with the C minetestmapper
I´ve installed it into /usr/local/bin chown root:root chmod rwx,r-x,r-x.
if I start it from rhe init.lua in the mod Folder theres an normal .png image in the textures Folder.
But if I try to run it from the commandline minetestmapper --backend redis -i (Worldpath) -o (png File) i only get an black image.
what i´m doing wrong?
thanks,
JPR

[JPRuehmann]

Thanks now it works from the commandline to.

But a teeny wheeny thing there is in the examples the "" for the colors not there.
without it is not working in the help -h it is described correctly.

Thanks,
JPR

[sfan5]

But a teeny wheeny thing there is in the examples the "" for the colors not there.
without it is not working in the help -h it is described correctly.

The # not working without a " is the fault of your shell, programs usually do not take this into accout in their help.

[JPRuehmann]

Sorry I was to fast.
only the first try worked than all tries produced a black File with 1.4kb in Size.
This Command
minetestmapper --geometry -500:-500+1000+1000 --bgcolor '#000000' --backend redis -i /home/minetest/.minetest/worlds/World/ -o ~/Downloads/World-Map.png
Produced this Output on the Commandline.

Code: Select all

Unknown nodes:
4seasons:desertsand_winter
4seasons:grass_autumn
4seasons:grass_spring
4seasons:ice_flowing
4seasons:ice_source
4seasons:leaves_autumn
4seasons:leaves_spring
4seasons:sand_winter
bedrock:bedrock
bedrock:glass
beer:beer_grape
beer:beer_hops
bitchange:mineninth_in_ground
default:apple
default:cactus
default:clay
default:desert_sand
default:desert_stone
default:dirt
default:dirt_with_grass
default:dry_shrub
default:grass_1
default:grass_2
default:grass_3
default:grass_4
default:grass_5
default:gravel
default:lava_flowing
default:lava_source
default:leaves
default:nyancat
default:nyancat_rainbow
default:papyrus
default:sand
default:stone
default:stone_with_coal
default:stone_with_copper
default:stone_with_gold
default:stone_with_iron
default:stone_with_mese
default:tree
default:water_flowing
default:water_source
farming:pumpkin
farming_plus:banana
farming_plus:banana_leaves
farming_plus:carrot
farming_plus:cocoa
farming_plus:cocoa_leaves
farming_plus:orange
farming_plus:potato
farming_plus:rhubarb
farming_plus:strawberry
farming_plus:tomato
fire_flowers:fireflower
flowers:dandelion_white
flowers:dandelion_yellow
flowers:geranium
flowers:rose
flowers:tulip
flowers:viola
gloopblocks:stone_mossy
glow:shrooms
glow:stone_with_worms
glowblocks:glowblockwhite_on
moreblocks:super_glow_glass
nature:blossom
smooth:dirt_stone
trail:dirt_with_grass_walked

And resulted in the File

[sfan5]

Make sure your colors.txt file is fine, it seems like the color entries for the nodes are not there.

[JPRuehmann]

Hello
Sorry I don´t understand what you are talking about.
What is a color.txt File, where must it located and what is its content?
At least where can I get it?
Thanks,
JPR