Minetestmapper rewritten into C++

Sokomine
Member
 
Posts: 3886
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

by Sokomine » Thu Jan 30, 2014 04:49

With the help of ShadowNinja and VanessaE, I modified a local copy of ShadowNinjas branch of the minetestmapper with the intention of getting a smaller file size where the buildings are easier to see. Of course that map looks far less good than the "normal" one - that is intended. My goal is to get a more abstract, much smaller map that helps players locate roads and towns.

Modifications done:
1. The colors where changed so that grass, dirt and papyrus got the grass-color. Sand, desert_sand, desert_stone, cactus and dry shrub all counts as desert_sand. Stone and gravel retain their color. All other content - possibly player-generated - is drawn in red.
2. The trees turned out to be annoying. Thus, leaves, apples, trees and torches are ignored/treated as air by the mapper.
3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

The result can be seen here: 1.7 MB map of Redcrabs server

It would be more like a real map if the streets where visible as such.
A list of my mods can be found here.
 

twoelk
Member
 
Posts: 1328
Joined: Fri Apr 19, 2013 16:19
Location: northern Germany
GitHub: twoelk
IRC: twoelk
In-game: twoelk

by twoelk » Thu Jan 30, 2014 11:31

now that looks really interesting!

I have been playing with the mapper for some while now, hoping not to have to change the code but have since come to the conclusion I can not avoid it to get the results I want.

But if somebody else is allready working on it ;-)

For one I wanted to get rid of the default color table that the program keeps defaulting to. For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest. If I delete everything else in the color.txt it takes the builtin values for grass,stone,etc. Setting them another color, such as white, blocks everything underneath.

So my sollution was either to define an ignore value such as is used for air or make the program use the values described in the external color.txt file only. (as far as I can see there is no ignore value, it is simply hardcoded to ignore air. With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.
 

Sokomine
Member
 
Posts: 3886
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

by Sokomine » Fri Jan 31, 2014 20:27

twoelk wrote:For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest.

You can invert the exclusion of leaves/apples/etc and let it exclude anything that is not default:rail. That's pretty easy. I did it for a test with torches. Right now, all these changes are done to the source code and cannot be passed as parameters. But then, those are special maps anyway, and compiling is fast. Much faster than the actual map generation.

twoelk wrote:With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

There has to be an internal distinction between air and vacuum. Try to find out the node names!

twoelk wrote:The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.

Oh yes. That sounds very reasonable. All those odd sky plattforms which are not visible from the ground but still block up view have puzzled me a lot. And there have been several players wondering how to get to the space ships visible on the map of Redcrabs.

[A while later]After experimenting a bit with it, drawing for example one layer at a time (nodes at height 0-15 only, or 16-32), part of it turned out to be intresting. Plotting the areas below ground does not help much as that would require inverting what is drawn - instead of normal nodes, air would suddenly become particulary intresting. Else it's all more or less stone.

I'll attach the colors.txt file and a list of the tiny modifications to ShadowNinjas branch that might help to experiment a bit with it.
Attachments
mapper-changes.zip
(58.98 KiB) Downloaded 78 times
Last edited by Sokomine on Fri Jan 31, 2014 20:27, edited 1 time in total.
A list of my mods can be found here.
 

User avatar
Krock
Developer
 
Posts: 4391
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

by Krock » Sat Mar 01, 2014 13:08

*BUMP*

It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?
Thanks in advance.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 05, 2014 17:13

LazyJ wrote:When positive x and/or positive z coords are entered into the geometry option, the cpp minetestmapper generates white space from 0,0 to the specified positive coord and then it renders the area image.

twoelk wrote:This would be a cool feature indeed and usefull for many things. Slicing at a given hight would not only allow something like hight=30 to exclude platforms above that vallue but hight=-130 would give me a layout of all my tunnels at that level (and show me all caves as well of course)

LazyJ wrote:Ok, another feature that would be nice (or perhaps it's a code line that can be tweaked as needed) is to specify a height or altitude to ignore things (kind of like how minetestmapper ignores the clouds).

Sokomine wrote:3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

Krock wrote:It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?


I have implemented the features that were requested
Here's my fork: https://github.com/sfan5/minetest-mapper-cpp
Last edited by sfan5 on Wed Mar 05, 2014 21:02, edited 1 time in total.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 20:42

Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 05, 2014 21:01

spillz wrote:Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)

That would require me to include many Minetest headers too..

Random news: I added LevelDB support, but some optiomisations to the code wouldn't hurt.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Fri Mar 14, 2014 21:09

I just made a better (automatically generateable) colors.txt for my fork.
Even though the colors.txt was automatically generated using a (pretty primitive) algoritm they still look nice.
Here's a comparison:
Image
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

LazyJ
Member
 
Posts: 591
Joined: Wed Sep 12, 2012 12:29
Location: Podunk, Nowhere, USA

by LazyJ » Sat Mar 22, 2014 03:59

I've been trying sfan5's fork of MinetestMapper C++.

The reuslts, so far, have been mixed but encouraging.



--backend sqlite3
I tried generating updates to the overview maps that I post in our Minetest.net Forums', LinuxGaming.us server thread. Our server is using the default, sqlite3 backend.

The four, 1,000x1,000 block, 'section' maps take about 2 minutes each when I have used Mireq's or, at present, VanessaE's MinetestMapper C++ fork. sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

The 'Main Section' overview map is 3,300x2,400. It usually takes about 5 minutes to render. When I tried to render it with sfan5's fork it took over 3 hours.



--backend leveldb
At first I kept getting 'Unknown map backend: leveldb' error messages.

The included README.rst file instructs to compile the minetestmapper by using:

Code: Select all
cmake .

make


To get the '--backend leveldb' to work I had to recompile minetestapper with:

Code: Select all
cmake . -DENABLE_LEVELDB=1

make


A 1,000x1,000, overview map, of a leveldb backend world, took about 5 minutes.

A 2,000x2,000, overview map, of the same leveldb backend world, took about 16 minutes.


Hopefully some of this info will help improve future updates to the minestmapper.

Thanks to Mireq, Sean884, Sokomine, VanessaE and sfan5 for progressing this vital Minetest tool. ;)
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Sat Mar 22, 2014 08:32

LazyJ wrote:sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

Could you try reverting https://github.com/sfan5/minetest-mapper-cpp/commit/5bf2039bbb0d1f27141f7962125cc3807659e056 and then trying again. my mapper should not be slower than mireq's or VanessaE's.

Edit: I have found out why mireq's/VanessaE's mapper is faster:
Image
(It does not generate the full map)
Last edited by sfan5 on Sat Mar 22, 2014 12:30, edited 1 time in total.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
Enke
Member
 
Posts: 469
Joined: Fri Nov 15, 2013 02:56
Location: The internet
GitHub: NANOsoldierEnke
IRC: Enke
In-game: Enke

by Enke » Sun Mar 23, 2014 02:36

Pretty.....

*drools
Lush8
ExtraVars for Red Eclipse

<Anarchid> my turn was still the most awesome, yielding all the cripples, two captured paranormals, and death rate of about 30%
<ORCACommander> Anarchid: you need to work harder
<ORCACommander> I am hereby putting you under review until you can increase the casualty rate
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

by Argos » Wed Mar 26, 2014 09:06

I added a few patches on top of sfan5's version:

  • Fixed the computation of block numbers to be included in the map (off-by-one)
  • Failure to open output file is now caught (instead of causing SEGV)
  • Fixed --min-y and --max-y (was broken)
  • Added option --verbose which:
    • reports world and map statistics and coordinate limits
    • reports database access statistics
  • Added option --forcegeometry, which creates a map of the requested geometry, even if the world is smaller
    (useful if the map is larger than minetestmapper will handle: generate the map in sections, with --forcegeometry, and glue those together)
  • Fixed a memory leak in the sqlite database code (which was also a minor performance issue)
  • Improved database performance:
    • by default, only blocks that are needed, are requested from the database now.
      leveldb should benefit greatly (I expect)
      as should sqlite when generating partial maps
    • bulk reading of one world row at a time is still an option for sqlite.
      I suspect it might be faster in some cases, depending on the map and on which part of it is being rendered.

Code on github

Miscellaneous info:

I tested my changes with an sqlite database on a system equipped with an SSD.
I'd like to hear about the performance effects when using an HDD.

I haven't gotten around to testing using a leveldb world yet. (feel free to beat me to it :-)

I noticed the code contains two licenses (LGPL 2.1 & BSD-2), without being clear about their respective statuses. That's something to be sorted out / clarified...
Last edited by Argos on Wed Mar 26, 2014 09:11, edited 1 time in total.
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 26, 2014 15:59

Argos wrote:I added a few patches on top of sfan5's version:[...]

I have merged some of them for now, I'll some more later.
Thanks for your contribution
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

Amaz
Member
 
Posts: 348
Joined: Wed May 08, 2013 08:26
GitHub: Amaz1
IRC: Amaz
In-game: Amaz

by Amaz » Wed Mar 26, 2014 16:11

sfan5, I've been trying to use your auto generate colors, how do I use the avgcolors.py? When I run it like python avgcolors.py, it outputs this: Usage: avgcolor.py <input> So I'm presuming that there is something that I need to add after the avgcolors.py? I'm using Ubuntu. And where should the path $(find /minetest/dir -type f -name $2) in the command file lead to?
Thanks!
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 26, 2014 16:37

I've clarified the instructions a bit.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

by Argos » Wed Mar 26, 2014 16:52

sfan5 wrote:I have merged some of them for now, I'll some more later.
Thanks for your contribution

Thanks.

Please let me know if there is anything I can contribute to getting them merged swiftly and smoothly.
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany

Re: Minetestmapper rewritten into C++

by JPRuehmann » Sat Apr 26, 2014 11:34

Hello
Are there future plans to add redis backend support?
Thanks,
JPR
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

Re: Minetestmapper rewritten into C++

by sfan5 » Sat Apr 26, 2014 13:02

JPRuehmann wrote:Are there future plans to add redis backend support?

I already wanted to it yesterday, but here is it:
https://github.com/minetest/minetestmap ... be61c84336
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany

Re: Minetestmapper rewritten into C++

by JPRuehmann » Sun Apr 27, 2014 23:11

Hello
I have a problem with the C minetestmapper
I´ve installed it into /usr/local/bin chown root:root chmod rwx,r-x,r-x.
if I start it from rhe init.lua in the mod Folder theres an normal .png image in the textures Folder.
But if I try to run it from the commandline minetestmapper --backend redis -i (Worldpath) -o (png File) i only get an black image.
what i´m doing wrong?
thanks,
JPR
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany

Re: Minetestmapper rewritten into C++

by JPRuehmann » Tue Apr 29, 2014 20:16

Thanks now it works from the commandline to.

But a teeny wheeny thing there is in the examples the "" for the colors not there.
without it is not working in the help -h it is described correctly.

Thanks,
JPR
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

Re: Minetestmapper rewritten into C++

by sfan5 » Tue Apr 29, 2014 20:40

JPRuehmann wrote:But a teeny wheeny thing there is in the examples the "" for the colors not there.
without it is not working in the help -h it is described correctly.

The # not working without a " is the fault of your shell, programs usually do not take this into accout in their help.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany

Re: Minetestmapper rewritten into C++

by JPRuehmann » Tue Apr 29, 2014 20:45

Sorry I was to fast.
only the first try worked than all tries produced a black File with 1.4kb in Size.
This Command
minetestmapper --geometry -500:-500+1000+1000 --bgcolor '#000000' --backend redis -i /home/minetest/.minetest/worlds/World/ -o ~/Downloads/World-Map.png
Produced this Output on the Commandline.
Code: Select all
Unknown nodes:
4seasons:desertsand_winter
4seasons:grass_autumn
4seasons:grass_spring
4seasons:ice_flowing
4seasons:ice_source
4seasons:leaves_autumn
4seasons:leaves_spring
4seasons:sand_winter
bedrock:bedrock
bedrock:glass
beer:beer_grape
beer:beer_hops
bitchange:mineninth_in_ground
default:apple
default:cactus
default:clay
default:desert_sand
default:desert_stone
default:dirt
default:dirt_with_grass
default:dry_shrub
default:grass_1
default:grass_2
default:grass_3
default:grass_4
default:grass_5
default:gravel
default:lava_flowing
default:lava_source
default:leaves
default:nyancat
default:nyancat_rainbow
default:papyrus
default:sand
default:stone
default:stone_with_coal
default:stone_with_copper
default:stone_with_gold
default:stone_with_iron
default:stone_with_mese
default:tree
default:water_flowing
default:water_source
farming:pumpkin
farming_plus:banana
farming_plus:banana_leaves
farming_plus:carrot
farming_plus:cocoa
farming_plus:cocoa_leaves
farming_plus:orange
farming_plus:potato
farming_plus:rhubarb
farming_plus:strawberry
farming_plus:tomato
fire_flowers:fireflower
flowers:dandelion_white
flowers:dandelion_yellow
flowers:geranium
flowers:rose
flowers:tulip
flowers:viola
gloopblocks:stone_mossy
glow:shrooms
glow:stone_with_worms
glowblocks:glowblockwhite_on
moreblocks:super_glow_glass
nature:blossom
smooth:dirt_stone
trail:dirt_with_grass_walked

And resulted in the File
Attachments
World-Map.png
(1.2 KiB) Not downloaded yet
 

User avatar
sfan5
Moderator
 
Posts: 3816
Joined: Wed Aug 24, 2011 09:44
Location: Germany
GitHub: sfan5
IRC: sfan5

Re: Minetestmapper rewritten into C++

by sfan5 » Wed Apr 30, 2014 07:48

Make sure your colors.txt file is fine, it seems like the color entries for the nodes are not there.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
JPRuehmann
Member
 
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany

Re: Minetestmapper rewritten into C++

by JPRuehmann » Wed Apr 30, 2014 08:20

Hello
Sorry I don´t understand what you are talking about.
What is a color.txt File, where must it located and what is its content?
At least where can I get it?
Thanks,
JPR
 

PreviousNext

Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 0 guests