Minetestmapper rewritten into C++

Post Reply
User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: Minetestmapper rewritten into C++

by Kilarin » Post

Evergreen wrote:the python one is still under /minetestdir/util/minetestmapper.py There just isn't a C++ mapper.
Not in my installation. At least I can't find it. I even pulled up synaptic package manager and browsed all installed files for minetest. Not util folder, no minetestmapper.py. Browsed through the .deb file, no minetestmapper.py.

Perhaps it just isn't in the dev builds? I'm running on:

I'm running: minetest_201406020414-0~2778~ubuntu12.04.1_i386.deb

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

Evergreen wrote:
Kilarin wrote:Am I understanding correctly that minetestmapper is no longer included with minetest, and I'll need to download one of the forks and compile it?
No, the python one is still under /minetestdir/util/minetestmapper.py There just isn't a C++ mapper.
never was in any windows build I know of, but I guess that is no surprise to anyone.
Would be nice if the link to the way outdated exe in the first post could be fixed or commented because it is somewhat missleading.

Just for the records: I usually make a subfolder called "mapper" in the world folder of interest. In that I put the minetest_mapper.exe, a version of colors.txt and a batch file I called "make_local_map.bat" which has the following content:

Code: Select all

minetest_mapper.exe -i .. -o World201406xx.png --drawscale  --draworigin --drawplayers  --geometry -6000:-6000+12000+12000
This will put a map file in the world folder, so I dont have to do some complicated map sorting of multiple worlds. Of course I have to rename the date part as I never got around to include a timestamp. I guess something like

Code: Select all

 echo %date%-%time%
in there should do the job.

As you may have noticed, I indeed have various mapper folders, each in a world folder I want a map of. This way I have all settings and tweaks for the specific world in question in a subfolder of the said world and as the mapper files are not really that big this seemed the easiest way - besides I'm too lazy to write sophisticated scripts.

User avatar
JPRuehmann
Member
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany
Contact:

Re: Minetestmapper rewritten into C++

by JPRuehmann » Post

Hi
One little thingy about your way.
The Idea with the Batch file in the acording World folder is good, to put the map in the World folder to is fine to.
But the colors file should only be there if you want to youse a special colors file for that world. The normal place for the colors file is besides the minetest.conf File in the Main Folder.
The exe File should be in a central folder that is searched by Windows so you can use the command minetestmapper from anywhere in the System without declaring the Path everytime, echo $PATH should tell you what paths are declared. The Windows Folder should be in there. Another way is to put the exe into the bin Folder in the minetest folder and add the path to the $Path Variable.
Thats because the Programms are frequently Updated and in this way you only have to keep track of one exe and one color file.
the special settings are in the batch File so they should function normaly.
hope that gives some help to.
have Fun,
JPR

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

interesting points but nope.
To explain a little, I have not installed MT on any computer locally. I use a run-in-place version I carry around with me on a portable device. I use relative paths for everything as all absolute paths change whenever I plug into some other machine. I usually avoid setting paths on the host machine, minimising the stuff I have to cleanup when I leave the computer. Having everything relevant in each world folder is usefull for quick moving stuff around like when I move to a new MT-dev version or sharing worlds with others and I have several colors.txt files for different purposes just as the batch files are different for various worlds.

I don't like having to type long paths that I have to memorize or scribble to some paper whenever I start a program with parameters that I can't remember anyways. What I have now is a one click sollution that is easily portable to any world folder and works without much input - until I start tweaking again of course.

User avatar
JPRuehmann
Member
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany
Contact:

Re: Minetestmapper rewritten into C++

by JPRuehmann » Post

Hello
that is a point.
But still then you can use a central minetestmapper File by putting the exe in the bin folder and call it in the batch files from there.
you can cäll a batch file from the autostart that ads the path to $PATH Variable two.
but that are only Ideas and oppinions for the other users that are using MT from the local HD.
Have Fun,
JPR

P.S.: I am using Linux.

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

I just couldn't leave the code doing only half the job, so here is a batch-file version that generates a file name for the resulting map that includes the date in ISO 8601 format. So just include this script snippet into whatever batch file you use to run the mapper with. Of course for running on Linux you may have to adjust it a little to run as bash script. ;-P

Code: Select all

SET isodate=%date:~6,4%%date:~3,2%%date:~0,2%T%time:~0,2%%time:~3,2%
SET isodate=%isodate: =0%

minetest_mapper.exe -i .. -o World-%isodate%.png --drawscale  --draworigin --drawplayers  --geometry -6000:-6000+12000+12000
this script assumes it resides in a subfolder of the world-folder together with the minetest_mapper.exe and the colors.txt and should generate a map file of the style "World-20140617T1801.png" in that same subfolder. The two dots let it navigate relative to it's current position. This way you can make a clean folder for this in some general place and simply copy it to all new worlds as you create them and the batch will always work regardless of any name change to the world it resides in.

User avatar
JPRuehmann
Member
Posts: 334
Joined: Fri Mar 21, 2014 21:40
Location: Germany
Contact:

Re: Minetestmapper rewritten into C++

by JPRuehmann » Post

Hello
Lastly you´ve sended me a tarball, Thanks.
But when will it get into the git Version?
Thanks,
JPR

Argos
Member
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Post

JPRuehmann wrote:Hello
Lastly you´ve sended me a tarball, Thanks.
But when will it get into the git Version?
Thanks,
JPR
As a matter of a fact, I just finished testing everything yesterday. I am currently making the new version release-ready (updating changelog, etc), and I hope to push it today.

---Update: I just pushed the new version.
Changes:
  • make install can now create a directory hierarchy
  • generates deb and/or rpm packages
  • '@include' directive for colors files
  • Fully transparent nodes (alpha=0) are treated specially
  • Air nodes can optionally be rendered as well (they should be defined as fully transparent)
  • Freeminer support (new-style database keys for leveldb)
---2nd update:
  • In addition to the existing way of computing transparency, which makes transparent colors darker and more opaque with depth, --drawalpha now also supports a second method which preserves transparency regardless of depth.

User avatar
LazyJ
Member
Posts: 687
Joined: Wed Sep 12, 2012 12:29
Location: Podunk, Nowhere, USA

Re: Minetestmapper rewritten into C++

by LazyJ » Post

Thanks for all you've done for this mod, Argos. ;)

I haven't used some of the fancier functions yet but it does a really nice job producing the maps for our server. \o/

User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: Minetestmapper rewritten into C++

by Kilarin » Post

using geometry, I can set horizontal starting coords and the width for each
--geometry 100:200+300+400
would map a region starting at horizontal coords 100,200, and going to 400,600, correct?

Is there any way to set a vertical limit on mapping? For example, to say that I want to map the same region as above, but only map coords with a height (minetests Y coord) starting at 100 and extending up to 200?

I'm asking because in floatlands/skytest it would be very handy to be able to control the vertical region being mapped since the floating islands overlap each other

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

edit: deleted nonsense answer
gah - if I could read I wouldn't write nonsense

User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: Minetestmapper rewritten into C++

by Kilarin » Post

twoelk wrote:if I could read I wouldn't write nonsense
I can read, and it's never stopped me. :)

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

Here is a little preview version of a batch file driven GUI for the windows version I am working on.
Batch-file-GUI for Minetest-Mapper using the WizApp gadget. (Dropbox link, so "rightclick-save-under" might not work)

Put the folder found in the zip into the minetestmapper folder so that it looks something like this:

Code: Select all

some_folder/
├── minetest_mapper.exe
└── wizapp/ (or some other name)
    └── wizapp.exe (the program supplying the eyecandy - not by me)
    └── runmapper.bat (my batch, start this to run the app)
The homepage of the WizApp gadget I used may be found here: http://wizapp.sf.net

To run it you will have to start the runmapper.bat. You can then choose a world anywhere on your system and choose a name (or use the one the app suggests) for the map. As for now you can toggle drawscale, drawplayers and draworigin. You cannot yet alter the geometry of the output or change the colors. These are things to come as well as saving some settings including an individual path to the minetest_mapper.exe.

As said before this is a preview and I am mostly interested wether this works on other windows versions and I have found all ways to crash it so far.

(edit)
So far it seems to work on Vista but will fail as is on XP. The fix to make it work in XP breaks some back and cancel buttons, correct error level handling just doesn't seem to work with this app under XP :-(

have fun
Last edited by twoelk on Sat Aug 02, 2014 12:27, edited 1 time in total.

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Minetestmapper rewritten into C++

by Krock » Post

twoelk wrote:Here is a little preview version of a batch file driven GUI for the windows version I am working on.

As said before this is a preview and I am mostly interested wether this works on other windows versions and I have found all ways to crash it so far.
have fun
Well, I can't get your tool working, it seems like it can't find a "goto point".

Code: Select all

E:\Programme\minetest\util\wizapp>start /w wizapp FB DIR
E:\Programme\minetest\util\wizapp>call wabatconfigs.bat
wizapp>set waoutput=E:\Programme\minetest\worlds\yapp
y2
wizapp>set waoutnum=
wizapp>set world_dir=E:\Programme\minetest\worlds\yap
py2
wizapp>set watext=
wizapp>set walistsep=
wizapp>set waoutput=
wizapp>if errorlevel 2 goto :cancel
wizapp>if errorlevel 1 goto :letsbegin
Das Sprungziel - letsbegin wurde nicht gefunden.
wizapp>
Btw, chrome marks the download as malicious
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

User avatar
Napiophelios
Member
Posts: 1035
Joined: Mon Jul 07, 2014 01:14
GitHub: Napiophelios
IRC: Nappi
In-game: Nappi

Re: Minetestmapper rewritten into C++

by Napiophelios » Post

I compiled something similar using a batch file I found here in the Minetest forums and a basic NSIS script

Image

Its not too fancy but it can speed things up a bit.Just started on this so I havent figured out how to
implement auto file names and defining multiple custom parameters is limited.

I currently have --drawscale --draworigin --drawplayers --geometry -6000:-6000+12000+12000 parameters as the default;
I found these parameters here in this topic and they work pretty good for me so far.

You will need to place the SFX.exe in your Minetest root folder (and also colors.txt if you have one)

Image
Attachments
MinetestMapper-Cpp_[NSIS-SFX].7z
Command-Line "GUI" for Minetest Mapper C++
(710.35 KiB) Downloaded 95 times

User avatar
addi
Member
Posts: 666
Joined: Thu Sep 20, 2012 03:16
GitHub: adrido
Location: Black-Forest, Germany

Re: Minetestmapper rewritten into C++

by addi » Post

before here are even more (more or less nice) gui's, just a question.
can somebody make a windows build of the minetestmapper first?
just because the downloads folder is empty

or is there another place to download it?

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Minetestmapper rewritten into C++

by Krock » Post

addi wrote:before here are even more (more or less nice) gui's, just a question.
can somebody make a windows build of the minetestmapper first?
just because the downloads folder is empty

or is there another place to download it?
I also asked for a new windows build, until now there's no new version available.
We windows users have to use This one
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

User avatar
addi
Member
Posts: 666
Joined: Thu Sep 20, 2012 03:16
GitHub: adrido
Location: Black-Forest, Germany

Re: Minetestmapper rewritten into C++

by addi » Post

Krock wrote:
addi wrote:before here are even more (more or less nice) gui's, just a question.
can somebody make a windows build of the minetestmapper first?
just because the downloads folder is empty

or is there another place to download it?
I also asked for a new windows build, until now there's no new version available.
We windows users have to use This one
uhh its nearly 1,5 years old :(
and it also does not support the new database backends

edit: Krock, you also provided some minetest builds. is it also possible to make and provide a minetest mapper build (maybe with including gui ;-))?

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: Minetestmapper rewritten into C++

by twoelk » Post

addi wrote:. . .
uhh its nearly 1,5 years old :(
and it also does not support the new database backends
yup, and as for now my batch-file is designed to work with the old one, sorry.

Have been playing with a new version of sfan5's fork but ran into some problems.
Simple stuff like having to include possible name variations for the exe or ignoring the screen output and more difficult stuff like checking if a backend is defined in the world.mt besides supporting all them wonderfull new parameters while I still haven't come up with a nice color chooser.

(edit)
Krock was the one that pointed out that the batch had difficulties running in XP

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Minetestmapper rewritten into C++

by Krock » Post

addi wrote:edit: Krock, you also provided some minetest builds. is it also possible to make and provide a minetest mapper build (maybe with including gui ;-))?
The codes contain some non-MSVC things, example:

https://github.com/minetest/minetestmap ... er.cpp#L11
https://github.com/minetest/minetestmap ... es.cpp#L10

EDIT: replace linux with non-MSVC
Last edited by Krock on Sat Aug 02, 2014 15:57, edited 1 time in total.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

sfan5
Moderator
Posts: 4094
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

Re: Minetestmapper rewritten into C++

by sfan5 » Post

Krock wrote:
addi wrote:edit: Krock, you also provided some minetest builds. is it also possible to make and provide a minetest mapper build (maybe with including gui ;-))?
The codes contain some linux-only things, example:

https://github.com/minetest/minetestmap ... er.cpp#L11
https://github.com/minetest/minetestmap ... es.cpp#L10
Those header files are present in MinGW.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

User avatar
Napiophelios
Member
Posts: 1035
Joined: Mon Jul 07, 2014 01:14
GitHub: Napiophelios
IRC: Nappi
In-game: Nappi

Re: Minetestmapper rewritten into C++

by Napiophelios » Post

Had a chance to work on this again tonight and made a few adjustments

MinetestMapper_SFX is a self-extracting archive created with NSIScript
that bundles a batch file and a copy of Minetest Mapper C++ together.
You can easily open with 7zip to view nsi and bat files used.

changes made in MinetestMapper_SFX v1.01:

-several gammar and punctuation errors within batch file fixed.
-Output files now have an option to add date time to end of file name.
-Option to rename pre-existing files with the same name by adding date time to end of file name instead of overwriting
-SFX Archive will now provide a colors.txt if one isnt present (skip if already there)
-removed splash screen and settings.ini functions
-various tweaks to both the html help & README files

To map a Minetest saved game world just place the MinetestMapper_SFX.exe file in you Minetest Root folder and launch.
Attachments
MinetestMapper_[NSIS-SFX1.01].7z
Command-Line "GUI" for Minetest Mapper C++
(817.65 KiB) Downloaded 106 times

Argos
Member
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Post

I have pushed a new update to my version of minetestmapper.

Notable new features are:
  • A height map (using different colors for different heights) can be generated instead of a regular map
  • The map can directly be generated in a reduced size. This allows overview maps of huge worlds to be easily created. In addition, this now also allows full world maps of such worlds to be actually generated (at a reduced size)..
  • A custom scale division can be specified.
For a full list of changes, consult the Changelog.

More infomation:

est31
Developer
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Post

Hi Argos, I like your mapper!
I have developed a small sloppy mapper script that calls your program to generate tiles. The main design idea behind this script was to not have one minetestmapper instance making a huge map and then cutting it into pieces ("top down") but to make minetestmapper generate many small tiles and then set them together for zoom out layers ("bottom up"). This approach allows to avoid keeping the whole map in RAM. minetestmapper and also the cutting tools other sloppy mappers use require some resources here.

For a small (600 MB), sqlite based map this works very well. You can try a live version here.

However, I had speed problems with an image of VanessaE's survival server(map download here, danger 7 GB). I ran this command:

Code: Select all

minetestmapper -i Survival_World/ --geometry -700,-2450+1000+1000 -o output
It renders only a small part of the map (1000x1000), but takes 40 seconds to execute. And that for a single tile, my standard config for a map has over 5000. My suspection is that most time is spent loading all area position hashes. I see this is the best way of doing it when you request a map of the whole world, which is usually only sparsely populated. But when you have many instances of minetestmapper called after another, they spend more time loading the keys than doing anything else. There are some options (ordered from easy to do to hard to do, but also by how much I think the solution helps solving the problem) to solve this:
-Adding a command line option that disables this loading, and instead tries to lookup the map for every required position
-When minetestmapper could store those keys in a file in a way that is optimised for fast access without having to load the whole file, it would greatly improve speed, too.
-If minetestmapper could be configured to generate many small tiles, prefferably multithreaded (but thats no priority), in one instance, and not store so much in RAM, it would be best.

What do you think, is the best option?

Argos
Member
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Post

est31 wrote:Hi Argos, I like your mapper!
Thanks!
I have developed a small sloppy mapper script that calls your program to generate tiles. The main design idea behind this script was to not have one minetestmapper instance making a huge map and then cutting it into pieces ("top down") but to make minetestmapper generate many small tiles and then set them together for zoom out layers ("bottom up"). This approach allows to avoid keeping the whole map in RAM. minetestmapper and also the cutting tools other sloppy mappers use require some resources here.
Good idea. And very nice interactive map !
My suspection is that most time is spent loading all area position hashes. I see this is the best way of doing it when you request a map of the whole world, which is usually only sparsely populated. But when you have many instances of minetestmapper called after another, they spend more time loading the keys than doing anything else.
Could be. In particular on a leveldb database. I think sqlite3 (which uses an index) and redis (which keeps everything in memory anyway) are pretty efficient at producing block lists. Does VanessaE's world use leveldb ?

But then, even for a leveldb database, I presume that the block list should be in the disk cache by the time the first mapper process terminates, so that subsequent instances don't need the disk access...

I know from my own tests that decompression of map blocks is often quite a large percentage of the execution time. Much larger than reading block numbers. But those tests were on larger images (I think 10000x10000), and I must confess I don't own a quad-core i7...

Nevertheless, whatever my results, YMMV. There are a few easy ways you could determine where your version actually spends time::
  • By far the easiest: use --verbose=2. As soon as it has printed the 'World Geometry', all map positions (not the blocks themselves) have been loaded, and analyzed. After the 'Map Output Geometry' has been printed, the rest of the time is spent loading blocks, unzipping them, generating the map and generating/writing the image.

    In order to make more exact measurements, you could use

    Code: Select all

    script -f -c '<minetestmapper command> --verbose=2' | awk '{printf "[%s] %s\n", strftime(), $0}'
    
    to log the output with timing information. If your awk complains, you may need to use gawk.
  • Use your OS's tools to analyze. On Linux, I used perf from linux-tools - which showed that unzipping was very time-consuming.
  • Instrument the code :-)
  • ...
It would be helpful if you could analyze where your version really spends it time...
There are some options (ordered from easy to do to hard to do, but also by how much I think the solution helps solving the problem) to solve this:
-Adding a command line option that disables this loading, and instead tries to lookup the map for every required position
-When minetestmapper could store those keys in a file in a way that is optimised for fast access without having to load the whole file, it would greatly improve speed, too.
-If minetestmapper could be configured to generate many small tiles, prefferably multithreaded (but thats no priority), in one instance, and not store so much in RAM, it would be best.
I agree. That last option would be a very nice feature. I have actually thought about such a feature, but not yet analyzed the impact, or even decided to do so :-)
An added advantage would be, that special care could be taken to ensure that the boundaries between tiles have correct shading. Now they probably don't, unless you generate larger images and cut the borders (1 pixel should suffice).
What do you think, is the best option?
The third option has definite architectural advantages. Except for ease of implementation indeed. I think the second option should be relatively painless. Even the cost of loading a 60MB to 120MB file of block numbers (for a HUGE world with 20M blocks) would probably be neglegible. The first option may indeed be easiest to implement, and verify for its benefit - which it might very well not have in a significant way... If it does have a significant benefit, it will probably be a quick win.

So the 'best' option depends on one's goals...

However, I do think there is an option that is easier still: you could consider generating 10240x10240 maps instead of 1024x1024. That reduces the number of times the blocks need to be read from the database by a factor 100, while still generating reasonably small maps (requiring maybe 0.5 GB of RAM per image).

WRT multithreading: I suspect the best way to make use of multithreading is to parallelize the decompression of map blocks. The easiest way to make use of multiprocessing though, at the moment, is probably to start several instances of minetestmapper in parallel :-)

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests