Minetestmapper rewritten into C++

est31
Developer
 
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Tue Feb 24, 2015 20:56

See this pastebin script for my output: http://pastebin.com/0bauwi8y . So it needs around 36 seconds between startup and printing "Map Output Geometry", and 3 seconds until it has generated an image.
Yes, VanessaE's map is leveldb.
 

LazyJ
Member
 
Posts: 585
Joined: Wed Sep 12, 2012 12:29
Location: Podunk, Nowhere, USA

Re: Minetestmapper rewritten into C++

by LazyJ » Wed Feb 25, 2015 00:22

Argos wrote:I have pushed a new update to my version of minetestmapper.

Notable new features are:
  • A height map (using different colors for different heights) can be generated instead of a regular map
  • The map can directly be generated in a reduced size. This allows overview maps of huge worlds to be easily created. In addition, this now also allows full world maps of such worlds to be actually generated (at a reduced size)..
  • A custom scale division can be specified.
For a full list of changes, consult the Changelog.

More infomation:


The documentation is good and useable and the mapper does an excellent job.

I've been using Argos'/Rogier-5's minetestmapper to create the overview maps for LinuxGaming ever since he released his fork. A very reliable program that does the job very well.

Many thanks, Argos, for the great work you've done. ;)
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Wed Feb 25, 2015 11:11

est31 wrote:[...] So it needs around 36 seconds between startup and printing "Map Output Geometry", and 3 seconds until it has generated an image.


There should be some room for improvement there :-)

I did a quick hack, and pushed a new branch ('WIP-mt') with a patch that disables prefetching the blocklist. Maybe you'd like to give it a try.

It does need more work before it is in a releasable state: the old behavior is no longer available, and it will also fail on freeminer worlds using leveldb.
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Wed Feb 25, 2015 14:28

The documentation is good and useable and the mapper does an excellent job.

I've been using Argos'/Rogier-5's minetestmapper to create the overview maps for LinuxGaming ever since he released his fork. A very reliable program that does the job very well.

Many thanks, Argos, for the great work you've done. ;)


Thanks LazyJ !
 

User avatar
rubenwardy
Moderator
 
Posts: 5790
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Minetestmapper rewritten into C++

by rubenwardy » Wed Feb 25, 2015 15:55

You may be interested in this pull request:

Split block position into separate fields in SQLite3 database - https://github.com/minetest/minetest/pull/1845
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Thu Feb 26, 2015 11:33

rubenwardy wrote:You may be interested in this pull request:

Split block position into separate fields in SQLite3 database - https://github.com/minetest/minetest/pull/1845


Looks quite interesting indeed. It would almost certainly benefit minetestmapper's mapping speed on smaller parts of a large world.

As you may have read, I started testing this with minetestmapper, but on some conversion tests I did between different database formats, which should give a good impression of relative sqlite read and write performances with and without the patch, the new format appears to have significantly worse performance...

I don't know about priorities for minetest, but if the performance can't be improved, personally, I'd be surprised if the patch made it...
 

est31
Developer
 
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Fri Feb 27, 2015 07:29

OK, far more faster for areas where there is no "miss": http://pastebin.com/cnSpNCGG However, when I run it on an area where everything is a "miss", I get an execution time of 57 seconds. So unfortunately that option doesn't help much. What about the second? A file that stores all (64 bit) block positions uncompressed for that map would need around 96 MiB. I'm sure gzip would improve this even further.
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Fri Feb 27, 2015 08:02

est31 wrote: However, when I run it on an area where everything is a "miss", I get an execution time of 57 seconds. So unfortunately that option doesn't help much.


Please, do use --min-y and --max-y to limit the vertical size of the area you are mapping... Minetestmapper is currently querying two orders of magnitude more blocks than it is mapping (36150 mapped blocks, 8.26M queried):
Code: Select all
Statistics:  blocks read: 8257398  (0 cached + 36150 uncached);  blocks rendered: 36150;  area rendered: 4032/4032  (1032192 nodes)


If you were to use --max-y 256 and --min-y -256 for instance, it would save about 8.2M needless queries, and generate the map almost instantaneously...

I don't suppose that you have stray blocks or other content near Y=32700 that must absolutely be mapped, along with other content that is at around level Y=0 :-)

Without limiting the Y direction, on an area where everything is a 'miss', the number of queries would probably double to 16.5M.
Given that the running time of the first area is about 30s, I'm not surprised the running time of the second (empty) area would be almost twice as much. With --min-y and --max-y, the 'empty' map would probably be generated in about 2*(almost instantaneously) :-)
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

est31
Developer
 
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Fri Feb 27, 2015 10:00

wow, that made it blazingly fast. thanks! Can you make that patch an option? I guess its still better for larger areas.
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Fri Feb 27, 2015 10:29

est31 wrote:wow, that made it blazingly fast. thanks! Can you make that patch an option?

I've already been doing some work on polishing it, integrating it with the existing functionality, and making it and 'official' option. But there's still work to do - it's more work than I expected...

I guess its still better for larger areas.

Probably not, actually...

Mapping a larger area with a lot of emtpy space causes a lot of non-existant nodes to be queried in vain, and a lot of needless CPU cycles to be wasted on that - as you found out by not limiting the Y range :-) That's the tradeoff. For areas that contain a small percentage of the world's nodes, and a high percentage of existing blocks, this patch speeds up mapping. In other cases it can slow it down tremendously, and it is better to limit the search by getting a block list first.

I tested it on VanessaE's world, mapping areas of 1k, 4k, 16k and 64k (i.e. the full world), and limiting the Y direction: Do compare the 'query' and 'read' columns, for the number of blocks queried, and the number of blocks actually read from the database:
Code: Select all
Backend    Size    Y  Opt   Query    Read  Render    Area   Total    User     Sys      Time  CPU    Input  Output
leveldb    1024   63   no    0.0M    0.0M    0.0M    0.0M    0.0M    61.1    12.5      1:19  92%  17.401M     520
leveldb    1024   63  yes    0.0M    0.0M    0.0M    0.0M    0.0M     2.3     0.1      0:03  80%   0.077M     520
leveldb    4096   63   no    0.2M    0.2M    0.2M    0.1M    0.1M    88.9    13.7      1:52  91%  17.656M     792
leveldb    4096   63  yes    0.3M    0.2M    0.2M    0.1M    0.1M    30.6     0.7      0:34  89%   0.401M     792
leveldb   16384   63   no    1.1M    1.1M    1.1M    0.3M    1.0M   199.5    14.7      3:50  92%  18.378M    1752
leveldb   16384   63  yes    7.0M    1.1M    1.1M    1.0M    1.0M   181.4     2.7      3:17  93%   1.504M    1752
leveldb   65536   63   no    1.3M    1.3M    1.3M    0.3M    6.5M   248.2    14.2      4:39  93%  18.613M    2088
leveldb   65536   63  yes  126.7M    1.3M    1.3M   16.0M   16.0M  1136.2     4.4     19:23  98%   2.320M    2168
leveldb   65536  255   no    3.0M    3.0M    3.0M    0.3M    6.5M   424.4    14.8      7:40  95%  18.944M    2096
leveldb   65536  255  yes  507.1M    3.0M    3.0M   16.0M   16.0M  3931.0     8.5   1:06:26  98%   3.480M    2184


BTW: This table also shows how extremely inefficient the leveldb database is at obtaining the block list... It probably just reads the entire database...
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

User avatar
jp
Member
 
Posts: 744
Joined: Wed Dec 18, 2013 09:03
Location: France
GitHub: kilbith
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Fri Feb 27, 2015 14:06

Hi JP !

jp wrote:Simple question : you intend to contribute on the official repo ?

I have tried to contribute, but apparently there was no interest in my patches :-(

One or two trivial ones got merged, and apart from that, there was no feedback.

...so I'm just maintaining this version...
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Mon Mar 02, 2015 08:59

I've put together windows builds (32 & 64 bit) of minetestmapper. Cross-compiled using MinGW on Linux. They start fine under Wine, and from what I hear, they produce nice maps as well on a real copy of Windows :-)

They can be downloaded here
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

User avatar
addi
Member
 
Posts: 658
Joined: Thu Sep 20, 2012 03:16
Location: Black-Forest, Germany
GitHub: adrido

Re: Minetestmapper rewritten into C++

by addi » Wed Mar 04, 2015 14:33

lot of thanks for your windows build. it works great :)
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Wed Mar 11, 2015 10:51

addi wrote:lot of thanks for your windows build. it works great :)

Thanks! (& good to hear additional confirmation that it works well a real copy of Windows)
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

User avatar
cHyper
Member
 
Posts: 842
Joined: Fri May 06, 2011 08:49
Location: Austria
GitHub: cHyper-0815OL
IRC: cHyper
In-game: cHyper cHyper_0815
 

est31
Developer
 
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Wed Mar 25, 2015 05:50

The official version of "--disable-blocklist-prefetch" doesn't have any speed improvements, only the one at WIP-mt. Does it really disable the prefetch?
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Wed Mar 25, 2015 23:58

est31 wrote:The official version of "--disable-blocklist-prefetch" doesn't have any speed improvements, only the one at WIP-mt. Does it really disable the prefetch?

That's puzzling... It should disable prefetch, and it should give a performance improvement on smaller parts of large worlds - comparable to the WIP-mt version... Did you use --min-y and --max-y in your tests ?

I just re-compiled the exact version that is on github, and re-ran a test on Vanessa's world to be certain, and I do see significant improvements, so could you provide more details about your tests ? Preferably the command-line and the output of minetestmapper with --verbose=2, with and without --disable-blocklist-prefetch, and preferably --backend set to an appropriate value. If you have /usr/bin/time installed, its output could be useful too (the shell built-in version of time would not provide much useful information).
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

est31
Developer
 
Posts: 173
Joined: Mon Dec 29, 2014 01:49

Re: Minetestmapper rewritten into C++

by est31 » Fri Mar 27, 2015 06:05

When I checked again, I couldn't reproduce it anymore. So it seems to be my mistake.
 

User avatar
12Me21
Member
 
Posts: 873
Joined: Tue Mar 05, 2013 00:36
Location: (Ignore all of my posts before 2018)
GitHub: 12Me21

Re: Minetestmapper rewritten into C++

by 12Me21 » Fri Mar 27, 2015 14:16

I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
 

User avatar
addi
Member
 
Posts: 658
Joined: Thu Sep 20, 2012 03:16
Location: Black-Forest, Germany
GitHub: adrido

Re: Minetestmapper rewritten into C++

by addi » Fri Mar 27, 2015 15:03

12Me21 wrote:I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
maybe because its a command-line tool?

https://github.com/Rogier-5/minetest-mapper-cpp/wiki/Downloads wrote:
  • Unpack it somewhere in your user directory. Do not place it in c:\Program Files\ or another 'standard' location for programs. That seems to cause problems. Alongside your minetest package should be just fine.
  • Start minetestmapper from a command-line.

    minetestmapper-20150301\bin\minetestmapper.exe
  • If all is well, it prints a long help message. Next, read the manual (included in the doc directory), and start mapping.
 

User avatar
12Me21
Member
 
Posts: 873
Joined: Tue Mar 05, 2013 00:36
Location: (Ignore all of my posts before 2018)
GitHub: 12Me21

Re: Minetestmapper rewritten into C++

by 12Me21 » Fri Mar 27, 2015 15:15

addi wrote:
12Me21 wrote:I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
maybe because its a command-line tool?

https://github.com/Rogier-5/minetest-mapper-cpp/wiki/Downloads wrote:
  • Unpack it somewhere in your user directory. Do not place it in c:\Program Files\ or another 'standard' location for programs. That seems to cause problems. Alongside your minetest package should be just fine.
  • Start minetestmapper from a command-line.

    minetestmapper-20150301\bin\minetestmapper.exe
  • If all is well, it prints a long help message. Next, read the manual (included in the doc directory), and start mapping.


I can't use the command line because I'm not an admin.
 

User avatar
Krock
Developer
 
Posts: 4341
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: Minetestmapper rewritten into C++

by Krock » Fri Mar 27, 2015 16:55

12Me21 wrote:I can't use the command line because I'm not an admin.

Create a new in the same directory as minetestmapper.exe: run.bat
Open it with notepad and save it with those contents:
Code: Select all
@echo off
minetestmapper.exe --help
pause


If you can't run that simple script, ask the system admin to allow it.
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

Argos
Member
 
Posts: 40
Joined: Tue Mar 04, 2014 21:47

Re: Minetestmapper rewritten into C++

by Argos » Fri Mar 27, 2015 18:56

Krock wrote:
12Me21 wrote:I can't use the command line because I'm not an admin.

Create a new in the same directory as minetestmapper.exe: run.bat
Open it with notepad and save it with those contents:
Code: Select all
@echo off
minetestmapper.exe --help
pause


If you can't run that simple script, ask the system admin to allow it.


Does cmd.exe need special privileges to run ?? I can't imagine that an administrator would disable cmd.exe, but allow regular users to install any software, like minetest or minetestmapper. That would also allow the user to install another shell / command processor, which trivially circumvents the cmd.exe restriction...

Anyway, if the script works, you could probably also use the following batch file to get a command-line:

Code: Select all
@echo off
cmd


I currently can't verify this on a windows box. If it doesn't work, you could search for cmd.exe on C:\ (it's probably in c:\windows\system32), and use the full location in the batch file. For example:

Code: Select all
@echo off
c:\windows\system32\cmd


Personally, I actually prefer creating and editing batch files to start minetestmapper (or really: the unix equivalent: shell scripts), instead of retyping the entire command-line every time. There is a short section on using batch files in the manual. It assumes you already have a command-prompt. If you don't, do add 'pause' as a last line, just like in Krock's example (I'll have to add a comment about that to the manual...)
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
 

twoelk
Member
 
Posts: 1324
Joined: Fri Apr 19, 2013 16:19
Location: northern Germany
GitHub: twoelk
IRC: twoelk
In-game: twoelk

Re: Minetestmapper rewritten into C++

by twoelk » Fri Mar 27, 2015 19:57

If you edit the link to the file name the Wizap script described on page 6 still works. The discussion above and below that entry also has some other interesting links and script examples.

Just make a plain txt file, copy some commands from the forum or write your own into it and then rename the file ending to *.bat
 

PreviousNext

Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 1 guest