Minetestmapper rewritten into C++
Re: Minetestmapper rewritten into C++
See this pastebin script for my output: http://pastebin.com/0bauwi8y . So it needs around 36 seconds between startup and printing "Map Output Geometry", and 3 seconds until it has generated an image.
Yes, VanessaE's map is leveldb.
Yes, VanessaE's map is leveldb.
Re: Minetestmapper rewritten into C++
The documentation is good and useable and the mapper does an excellent job.Argos wrote:I have pushed a new update to my version of minetestmapper.
Notable new features are:For a full list of changes, consult the Changelog.
- A height map (using different colors for different heights) can be generated instead of a regular map
- The map can directly be generated in a reduced size. This allows overview maps of huge worlds to be easily created. In addition, this now also allows full world maps of such worlds to be actually generated (at a reduced size)..
- A custom scale division can be specified.
More infomation:
I've been using Argos'/Rogier-5's minetestmapper to create the overview maps for LinuxGaming ever since he released his fork. A very reliable program that does the job very well.
Many thanks, Argos, for the great work you've done. ;)
My blog: LazyJ's Minetest World
Re: Minetestmapper rewritten into C++
There should be some room for improvement there :-)est31 wrote:[...] So it needs around 36 seconds between startup and printing "Map Output Geometry", and 3 seconds until it has generated an image.
I did a quick hack, and pushed a new branch ('WIP-mt') with a patch that disables prefetching the blocklist. Maybe you'd like to give it a try.
It does need more work before it is in a releasable state: the old behavior is no longer available, and it will also fail on freeminer worlds using leveldb.
Re: Minetestmapper rewritten into C++
Thanks LazyJ !The documentation is good and useable and the mapper does an excellent job.
I've been using Argos'/Rogier-5's minetestmapper to create the overview maps for LinuxGaming ever since he released his fork. A very reliable program that does the job very well.
Many thanks, Argos, for the great work you've done. ;)
- rubenwardy
- Moderator
- Posts: 6978
- Joined: Tue Jun 12, 2012 18:11
- GitHub: rubenwardy
- IRC: rubenwardy
- In-game: rubenwardy
- Location: Bristol, United Kingdom
- Contact:
Re: Minetestmapper rewritten into C++
You may be interested in this pull request:
Split block position into separate fields in SQLite3 database - https://github.com/minetest/minetest/pull/1845
Split block position into separate fields in SQLite3 database - https://github.com/minetest/minetest/pull/1845
Re: Minetestmapper rewritten into C++
Looks quite interesting indeed. It would almost certainly benefit minetestmapper's mapping speed on smaller parts of a large world.rubenwardy wrote:You may be interested in this pull request:
Split block position into separate fields in SQLite3 database - https://github.com/minetest/minetest/pull/1845
As you may have read, I started testing this with minetestmapper, but on some conversion tests I did between different database formats, which should give a good impression of relative sqlite read and write performances with and without the patch, the new format appears to have significantly worse performance...
I don't know about priorities for minetest, but if the performance can't be improved, personally, I'd be surprised if the patch made it...
Re: Minetestmapper rewritten into C++
OK, far more faster for areas where there is no "miss": http://pastebin.com/cnSpNCGG However, when I run it on an area where everything is a "miss", I get an execution time of 57 seconds. So unfortunately that option doesn't help much. What about the second? A file that stores all (64 bit) block positions uncompressed for that map would need around 96 MiB. I'm sure gzip would improve this even further.
Re: Minetestmapper rewritten into C++
Please, do use --min-y and --max-y to limit the vertical size of the area you are mapping... Minetestmapper is currently querying two orders of magnitude more blocks than it is mapping (36150 mapped blocks, 8.26M queried):est31 wrote: However, when I run it on an area where everything is a "miss", I get an execution time of 57 seconds. So unfortunately that option doesn't help much.
If you were to use --max-y 256 and --min-y -256 for instance, it would save about 8.2M needless queries, and generate the map almost instantaneously...Code: Select all
Statistics: blocks read: 8257398 (0 cached + 36150 uncached); blocks rendered: 36150; area rendered: 4032/4032 (1032192 nodes)
I don't suppose that you have stray blocks or other content near Y=32700 that must absolutely be mapped, along with other content that is at around level Y=0 :-)
Without limiting the Y direction, on an area where everything is a 'miss', the number of queries would probably double to 16.5M.
Given that the running time of the first area is about 30s, I'm not surprised the running time of the second (empty) area would be almost twice as much. With --min-y and --max-y, the 'empty' map would probably be generated in about 2*(almost instantaneously) :-)
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
Re: Minetestmapper rewritten into C++
wow, that made it blazingly fast. thanks! Can you make that patch an option? I guess its still better for larger areas.
Re: Minetestmapper rewritten into C++
I've already been doing some work on polishing it, integrating it with the existing functionality, and making it and 'official' option. But there's still work to do - it's more work than I expected...est31 wrote:wow, that made it blazingly fast. thanks! Can you make that patch an option?
Probably not, actually...I guess its still better for larger areas.
Mapping a larger area with a lot of emtpy space causes a lot of non-existant nodes to be queried in vain, and a lot of needless CPU cycles to be wasted on that - as you found out by not limiting the Y range :-) That's the tradeoff. For areas that contain a small percentage of the world's nodes, and a high percentage of existing blocks, this patch speeds up mapping. In other cases it can slow it down tremendously, and it is better to limit the search by getting a block list first.
I tested it on VanessaE's world, mapping areas of 1k, 4k, 16k and 64k (i.e. the full world), and limiting the Y direction: Do compare the 'query' and 'read' columns, for the number of blocks queried, and the number of blocks actually read from the database:
Code: Select all
Backend Size Y Opt Query Read Render Area Total User Sys Time CPU Input Output
leveldb 1024 63 no 0.0M 0.0M 0.0M 0.0M 0.0M 61.1 12.5 1:19 92% 17.401M 520
leveldb 1024 63 yes 0.0M 0.0M 0.0M 0.0M 0.0M 2.3 0.1 0:03 80% 0.077M 520
leveldb 4096 63 no 0.2M 0.2M 0.2M 0.1M 0.1M 88.9 13.7 1:52 91% 17.656M 792
leveldb 4096 63 yes 0.3M 0.2M 0.2M 0.1M 0.1M 30.6 0.7 0:34 89% 0.401M 792
leveldb 16384 63 no 1.1M 1.1M 1.1M 0.3M 1.0M 199.5 14.7 3:50 92% 18.378M 1752
leveldb 16384 63 yes 7.0M 1.1M 1.1M 1.0M 1.0M 181.4 2.7 3:17 93% 1.504M 1752
leveldb 65536 63 no 1.3M 1.3M 1.3M 0.3M 6.5M 248.2 14.2 4:39 93% 18.613M 2088
leveldb 65536 63 yes 126.7M 1.3M 1.3M 16.0M 16.0M 1136.2 4.4 19:23 98% 2.320M 2168
leveldb 65536 255 no 3.0M 3.0M 3.0M 0.3M 6.5M 424.4 14.8 7:40 95% 18.944M 2096
leveldb 65536 255 yes 507.1M 3.0M 3.0M 16.0M 16.0M 3931.0 8.5 1:06:26 98% 3.480M 2184
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
Re: Minetestmapper rewritten into C++
Simple question : you intend to contribute on the official repo ?
Re: Minetestmapper rewritten into C++
Hi JP !
One or two trivial ones got merged, and apart from that, there was no feedback.
...so I'm just maintaining this version...
I have tried to contribute, but apparently there was no interest in my patches :-(jp wrote:Simple question : you intend to contribute on the official repo ?
One or two trivial ones got merged, and apart from that, there was no feedback.
...so I'm just maintaining this version...
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
Re: Minetestmapper rewritten into C++
I've put together windows builds (32 & 64 bit) of minetestmapper. Cross-compiled using MinGW on Linux. They start fine under Wine, and from what I hear, they produce nice maps as well on a real copy of Windows :-)
They can be downloaded here
They can be downloaded here
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
- addi
- Member
- Posts: 666
- Joined: Thu Sep 20, 2012 03:16
- GitHub: adrido
- Location: Black-Forest, Germany
Re: Minetestmapper rewritten into C++
lot of thanks for your windows build. it works great :)
Re: Minetestmapper rewritten into C++
Thanks! (& good to hear additional confirmation that it works well a real copy of Windows)addi wrote:lot of thanks for your windows build. it works great :)
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
Re: Minetestmapper rewritten into C++
+1 awesome ...
Re: Minetestmapper rewritten into C++
The official version of "--disable-blocklist-prefetch" doesn't have any speed improvements, only the one at WIP-mt. Does it really disable the prefetch?
Re: Minetestmapper rewritten into C++
That's puzzling... It should disable prefetch, and it should give a performance improvement on smaller parts of large worlds - comparable to the WIP-mt version... Did you use --min-y and --max-y in your tests ?est31 wrote:The official version of "--disable-blocklist-prefetch" doesn't have any speed improvements, only the one at WIP-mt. Does it really disable the prefetch?
I just re-compiled the exact version that is on github, and re-ran a test on Vanessa's world to be certain, and I do see significant improvements, so could you provide more details about your tests ? Preferably the command-line and the output of minetestmapper with --verbose=2, with and without --disable-blocklist-prefetch, and preferably --backend set to an appropriate value. If you have /usr/bin/time installed, its output could be useful too (the shell built-in version of time would not provide much useful information).
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
Re: Minetestmapper rewritten into C++
When I checked again, I couldn't reproduce it anymore. So it seems to be my mistake.
- 12Me21
- Member
- Posts: 873
- Joined: Tue Mar 05, 2013 00:36
- GitHub: 12Me21
- Location: (Ignore all of my posts before 2018)
Re: Minetestmapper rewritten into C++
I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
- addi
- Member
- Posts: 666
- Joined: Thu Sep 20, 2012 03:16
- GitHub: adrido
- Location: Black-Forest, Germany
Re: Minetestmapper rewritten into C++
maybe because its a command-line tool?12Me21 wrote:I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
https://github.com/Rogier-5/minetest-mapper-cpp/wiki/Downloads wrote:
- Unpack it somewhere in your user directory. Do not place it in c:\Program Files\ or another 'standard' location for programs. That seems to cause problems. Alongside your minetest package should be just fine.
- Start minetestmapper from a command-line.
minetestmapper-20150301\bin\minetestmapper.exe- If all is well, it prints a long help message. Next, read the manual (included in the doc directory), and start mapping.
- 12Me21
- Member
- Posts: 873
- Joined: Tue Mar 05, 2013 00:36
- GitHub: 12Me21
- Location: (Ignore all of my posts before 2018)
Re: Minetestmapper rewritten into C++
I can't use the command line because I'm not an admin.addi wrote:maybe because its a command-line tool?12Me21 wrote:I can't get it to work. When I click on the .exe file, a window pops up for a 100th of a second and then disappears.
https://github.com/Rogier-5/minetest-mapper-cpp/wiki/Downloads wrote:
- Unpack it somewhere in your user directory. Do not place it in c:\Program Files\ or another 'standard' location for programs. That seems to cause problems. Alongside your minetest package should be just fine.
- Start minetestmapper from a command-line.
minetestmapper-20150301\bin\minetestmapper.exe- If all is well, it prints a long help message. Next, read the manual (included in the doc directory), and start mapping.
- Krock
- Developer
- Posts: 4650
- Joined: Thu Oct 03, 2013 07:48
- GitHub: SmallJoker
- Location: Switzerland
- Contact:
Re: Minetestmapper rewritten into C++
Create a new in the same directory as minetestmapper.exe: run.bat12Me21 wrote:I can't use the command line because I'm not an admin.
Open it with notepad and save it with those contents:
Code: Select all
@echo off
minetestmapper.exe --help
pause
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
Re: Minetestmapper rewritten into C++
Does cmd.exe need special privileges to run ?? I can't imagine that an administrator would disable cmd.exe, but allow regular users to install any software, like minetest or minetestmapper. That would also allow the user to install another shell / command processor, which trivially circumvents the cmd.exe restriction...Krock wrote:Create a new in the same directory as minetestmapper.exe: run.bat12Me21 wrote:I can't use the command line because I'm not an admin.
Open it with notepad and save it with those contents:If you can't run that simple script, ask the system admin to allow it.Code: Select all
@echo off minetestmapper.exe --help pause
Anyway, if the script works, you could probably also use the following batch file to get a command-line:
Code: Select all
@echo off
cmd
Code: Select all
@echo off
c:\windows\system32\cmd
My mods & tools:
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
My fork of Minetestmapper - much improved - features, manual, Windows downloads
[MOD] Wrench (rotate:wrench) - improved screwdriver alternative - manual, download
-
- Member
- Posts: 1482
- Joined: Fri Apr 19, 2013 16:19
- GitHub: twoelk
- IRC: twoelk
- In-game: twoelk
- Location: northern Germany
Re: Minetestmapper rewritten into C++
If you edit the link to the file name the Wizap script described on page 6 still works. The discussion above and below that entry also has some other interesting links and script examples.
Just make a plain txt file, copy some commands from the forum or write your own into it and then rename the file ending to *.bat
Just make a plain txt file, copy some commands from the forum or write your own into it and then rename the file ending to *.bat
Who is online
Users browsing this forum: No registered users and 18 guests