[Tool] mtmapprune [mtmapprune] Prune your map.sqlite fast!

Post Reply
sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

[Tool] mtmapprune [mtmapprune] Prune your map.sqlite fast!

by sofar » Post

Project website: https://github.com/minetest-tools/mtmapprune
License: LGPL-2.1+ MIT
Binaries: Linux X86_64: https://github.com/minetest-tools/mtmapprune/releases
Latest version: https://github.com/minetest-tools/mtmap ... x86_64.zip
Windows builds are available for some releases. Check the github releases to see which ones are.

Prunes your map.sqlite and deletes blocks outside a specified range.

Usage:

Code: Select all

mtmapprune map.sqlite max_x [max_y [max_z [min_x min_y min_z]]]

If omitted, `max_y` and `max_z` default to the value of the `max_x` limit. The `min_*` limits will default to the negative values of the `max_*` limits. The limits are node positions, not "block positions".

Example: to prune all nodes with x and z > 1000, and x and z < -1000 and y > 200, and y < 200, use:

mtmapprune map.sqlite 1000 200
mtmapprune works really fast by marking all the blocks outside the reserve region as deleted, and then vacuuming the sqlite db. This makes sqlite do all the work and removes all the empty unused space from your sqlite db file.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

v2 adds full support of pruning to any arbitrary (xyz)-(xyz) box coordinate set, while maintaining backwards compatibility with the way that v1 uses the parameters. So if you now have a box that's in all negative coordinates, or doesn't include the origin (0,0,0), it still works.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Fixer » Post

Windows build please!

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by twoelk » Post

ooh,
please expand this to a mt-map-database manupilating tool.
stuff like deleting certain nodes, moving map parts around or even merging maps would be cool

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Fixerol wrote:Windows build please!
Completely untested, but, give this a try:

https://github.com/minetest-tools/mtmap ... -win64.zip

[edit: fixed link]

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

twoelk wrote:ooh,
please expand this to a mt-map-database manupilating tool.
stuff like deleting certain nodes, moving map parts around or even merging maps would be cool
Deleting specific nodes? Probably not, since this code avoids entirely looking at the content. However, moving blocks around would actually be possible. Not sure if I want to go there, though, but it's certainly possible to do just that.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Fixer » Post

>mtmapprune-v3-x86_64.zip

This one is for linux, ELF file.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Fixerol wrote:>mtmapprune-v3-x86_64.zip

This one is for linux, ELF file.
whoops, fixed link for ya.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Fixer » Post

Gives me error at launch when trying to prune:
Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
Win7sp1

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Fixerol wrote:Gives me error at launch when trying to prune:
Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
Win7sp1
Looks like the latest version fixes that issue as well.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Gael de Sailly » Post

It doesn't work for me
(using v3 binary on Ubuntu 16.04 and Minetest 0.4.16)

The script runs, but nothing is deleted.

First time I run it:

Code: Select all

map.sqlite: removed 47368 of 47548 blocks (limits: [330, 0, -160]-[460, 50, -90])
Second time:

Code: Select all

map.sqlite: removed 0 of 180 blocks (limits: [330, 0, -160]-[460, 50, -90])
The deletion seems to be taken into account, but the file size is not reduced.
Just realize how bored we would be if the world was perfect.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Gael de Sailly wrote:It doesn't work for me
(using v3 binary on Ubuntu 16.04 and Minetest 0.4.16)

The script runs, but nothing is deleted.

First time I run it:

Code: Select all

map.sqlite: removed 47368 of 47548 blocks (limits: [330, 0, -160]-[460, 50, -90])
Second time:

Code: Select all

map.sqlite: removed 0 of 180 blocks (limits: [330, 0, -160]-[460, 50, -90])
The deletion seems to be taken into account, but the file size is not reduced.
Can you post the exact command line used in each run?

What is the size of the file before you ran it the first time, and after?

It's not a script, it's an actual binary.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Gael de Sailly » Post

sofar wrote:Can you post the exact command line used in each run?
I was in the world directory and I ran this twice:

Code: Select all

~/Téléchargements/mtmapprune-v3/mtmapprune map.sqlite 460 50 -90 330 0 -160
sofar wrote:
Gael de Sailly wrote:What is the size of the file before you ran it the first time, and after?
20.5 MB before, 20.5 MB after.
The modification date is not even updated.
Just realize how bored we would be if the world was perfect.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Gael de Sailly wrote:
sofar wrote:Can you post the exact command line used in each run?
I was in the world directory and I ran this twice:

Code: Select all

~/Téléchargements/mtmapprune-v3/mtmapprune map.sqlite 460 50 -90 330 0 -160
sofar wrote:
Gael de Sailly wrote:What is the size of the file before you ran it the first time, and after?
20.5 MB before, 20.5 MB after.
The modification date is not even updated.
*shrug*

maybe can you upload the map.sqlite (gzip it first) to this thread, so I can figure out the issue?

Note that mtmapprune only deletes *blocks* (80x80x80!) and you are cutting down to a really small area, so it may just be that the pruning algorithm leaves a ton of blocks around - your border limits may just be causing some confusion. But, I'd like to see your map to make sure that is the case.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

v4 posted. There is a windows build as well. Please test.

SiliconPenguin
Member
Posts: 12
Joined: Sun Sep 22, 2019 08:56

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by SiliconPenguin » Post

Thank you so much for this tool, sofar! It took me a loooooooonnnngggg time to find it. As a matter of fact, it was your comment on someone else's post regarding resetting a world where you provided a link to this thread, yeah, that's how I found this wonderful tool.

mtmapprune worked excellently for me. It pruned a 1.6 GB map.sqlite map down to less then 200MB. It wasn't the file size that was important to me. I had added some mods that added new ores and trees, and I had also explored a very large part of the map. I wanted to "unexplore" the entire map, other than a limited space, so that the new ores and trees would spawn in in nearby regions.

mtmapprune was a little confusing to use at first because the example was too abbreviated. Also, when I use F5, I get those 3 coordinates: East/West - Height - North/South. I got a little confused because when I look at a map, I can see an X and a Y, and my Z becomes my depth. So I had to run mtmapprune twice to get the results I desired, but that was me trying to be too smart I guess. ;)

Anyway, the only thing I'm disappointed about in regards to this tool is how hard it was to find it!

User avatar
Grigor
Member
Posts: 49
Joined: Sun Dec 01, 2019 05:55

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Grigor » Post

Hey everybody! My first post.
I've got this small map, 2000 x 2000 / mapgen v6, which I've made into an island with a combination of sfan5's nuke mod (modified) and worldedit, working in creative mode. The question: Is there a way to extend the edges of the map with default:water_source blocks? It'd be nice to move the horizon beyond the haze without blowing up too much more of the landscape!
map.jpg
map.jpg (757.73 KiB) Viewed 1056 times

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Grigor wrote:Hey everybody! My first post.
I've got this small map, 2000 x 2000 / mapgen v6, which I've made into an island with a combination of sfan5's nuke mod (modified) and worldedit, working in creative mode. The question: Is there a way to extend the edges of the map with default:water_source blocks? It'd be nice to move the horizon beyond the haze without blowing up too much more of the landscape!
map.jpg
Yes, just create a mapgen that generates default_water_source in ungenerated mapblocks, or air above 0. This is something a mapgen mod needs to do, not this program. Note: making a mapgen isn't too difficult, but outside of scope of this thread - please open a separate thread or look for existing mapgens that do something similar.

User avatar
Grigor
Member
Posts: 49
Joined: Sun Dec 01, 2019 05:55

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Grigor » Post

Thanks sofar; does this mean I can extend the map I've got, or would I need to start from scratch? Will post this is the relevant section also.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by sofar » Post

Grigor wrote:Thanks sofar; does this mean I can extend the map I've got, or would I need to start from scratch? Will post this is the relevant section also.
mapgen can be changed and only ever affects *ungenerated* blocks. This means that any block you preserve with mtmapprune will also be preserved by mapgen.

User avatar
Grigor
Member
Posts: 49
Joined: Sun Dec 01, 2019 05:55

Re: [Tool] mtmapprune [mtmapprune] Prune your map.sqlite fas

by Grigor » Post

Problem solved by editing map_meta.txt. See viewtopic.php?f=12&t=23770&p=363389#p363389

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests