Page 1 of 2

Un-explore the world ?

Posted: Thu Jul 17, 2014 18:22
by Marshall_maz
Hi everyone ,

Is there a way to un-explore unused part of the world and get it removed from the world database.? ( As if you have never been to that part of the world).

When I started my world I did a lot of exploring opening big parts of the world. There is currently big parts of the world unused. Now when I add mods that add things like new trees etc. it will only be created when I get to unexplored areas and that part of the world get's created and added into the database. So now I have to travel very far to get any newly added items as I have to travel to unexplored areas for them to get created. So I would rather un-explore the current unused areas of the world ( if at all possible ) and re-explore them after adding new mods. Starting a new world is not an option as there is way too many work and buildings already put into this world.

Thanx

Re: Un-explore the world ?

Posted: Thu Jul 17, 2014 18:27
by Krock
There was already the idea to not save unchanged chunks.I don't know about the progress of this.
For sqlite3 databases (default), you can use a sqlite3 database editor. A MT-specific tool for this doesn't exist AFAIK.

Re: Un-explore the world ?

Posted: Thu Jul 17, 2014 20:26
by solars
For my Karsthafen map, I had written a quick script to shrink my map. I wrote it only for my personal use, so its quick and dirty... In this version, I have translatet the error-messages to english.

Code: Select all

#!/usr/bin/perl

use strict;
use CGI;
use DBI qw(:sql_types);
my $y;
my $z;
my $a;
my $b;
my $cgi=CGI->new();
my $Game=$cgi->param('game')?$cgi->param('game'):'.';
my $Out=$cgi->param('out')?$cgi->param('out'):'map.sqlite';

if(-e "$Game/map.sqlite" && -e "$Game/savegame.txt"){
  my $dbGame=DBI->connect("dbi:SQLite:$Game/map.sqlite",'','')||die 'Cannot open map.sqlite';
  my $dbSave=DBI->connect('dbi:SQLite:dbname=:memory:','','')||die 'Cannot create inmemory database';
  my $dbInput=$dbSave->prepare("CREATE TABLE `blocks` (`pos` INT NOT NULL PRIMARY KEY,`data` BLOB)");
  $dbInput->execute();

  open(FILE,"<$Game/savegame.txt")||die 'Cannot open savegame.txt';
  foreach(<FILE>){
    my @line=split(/\,/,$_);
    if($line[5]){
      $line[0]=int($line[0]/16-1);
      $line[1]=int($line[1]/16+1);
      $line[2]=int($line[2]/16-1);
      $line[3]=int($line[3]/16+1);
      $line[4]=int($line[4]/16-1);
      $line[5]=int($line[5]/16+1);
      print 'Writing ',$line[6];
      for(my $x=$line[0];$x<=$line[1];$x++){
	for(my $y=$line[2];$y<=$line[3];$y++){
	  for(my $z=$line[4];$z<=$line[5];$z++){
	    my $pos=minetest_get_block_as_integer($x,$y,$z);
	    my $dbData=$dbGame->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
	    $dbData->execute($pos);
	    my $rows=$dbData->fetchall_arrayref;
	    if($rows->[0][1]){
	      my $dbTest=$dbSave->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
	      $dbTest->execute($pos);
	      my $test=$dbTest->fetchall_arrayref;
	      unless($test->[0][1]){
		$dbInput=$dbSave->prepare("INSERT INTO `blocks`(pos,data) VALUES (?,?)");
		$dbInput->execute($rows->[0][0],$rows->[0][1]);
	      }
	    }
	  }
	}
      }
    }
  }
  print "\n";
  $dbSave->sqlite_backup_to_file($Out);
  $dbSave->disconnect();
  $dbGame->disconnect();
}else{
  print "Minetest savegame with savegame.txt not found.\n\nUsage of this script:\n";
  print "nt_clean.pl game=[path to the savegame directory] out=[path for the new database file]\n\n";
  print "Width no use of the game parameter, the data is read from the actual directory.\nWidth no use of the out parameter, the database is written to 'map.sqlite'.\n\n";
  print "If you use the script from the savegame directory with no set parameters, the game database will be overwritten with no warning!\n\n";
  print "The file savegame.txt is a CSV file with no quotation marks and with comma as separator and following table columns:\n";
  print "1 - x from\n2 - x to\n3 - y from\n4 - y to\n5 - z from\n6 - z to\n7 - Name of the block\n\n";
}


sub minetest_unsigned_to_signed{
  if($_[0]<$_[1]){
    return $_[0];
  }else{
    return ($_[0]-2*$_[1]);
  }
}

sub minetest_get_integer_as_block{
  my $int=shift;
  my @return;
  $return[0]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[0])/4096);
  $return[1]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[1])/4096);
  $return[2]=minetest_unsigned_to_signed($int%4096,2048);
  return @return;
}

sub minetest_get_block_as_integer{
  return $_[2]*16777216+$_[1]*4096+$_[0];
}
If you save the code to 'mt_clean.pl' and put a savegame.txt to the directory with your map.sqlite-file, you can use the script with (if your savefile is in '.minetest/worlds/world' and a copy for the new version in '.minetest/worlds/newworld':
perl mt_clean.pl game=.minetest/worlds/world out=.minetest/worlds/newworld/map.sqlite

Bevor you use the script, you have to put a savegame.txt-file in the directory with your map.sqlite-file. As an example:

Code: Select all

-100,100,-20,50,-100,100,Center of the World
20,300,200,250,-120,-50,My Skycastle
Every Line is a definition of a area, that should be saved in the new database. The second line is an area from 20,200,-120 to 300,250,-50.
It will save a bigger area as you write in the file, it put the 16x16x16 Fields big blocks in the new database. But big landscapes outside this areas will be unexplored after that.

I hope this script can help you. It works from version 0.4.10 of minetest. In older versions, the landscape-generator destroys the buildings.

PS: Dont use the script form the Savegame-Directory without saving the map beforehand. You can use the script without parameter from the path with your map.sqlite-File and it will overwrite it with the belittled file without asking. With an faulty savegame.txt, your map can be destroyed. To copy the map and use the script on the copy without parameters make the new map in a simple way.

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 07:17
by Marshall_maz
I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 08:26
by Krock
Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.
Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 08:30
by Marshall_maz
Krock wrote:
Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.
Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.
But if I do that I loose everything already build in the map ?

Mine is 185mb , we are 2 users playing in the game and both of us already has lots of buildings and mines and other stuff. Don't want to loose all that

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 13:25
by solars
Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.
Is your perl without the dbi-modules?

Then you can install the dbi modules. At debian, you can do it as root with

Code: Select all

aptitude install libdbi-perl
The dbi-perl package will be found in any linux-distribution.
If not, you can download ist there: http://dbi.perl.org/

And the path at the parameter 'game' must be right, if not, the script can't found the sqlite-database of your map.

Your world is 185mb big? My actual shrinked map is 50mb big... The original map: 6358mb!!! Without my script, I havn't a chance to build a download.

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 20:10
by prestidigitator
I think this would be a GREAT addition to the API. We have a minetest.forceload_block(pos) call now. I think it would work well to also have:
  • minetest.delete_block(pos)
    Marks the block containing pos for deletion from the database when it would normally be unloaded (may not be until server restart). This will not happen while there are active players in the nearby area. Removes the block from the forceload list if it is currently in it (also, adding a block to the forceload list cancels deletion).
  • minetest.cancel_delete_block(pos)
    Unmarks the block containing pos for deletion. Has no effect if the block has already been deleted. Simply serves to cancel a pending delete.
  • minetest.register_on_blockdelete(function(pos))
    Registers a callback to be called when (after) the block is actually deleted. On server startup, all mods are initialized before these callbacks are made.
Small side note though: It may not help space consumption much, but for pure-Lua mapgens you could effectively do this by having a hook in the mod called both during map generation and when someone wants to "regenerate" an area to clear changes, etc. Yet another plus for doing mapgen entirely in Lua. You could probably even fake an "ongenerate" callback to other mods....

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 21:30
by Sokomine
Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.

Re: Un-explore the world ?

Posted: Fri Jul 18, 2014 22:42
by prestidigitator
Sokomine wrote:Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.
True, MOST of the time you probably wouldn't want to do it on its own. You'd probably also want to make use of events like on_dignode and on_placenode and maybe even functions like minetest.find_nodes_in_area and/or minetest.get_objects_inside_radius. However, adding to the actual API isn't about constraining possibilities, so better to add functionality that can be used generally than to add something very special-purpose. Who is to say someone wouldn't want to create a mod that allows an admin to blast away sections of the map even if players HAVE built there? Let them build in safeguards (such as warnings to players or cancelling the block deletion on certain events or requesting a custom mapgen to compare against what it WOULD generate for the block) if they like.

Re: Un-explore the world ?

Posted: Sun Aug 31, 2014 09:17
by mike
It would be easier to remove big chunks from the world database instead of excact coordinates.

If you have 5 areas where people act, then you should preserve them.
I havent looked into the sqlite3 yet but if the world is a part of chunks you could just preserve some and delete the rest.

Re: Un-explore the world ?

Posted: Thu Oct 02, 2014 20:09
by lag01
This simple Python script is as far is i got now:

Code: Select all

#!/usr/bin/env python

import sqlite3
import time

source = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite'
target = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite.rebuilded'

sourceconn = sqlite3.connect(source)
targetconn = sqlite3.connect(target)
sourcecursor = sourceconn.cursor()
targetcursor = targetconn.cursor()
targetcursor.execute("CREATE TABLE IF NOT EXISTS `blocks` (`pos` INT NOT NULL PRIMARY KEY, `data` BLOB);")

i=0
for row in sourcecursor.execute("SELECT `pos`, `data` "+" FROM `blocks`;"):
    pos=getIntegerAsBlock(row[0])
    if pos[0]**2 + pos[2]**2 < (1600/16)**2 and pos[1]>(-60/16):    #1600 blocks in each direction and 60 blocks deep
        targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
        if ++i > 100:
            targetconn.commit()
            time.sleep(1)
            i=0

targetconn.commit()
sourceconn.close()
targetconn.close()
I think it is possible to use mapformat and minetestmapper as example to look inside blocks and actually sort them by their content.

One problem i found is that different mapgen versions are generating world differently, so boundary between old map blocks and new map, can look ugly.

Re: Un-explore the world ?

Posted: Sat Oct 11, 2014 20:39
by lag01
I did that! Github :)

Re: Un-explore the world ?

Posted: Sat Oct 11, 2014 21:18
by mike
how does it work?
can you set some areas which should be preserved?

Re: Un-explore the world ?

Posted: Sat Oct 11, 2014 22:11
by lag01
mike wrote:how does it work?
can you set some areas which should be preserved?
It preserves round area around map center:
remap.py

Code: Select all

if pos[0]**2 + pos[2]**2 < (160/16)**2 and pos[1]>(-60/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
Also it preserves blocks(16x16x16 cubes), which contains cobble, chests etc.

If that not enough for you, then you need to alter script code. For example add:

Code: Select all

elif pos[0]>=(200/16) and pos[0]<=(250/16) and pos[2]>=(100/16) and pos[2]<=(150/16) and pos[1]>(-16/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
That will save area around x 200 and z 100 and y above ~0.

Re: Un-explore the world ?

Posted: Sat Oct 11, 2014 23:12
by mike
it would be nice if there would be a special preserve node.
if you place such a node in an area and set the max distance of protection it would work
so you can go to your important "world" inside your world and set this node (admin only).
then the script knows what to protection and its visual :)
its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 14:42
by lag01
mike wrote: its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.
Well, finding nearby blocks could be little harder that what i have already coded, especially with variable distance.
I will look what i can do, meantime you can test if current code do the job for you?

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 16:18
by mike
iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 16:35
by lag01
mike wrote:iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.
I think, current script can save most parts of all 5 your worlds, you just need to specify correct path to your sqlite file and file where to copy data (i hope you use sqlite?)
Otherwise i can write code with coordinates you specified.

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 17:04
by mike
i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 17:50
by lag01
mike wrote:i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.
Ok, i sent you PM with script with your coordinates calculated.
You did not answered if you tested original script, i think that is mean of you...

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 18:13
by mike
which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.

Re: Un-explore the world ?

Posted: Sun Oct 12, 2014 19:01
by lag01
mike wrote:which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.
You can download zip from Github https://github.com/AndrejIT/map_unexplo ... master.zip
Then you need to write correct path to your minetest world folder in two places, something like

Code: Select all

source = r'~/.minetest/worlds/world1/map.sqlite'
About DB damage i have not though... DB table is quite simple, script just takes data as string from one DB and writes same unchanged string to another DB. The only chance that something will be broken is if sqlite and python handle string encodings incorrectly, on my computer that is not happening.

Re: Un-explore the world ?

Posted: Tue Oct 14, 2014 16:53
by aldobr
minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.

Re: Un-explore the world ?

Posted: Tue Oct 14, 2014 18:20
by lag01
aldobr wrote:minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.
Would be good addition. That could help to preserve blocks(chunks) where no player crafted blocks are added, for example tunnel parts where is no torches or buildings built completely from natural resources like dirt, sand, raw wood etc.

Upd.
Actually there already is "timestamp" field, i wonder what that do...
Upd2.
Bot we need to deal with blocks modified only by water/lava flow then :(