Un-explore the world ?

Marshall_maz
Member
Posts: 249
Joined: Mon Jul 14, 2014 17:13
In-game: Mazal
Location: Cullinan, South-Africa

Un-explore the world ?

by Marshall_maz » Post

Hi everyone ,

Is there a way to un-explore unused part of the world and get it removed from the world database.? ( As if you have never been to that part of the world).

When I started my world I did a lot of exploring opening big parts of the world. There is currently big parts of the world unused. Now when I add mods that add things like new trees etc. it will only be created when I get to unexplored areas and that part of the world get's created and added into the database. So now I have to travel very far to get any newly added items as I have to travel to unexplored areas for them to get created. So I would rather un-explore the current unused areas of the world ( if at all possible ) and re-explore them after adding new mods. Starting a new world is not an option as there is way too many work and buildings already put into this world.

Thanx

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Un-explore the world ?

by Krock » Post

There was already the idea to not save unchanged chunks.I don't know about the progress of this.
For sqlite3 databases (default), you can use a sqlite3 database editor. A MT-specific tool for this doesn't exist AFAIK.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

User avatar
solars
Member
Posts: 676
Joined: Sat Jul 20, 2013 15:16
Contact:

Re: Un-explore the world ?

by solars » Post

For my Karsthafen map, I had written a quick script to shrink my map. I wrote it only for my personal use, so its quick and dirty... In this version, I have translatet the error-messages to english.

Code: Select all

#!/usr/bin/perl

use strict;
use CGI;
use DBI qw(:sql_types);
my $y;
my $z;
my $a;
my $b;
my $cgi=CGI->new();
my $Game=$cgi->param('game')?$cgi->param('game'):'.';
my $Out=$cgi->param('out')?$cgi->param('out'):'map.sqlite';

if(-e "$Game/map.sqlite" && -e "$Game/savegame.txt"){
  my $dbGame=DBI->connect("dbi:SQLite:$Game/map.sqlite",'','')||die 'Cannot open map.sqlite';
  my $dbSave=DBI->connect('dbi:SQLite:dbname=:memory:','','')||die 'Cannot create inmemory database';
  my $dbInput=$dbSave->prepare("CREATE TABLE `blocks` (`pos` INT NOT NULL PRIMARY KEY,`data` BLOB)");
  $dbInput->execute();

  open(FILE,"<$Game/savegame.txt")||die 'Cannot open savegame.txt';
  foreach(<FILE>){
    my @line=split(/\,/,$_);
    if($line[5]){
      $line[0]=int($line[0]/16-1);
      $line[1]=int($line[1]/16+1);
      $line[2]=int($line[2]/16-1);
      $line[3]=int($line[3]/16+1);
      $line[4]=int($line[4]/16-1);
      $line[5]=int($line[5]/16+1);
      print 'Writing ',$line[6];
      for(my $x=$line[0];$x<=$line[1];$x++){
	for(my $y=$line[2];$y<=$line[3];$y++){
	  for(my $z=$line[4];$z<=$line[5];$z++){
	    my $pos=minetest_get_block_as_integer($x,$y,$z);
	    my $dbData=$dbGame->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
	    $dbData->execute($pos);
	    my $rows=$dbData->fetchall_arrayref;
	    if($rows->[0][1]){
	      my $dbTest=$dbSave->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
	      $dbTest->execute($pos);
	      my $test=$dbTest->fetchall_arrayref;
	      unless($test->[0][1]){
		$dbInput=$dbSave->prepare("INSERT INTO `blocks`(pos,data) VALUES (?,?)");
		$dbInput->execute($rows->[0][0],$rows->[0][1]);
	      }
	    }
	  }
	}
      }
    }
  }
  print "\n";
  $dbSave->sqlite_backup_to_file($Out);
  $dbSave->disconnect();
  $dbGame->disconnect();
}else{
  print "Minetest savegame with savegame.txt not found.\n\nUsage of this script:\n";
  print "nt_clean.pl game=[path to the savegame directory] out=[path for the new database file]\n\n";
  print "Width no use of the game parameter, the data is read from the actual directory.\nWidth no use of the out parameter, the database is written to 'map.sqlite'.\n\n";
  print "If you use the script from the savegame directory with no set parameters, the game database will be overwritten with no warning!\n\n";
  print "The file savegame.txt is a CSV file with no quotation marks and with comma as separator and following table columns:\n";
  print "1 - x from\n2 - x to\n3 - y from\n4 - y to\n5 - z from\n6 - z to\n7 - Name of the block\n\n";
}


sub minetest_unsigned_to_signed{
  if($_[0]<$_[1]){
    return $_[0];
  }else{
    return ($_[0]-2*$_[1]);
  }
}

sub minetest_get_integer_as_block{
  my $int=shift;
  my @return;
  $return[0]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[0])/4096);
  $return[1]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[1])/4096);
  $return[2]=minetest_unsigned_to_signed($int%4096,2048);
  return @return;
}

sub minetest_get_block_as_integer{
  return $_[2]*16777216+$_[1]*4096+$_[0];
}
If you save the code to 'mt_clean.pl' and put a savegame.txt to the directory with your map.sqlite-file, you can use the script with (if your savefile is in '.minetest/worlds/world' and a copy for the new version in '.minetest/worlds/newworld':
perl mt_clean.pl game=.minetest/worlds/world out=.minetest/worlds/newworld/map.sqlite

Bevor you use the script, you have to put a savegame.txt-file in the directory with your map.sqlite-file. As an example:

Code: Select all

-100,100,-20,50,-100,100,Center of the World
20,300,200,250,-120,-50,My Skycastle
Every Line is a definition of a area, that should be saved in the new database. The second line is an area from 20,200,-120 to 300,250,-50.
It will save a bigger area as you write in the file, it put the 16x16x16 Fields big blocks in the new database. But big landscapes outside this areas will be unexplored after that.

I hope this script can help you. It works from version 0.4.10 of minetest. In older versions, the landscape-generator destroys the buildings.

PS: Dont use the script form the Savegame-Directory without saving the map beforehand. You can use the script without parameter from the path with your map.sqlite-File and it will overwrite it with the belittled file without asking. With an faulty savegame.txt, your map can be destroyed. To copy the map and use the script on the copy without parameters make the new map in a simple way.

Marshall_maz
Member
Posts: 249
Joined: Mon Jul 14, 2014 17:13
In-game: Mazal
Location: Cullinan, South-Africa

Re: Un-explore the world ?

by Marshall_maz » Post

I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.

User avatar
Krock
Developer
Posts: 4650
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: Un-explore the world ?

by Krock » Post

Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.
Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

Marshall_maz
Member
Posts: 249
Joined: Mon Jul 14, 2014 17:13
In-game: Mazal
Location: Cullinan, South-Africa

Re: Un-explore the world ?

by Marshall_maz » Post

Krock wrote:
Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.
Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.
But if I do that I loose everything already build in the map ?

Mine is 185mb , we are 2 users playing in the game and both of us already has lots of buildings and mines and other stuff. Don't want to loose all that

User avatar
solars
Member
Posts: 676
Joined: Sat Jul 20, 2013 15:16
Contact:

Re: Un-explore the world ?

by solars » Post

Marshall_maz wrote:I tried this script but doesn't work. I get sqlite db modules failures.
Is your perl without the dbi-modules?

Then you can install the dbi modules. At debian, you can do it as root with

Code: Select all

aptitude install libdbi-perl
The dbi-perl package will be found in any linux-distribution.
If not, you can download ist there: http://dbi.perl.org/

And the path at the parameter 'game' must be right, if not, the script can't found the sqlite-database of your map.

Your world is 185mb big? My actual shrinked map is 50mb big... The original map: 6358mb!!! Without my script, I havn't a chance to build a download.

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: Un-explore the world ?

by prestidigitator » Post

I think this would be a GREAT addition to the API. We have a minetest.forceload_block(pos) call now. I think it would work well to also have:
  • minetest.delete_block(pos)
    Marks the block containing pos for deletion from the database when it would normally be unloaded (may not be until server restart). This will not happen while there are active players in the nearby area. Removes the block from the forceload list if it is currently in it (also, adding a block to the forceload list cancels deletion).
  • minetest.cancel_delete_block(pos)
    Unmarks the block containing pos for deletion. Has no effect if the block has already been deleted. Simply serves to cancel a pending delete.
  • minetest.register_on_blockdelete(function(pos))
    Registers a callback to be called when (after) the block is actually deleted. On server startup, all mods are initialized before these callbacks are made.
Small side note though: It may not help space consumption much, but for pure-Lua mapgens you could effectively do this by having a hook in the mod called both during map generation and when someone wants to "regenerate" an area to clear changes, etc. Yet another plus for doing mapgen entirely in Lua. You could probably even fake an "ongenerate" callback to other mods....

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: Un-explore the world ?

by Sokomine » Post

Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.
A list of my mods can be found here.

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: Un-explore the world ?

by prestidigitator » Post

Sokomine wrote:Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.
True, MOST of the time you probably wouldn't want to do it on its own. You'd probably also want to make use of events like on_dignode and on_placenode and maybe even functions like minetest.find_nodes_in_area and/or minetest.get_objects_inside_radius. However, adding to the actual API isn't about constraining possibilities, so better to add functionality that can be used generally than to add something very special-purpose. Who is to say someone wouldn't want to create a mod that allows an admin to blast away sections of the map even if players HAVE built there? Let them build in safeguards (such as warnings to players or cancelling the block deletion on certain events or requesting a custom mapgen to compare against what it WOULD generate for the block) if they like.

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

It would be easier to remove big chunks from the world database instead of excact coordinates.

If you have 5 areas where people act, then you should preserve them.
I havent looked into the sqlite3 yet but if the world is a part of chunks you could just preserve some and delete the rest.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

This simple Python script is as far is i got now:

Code: Select all

#!/usr/bin/env python

import sqlite3
import time

source = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite'
target = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite.rebuilded'

sourceconn = sqlite3.connect(source)
targetconn = sqlite3.connect(target)
sourcecursor = sourceconn.cursor()
targetcursor = targetconn.cursor()
targetcursor.execute("CREATE TABLE IF NOT EXISTS `blocks` (`pos` INT NOT NULL PRIMARY KEY, `data` BLOB);")

i=0
for row in sourcecursor.execute("SELECT `pos`, `data` "+" FROM `blocks`;"):
    pos=getIntegerAsBlock(row[0])
    if pos[0]**2 + pos[2]**2 < (1600/16)**2 and pos[1]>(-60/16):    #1600 blocks in each direction and 60 blocks deep
        targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
        if ++i > 100:
            targetconn.commit()
            time.sleep(1)
            i=0

targetconn.commit()
sourceconn.close()
targetconn.close()
I think it is possible to use mapformat and minetestmapper as example to look inside blocks and actually sort them by their content.

One problem i found is that different mapgen versions are generating world differently, so boundary between old map blocks and new map, can look ugly.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

I did that! Github :)

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

how does it work?
can you set some areas which should be preserved?

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

mike wrote:how does it work?
can you set some areas which should be preserved?
It preserves round area around map center:
remap.py

Code: Select all

if pos[0]**2 + pos[2]**2 < (160/16)**2 and pos[1]>(-60/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
Also it preserves blocks(16x16x16 cubes), which contains cobble, chests etc.

If that not enough for you, then you need to alter script code. For example add:

Code: Select all

elif pos[0]>=(200/16) and pos[0]<=(250/16) and pos[2]>=(100/16) and pos[2]<=(150/16) and pos[1]>(-16/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
That will save area around x 200 and z 100 and y above ~0.

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

it would be nice if there would be a special preserve node.
if you place such a node in an area and set the max distance of protection it would work
so you can go to your important "world" inside your world and set this node (admin only).
then the script knows what to protection and its visual :)
its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

mike wrote: its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.
Well, finding nearby blocks could be little harder that what i have already coded, especially with variable distance.
I will look what i can do, meantime you can test if current code do the job for you?

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

mike wrote:iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.
I think, current script can save most parts of all 5 your worlds, you just need to specify correct path to your sqlite file and file where to copy data (i hope you use sqlite?)
Otherwise i can write code with coordinates you specified.

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

mike wrote:i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.
Ok, i sent you PM with script with your coordinates calculated.
You did not answered if you tested original script, i think that is mean of you...

mike
Member
Posts: 46
Joined: Wed Sep 26, 2012 01:15

Re: Un-explore the world ?

by mike » Post

which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

mike wrote:which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.
You can download zip from Github https://github.com/AndrejIT/map_unexplo ... master.zip
Then you need to write correct path to your minetest world folder in two places, something like

Code: Select all

source = r'~/.minetest/worlds/world1/map.sqlite'
About DB damage i have not though... DB table is quite simple, script just takes data as string from one DB and writes same unchanged string to another DB. The only chance that something will be broken is if sqlite and python handle string encodings incorrectly, on my computer that is not happening.

User avatar
aldobr
Member
Posts: 316
Joined: Sun Nov 25, 2012 05:46

Re: Un-explore the world ?

by aldobr » Post

minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.

User avatar
lag01
Member
Posts: 321
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag
Contact:

Re: Un-explore the world ?

by lag01 » Post

aldobr wrote:minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.
Would be good addition. That could help to preserve blocks(chunks) where no player crafted blocks are added, for example tunnel parts where is no torches or buildings built completely from natural resources like dirt, sand, raw wood etc.

Upd.
Actually there already is "timestamp" field, i wonder what that do...
Upd2.
Bot we need to deal with blocks modified only by water/lava flow then :(

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 18 guests