almost infinite map

Post Reply
User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

almost infinite map

by Walker » Post

original Thread: viewtopic.php?f=5&t=9183

i havent´n read the complet thread, sorry, but i have a nice idea ^^

Minetest storages MULTIPLE map.splite(s) ^^

like so:

if your starting the game, minetest create map.splite and fill it with data
intern minetest handle the map als sector 0/0

if the player reach Y to 31000 or -31000, that is the and, like now

BUT: if the player reach X or Z by 31000 or -31000, minetest a a new File
e.g. the player reach the point (0,0,33000), minetest div the 33000 by 32768, this is 1 and a rest of 232
extern mintest say player is at (0,0,33000) but intern minetest create a secound file with this schema:
map(-/+)%02dX(-/+)%02dZ, in this e.g. map+00+01.sqlite and put the player in this file to (0,0,232)

i you think, "hey, this sounds like Sectors", yes, this is, what i want ^^

the Map has intern secors by 16x16x16, but the map itself is also ONE sector of 65536x65536x65536
and can dynamicly put together by the Server in X and Z dimensions ;)

------ Da mein Englisch SCHEIßE ist nochmal in Deutsch ^^ ------

Ich hab den gesamten Thread nicht gelesen, sorry, aber ich hab ne gute idee ^^

Minetest speichert MEHRERE map.sqlite(s) ^^

so ungefähr:

wenn du das spiel startest erstellt minetest eine Datei map.sqlite und füllt sie mit Daten
intern behandelt Minetest diese Map als Sektor 0/0

wenn der Player, in Y Richtung, 31000 oder -31000 erreicht ist schluss, so wie jetzt ja auch schon

ABER: wenn der Spieler in X oder Z Richtung 31000 oder -31000 erreicht, erstellt minetest eine neue Datei
z.B. der Spieler erreicht den Punkt (0,0,33000), dann teilt Minetest die 33000 durch 32768, das sind 1 und 232 rest
nach außen sagt Meinerest, der spieler ist bei (0,0,33000) aber intern erstellt Minetest eine zweite Datei nach folgendem Schema:
map(-/+)%02dX(-/+)%02dZ, in diesem Beispiel map+00+01.sqlite und setzt den spieler in dieser Datei nach (0,0,232)

wenn sie jetzt denken "hei, DU ARSCH, das hasst du ja bei ikea geklaut", dann lies den Englischen text
... ... ...
wenn sie jetzt denken "hei, das klingt ja wie Sektoren", ja, dass ist das, was ich möchte ^^

die Map hat intern Sektoren von 16x16x16, aber die Map selbst ist halt auch NUR ein Sektor von 65536x65536x65536
und kann vom Server dynamisch in X und Z Richtung zusammengesetzt werden ;)

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

because the map files internally are still the same, they are 100% COMPATIBLE, the server does it all by itself, and since sector 0/0 is called simply map.sqlite, existing maps can be opened and if the borders are not exceeded, new maps can also be opened by old server;)

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

voxelproof wrote:
Walker wrote: because the map files internally are still the same, they are 100% COMPATIBLE, the server does it all by itself, and since sector 0/0 is called simply map.sqlite, existing maps can be opened and if the borders are not exceeded, new maps can also be opened by old server;)
I get the idea, but it's been already discussed in some thread a year or more before. Your proposal is tantamount to multi-dimensional worlds, with particular dimensions (map.sqlite data) accessible from the in-game level. It'd be very interesting of course but as far as I recollect the discussion there's some essential technical obstruction making it infeasible atm. The same idea implemented to creating coherent large flat one-dimensional worlds would moreover need general overhaul of map generation algorithms and this makes this all the more difficult to achieve.

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

Linuxdirk wrote:This would cause 0,0,0;0,0,0 (or whatever the separator instead of ; will be) being a valid position causing everything what was done so far break.

Since devs fear even minor breakage this won’t never happen.

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

ShadMOrdre wrote:This can easily happen, with full dev support, provided the following condition can be at least somewhat provided.

1. Provide sample code that the core devs can test, and perhaps approve. Follow all coding guidelines set forth by the dev team, and create a pull request. Some of this could might even possibly be handled simply by reworking the SQL code. This might be one of the ways to avoid breakage, and could see the quickest implementation. If you instead code this in C++, if you can avoid breakage, you could see rather quick adoption of the code. If the only solution is in fact breakage, then the code would only be merged into the next major release development branch.

The reason for the existing limits is a 32bit/64bit thing. With some reworking of some critical parts of the engine and perhaps the database, I believe this could be achieved.

I thinks it might be easier than we think, if enough of the right folks just think it through. The devs are already pretty swamped, so if something like this were to happen, it is going to have to be one of or a team of us to simply make the code and pull requests.

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

Linuxdirk wrote:This would cause 0,0,0;0,0,0 (or whatever the separator instead of ; will be) being a valid position causing everything what was done so far break.

Since devs fear even minor breakage this won’t never happen.

eben NICHT
1. währe es nur 0,0;0,0,0, da die Sektoren NUR in X und Z Richtung vorgesehen sind ( wobei natürlich Y auch implementieren könnte, stell ich mir aber mit den Biomen schwierig von, die mögen Höhenlagen außerhalb von +/-31000 bestimmt nicht ... keine Ahnung ^^ )
2. ist das nach außen transparent, das heißt, die Syntax ist weiterhin x,y,z nur das x und z größer beziehungsweise kleiner werden können

die Einteilung in Sektoren soll ja das Backend ( die Engine ) ALLEINE machen, also On-The-Fly ^^

--------- [ Google Translate ] ------------

just NOT
1. It would only be 0.0, 0.0.0, since the sectors are ONLY in the X and Z directions (which of course could implement Y as well, but with the biomes I find it difficult to think of those that like altitudes outside of + / -31000 certainly not ... no idea ^^)
2. is this transparent to the outside, that is, the syntax is still x, y, z only the x and z can be larger or smaller

the division into sectors should make the backend (the engine) ALONE, so on-the-fly ^^

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

ShadMOrdre wrote:This can easily happen, with full dev support, provided the following condition can be at least somewhat provided.

1. Provide sample code that the core devs can test, and perhaps approve. Follow all coding guidelines set forth by the dev team, and create a pull request. Some of this could might even possibly be handled simply by reworking the SQL code. This might be one of the ways to avoid breakage, and could see the quickest implementation. If you instead code this in C++, if you can avoid breakage, you could see rather quick adoption of the code. If the only solution is in fact breakage, then the code would only be merged into the next major release development branch.

The reason for the existing limits is a 32bit/64bit thing. With some reworking of some critical parts of the engine and perhaps the database, I believe this could be achieved.

I thinks it might be easier than we think, if enough of the right folks just think it through. The devs are already pretty swamped, so if something like this were to happen, it is going to have to be one of or a team of us to simply make the code and pull requests.

ich denke auch so,

jedoch weiß ich ich nicht, wie weit die Limitierung auf 16bit vorgetrungen ist, wenn es wirklich nur durch die Schnittstelle zum Backend begrenzt wird, ist es ein Leichtes

Wenn die Engine aber die 16bit interpretierung im gesammten Code hat ( z.B. durch Bitshifting-Methoden, die nur auf 16bit ausgelegt sind ), wird das ganze schon schwiriger ^^

------ [ Google Translate ] -----------

I think so, too,

However, I do not know how far the limitation has been made to 16bit, if it is really limited only by the interface to the backend, it is easy

But if the engine has the 16bit interpretation in the whole code (for example, by bit-shifting methods, which are only set to 16bit), the whole thing gets more and more busy ^^

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

Desweiteren könnte man ja mit der Namenskonvention gleich weitere Feature hinzufügen, wie Dimensionen, mehrere Maps pro Welt etc. diese werden einfach als ebenen in die Weltstruktur hinzugefügt ...

ich habe z.B. für ein eigenes Programm folgende Konversation benutzt:

Code: Select all

tilefile$ = worldpath$+"/d"+Str(dimension)+"/s"+Str(escX)+"/ss"+Str(escY)+"/x"+Str(x)+"/y"+Str(y)+"/z"+Str(z)+".sqlite"
// z.B. tilefile$ = "/home/knoppix/.puretest/worlds/testword/d1/s-3/ss5/x43/y-400/z0.sqlite
man sieht schon, das man quasi beliebig viele Verzeichnis-ebenen dazwischen knallen kann ;)

natürlich könnte man erstmal mit X/Z anfange ( wie im ersten Beitrag beschrieben ), mann kann das System aber auch für spätere Erweiterungen offen halten, solange mann eines beachtet: Neu Features brauchen IMMER eindeutige bezeichnungen ( "d", "s", "ss", "x", "y", "z" usw. )

wenn man also z.B. folgende Struktur, aus dem ersten Beitrag, hat:

Code: Select all

map+13-21.sqlite
dieser wird nun in eine Verzeichnisstruktur übersetzt:

Code: Select all

map/x13/z-21.sqlite
nun wird eine Y-Dimension hinzugefügt, wobei Meinetest merkt, das es in den "x??"-Verzeichnissen keine "y??"-Verzeichnisse gibt und demnach alle Unterverzeichnisse in eine Unter-ebene verschiebt:

Code: Select all

mkdir /map/x13/y0
mv map/x13/z-21.sqlite map/x13/y0/z-21.sqlite
und nun entscheidet sich das Minetest-Team, Jahre später, noch Dimensionen, wie Hölle und Himmel, hinzuzufügen ...
kein Problem, die neue Verzeichnis Struktur heißt einfach "d".
dabei merkt Minetest wieder ... "d??" gibt es unter "map" nicht ... also updaten:

Code: Select all

mkdir /map/d0
mv map/x13/y0/z-21.sqlite map/d0/x13/y0/z-21.sqlite
Und so weiter und so fort ^^

--------- [ Google Translate ] ------------

Furthermore, one could indeed add more feature with the naming convention, like dimensions, several maps per world, etc. these are simply added as levels in the world structure ...

i have e.g. used for its own program the following conversation:

Code: Select all

tilefile$ = worldpath$+"/d"+Str(dimension)+"/s"+Str(escX)+"/ss"+Str(escY)+"/x"+Str(x)+"/y"+Str(y)+"/z"+Str(z)+".sqlite"
// z.B. tilefile$ = "/home/knoppix/.puretest/worlds/testword/d1/s-3/ss5/x43/y-400/z0.sqlite
you can already see that you can pop almost any number of directory levels in between;)

Of course you could start with X / Z first (as described in the first post), but you can also keep the system open for later extensions, as long as one notes: New features ALWAYS need unique identifiers ("d", "s", "ss", "x", "y", "z" etc.)

so if you following structure, from the first post, has:

Code: Select all

map+13-21.sqlite
this is now translated into a directory structure:

Code: Select all

map/x13/z-21.sqlite
Now a Y-dimension is added, where Meinetest notices that there are no "y??" directories in the "x??" directories and therefore shifts all subdirectories to a sub-level:

Code: Select all

mkdir /map/x13/y0
mv map/x13/z-21.sqlite map/x13/y0/z-21.sqlite
and now, years later, the Minetest team decides to add dimensions like hell and sky ...
no problem, the new directory structure is simply called "d".
while Minetest notices again ... "d??" there is not under "map" ... so update:

Code: Select all

mkdir /map/d0
mv map/x13/y0/z-21.sqlite map/d0/x13/y0/z-21.sqlite
And so on and so on ^^

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

wichtig hierbei ist natürlich, EGAL was hinzugefügt wird, minetest muss es immer aus sicht des Spielers / der Clienten abstrahieren:
X/Y/Z = originale ingame position ( auch inerhalb einer SQLite-Datei ) ... x/y/z/d = Verzeichnisstruktur

Code: Select all

X += x*65535
Y += y*65535
Y += d*65536*65536
Z += z*65535

Code: Select all

// posX/Y/Z = absolute Position
// worldX/Y/Z/D = Werte von den  Verzeichnisstrukturen
// innerX/Y/Z = Werte innerhalb der SQLite-Dateien
//
// D = dimension ( hölle, himmel, ende, usw. )
//
posX = worldX<<16 | (innerX + 32768)              // da die X-position innerhalb einer SQLite nicht von 0 sondern von -32768 startet
posY = worldD<<32 | worldY<<16 | (innerY + 32768) // da die X-position innerhalb einer SQLite nicht von 0 sondern von -32768 startet
posZ = worldZ<<16 | (innerZ + 32768)              // da die X-position innerhalb einer SQLite nicht von 0 sondern von -32768 startet
--------- [ Google Translate ] ------------

Of course it is important, what is added, minetest always has to abstract it from the view of the player (s):
X/Y/Z = original ingame position (also inside a SQLite file) ... x/y/z/d = directory structure

Code: Select all

X += x*65535
Y += y*65535
Y += d*65536*65536
Z += z*65535

Code: Select all

// posX/Y/Z = absolute position
// worldX/Y/Z/D = values from the directory structures
// innerX/Y/Z = values within the SQLite files
//
// D = dimension (hell, heaven, end, etc.)
//
posX = worldX<<16 | (innerX + 32768)              // because the X-position within a SQLite does not start from 0 but from -32768
posY = worldD<<32 | worldY<<16 | (innerY + 32768) // because the X-position within a SQLite does not start from 0 but from -32768
posZ = worldZ<<16 | (innerZ + 32768)              // because the X-position within a SQLite does not start from 0 but from -32768

User avatar
paramat
Developer
Posts: 3700
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat
IRC: paramat
Location: UK

Re: almost infinite map

by paramat » Post

'almost infinite' is a nonsense phrase =)

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

"almost" means, that you can reach over 32^3 and more blocks
to travel this distance ( without cheating ) you need ... nearly forever ?
so yes, if you starts from center, the map feels "infinite"

^^

User avatar
Pyrollo
Developer
Posts: 385
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: almost infinite map

by Pyrollo » Post

Regarding to the amount of work that should be needed to make that change, does the gameplay worth it ?
A "simple" rework of the map format and coordinate (which would be IMO much simpler that a complex tiled system of databases) to have larger worlds could have the same effect.

Anyway, when you look at a server map, 90% of it is unexplored. For me, it means that the map is fairly large enough for a good gameplay.

For me axes of improvement would more be in "multiple dimension" / parallel worlds. And even that could be achieved without increasing the map size (with horizontal slices).
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

User avatar
voxelproof
Member
Posts: 1087
Joined: Sat Aug 05, 2017 08:13
Location: Europe

Re: almost infinite map

by voxelproof » Post

paramat wrote:'almost infinite' is a nonsense phrase =)
Mathematically, yes. But there's a point of using term of 'infinity' on psychological grounds anyway. I argue that 'practically infinite' (open in-game worlds or any other content) means 'not practically explorable within a player's lifetime'. Since current MT worlds are definitely 'practically explorable' they don't fit the definition. A different matter is for what sake such enormous spaces should be implemented; sometimes (and predominately) it may be counterproductive from the point of view of player's experience. Very often overly spacious worlds create different curious psychological effects -- feeling of 'emptiness' (when content doesn't fill them sufficiently) or desolation (what can be perhaps best understood when playing Space Engine -- the tremendous available space makes unprepared wanderer feel lost).
To miss the joy is to miss all. Robert Louis Stevenson

User avatar
Hume2
Member
Posts: 709
Joined: Tue Jun 19, 2018 08:24
GitHub: Hume2
In-game: Hume2
Location: Czech Republic

Re: almost infinite map

by Hume2 » Post

paramat wrote:'almost infinite' is a nonsense phrase =)
It does make sense. For example, in physics, almost infinite resistance means that the resistance is so big so it can't be measured by used device. Almost infinite means that something is so big that it can be aproximated by infinity for easier calculations. It can also mean that it is known to be finite but it is so big so it can't be measured by the used method.
If you lack the reality, go on a trip or find a job.

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: almost infinite map

by ShadMOrdre » Post

Almost infinite and gameplay value.

Almost infinite isn't even good enough for those of us who like to simply explore spaces from time to time. Some of us want truly infinite for reasons that go far beyond using MT as a simple game engine. MTs potential is not realized by ANY of us. As a DB backend, a concept entirely out of scope with MT as a game engine, MT has the potential to excel in ways that Oracle and MS would drool over. Yes, I know that MT uses SQLite. This is not lost on me. I've spent years in the industry, as a db pro. It's in the processing. Maybe as middleware. But still valid. Just think....

As for gameplay value, well, if you are playing a non-game, with non-game play value, then adding anything to nothing will still result in nothing, IMHO. Don't get me wrong here. MT Game does what it is supposed to do, (Display the potential of MT and maybe attract MC expats), and it does so well. It is light, and is a fairly good representation of MTs capabilities as a MC like game engine. MineClone does the same.

In my lib_ecology mod, I have just provided a quick, lag-light way to provide up to 25600 ecosystems, simply for those of us who want to see more "world" than that which seems to be geared towards regurgitated minecrap gameplay. I have a feeling, if I actually generated that many ecosystems, I would not be able to see them all on a single map, only being able to explore some of them in certain worlds, depending on elevation and things like that.

MT is an AMAZING engine, capable of processing data quite capably and efficiently. Limiting it to only providing MC style thinking, and thus gameplay, is missing the point entirely, IMHO.

MT is capable of a depth that a Java based MC would never be able to provide. MS misspent $2b on that one.

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: almost infinite map

by twoelk » Post

paramat wrote:'almost infinite' is a nonsense phrase =)
maybe
"only limited by available hardware"
might be a more realistic goal

User avatar
voxelproof
Member
Posts: 1087
Joined: Sat Aug 05, 2017 08:13
Location: Europe

Re: almost infinite map

by voxelproof » Post

ShadMOrdre wrote:
MT is an AMAZING engine, capable of processing data quite capably and efficiently. Limiting it to only providing MC style thinking, and thus gameplay, is missing the point entirely, IMHO.
That's 100% true and it's also my deep feeling -- frankly I think all those threads "What MT is still missing over MC" are here just to fix the feeling of a second-hand copy of a revered brand. MT contributors should forget about MC at some point especially since it's now obviously being reverse-developed by its current owners.
MT is capable of a depth that a Java based MC would never be able to provide. MS misspent $2b on that one.
Probably they didn't buy it to make it an ultimate perfect game indeed. People don't understand how corporate economics works if they think that every asset a company possesses automatically has to be improved and developed into something more perfect. Sometimes it's in the best interest of a firm to buy things just to shut them down. And this is the case of Minecraft -- 'killing it softly' (e.g. by the newest AR version which is apparently set to fail) releases huge swathes of playerbase for more profitable and short-lived AAA trash.
To miss the joy is to miss all. Robert Louis Stevenson

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: almost infinite map

by ShadMOrdre » Post

Hmmmm.....

I don't think M$ wants to kill MC. The IP built around it is Disneyesque at this point. Maybe M$ should sell the "child" friendly part of MC to Disney....

No... I think M$ is smarter than that. They did not spend $2b on a video game property. This was not the point at all. It's a nice addition to the X brand console, but they had other ideas.

I thinks M$ spent $2b on a very efficient, albeit Java based, data crunching algorithm. But this is all getting OT.

As for the infinite worlds idea, I just want to know that I can explore and not run into a wall. I do not expect to go 1 trillion miles in my MT world, because even teleporting the distance to make up for light speed, it wouldn't matter. I know at some point, the noise that generates the terrain will either repeat, or just get redundant.

I don't expect or even want to go 1 trillion miles, but I still would like to think that I could. And...as MT develops, as I pray it does, I would also hope that it will grow to take advantage of the new techs that will invariably come about. Like, maybe using some better way to determine map position, than just relying on a 16bit value, with limited value.

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: almost infinite map

by sorcerykid » Post

My two cents:

Personally, I think the higher priority is to overhaul the legacy database format. It doesn't make sense to make an "infinite map" if the underlying architecture is itself is outdated. I mean mapblock data is still being stored as a BLOB and mapblock positions are calculated using a highly obfuscated formula -- both of these are relics from back when Minetest 0.1 used to store mapblocks as flat files.

Ensuring that the underlying database architecture itself is future-proof, in my view, is an issue that needs to be rectified first.
Last edited by sorcerykid on Tue May 21, 2019 16:19, edited 1 time in total.

User avatar
Walker
Member
Posts: 1816
Joined: Tue Oct 03, 2017 09:22
In-game: Walker
Contact:

Re: almost infinite map

by Walker » Post

sorcerykid wrote:obfuscated formula
?

i think its only like

Code: Select all

pos = X<<16|Y
?

User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: almost infinite map

by sorcerykid » Post

No that would only work if the mapblock positions are unsigned. The actual formula is

Code: Select all

local function to_signed( val )
        return val < 2048 and val or val - 2 * 2048
end

function decode_pos( index )
        local x = to_signed( index % 4096 )
        index = math.floor( ( index - x ) / 4096 )
        local y = to_signed( index % 4096 )
        index = math.floor( ( index - y ) / 4096 )
        local z = to_signed( index % 4096 )
        return { x = x, y = y, z = z }
end

User avatar
Pyrollo
Developer
Posts: 385
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: almost infinite map

by Pyrollo » Post

sorcerykid wrote:My two cents:

Personally, I think the higher priority is to overhaul the legacy database format. It doesn't make sense to make an "infinite map" if the underlying architecture is itself is outdated. I mean mapblock data is still being stored as a BLOB and mapblock positions are calculated using a highly obfuscated formula -- both of these are relics from back when Minetest 0.1 used to store mapblocks as flat files.

Ensuring that the underlying database architecture itself is future-proof, in my view, is an issue that needs to be rectified first.
Database format is a good question. Today all is stored in a blob (map nodes + metadata + entities) indexed by an integer containing the block coord.

On a relational database point of view it would be much better to have this data in rationalized bunch of tables. It could be easy and probably efficient.

But, this is only for relational databases. MT supports non relational databases like leveldb and redis. I guess that the main reason all is kept in a single two fields table.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests