one world DB, but more mts using ?

Post Reply
User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

one world DB, but more mts using ?

by Festus1965 » Post

IT was some last brain flash before I fall asleep last night.

As it is possible to use external Database Backends and they normally are able to handle more then one "access" I thought about that.

So I wonder if it would be possible to use an mts 0.4.17.1 server and also an mts 5.0.0 server access the same external world.DB,
as only share world blocks, (or also players, etc.)
and even to let old gamers play without technic in a server called "Wildes Land" (0.4.16) where also that spawn is its on,
and another server "Celine in Thailand" (0.4.17.1) with other options and spawn ...
even more than "Celines Wildes Thailand" (5.0.0)

But that how is not the discuss basic,

it is just about would that one DB but maybe 2 access minetest servers be possible ?

From whatever know yet (had to admin an oracle db for 3 years with 1500 users) I say yes !
(sure there will come also the issue about both mts, and each a gamer just are same block "digging" or "placing" = first comes first as also in one server)
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

wziard
Member
Posts: 127
Joined: Mon Oct 29, 2018 19:12

Re: one world DB, but more mts using ?

by wziard » Post

I think that will give a totally new feel to the term 'race condition' :-D

Try it! It will probably have loads of bugs, but it's an interesting experiment.

User avatar
texmex
Member
Posts: 1753
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: tacotexmex

Re: one world DB, but more mts using ?

by texmex » Post

You should experiment with this! Imagine multiple servers doing completely different things but in, say, different parts of the same world.

User avatar
Linuxdirk
Member
Posts: 3218
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: one world DB, but more mts using ?

by Linuxdirk » Post

texmex wrote:You should experiment with this! Imagine multiple servers doing completely different things but in, say, different parts of the same world.
When a user travels from one part of the world to another part how will they see the other users? How does the client know a node was changed when the client already loaded that specific node?

This is actually a nice idea, but as long a ss servers are not able to communicate with each other this can only be a nice idea.

User avatar
texmex
Member
Posts: 1753
Joined: Mon Jul 11, 2016 21:08
GitHub: tacotexmex
In-game: tacotexmex

Re: one world DB, but more mts using ?

by texmex » Post

They won't. It would be what game makers call mingleplayer. Ghosts of other dimensions of the same world building and digging. What the client should know I've no idea about =)

User avatar
Linuxdirk
Member
Posts: 3218
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: one world DB, but more mts using ?

by Linuxdirk » Post

texmex wrote:What the client should know I've no idea about =)
The client needs to know when a node changes. This is usually triggered by the client manipulating the node (or when some function changes a node). I am not sure if the client proactively queries the database about node changes for nodes that are already loaded.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

the client gets as before the changes he did him/her-self, and will be send also new from server like now,

the question is, how does the server know that a node, block ! near a gamer changed from another gamer on his own server and see the need to send him/her that also as set in conf about distance to actualize ... normal version,
that is working we know

and then how would another server realize that a block got changes even without his own doing, and will go to send them to his clients also ... would be that 2 server using version
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: one world DB, but more mts using ?

by sofar » Post

This isn't possible, at least not without changes to how MT uses the database.

You *could* use postgresql and store the world in the same table and have 2 minetest servers connect to it. That part works already.

What doesn't work is that neither minetest server knows when the other makes changes to the world. That part would require a shared lock between the 2 servers, and some sort of journal or transactional system to make sure that both worlds understand what changes happened on the other server.

The way it works with a single server is much more simple. When blocks are loaded, the server keeps track in memory if itself changes it, and just writes the changes back after a while.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

so it IS possible,
* if the spawns and action areas of the different servers are separated,
* or mainly gamers of different savers are not changing nodes/blocks in other areas
* or the gamer let forget his clients store (mapblock unload timeout) and forces a reaload of the area soon ?
* or limit down the Mapblock limit AND leaves area to force new load
* or i think there is a server setting to force reload (also may rise the network traffic)

when I get how to use one of the two DB that allow multiple access, I sure will give it a try as I see big advantages.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: one world DB, but more mts using ?

by sofar » Post

Festus1965 wrote: when I get how to use one of the two DB that allow multiple access, I sure will give it a try as I see big advantages.
There's nothing to try right now. None of the "exclusions" you list are currently safe enough to do this. You will corrupt the map, guaranteed.

Astrobe
Member
Posts: 571
Joined: Sun Apr 01, 2018 10:46

Re: one world DB, but more mts using ?

by Astrobe » Post

Hmmm... A Minetest build that could act simultaneously as a server and as a client, so it could use a pseudo-player as a proxy to mirror the actions of its player to the main server? Just a little less crazy.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

first test worked

postgresql DB
to 5.0 server with support
mt file with both same connect string, so on mts had to use via local network

both on same p[lace saw same environment, same flower, same position
but when one digged it away, second didn't see a change ... but a start ...

but that is all what I wanted for my idea

as now I can do
* server "Wildes Land" with old spawn 3 position
* server "Celine in Thailand" with old spawn 4 position
* server "Celines Wildes Thailand" as now for new gamers
if i want ... still a lot of test to do
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Linuxdirk
Member
Posts: 3218
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: one world DB, but more mts using ?

by Linuxdirk » Post

Instead of having multiple servers running on the same database basically corrupting the map over time without any benefit a much cooler thing would be having one server being able to provide multiple maps to the players ... :)

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

Linuxdirk wrote:Instead of having multiple servers running on the same database basically corrupting the map over time without any benefit a much cooler thing would be having one server being able to provide multiple maps to the players ... :)
again, someone shooting against an idea.

As I proofed now that it is possible to use ONE postgresql db for mind 2 servers, it again that corrupting map.
(What code might be responsible for this and who a good db should handle it ?)

I will proof it this days. Also, but what I saw in that short part was expected.
* Players from not same server cannot see each other,
* changes in front the other server gamer are not seen directly the dig, or build - as the other server don't know it yet. But I am sure when i start it again, and 2nd server gamer goes same position, should see it then, if not, why and how to solve, ... if needed. As for server 2 there was no need to reload data, it is clear for me yet, not to see changes from 1st gamer.

MY first Idea for now:
* Upgrade my main server with still 0.4.16 to 0.4.17 with postgresql, and migrate map and players
* then use the 5.0.0-rc-fee server to take this data and serve it also on the new version server.

That is enough as can check diffs from old to new, and even test some builds, actions ...
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Linuxdirk
Member
Posts: 3218
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: one world DB, but more mts using ?

by Linuxdirk » Post

Festus1965 wrote:As I proofed now that it is possible to use ONE postgresql db for mind 2 servers,
That was never doubted.
Festus1965 wrote:(What code might be responsible for [map corruption] and who a good db should handle it ?)
There is no specific code. It is the overall concept of how the client operates. There is no way to fix this except rewriting almost the whole system how the client manipulates the world and uses the database.

Please just stop. It is impractical and currently impossible to be done properly. There is no benefit in this. There is no simple way to make it properly work.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

ok, i read, agree most.

2nd test:
* changes of player 1 are not seen of player 2 (server 2)
* after relog from gamer 2 changes also net seen,
* after restart server 2, player 2 (server 2) get the map after login like it was changed from player 1 (server 1) before.

I see some issue to use this.
* An open server with no interact, show what we have, mods, etc. - if player is interested, he need to find the hidden servers (not public list) address to take part ...
* Areas are so far from each other, that gamers might not meet also , even diff mods are possible (i could let old gamers, with only the old mods to there old spawn 2), what is far away, so easier to remember that is my old server (name.)
= server "Wildes Land", with less mods, spawn2
= server "Celines Wildes Thailand" like now spawn5
maybe = server for the kids only "Celine in Thailand" spawn4 - so my kids might not interfere the serverslist
* much more ideas ... like separate areas and languages ...
but testing further ...
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: one world DB, but more mts using ?

by ShadMOrdre » Post

This is actually an interesting experiment. There is absolutely no reason he should stop, unless, you are affected, which I don't believe, you are.

Festus,

The mapserver project uses dual access. While MT is running, the mapserver accesses the database readonly. This might be the only way to prevent data corruption, but of course means, that the readonly MT cannot make changes to the world.

I do not know the engine side of how MT handles database locks, I would defer to Sofar here, but if MT locks the database only at the transaction level, ie, is ACID compliant, then each instance of MT should be able to write to the database, with record level locking occuring only at the time of commit.

Remember, you probably should only be conducting this experiment on a test world, that you do not care about.

Please keep posting your results.

Shad

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: one world DB, but more mts using ?

by sofar » Post

Linuxdirk wrote:Instead of having multiple servers running on the same database basically corrupting the map over time without any benefit a much cooler thing would be having one server being able to provide multiple maps to the players ... :)
this is already possible. If you use postgresql, you can specify the name of the table to use. Therefore, you can have multiple tables in a single database.

From the wiki:
world.mt for PostgreSQL

pgsql_connection -- PostgreSQL connection string for world map | host=<db_host> user=<db_user> password=<db_password> dbname=<db_name>

pgsql_player_connection -- PostgreSQL connection string for players | host=<db_host> user=<db_user> password=<db_password> dbname=<db_name>

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

one world DB, but more mts using ?

by Festus1965 » Post

so far, between a 5.0.0 mtserver and his same machine made postgresql and a 2nd machine with another 5.0.0-minetestserver it worked so far.

today
* I upgraded the main using server from 0.4.16 to 0.4.17.1 with
* compiled without LuaJit, but with postgresql support
* first migrate map and players to postgres,
* migrate auth to sqlite3
and it is running well yet. (about 0.1 - 0.15 lag more yet - need to tune db and move to faster SSD also later)

SOLVED
forget that here behind!
I solved as it was a wrong string databaseconnect.
(the 5.0.0 was write, but i migratet on wrong db in 0.4.17.1 - blame on me)
But I cant connect from 2nd machine with minetestserver, or lets say, it is connection as no error or failure, but I cant see the map. Is there a change from 0.4.17.1 postgresql format to 5.0.0 ?

postgresql server is open for IP, port and allow md5, like I did at other version also. ufw ok, otherwise mts 5.0.0 2 server would complain, but nothing ... hmm
Last edited by Festus1965 on Sat Feb 09, 2019 01:01, edited 2 times in total.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Linuxdirk
Member
Posts: 3218
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: one world DB, but more mts using ?

by Linuxdirk » Post

sofar wrote:From the wiki:
Maybe I'm a bit stupid but with keywords I think that are sufficient ("database", "back-end", "postgresql", etc.) I can't find any information on this.

ShadMOrdre
Member
Posts: 1118
Joined: Mon Dec 29, 2014 08:07
Location: USA

Re: one world DB, but more mts using ?

by ShadMOrdre » Post

Festus,

There is a breaking change between v0.4.16/17 and v5.0.0. This may also affect why you are having issues connecting different servers to the same database. Please try using the same server version for each MT instance.

As for getting this concept working in the larger sense, keep in mind that MT was written using SQLite3, and thus is subject to its limitations. While Postgresql is far more robust, engine changes need to be made to accommodate the differences.

Maybe I should peek at the engine code and see if I can help here. I'll admit to some limitations as far as my C++, but am rock solid on SQL. If I'm lucky, there still might be something useful in me.

twoelk
Member
Posts: 1482
Joined: Fri Apr 19, 2013 16:19
GitHub: twoelk
IRC: twoelk
In-game: twoelk
Location: northern Germany

Re: one world DB, but more mts using ?

by twoelk » Post

while I don't really think having several servers writing to the same map database at the same time is within minetests scope of design I could imagine sharing other resources more usefull.

All my minetest installations for example save screenshots to the same folder parallel to the minetest folders. I could imagine being able to share for example favorites or mods, or even world folders could be way more usefull and above all easilly done compared to the hickhack that servers might get into when deciding whose writing to the map database is the more valid and which server might have to wait while others write to file. I know that some of this is allready possible but it would be nice to have this be more easier and user friendly. The central media server some servers point to when a client joins is allready a similar solution.

As to multible servers managing the same world I would think a server grid concept might be the more interesting thing to do. Several servers each managing only a certain part of a larger map and having some sort of easy and fast way of handing over a user between each other. I guess the easiest way to do this would be to have each server manage an island and let the borders be in the ocean. No new concept of course. It has been discussed in this forum before and there are quite a few other games that have used some sort of grid servers system for years now out there.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: one world DB, but more mts using ?

by Festus1965 » Post

ShadMOrdre wrote:Festus,

There is a breaking change between v0.4.16/17 and v5.0.0. This may also affect why you are having issues connecting different servers to the same database. Please try using the same server version for each MT instance.
My idea was to share same database also for 5.0.0 to see what is running, on 5.0.0 without changes, easier to compare actual world - so need to take backup again for the 5.0.0 check.

SOLVED !
I did a so easy but main mistake, I don't want to talk about.
Access is possible, 5.0.0-rc1-server 2 is accessing 0.4.17.1-server 1 postgresql database on server 1 via network.

twoelk wrote:... As to multible servers managing the same world I would think a server grid concept might be the more interesting thing to do. Several servers each managing only a certain part of a larger map and having some sort of easy and fast way of handing over a user between each other....
That sounds more like my idea, using several servers, on same database in different areas.
* giving access to original old "Wildes Land" (Spawn 1+2), with only this old mods, and use the separated area data, to shelter all the other areas.
* giving access to old "Celine in Thailand" (area Spawn 4), and also this area dat to block other zones from changing

so even would be able to offer diff language servers and zones, but always able to visit the neighbours, but without change.

So mainly a copy of all, but sheltered all, for public view and see if worth to use less known (not mobile client flooded) server for real access.

Somewhere this direction I am still testing.

now learning rename, backup, restore db ...
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests