How to run server and client(s) synchronously

For people working on the C++ code.
Post Reply
rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

How to run server and client(s) synchronously

by rkla » Post

Hi,

with a few people at EleutherAI we are working on a minetest fork that can be used for testing AI systems in complex environments.

In order to speed up / slow down minetest for the purpose of training these AIs it would be beneficial to have server and client(s) run synchronously and maybe even in the same process.

The relevant loops seem to be in server.cpp and game.cpp
Would a simple approach like communicating a time step between these two work?
What other ways can you think of that might work?

User avatar
Krock
Developer
Posts: 4649
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: How to run server and client(s) synchronously

by Krock » Post

Use the server to send a dtime multiplier via network. That will affect the physics, however it cannot change functionalities that directly take the system millisecond/microsecond time.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: How to run server and client(s) synchronously

by rubenwardy » Post

I've written a script here that runs headless clients. It's quite expensive on resources though. https://gitlab.com/rubenwardy/classroom ... s/start.sh

It might be better to use something like Hydra, which is a framework to make headless clients without any rendering etc
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Krock wrote:
Sun Feb 05, 2023 21:08
Use the server to send a dtime multiplier via network. That will affect the physics, however it cannot change functionalities that directly take the system millisecond/microsecond time.
That sounds promising, because we mainly care about the physics. What features are timed based on the client system time?

In terms of implementing this, I guess we can just add a command to `networkprotocol.h`?
Or is there maybe already a generic broadcast command?
rubenwardy wrote:
Sun Feb 05, 2023 21:14
I've written a script here that runs headless clients. It's quite expensive on resources though. https://gitlab.com/rubenwardy/classroom ... s/start.sh

It might be better to use something like Hydra, which is a framework to make headless clients without any rendering etc
Thanks for sharing!
We have modified parts of the rendering pipeline to render offscreen and hide the window.
But haven't actually checked yet whether it is more performant than using xvfb.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

rkla wrote:
Sat Feb 04, 2023 19:09
How to run server and client(s) synchronously
Can you explain what that mean ?

As the client always gets data, AFTER server is ready to process them, and deliver a client can never be on the same state as a server,
* as server own lag (time to process)
* network transport (even if server and some clients run on same machine and save rtt)
with some default settings changed I get max lag of 0.033 as of values in minetest.conf, but you need 0s difference I read out of "synchronously"

This why I ask about "synchronously" or what you mean to see.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
rubenwardy
Moderator
Posts: 6972
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: How to run server and client(s) synchronously

by rubenwardy » Post

rkla wrote:
Sun Feb 05, 2023 23:33
Thanks for sharing!
We have modified parts of the rendering pipeline to render offscreen and hide the window.
But haven't actually checked yet whether it is more performant than using xvfb.
There's probably a reason why I didn't just use a null device but I don't remember what it is
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Festus1965 wrote:
Sun Feb 05, 2023 23:51
rkla wrote:
Sat Feb 04, 2023 19:09
How to run server and client(s) synchronously
Can you explain what that mean ?
It's a good question.
I think the main problem I would like to fix is missing information on the client side due to lag in the asynchronous communication.
For example, missing sections of the map or wrong inventory state etc. because client is updating without waiting for the missing information to arrive.

But this doesn't mean that no lag is allowed, I just want to delay the client updates until the server tells them it's their turn to update.
As a byproduct this would allow setting a desired update frequency, i.e. slow down/speed up the game (up to hardware and network limitations).

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

rkla wrote:
Mon Feb 06, 2023 00:33
It's a good question.
I think the main problem I would like to fix is missing information on the client side due to lag in the asynchronous communication.
For example, missing sections of the map or wrong inventory state etc. because client is updating without waiting for the missing information to arrive.
* Missing section on a map are likely more a problem of slow and low server disc and missing cache - of default DB.
* Missing information: yes from both server lag, and distance rtt like you build a stone, client know that was ok and put it / show the stone ... but as of settings missing data, client might take stone away as no confirmation, and later stone comes back, after data from server arrive.
I have a lot of Videos from this behavior on several server, what can lead to death, if you are just inside the stones you dig and 'detected' then as cheater with hurt to death.

Then you have to play with some settings in server:minetest.conf as

Code: Select all

# dedicated_server_step = 0.09
dedicated_server_step = 0.033
# server_step = 0.05
server_step = 0.033
not sure anymore which of them force a faster sending of data.

It will be a bit a problem as sending new data can only be done, when all new data for server are worked out and new results are ready to share with all clients.
So to avoid client lag, this should be optimized and kept very low.

Of what distance between server and clients we talk ? just nearby like me on my server is 1 ms rtt, but - as I wrote somewhere else - we need to be aware that we always play in the past ... me to EU 200ms.

Client has so far I see two things after receive, update own data and then 'send' to screen (fps).

So far, IF you are sure that server always have same lag, can adjust the data net sending time with a similar repeating time,
I let rtt out
and as of fE 60 Hz mean every 16.67 ms a new frame can be shown, that would be some option to relay on with
* server should be ready faster,
* server should send date in same frequency - that no further delay is there.

The problem, if you have a bigger distance between server and client, and you might NOT let the client update itself, before the same confirming data from server is there, might be a problem for gaming, as build nodes will be missing, and not available for further build on that one.

Sure depends on what data / usage that will be used.

But there are typical limits like fps 60 (default most) with easy back calculated to what speed the server might have to be ready and send. If we can see this in real is another thing.

That the client might be able to process even every 8 or 4ms new data and work them out, possible.

IF server need too long to get ready, I think useless same data might be send, but I am not sure.


Some hints on my 7 years gaming, own server and play on distant server:
The most and first impact to asynchronicity it a slow server itself, to weak hardware (CPU RAM Cache speed),
and maybe clients with weak own CPU RAM not able to keep up to process (I don't mean show them on screen = another problem of gamer)
The lost on the way is NOT that problem in most cases, as the client can predict a bit,
BUT sure it is difficult to play on slow server (lag) much MORE than play on distant server (rtt).
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Desour
Member
Posts: 1469
Joined: Thu Jun 19, 2014 19:49
GitHub: Desour
IRC: Desour
In-game: DS
Location: I'm scared that if this is too exact, I will be unable to use my keyboard.

Re: How to run server and client(s) synchronously

by Desour » Post

The easiest thing to do is probably to just open a tcp socket or a pipe for each client and for the server, and block on it for a message at some point in the client and server loop. Then you can control from a separate process when each thing should do its next step.
he/him; Codeberg; GitHub; ContentDB; public personal TODO list; "DS" is preferred (but often too short)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

DS-minetest wrote:
Mon Feb 06, 2023 22:37
The easiest thing to do is probably to just open a tcp socket or a pipe for each client and for the server, and block on it for a message at some point in the client and server loop. Then you can control from a separate process when each thing should do its next step.
Thanks for suggesting that. I thought that would lead to a lot of trouble, but actually it worked out of the box. Now I just call the code from AsyncServerStep within the dedicated server loop and block client updates while server is updating and vice versa through a separate tcp connection.

I also send a dtime to clients as suggested earlier to be able to define how much in-game time passes in each update.
However, I am not entirely sure that I undertood the dtime that is used in the server loop.
Is it an ingame time or related to network timing?

Another problem that I encountered is periodic spikes of the server update time to around 1s. In this plot those are the spikes toward the bottom that occur roughly every 5s or 100 steps: https://drive.proton.me/urls/4GJ35JJCFR#hn6vrHGBTlI1
Btw, the dtime that the server sends is 0.05.
I am not quite sure where they come from. I speculate that it is some expensive function call that is only necessary infrequently. Of course in the asynchronous case this isn't a problem, as it is running in the server thread, but for my case I would probably want to split this calculation up over multiple time steps...
Do you have any idea which process could cause these spikes?

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

every 4.7 s or so is the saving of the map - check that - this spike was found by several other server admins before.
play with that saving time and check if the spike also moves.

dtime from serverstep - you can also change that if needed
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Festus1965 wrote:
Sat Feb 11, 2023 22:29
every 4.7 s or so is the saving of the map - check that - this spike was found by several other server admins before.
play with that saving time and check if the spike also moves.

dtime from serverstep - you can also change that if needed
Thanks! Yes, it is indeed the save interval of the map, from settings:
server_map_save_interval = 5.3

I don't need to save the map state at all, so I will set this to a very large number.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

rkla wrote:
Sat Feb 11, 2023 23:12
Thanks! Yes, it is indeed the save interval of the map, from settings:
server_map_save_interval = 5.3

I don't need to save the map state at all, so I will set this to a very large number.
if you use writing cache at server unit (sysctl.conf vm.dirty_... settings) to use free RAM as cache, you will not feel it anymore as on my server also - but if not need, then ok
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Festus1965 wrote:
Sat Feb 11, 2023 23:17
if you use writing cache at server unit (sysctl.conf vm.dirty_... settings) to use free RAM as cache, you will not feel it anymore as on my server also - but if not need, then ok
There is another common source of spikes, albeit smaller ones, which in the profiler shows up as "map timer and unload".
I found there is a related setting "server_unload_unused_data_timeout", but that does not allow one to limit the compute time.
Do you maybe know a trick to limit these spikes?
another solution might be to just offload such calls to other threads to not block on them..

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

rkla wrote:
Sun Feb 12, 2023 01:10
There is another common source of spikes, albeit smaller ones, which in the profiler shows up as "map timer and unload".
I found there is a related setting "server_unload_unused_data_timeout", but that does not allow one to limit the compute time.
Do you maybe know a trick to limit these spikes?
another solution might be to just offload such calls to other threads to not block on them..
server_unload_unused_data_timeout
is the time set that minetest itself (own cache) forget about mapblocks,
and if needed as of a player enter or force another mapblock to load
it (a mapblock) need to be read again from the slow disc, instead just take if from own ram cache

= so to keep a mapblock in mt cache is faster, and avoid a slow reload from slow disc
= server_unload_unused_data_timeout with higher amount is better

but you need to check your server memory (free -m) if there is enough RAM to be used for this cache, and slow rise the value at server_unload_unused_data_timeout as of possible and need

another option to avoid slow disc access is also to use (here works on ubuntu the sysctl.conf with vm.dirty... to use free ram as system cache, wider usage than only mapblocks


also consider to change to faster database instead of sqlite3 default, like to postgresql as it IS faster (I use it) and also allow to keep that data (mapblock and player) also in internal set higher cache

depending on what your using the mtserver, also consider to reduce the "# active_block_range = 4 (default)" in the minetest.conf, as 4 mean each side of player +4 mapblocks is a diameter of 9 * 9 * 9 mapblocks keep active is too much as I know.
I work with value = 2 and that is fine.. mean a range of 5*5*5 is always active around player and enough for any game and lower the impact a lot.


otherwise activate build in mod profiler and take a extract with "/profiler save" and look if any of the mods including default shows high peaks, as can see it there.
otherwise use htop for
* check if it is server or system caused,
* and also can use htop to give mt server a higher priority to run, I use 18 for server, and 19 for database, so all other task have 20 and are after mt if CPU is in doubt what to do first
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Thanks! I have changed settings as you suggested and it is faster now.

I had another question when looking at transmitting information from server to client via mod channels.
For example if I send some information in every global step of the server mod, most of the times I can receive a message in every step, but sometimes a step is skipped.
Is it possible to synchronize this as well? It seems it should be possible but since this goes through the lua interface it is hard to debug where a delay of the message is produced.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

Maybe ! ...
It is not possible to send every step if the server is not ready processing the data, and so a send is blocked ? as it can't send half prepared data ?
Here I am guessing, as mentioned before ... the limit of speeding sending new data is limited by its state to be ready for ?
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

Okay, so I have tracked down the problem.
The server does always send out the packets, but they sometimes don't arrive in time for the client to receive them. I checked this by pausing the game loop before the client receives packets and found that pausing for around 50ms ensures that the packets I need arrive.
The problem with that is of course that it again limits frame rate. And it seems like the number of packets that arrive varies a lot between steps.
So I think there are two possible solutions:
1. count the number of packets sent by the server each time step and let the client wait until this number of packets arrives.
2. don't use mod channels at all, just transmit the information I need via the blocking tcp socket

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: How to run server and client(s) synchronously

by Festus1965 » Post

My thought is where is this 0.050s or 50 ms coming from ?
??? # server_step = 0.05

What is the rtt between your server and client ?

Isn't the server continues sending packages depending on the
# dedicated_server_step = 0.09
# server_step = 0.05
settings ? And just some of them 'we' as client wait for or realize was changing something we could realize ?
... as fE the direction we look to ... (grass or a tree have grown inside the distance of server is aware to need to tell us this change as of our position)

interesting to follow this experiment !
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

rkla
New member
Posts: 9
Joined: Sat Feb 04, 2023 18:42
GitHub: rkla

Re: How to run server and client(s) synchronously

by rkla » Post

The sync mode now works pretty well.
I am able to control the ingame time (which is a lot of fun to play around with, e.g. setting dtime = 0.005 you get slow motion feeling, where as dtime = 0.1 you get double time).
I can get the information I need from server mods, read it out through lua global variables and send it to the client every step through the blocking tcp connection.

The only bugs that remain, seem to be related to loading/generating the map.
For example, if I run to the end of the map for a few minutes the game will segfault at some point.
This might also be unrelated to the client server communication and due to the minetest.conf settings I am using to improve performance:

Code: Select all

default_privs = interact, shout, debug
developer_mode = false
enable_client_modding = true
show_debug = false
csm_restriction_flags = 0
enable_mod_channels = true
debug_log_level = verbose
profiler_print_interval = 0
enable_sound = false

# below are the settings that might be causing the problems
server_map_save_interval = 1000000
server_unload_unused_data_timeout = 1000000
client_unload_unused_data_timeout = 1000000

active_block_range = 4
abm_time_budget = 0.01
abm_interval = 0.1
active_block_mgmt_interval = 4.0

num_emerge_threads = 0
full_block_send_enable_min_time_from_building = 0.
max_block_send_distance = 100
max_block_generate_distance = 100
cache_block_before_spawn = true
IIRC the map loading/generation is done in separate "emerge" threads, so I wonder whether this could be at odds with the synchronization attempt.
I guess since my overall goal is that the map is always loaded/generated in time before the client renders a partly missing chunk I will try to run this code in the main loop and see if that helps.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests