How to check real lag of server?

User avatar
bosapara
Member
 
Posts: 397
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

How to check real lag of server?

by bosapara » Thu Jan 10, 2019 14:55

How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag

Code: Select all
minetest.place_schematic({x = pos.x - 3,y = pos.y - 4,z = pos.z - 3},minetest.get_modpath("skyblock").."/schems/island.mts","0",nil,false)   



Image
 

User avatar
sorcerykid
Member
 
Posts: 911
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: How to check real lag of server?

by sorcerykid » Thu Jan 10, 2019 17:04

I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.

Image
 

User avatar
bosapara
Member
 
Posts: 397
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 17:53

sorcerykid wrote:I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.


Already read about your mod when searched hud mod that show lag / delay

Glad to see someone take care about how to check real lag
 

sofar
Developer
 
Posts: 1843
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 18:23

bosapara wrote:How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag


"lag" is a subjective term. Many people have different understanding of what it actually means or how it is defined.

From a most popular-definition type perspective, "lag" is the delay between user interactions and the results of those showing up on the client after the server has processed them. It is mathematically limited to never be less than RTT, the "round trip time" of network packets between the client and server (which is twice the one-way time needed).

https://en.wikipedia.org/wiki/Lag

From this definition you can understand that lag is player-related. Each player will have a different value for lag. Minetest displays the RTT value with F5.

For players, this is the most important definition of lag, since it defines how quickly their interactions take effect when they play.

For server owners, the common lag definition isn't very useful since it is almost entirely out of the sphere of influence for them to do much about it. If players connect from Australia to a server in Europe, those players will have 200ms of lag no matter what. For this reason, server owners generally avoid looking at RTT and instead look for data that describes "how fast does the server handle changes". This can be measured in various ways:

- how long does the engine take to process a server step
- how long does lua take to process a server step
- how long does it take for network packets to get sent a reply packet

Unfortunately, each server step isn't the same. Sometimes the server needs to process lots of lua code, sometimes the server just needs to handle a few player movement packets, and sometimes the server needs to modify lots of mapblocks and generate a few hundred thousand new nodes. So unlike RTT, which should be reasonably stable (unless you roam from country to country somehow, or are on the ISS), the server equivalent can flop wildly around from nanoseconds to sometimes over a second.

For this reason you can't just look at the latest value of "server execution time", since it might be 1000x bigger or smaller on the next iteration. Therefore, the MT engine exposes an "averaged" metric called "max_lag" that uses a sliding average type function. This has the benefit of showing you the "generally worst" performance of the server, but if occasional spikes aren't a real problem, it's not very useful. It also includes the minimum time step value, so it can never be 0.

Image
(Inside the Box live max_lag graph)

There are many ways to define server lag in other, more useful ways, but keep in mind that aside from raw data, every representation will need to do some statistical processing on it and therefore you're always looking at a derivative of the data. Sorcerykid's version is a very good approach to making server lag better exposed to server operators. But care must be taken to make sure the wrong interpretation isn't given to the data. For instance, a common pitfall is to use mod profiling which doesn't take ABM processing time into account, since mod profiling only counts Lua execution time, but ABM processing overhead is mostly accounted for in the engine. Schematic placement can be laggy, too, just ask rubenwardy about CTF map changes.
 

User avatar
bosapara
Member
 
Posts: 397
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 19:24

Sofar,

Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.

With /status - lag show me some weird numbers even when server is not laggy, or nothing when some mod is too laggy (profiler).

In other words, I’m disappointed with the default check features, just the meaningless features as for players and for server owners.

Will wait better implementation of lag check features:)
 

sofar
Developer
 
Posts: 1843
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 20:01

bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.


If you didn't understand what I wrote, please tell me what parts of my post were not understood.
 

User avatar
bosapara
Member
 
Posts: 397
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 20:25

sofar wrote:
bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.


If you didn't understand what I wrote, please tell me what parts of my post were not understood.


Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).
 

IhrFussel
Member
 
Posts: 57
Joined: Tue Nov 22, 2016 12:54
GitHub: IhrFussel
IRC: IhrFussel
In-game: IhrFussel

Re: How to check real lag of server?

by IhrFussel » Thu Jan 10, 2019 21:17

bosapara wrote:Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).


Looks like you did not understand at all what he said.

The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.

So if your server lags even once for 1 second, it will take several minutes for max_lag to reach regular numbers again. sorcerykid's mod will likely report realtime numbers (I didn't look at the code but I'm guessing it uses a globalstep callback to determine those values).
 

User avatar
bosapara
Member
 
Posts: 397
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 21:34

IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.


Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.
 

sofar
Developer
 
Posts: 1843
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 21:43

bosapara wrote:
IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.


Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.


As a player, you should just look at RTT. You have little to no control over server execution performance, but you can choose a server that is closer to your location.

max_lag is only of value to server owners who want to improve their server, and has some value in that. It's not a bogus value as many people suggest, just largely misunderstood by almost everyone.
 


Return to Problems



Who is online

Users browsing this forum: Bing [Bot] and 2 guests