How to check real lag of server?

User avatar
bosapara
Member
 
Posts: 412
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

How to check real lag of server?

by bosapara » Thu Jan 10, 2019 14:55

How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag

Code: Select all
minetest.place_schematic({x = pos.x - 3,y = pos.y - 4,z = pos.z - 3},minetest.get_modpath("skyblock").."/schems/island.mts","0",nil,false)   



Image
 

User avatar
sorcerykid
Member
 
Posts: 1001
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: How to check real lag of server?

by sorcerykid » Thu Jan 10, 2019 17:04

I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.

Image
 

User avatar
bosapara
Member
 
Posts: 412
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 17:53

sorcerykid wrote:I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.


Already read about your mod when searched hud mod that show lag / delay

Glad to see someone take care about how to check real lag
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 18:23

bosapara wrote:How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag


"lag" is a subjective term. Many people have different understanding of what it actually means or how it is defined.

From a most popular-definition type perspective, "lag" is the delay between user interactions and the results of those showing up on the client after the server has processed them. It is mathematically limited to never be less than RTT, the "round trip time" of network packets between the client and server (which is twice the one-way time needed).

https://en.wikipedia.org/wiki/Lag

From this definition you can understand that lag is player-related. Each player will have a different value for lag. Minetest displays the RTT value with F5.

For players, this is the most important definition of lag, since it defines how quickly their interactions take effect when they play.

For server owners, the common lag definition isn't very useful since it is almost entirely out of the sphere of influence for them to do much about it. If players connect from Australia to a server in Europe, those players will have 200ms of lag no matter what. For this reason, server owners generally avoid looking at RTT and instead look for data that describes "how fast does the server handle changes". This can be measured in various ways:

- how long does the engine take to process a server step
- how long does lua take to process a server step
- how long does it take for network packets to get sent a reply packet

Unfortunately, each server step isn't the same. Sometimes the server needs to process lots of lua code, sometimes the server just needs to handle a few player movement packets, and sometimes the server needs to modify lots of mapblocks and generate a few hundred thousand new nodes. So unlike RTT, which should be reasonably stable (unless you roam from country to country somehow, or are on the ISS), the server equivalent can flop wildly around from nanoseconds to sometimes over a second.

For this reason you can't just look at the latest value of "server execution time", since it might be 1000x bigger or smaller on the next iteration. Therefore, the MT engine exposes an "averaged" metric called "max_lag" that uses a sliding average type function. This has the benefit of showing you the "generally worst" performance of the server, but if occasional spikes aren't a real problem, it's not very useful. It also includes the minimum time step value, so it can never be 0.

Image
(Inside the Box live max_lag graph)

There are many ways to define server lag in other, more useful ways, but keep in mind that aside from raw data, every representation will need to do some statistical processing on it and therefore you're always looking at a derivative of the data. Sorcerykid's version is a very good approach to making server lag better exposed to server operators. But care must be taken to make sure the wrong interpretation isn't given to the data. For instance, a common pitfall is to use mod profiling which doesn't take ABM processing time into account, since mod profiling only counts Lua execution time, but ABM processing overhead is mostly accounted for in the engine. Schematic placement can be laggy, too, just ask rubenwardy about CTF map changes.
 

User avatar
bosapara
Member
 
Posts: 412
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 19:24

Sofar,

Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.

With /status - lag show me some weird numbers even when server is not laggy, or nothing when some mod is too laggy (profiler).

In other words, I’m disappointed with the default check features, just the meaningless features as for players and for server owners.

Will wait better implementation of lag check features:)
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 20:01

bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.


If you didn't understand what I wrote, please tell me what parts of my post were not understood.
 

User avatar
bosapara
Member
 
Posts: 412
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 20:25

sofar wrote:
bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.


If you didn't understand what I wrote, please tell me what parts of my post were not understood.


Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).
 

IhrFussel
Member
 
Posts: 62
Joined: Tue Nov 22, 2016 12:54
GitHub: IhrFussel
IRC: IhrFussel
In-game: IhrFussel

Re: How to check real lag of server?

by IhrFussel » Thu Jan 10, 2019 21:17

bosapara wrote:Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).


Looks like you did not understand at all what he said.

The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.

So if your server lags even once for 1 second, it will take several minutes for max_lag to reach regular numbers again. sorcerykid's mod will likely report realtime numbers (I didn't look at the code but I'm guessing it uses a globalstep callback to determine those values).
 

User avatar
bosapara
Member
 
Posts: 412
Joined: Fri Apr 07, 2017 08:49
GitHub: bosapara
In-game: Jozet

Re: How to check real lag of server?

by bosapara » Thu Jan 10, 2019 21:34

IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.


Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.
 

sofar
Developer
 
Posts: 1997
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Thu Jan 10, 2019 21:43

bosapara wrote:
IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.


Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.


As a player, you should just look at RTT. You have little to no control over server execution performance, but you can choose a server that is closer to your location.

max_lag is only of value to server owners who want to improve their server, and has some value in that. It's not a bogus value as many people suggest, just largely misunderstood by almost everyone.
 

User avatar
Festus1965
Member
 
Posts: 729
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
In-game: Thomas Explorer

Re: How to check real lag of server?

by Festus1965 » Mon Jan 28, 2019 11:32

sofars explanation is very well, and I share his point.

So if you or anyone check for lag,
first he should understand
    * time of data send between server location and client location (gamer)
    * time the server need to process the incoming date (change position, build, etc) and process* them, and send them out
    * time or ability the clients hardware (CPU, RAM speed, GPU and game Settings) is able to show the received data

THEN the shown lag of /status or /lagcheck or /lag make more sence.

For Excample:
    * MY Server is in Thailand, but very fast, myself logging into the server via home GB Connection show RTT of 0.001 as I have no distance for the data to travel. (Connected via fiber 50 mb / 20 mb to internet)
    * The Server itself is also very fast: i7-4770 CPU, all 4 Memory slots used (= faster RAM Speed) and a 2.200 read write M.2 SSD, instead of slow normal 250 HDD readings, also keep most needed parts in fast RAM cache
    and as my
    * Gaming Maschine is even same CPU, 4 memory slots used, and has GPU GTX 960 2 GB (but 3 monitors) - and remember good working GPU driver is important
    --- so for me my own server is really fast, and i only realize lag when new blocks are generated (lag most near 0.2 even with moretrees, technic, monsters, ... but then getting to 2.5)
    --- but if I would sit at my old home in Germany using same PC there, might not change any speed of my Rig or the server, but just the data between have to travel about 7.000 km fiber, what is a not change fact of 220 ms about.
    So THIS lag might be a network-transport lag.
And also clients using Wifi, or even have a slow or bad Internet connection (old cable, using 4G, or Wifi at home) have or make there own lag ... not knowing it.

Easy to check your "ping" to the server you like in a dos / cmd box with "ping address" (no port)

So if you have a short ping as inside Germany or USA you and server location (should be about 20-40 ms then), and it is laggy, you might think about your rig or the server, the servers mod and if a lot of players are there - and just ONE of them going to new area (mean activation mapgen) you will get lag again (here from the server).

So what I often here about "this server lags" I don't listen to it anymore - as this gamer sure doesn't understand this basics.

about:
* fake-lag ... the real lag made a break or stop might just happen for 5 seconds, but the algorithm of the servers showed lag is very slow getting down, it is not the reality, that why sometime you feel good, even lag show bad
* like sofar wrote: chose a nearby server - check it with ping

And to answer the starting question:
How to check real lag of server?
the shown lag of server (/status or at me with hud) IS THE REAL SERVER LAG, mean the time the server need to do his work, not mention what clients do and how far they are.

So when the /status lag is low, but you feel lag, see the RTT, then you might understand better and think about distance from you and server, or flooded network.
I own minetest.one and offer a Minetestserver called Thailand Minetest.one (5.1.0-dev) on server.minetest.one.
 


Return to Problems



Who is online

Users browsing this forum: No registered users and 3 guests