[solved] How to check real lag of server?

Post Reply
User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

[solved] How to check real lag of server?

by bosapara » Post

How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag

Code: Select all

minetest.place_schematic({x = pos.x - 3,y = pos.y - 4,z = pos.z - 3},minetest.get_modpath("skyblock").."/schems/island.mts","0",nil,false)	

Image
Last edited by bosapara on Sat Sep 12, 2020 07:18, edited 1 time in total.

User avatar
sorcerykid
Member
Posts: 1462
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: How to check real lag of server?

by sorcerykid » Post

I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.

Image

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

sorcerykid wrote:I actually developed a mod for this purpose last year. I just haven't had time to fully refine the API, due to other life priorities. However, it's entirely functional and has been running successfully on my test server.

Currently it calculates peak, mean, and standard deviation of lag over short and long term. It also provides an improved server uptime function with millisecond accuracy (for cases where a monotonic clock would be problematic). Performance metrics are retained in a stack for easy polling by other mods or exportation to json file. In addition, it interfaces directly with my Polygraph mod, so in-game data visualization is easy. So far it's proven to be very useful for benchmarking of mods, which is why I have it on my test server :)

I'll see what I can do to get a beta version released by sometime next week.
Already read about your mod when searched hud mod that show lag / delay

Glad to see someone take care about how to check real lag

sofar
Developer
Posts: 2132
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Post

bosapara wrote:How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly

An example when im using code for island spawn - i have lag up (from /status) for 1 sec (from 0.2), but im not sure its real lag
"lag" is a subjective term. Many people have different understanding of what it actually means or how it is defined.

From a most popular-definition type perspective, "lag" is the delay between user interactions and the results of those showing up on the client after the server has processed them. It is mathematically limited to never be less than RTT, the "round trip time" of network packets between the client and server (which is twice the one-way time needed).

https://en.wikipedia.org/wiki/Lag

From this definition you can understand that lag is player-related. Each player will have a different value for lag. Minetest displays the RTT value with F5.

For players, this is the most important definition of lag, since it defines how quickly their interactions take effect when they play.

For server owners, the common lag definition isn't very useful since it is almost entirely out of the sphere of influence for them to do much about it. If players connect from Australia to a server in Europe, those players will have 200ms of lag no matter what. For this reason, server owners generally avoid looking at RTT and instead look for data that describes "how fast does the server handle changes". This can be measured in various ways:

- how long does the engine take to process a server step
- how long does lua take to process a server step
- how long does it take for network packets to get sent a reply packet

Unfortunately, each server step isn't the same. Sometimes the server needs to process lots of lua code, sometimes the server just needs to handle a few player movement packets, and sometimes the server needs to modify lots of mapblocks and generate a few hundred thousand new nodes. So unlike RTT, which should be reasonably stable (unless you roam from country to country somehow, or are on the ISS), the server equivalent can flop wildly around from nanoseconds to sometimes over a second.

For this reason you can't just look at the latest value of "server execution time", since it might be 1000x bigger or smaller on the next iteration. Therefore, the MT engine exposes an "averaged" metric called "max_lag" that uses a sliding average type function. This has the benefit of showing you the "generally worst" performance of the server, but if occasional spikes aren't a real problem, it's not very useful. It also includes the minimum time step value, so it can never be 0.

Image
(Inside the Box live max_lag graph)

There are many ways to define server lag in other, more useful ways, but keep in mind that aside from raw data, every representation will need to do some statistical processing on it and therefore you're always looking at a derivative of the data. Sorcerykid's version is a very good approach to making server lag better exposed to server operators. But care must be taken to make sure the wrong interpretation isn't given to the data. For instance, a common pitfall is to use mod profiling which doesn't take ABM processing time into account, since mod profiling only counts Lua execution time, but ABM processing overhead is mostly accounted for in the engine. Schematic placement can be laggy, too, just ask rubenwardy about CTF map changes.

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

Sofar,

Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.

With /status - lag show me some weird numbers even when server is not laggy, or nothing when some mod is too laggy (profiler).

In other words, I’m disappointed with the default check features, just the meaningless features as for players and for server owners.

Will wait better implementation of lag check features:)

sofar
Developer
Posts: 2132
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Post

bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.
If you didn't understand what I wrote, please tell me what parts of my post were not understood.

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

sofar wrote:
bosapara wrote:Even to be a noob in minetest development (me), im not shure that i can check server lag with default /status or check mod with profiler.
If you didn't understand what I wrote, please tell me what parts of my post were not understood.
Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).

IhrFussel
Member
Posts: 69
Joined: Tue Nov 22, 2016 12:54
GitHub: IhrFussel
IRC: IhrFussel
In-game: IhrFussel

Re: How to check real lag of server?

by IhrFussel » Post

bosapara wrote: Ive understood all what you wrote, thanx for explaining.

Again just tell what I mean: I havent not one tool to check server lag (as server owner) in case of performance mod or "laggy' mod (to explain more correctly).
Looks like you did not understand at all what he said.

The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.

So if your server lags even once for 1 second, it will take several minutes for max_lag to reach regular numbers again. sorcerykid's mod will likely report realtime numbers (I didn't look at the code but I'm guessing it uses a globalstep callback to determine those values).

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.
Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.

sofar
Developer
Posts: 2132
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: How to check real lag of server?

by sofar » Post

bosapara wrote:
IhrFussel wrote:The value you see in /status is max_lag which is a sliding average meaning it goes down slowly after 1 lag spike.
Main problem - server show 'fake lag' with /status even when i havent any lags or it doesnt show lags when server has huge lags.
As a player, you should just look at RTT. You have little to no control over server execution performance, but you can choose a server that is closer to your location.

max_lag is only of value to server owners who want to improve their server, and has some value in that. It's not a bogus value as many people suggest, just largely misunderstood by almost everyone.

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: How to check real lag of server?

by Festus1965 » Post

sofars explanation is very well, and I share his point.

So if you or anyone check for lag,
first he should understand
  • * time of data send between server location and client location (gamer)
    * time the server need to process the incoming date (change position, build, etc) and process* them, and send them out
    * time or ability the clients hardware (CPU, RAM speed, GPU and game Settings) is able to show the received data
THEN the shown lag of /status or /lagcheck or /lag make more sence.

For Excample:
  • * MY Server is in Thailand, but very fast, myself logging into the server via home GB Connection show RTT of 0.001 as I have no distance for the data to travel. (Connected via fiber 50 mb / 20 mb to internet)
    * The Server itself is also very fast: i7-4770 CPU, all 4 Memory slots used (= faster RAM Speed) and a 2.200 read write M.2 SSD, instead of slow normal 250 HDD readings, also keep most needed parts in fast RAM cache
    and as my
    * Gaming Maschine is even same CPU, 4 memory slots used, and has GPU GTX 960 2 GB (but 3 monitors) - and remember good working GPU driver is important
  • --- so for me my own server is really fast, and i only realize lag when new blocks are generated (lag most near 0.2 even with moretrees, technic, monsters, ... but then getting to 2.5)
    --- but if I would sit at my old home in Germany using same PC there, might not change any speed of my Rig or the server, but just the data between have to travel about 7.000 km fiber, what is a not change fact of 220 ms about.
    So THIS lag might be a network-transport lag.
And also clients using Wifi, or even have a slow or bad Internet connection (old cable, using 4G, or Wifi at home) have or make there own lag ... not knowing it.

Easy to check your "ping" to the server you like in a dos / cmd box with "ping address" (no port)

So if you have a short ping as inside Germany or USA you and server location (should be about 20-40 ms then), and it is laggy, you might think about your rig or the server, the servers mod and if a lot of players are there - and just ONE of them going to new area (mean activation mapgen) you will get lag again (here from the server).

So what I often here about "this server lags" I don't listen to it anymore - as this gamer sure doesn't understand this basics.

about:
* fake-lag ... the real lag made a break or stop might just happen for 5 seconds, but the algorithm of the servers showed lag is very slow getting down, it is not the reality, that why sometime you feel good, even lag show bad
* like sofar wrote: chose a nearby server - check it with ping

And to answer the starting question:
How to check real lag of server?
the shown lag of server (/status or at me with hud) IS THE REAL SERVER LAG, mean the time the server need to do his work, not mention what clients do and how far they are.

So when the /status lag is low, but you feel lag, see the RTT, then you might understand better and think about distance from you and server, or flooded network.

added:
The ONLY REAL lag the server can show is his own lag (mean get data and offer them), as every player with his distance and lousy network connection is DIFFERENT.

YOUR gamer lag depends basically on your distance to the server and the time your
* wish data (build, dig, movement) had to get the way through wifi, router, provider netork, internet to the server,
* worked out there in the real server lag time
* and send back the same way again to reach your unit
* and need to be worked out by your CPU and GPU
AND that lg is different.

When press F5 you might see your RTT, and if this is over 50 the server is already out of own county, and if more than 200 maybe take another one.

And as the serverlist ping is cheating (as its base is always distance to a server from the Netherlands server itself)

then better check one a month with Minetest-ping-sort and take only those servers with less than 200, better less than 100 ping. (100 ping = .1 RTT)

and if your using Wifi ... all problems are just there !
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

After a long time, I found a reason of lag and a solution.

If you need to delete about >40000 blocks or kind of it, never use "minetest.delete_area".

Instead of it use "minetest.get_voxel_manip()"

In result i had 0.023 seconds delay for island update instead of 1.3 sec with "minetest.delete_area".

I hope this information will be useful.

Image

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: How to check real lag of server?

by Festus1965 » Post

I am sorry, but a real lag is a steady all time server online similar lag as of using the game.

Your description is a act of special order for some time (maybe every 1 hour ?), but not running all the time.
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

Festus1965 wrote:
Fri Sep 11, 2020 01:26
I am sorry, but a real lag is a steady all time server online similar lag as of using the game.

Your description is a act of special order for some time (maybe every 1 hour ?), but not running all the time.
While server was online, for island update (in Skyblock mode) server got a lag each join of new player, +1.3 sec.

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: How to check real lag of server?

by Festus1965 » Post

bosapara wrote:
Fri Sep 11, 2020 11:42
While server was online, for island update (in Skyblock mode) server got a lag each join of new player, +1.3 sec.
Are you joking me and us all again, during an UPDATE, ARE you updating all day 24 hours long ...

I do updates when no-one is online,
When I do backup I tell the guys, if don't like they should go offline or wait that 10 minutes.

I understand again, why I stop this here ... OMG.
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

Festus1965 wrote:
Fri Sep 11, 2020 13:30
I understand again, why I stop this here ... OMG.
Probably you misunderstood what is Skyblock mode.

Skyblock mod reset / update the area with island each new player join. Area is about > 40000 blocks, so lags sometimes happens if we use minetest.delete_area.

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: How to check real lag of server?

by Festus1965 » Post

bosapara wrote:
Fri Sep 11, 2020 13:45
Probably you misunderstood what is Skyblock mode.

Skyblock mod reset / update the area with island each new player join. Area is about > 40000 blocks, so lags sometimes happens if we use minetest.delete_area.
THEN I found the source of lag ... it's only you, with your configuration of the game

you accept the lag or you change your crazy config ...
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

Festus1965 wrote:
Sat Sep 12, 2020 01:05
THEN I found the source of lag ... it's only you, with your configuration of the game
you accept the lag or you change your crazy config ...
Are you troll or just make your message stats better with random messages?

________________

Solution of topic already found, for huge areas better to use "minetest.get_voxel_manip()" instead of "minetes.delete_area".

Can be useful for many server owners who update a huge areas several times a day.

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: How to check real lag of server?

by Festus1965 » Post

bosapara wrote:
Sat Sep 12, 2020 07:17
Are you troll or just make your message stats better with random messages?
NO troll - just I realize the Thread theme and are smiling about "How to check real lag of server?" and not how to optimize your specific problem.
Typically I might tell you open another Thread for this then.
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: How to check real lag of server?

by bosapara » Post

Festus1965

The meaning of the topic is in the first post of this thread.


Image

Festus1965
Member
Posts: 1435
Joined: Sun Jan 03, 2016 11:58
In-game: bye bye

Re: [solved] How to check real lag of server?

by Festus1965 » Post

bosapara wrote:
Thu Jan 10, 2019 14:55
How to check real lag of server? Any mods or tricks for server owners?

Unfortunately /status and profiler can't show it correctly
That post already was enough to read ... when you dont "trust" a value that is at all servers on the same base ... what do you expect then ?

And only, when all servers use same base, and gamer can so compare it ... it is still trusted by me.

The most problem was that gamers mix server lag with the personal lag, or think when they have lag, forgot there own music video in background, or father just looking a porn.

But when you even mistrust the until now working ones ...

In your case, as I had similar thoughts long time ago:
* a measurement with no gamer: mean empty running server lag
* a measurement with 5 or 10 gamers without join peak
* and a hint to max lag as of new gamer joining

whatever ... but as of this years I know already that a overall minetest solution is not been seen ... just see your mistrust in the working ones, and with your special lag making solution.

Even my server peaks a bit with a join, learned that spawn point should not be in a jungle forest or between a lot of machines, etc.
THIS is a solid knowledge from most server managers here I think, but not you ... ?
Minetest has no future

User avatar
bosapara
Member
Posts: 618
Joined: Fri Apr 07, 2017 08:49

Re: [solved] How to check real lag of server?

by bosapara » Post

Festus1965,

Everything you write does not matter when we talk about complex mods that use timers.

No profiler, or /status will never show you the delay / lag.

________________

Only 100% of players who tell you "I can't open a chest within 5 secs" can help you to detect lags.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests