[Server 5.2.0-dev] Asia Thailand Minetest.One P30001 lag 0.1

[Festus1965]

some might have missed it,
some might have known that this was needed,

used game founded by Hectic and his own mods and enlarged by Festus1965 with technic, and pipeworks and more,

got an own name:

game: "BrainCraft"

hunger and monsters are off, as most used from builders and kids

[Festus1965]

Backup created: 14.879.390.848 Byte and also saved external

Server is/was all time still online


added evening:
after a lot of my daugthers friends was here at home, all PC used, and got 10 gamer in max, = place two server list for 1 hour - just after Pandorabox

[Festus1965]

Backup created: 14.892.782.775 Byte and also saved external

Server is/was all time still online

[Festus1965]

after electricity down after 7:30 ICT (0:30 UTC) for 2.5 hours,

Server is back on,
map-server is back on

added 2.11.2019:
/profiler save results under 5.1.0:
* avg:
** technic 29.9% (as I most stand there in my big technic space ship, when not active),
** areas 12.5% (?),
** mobs_animal 11.6%,
** pipeworks 11.1%,
** mesecons 10.8%,
** player_api 4.8%
missing mobs_monster as mod off, and so other mods get automatic more % as of less 100% compare with included mobs_monster
why areas so high ?
* max:
** pipeworks 99,9%,
** technic 99,5%,
** mesecons and mobs_animal 99.8 %,
** areas 99,3,
** player api 99,2%
= no of this mod takes really much more peak in the top 6, so for me mean nothing to worry here = balanced

lag can rise during login to 0.68 max,
and most stay around 0.25 or less

[Festus1965]

Backup created: 14.894.814.313 (+ 2.031.538 von 14.892.782.775) Byte and also saved external

Server is/was all time still online

[Festus1965]

prepared some updates:
* [mod]travelnet
new:
* [mod]xp_redo
* [mod]xp_redo_ranks_ores
* [mod]ranktools (make everything not work) not active

update:
Server restarted,
* travelnet works (remove station now)
* xp_redo (rank) works

[Festus1965]

server is back to version 5.0.1 as of lag by emitter, cant be handled by 5.1.0. - see: Video

Server is on with 5.0.1, but still lag near 1.4,
I try to finish that work and stop emitter then final.

added:
after emitter off, lag back down to 0.25 and more, fast

[Festus1965]

report after 3 days with V5.0.1 and new mod
* good, no crash
* lag most under 0.3, at 22 gamer until 0.5

profiler 2019.11.07 10.00
* 1. - technic - 29.7% (max 99,9 - 2.)
* 2. - pipeworks - 23.8% (max 99.8 - 5.)
* 3. - areas - 10.3% (max 99.5 - 7.)
* 4. - mesecons - 7.2% (max 99,9 - 2.)
* 5. - default - 4.6% (max 99,5 - 7.)
* 6. - player_api - 4.2% (max 99,4 - 10.)
* 7. - unified_inventory - 3.4% (max 99,5 - 7.)
* 8. - mobs_animals - 3.1% (max 99.6 - 8.)
* 9. - xp_redo - 2.6% (max 99,4 - 9.)
* 10. - time - 2.4% (max 99.2 - 13.)
...
* 14. - biome_lib - 0.7% (max 100 - 1.)
...
* 26. - signs_lib - >1% (max 99.9 - 2.)

Estimated as of profiler top technic:
I am often in the SpaceShip and activating all technic.

... until now XP_redu ranking only at German version.

added:
* yeah, I guess I found the reason of raising lag at 5.1.0 as I got lag on 5.0.1 back from 0.5 down to 0.2 - need a bit more proof how, and where to change settings

Server back online
* [mod] homedecor active now

[Festus1965]

prepared changes for next restart:
* new [mod] homedecor (+/...) now activating
* new [mod] unified_inventory_plus (good help)
* [mod] playeranim true - again, check lag

As just was no one in,
Server with new mods restarted.

mod work all, lag 0.24 just now ... with 7 gamer

[Festus1965]

further set changes to next restart:
* active_block_range = 1 (should still be enough)
* protector_radius = 8 (to "synchronise" protection areas with mapblock size)

[Festus1965]

Backup created: 15.121.866.303 (+ 227.051.990 Byte von 14.894.814.313 ) Byte

Server is/was all time still online

[Festus1965]

I crashed the server,
a Itemframe and a plant dont like each other

Server is back online (automatic, after about 25 sec)
sew setings are active,
= lag now about 0.16

added:
lag down to 0.13 with one gamer,
my spaceship is basic ready, started first mining and rubbertree cut, so see the automatic further crafting ... so far ok
the setting 8 for protectorblocks is a bit "funny" have to investigate there if ...

added 2:
as of missing option to craft copper_wire and motor, I had to update homedecor_pack and basic_materials
so far I see a lot of items ...

Server back online

[Festus1965]

thanks to BB
Monitoring: monitoring.minetest.land/...minetest.one is working.
Now you can see the truth about ... whatever

things I see yet:
* also in monitoring my server is very low in lag, even I let 14 gamer in ... still nice low ~350

* a message im terminal

Code: Select all

[monitoring] globalstep took 265318 us in mod *builtin*

about every 30 min ... reported/question here ... I remember a peak at pandorabox.io about every 30 min, at my server nothing ...

* when dig a blinky and set a adjustable blinky for it (even now with mesecons onto 4 stack injector), I see a peak at mesecons ... but cant reproduce it

* the biggest sean lag peak was 919 (before see just 515) with same time new gamer joined (took border up/close then again as before restart)

[Festus1965]

confirmed:

* a message im terminal

Code: Select all

[monitoring] globalstep took 265318 us in mod *builtin*

about every 30 min ... reported/question here ... I remember a peak at pandorabox.io about every 30 min, at my server nothing ...

... but I know now what is this peak: [mod] free_lj_mem ... was set to every 30 min, I set to 60 min, but know

AND
* also when a gamer try to join (at me blocked with [mod] border) or get blocked by [mod] no-guests and sometime also successfully join (but not everyone)
= confirm lag peak (as I have seen OFTEN at pandorabox.io but admin seamed not to believe me)

so the black peter goes to ... devs, builtin part ...

[Festus1965]

gamer "Baobab" bannend for 1 year,
as breaking in protected house, and took honeypot (I placed there 1 of 2 in map) = jailed

some todays profiler result in german thread ...

[Festus1965]

Sorry,
I updated to wrong time and as postgresql was involved, mts crashed.

Server is (was after 25 sec) back!

Backup also done: 15.131.407.274 Byte before (15.121.866.303 Byte = plus 9.540.971 Byte or 0.06% more). Doesn't seam to be too much explorers.

[Festus1965]

Server Hardware restarted after even some system OS update

changes in mts:
* new mod news (use with /news)
* updated mod mapserver (still not in use)
* deactivated mod: free_lj_mem (lag peaks), and profiler
* .conf new: password for server
* .mt change mapserver onto postgresql

Server back online
lag one gamer 0.14 !

added: 22.11.
* as preparing for weekend with more kids tested with 22 gamer in = avg lag just under 500ms (during Pandorabox.io is with 12 gamer arround 600 ms and more)
* had seen just 0.12 with just admin in also
* all gamer now activate no any nature, all up at spaceship, so just technic, mesecons (but no blinky) , and pipework (but all no work to do)
* so I have a test (2nd server) in mind without technic, pipeworks, mesecons, ... for just the kids = less lag on oine server, lass lag for kids as some things dont use

[Festus1965]

Backup also done: 15.210.153.228 Byte before (15.131.407.274 Byte = plus 88.745.954 Byte or 0.58% more).

Server kept online

[Festus1965]

I guess (proofed on this server) I found a way to reduce peaks.
As of sit long time in front watching monitoring, I saw two spots on this server:
* at or to /spawn - (but also the build-in mod make cpu work) , and
* my /home (sure other places often visited and special surrounding as technic or wood)

After I placed at /spawn a WorldAnchor to keep that area (take same radius as gamer) in memory, then the peaks was gone. The 2nd at my huge base also reduced or removed that peak sometime came up.
Even I have huge cache, and keep memory mts data for 24hours, some places seam to need this help.

[Festus1965]

just energy down, just too long for the batts

Server just restarting, and online

[Festus1965]

Backup also done: 15.213.008.615 Byte

Server kept online

[Festus1965]

* had a crash last nicht, after Logout fom too much Gamer same time ?
* even changed setings blocks_send to standard (mean about reduce 50% of value before)
* wanna take off 3 world_anchors (but they reduce lag when people join, like at spawn, or often visited places)
* wanna also test no forceload of switching stations as at Pandorabox

but that will take some time, each test about 24-28 hours, as also told the kids to stay online as possible, even it is very cold here, and warm from pc is welcome

added:
sure interesting as now after
* server updated from 5.0.1 to 5.2.0-dev
* and the 3 worldanchors I know deleted
: still 111 forceloaded blocks .... and lag I have to see after some hours with the 28 loggins again

next: I guess tomorrow restart with mod switching stations part forceload = off

[Festus1965]

Test continue
* server crashed with my easy fast fix try to take out forceload at switching stations, redu (so no change here)
* server show now only 3 forceload, after just delete forceload.txt file in world folder
= with yet 26 gamer in, lag is only 380 instead of 530 before with 28 gamer...

after one hour:
* 28 gamer, 410 ms lag = about 25% reduce lag, just delete forceload.txt one time to clean up.
* remind: here strong 4 Core / 8 thread acpu and real 32 GB ram, with m.s SSD - how much faster will slow server get ?

after five hours:
* now 4 instead of 4 mapblocks forceloaded - one gamer was in, maybe him ... hmmm
* lag still around/under 400 ms = speed up is permament in the same conditions

test:
* after deleting (dig) a switching-station and the cable below, the forceloaded blocks was NOT getting one less.
? failure ?

[techniX]

* after deleting (dig) a switching-station and the cable below, the forceloaded blocks was NOT getting one less.

We faced the same issue on our server. In our case it was even worse, because we have jumpdrive ships, so all the places where our ships have been were forceloaded.

So we did the following:
1. Modified switching station code: GitHub link to the corresponding commit
2. Stopped the server, removed force_loaded.txt from the world folder, and started server again.

As a result, number of forceloaded mapblocks dropped from 50 (which was our hard limit) to 23.

Hope this helps :)

UPD: the issue with the switching station is that it enables forceload in ABM function *each time* when it runs, and Minetest internal implementation of forceload has a counter, so this is what actually happens:
- player places switching station. ABM is registered
- ABM runs with specified interval, increasing forceload counter with each run via minetest.forceload_block
- Time passes. Let's say our forceload counter is 999 now.
- player removes switching station.
- minetest.free_forceload_block is called, decreasing the counter by 1.
- because the counter is 998 now, i.e. nonzero, Minetest still thinks this mapblock is forceloaded.
My "fix" makes switching station call forceload_block only once, when ABM runs for the first time for this block.

[Festus1965]

Thanks for the response,
I am not sure if I understood your explanation right, as I like the concept of still running switching station
* MOST when gamer is on,
* but better always than
* off as long I am not there.

My intention after that 8 hours today was like, as soon an existing switching station was "existentially" activated again, as a gamer activate that MapBlock, from then this switching station is on all time again, until I delete forceloaded.txt again.

That one switching station plus later might have been that of my daughter as she came in. So I try to make sure this was the one more, as - and that is the part I am confused - my count of force-loaded mapblocks was for weeks stable with 111 plus the 2 world-anchor I had to speed up logins, or teleportation to very often used places (/spawn and a city place = worked very well)
MEAN: that is the part I dont understand the part of counter ...

The Idea to delete the forceloaded.txt was so far right from my logical review about what I did, and what happened,
with the problem to step into the next issue here ... boring.

But, so far, with just deleting the forceloaded.txt some time I win already so much reduce of lag, that I am not in hurry yet.

But I keep the idea in head, and will see what time and more knowledge might do with it.

Best version might be, switching stations would work only, when gamer in online, where-ever he/she is ... but that is another part, as to make sure that they get deleted when rebuild/dug down.

So Servers admins with technic, here is a way (quite fast) to speed up our servers, and even only that switching stations might get back forceloaded that have been visited, mean that area is used by owner or neighbour - but very much "dead" builds are silent (= less lag) now!

This server now:
* 0 gamer / empty about 0.06
* 4 gamer, 3 kids running around, build 0.13
* 28 gamer, silent most same place up without biome, overlapping activated mapblocks 0.40 lag
lower I have never been with mesecons, technic even pipeworks and areas... even monitoring