minetest freeze - "the silence" chapter 9000

[Walker]

--- [ english by google-translate ] ---

I had the following problem two times:

- suddenly all players fly out due to timeout
- Minetest responds to nothing more
- There are no information in the log
- In the console is no error (the process freeze almost easy)
- No hard disk utilization
- No CPU utilization
- no load load (who knows about Unix, knows what I mean)

The problems started when I upgraded my servers from Minetest 5.0.1 to 5.1.0 and installed random_messages (a mod) ...
I have disabled the mod now and hope that the symptoms go away

but now a par ask in my head:

- Is this really a bug in MT-5.1, which only I have (since nobody in the forum has mentioned it yet ...)
- or can bring a mod minetest COMPLETE to the freeze? (So that it does not crash nor ANY message brings or responds to NOTHING)

--- [ german ] ---

ich hatte gerade zwei mal folgendes Problem:

- auf einmal fliegen alle Spieler wegen Timeout raus
- Minetest reagiert auf nichtsmehr
- Im Log stehen keine infos
- In der Konsole kommt kein Fehler ( der Prozess freezt quasi einfach )
- Keine Festplatte auslastung
- Keine CPU auslastung
- kein "load" auslastung ( wer ahnung von Unix hat, weiß, was ich meine )

die probleme begannen als ich meine Server vom Minetest 5.0.1 auf 5.1.0 upgadatet habe und random_messages ( eine mod ) installiert habe ...
Ich habe die mod nun deaktiviert und hoffe das die symptome weggehen

doch nun ergeben sich ein par fragen in meinem Kopf:

- ist das wirklich ein bug in MT-5.1, den nur ich habe ( da es hier im Forum noch niemand bis jetzt erwähnt hat ... )
- oder kann eine Mod Minetest KOMPLETT zum freeze bringen ? ( also das es weder abstürtzt noch IRGENDEINE meldung bringt bzw. auch auf NICHTSMEHR reagiert )

[Walker]

--- [ english ] ---

ok, even without the mod is minetest 5.1 again freezed ...

So it is the update from MT5 to MT5.1

Is there a method to analyze such a "error" (I see no error message)?

--- [ german ] ---

ok, auch ohne der mod ist minetest 5.1 wieder gefreezt ...

es liegt also am Update von MT5 auf MT5.1

gibt es eine methode einem solchen "Fehler" ( ich sehe ja keine Fehlermeldung ) zu analysieren ?

[Walker]

--- [ english ] ---

another info: when I press directly in the terminal Ctrl-C, come the four lines:

Code: Select all

2019-10-27 11:43:25: [Main]: INFO: signal_handler(): Ctrl-C pressed, shutting down.
2019-10-27 11:43:25: ACTION[Main]: Server: Shutting down
2019-10-27 11:43:25: ACTION[Main]: [doc] Server shuts down. Player data is about to be saved.
2019-10-27 11:43:26: ACTION[Main]: [doc] Wrote player data into /media/sda3/Minetest/server4/.minetest/worlds/MineClone2/doc.mt.

but then nothing more.
neither minetest quits nor is there another message
So I have to kill Minetest always with two Ctrl-C

--- [ german ] ---

noch eine Info: wenn ich dierekt im terminal Strg-C drücke, kommen die vier Zeilen:

Code: Select all

2019-10-27 11:43:25: [Main]: INFO: signal_handler(): Ctrl-C pressed, shutting down.
2019-10-27 11:43:25: ACTION[Main]: Server: Shutting down
2019-10-27 11:43:25: ACTION[Main]: [doc] Server shuts down. Player data is about to be saved.
2019-10-27 11:43:26: ACTION[Main]: [doc] Wrote player data into /media/sda3/Minetest/server4/.minetest/worlds/MineClone2/doc.mt.

aber dann kommt nichtsmehr.
weder beendet sich Minetest noch gibt es eine weitere meldung
ich muss also Minetest immer mit zweimaligem Strg-C abwürgen

[Sokomine]

ich muss also Minetest immer mit zweimaligem Strg-C abwürgen

Das kenne ich von Fällen, in denen ich versehentlich Endlosschleifen oder dergleichen eingebaut habe. Das geht dann aber üblicherweise mit hoher CPU-Auslastung einher.

[Walker]

genau das ist ja mein Problem, es ist GARKEINE Auslastung vorhanden, so wie wenn der prozess pausiert wurde ...

sowas hab ich echt noch nie erlebt ^^

[FreeGamers]

I updated to the version in Debian backports and have been combating an issue similar to this.

I get timeouts after a couple hours that don't show any errors.

I have to manually restart the server to allow clients to reconnect.

I have a stable environment and a dev environment that both run the same versions of software and mods.

However, this only happens on my stable server which is on an older processor (but quite capable quad core processor) and has clients connecting.

I'm still troubleshooting this. I don't believe its related to mods, memory corruption, file system corruption, or networking.

The observable behavior I can see is that after about 2 hours, the game will timeout for all players. Restarting the minetest-server service takes a minute or so. Server works fine after a reboot. The log shows no unusual entries. I've tried many different config settings and different things but no answer to this yet.

I've reenabled hyper-threading on my Intel CPU and am going to observe if the additional threads alleviate this issue at all. If so it seems it might be performance related.

However, I had no such issues until I upgraded onto version 5.1.0 this weekend.

Originally my game would not even keep clients connected for more than 10 seconds. Once I deleted the world file completely it would work, but only for a few hours. I'm not sure why.

[FreeGamers]

Did you ever find a fix for this in the 2 months since you made this thread? If throwing more threads and virtual cores at this problem fixes it, its clearly a workaround not a solution.

I wonder if it is:
https://github.com/minetest/minetest/issues/9111

And patched in 5.2.0-dev via these pulls...
https://github.com/minetest/minetest/pull/9114
https://github.com/minetest/minetest/pull/9135

I'm really annoyed by this bug. I spent most of my weekend dealing with it.

[FreeGamers]

Yeah hyperthreading didn't fix this. I'm not surprised by that and I've disabled HT again. Looking into solutions for this now...

[Festus1965]

I had this case two times but saw definitely another reason:
* some more gamer log out SAME time,
as often kids at me leave together as wanna play something else, so have 3-5 kids leave together, and at this cases I had two time this freeze also, but with clear signs that network got a problem, as I had to restart the hardware as of network, or just disable network and enable again.

[FreeGamers]

I dont believe this is whats happening, but I appreciate the suggestion. I'll keep an eye open for that.

[Walker]

Did you ever find a fix for this in the 2 months since you made this thread?

no, sorry

[Festus1965]

also es passiert AUCH mit MT 5.1, dann
* kann es eine andere mod sein, die vor dem Auftreten ein update erhielt, oder
* MT 5.1.0 auf 5.2.0 und testen ... 5.2.0 läuft bei mir gut, ohne Nachteile, und eben seither auch ohne freeze.
sonst wenn zu oft erstarrt, zurück zu 5.0.1 ?

[Walker]

sonst wenn zu oft erstarrt, zurück zu 5.0.1 ?

so hab ich es bei 5.1.0 gemacht und werde es wohl auch so bei 5.2 machen ^^

ansonsten ja, probieren geht über studieren ... freu mich schon auf 5.2 :)

[FreeGamers]

Here are some entries from my log today. I checked it when I was at work and remotely restarted... You can clearly see the server just timed out and stopped generating logs.

What version of GNU+Linux do you guys run? Debian Backports just got the 5.1.0 packages this week, I updated on Friday and its been headache since...

You can see the timestamps where it timed out and it wasn't dealt with until a couple hours later. No logs generated in an almost two hour period until the restart command was given. These are real players just replaced names.

Code: Select all

2019-12-16 14:41:40: ACTION[Server]: USER1 digs default:stone at (-1607,-104,93)
2019-12-16 14:41:40: WARNING[Server]: active block modifiers took 201ms (processed 443 of 1507 active blocks)
2019-12-16 14:41:41: ACTION[Server]: One digs default:stone_with_copper at (-1434,-292,271)
2019-12-16 14:41:41: ACTION[Server]: USER2 takes christmas_holiday_pack:garland_white from chest at (-1545,12,213)
2019-12-16 14:41:41: WARNING[Server]: active block modifiers took 201ms (processed 656 of 1522 active blocks)
2019-12-16 14:41:41: ACTION[Server]: USER1 digs default:stone at (-1608,-104,93)
2019-12-16 14:41:42: WARNING[Server]: active block modifiers took 201ms (processed 595 of 1522 active blocks)
2019-12-16 14:41:43: ACTION[Server]: USER1 digs default:stone at (-1608,-105,93)
2019-12-16 14:41:43: ACTION[Server]: One digs default:stone_with_copper at (-1432,-293,271)
2019-12-16 14:41:43: WARNING[Server]: active block modifiers took 201ms (processed 541 of 1522 active blocks)
2019-12-16 14:41:44: ACTION[Server]: USER1 digs default:stone at (-1607,-105,93)
2019-12-16 14:41:44: WARNING[Server]: active block modifiers took 201ms (processed 546 of 1522 active blocks)
2019-12-16 14:41:44: ACTION[Server]: One digs default:stone_with_copper at (-1434,-293,271)
2019-12-16 14:41:44: ACTION[Server]: USER2 takes christmas_holiday_pack:garland_yellow from chest at (-1545,12,213)
2019-12-16 14:41:45: ACTION[Server]: USER1 digs default:stone at (-1607,-106,93)
2019-12-16 14:41:45: WARNING[Server]: active block modifiers took 201ms (processed 530 of 1522 active blocks)
2019-12-16 14:41:46: WARNING[Server]: active block modifiers took 201ms (processed 520 of 1522 active blocks)
2019-12-16 14:41:46: ACTION[Server]: USER1 digs default:stone at (-1608,-106,93)
2019-12-16 14:41:47: WARNING[Server]: active block modifiers took 201ms (processed 567 of 1521 active blocks)
2019-12-16 14:41:47: ACTION[Server]: One places node default:torch_ceiling at (-1433,-290,272)
2019-12-16 14:41:47: ACTION[Server]: USER1 digs default:stone at (-1608,-107,93)
2019-12-16 14:41:47: ACTION[Server]: Moto moves dye:green to chest at (666,45,666)
2019-12-16 14:41:48: WARNING[Server]: active block modifiers took 201ms (processed 600 of 1521 active blocks)
2019-12-16 14:41:48: ACTION[Server]: USER2 places node christmas_holiday_pack:garland_yellow at (-1547,14,210)
2019-12-16 14:41:49: ACTION[Server]: USER1 digs default:stone at (-1607,-107,93)
2019-12-16 14:41:49: WARNING[Server]: active block modifiers took 201ms (processed 555 of 1498 active blocks)
2019-12-16 16:32:00: [Main]: INFO: signal_handler(): got SIGTERM, shutting down.
2019-12-16 16:32:00: ACTION[Main]: Server: Shutting down
2019-12-16 16:32:00: ACTION[Main]: [mana] Server shuts down. Rescuing data into mana.mt
2019-12-16 16:32:00: ACTION[Main]: [mana] Wrote mana data into /var/games/minetest-server/.minetest/worlds/world/mana.mt.

[FreeGamers]

Here is the next session's complete log (see attached), I've removed player names and IP addresses.

The last input visible on the server occurs at time stamp.
2019-12-16 17:06:57: ACTION[Server]: USER2 digs default:dirt_with_grass at (-1607,94,45)


Could the ABM processing be the issue here? It was never an issue before nothing drastically changed with active block modifiers on my end of the server code. I use a lot of ABM's for mob spawning.

Is the solution to this to optimize ABM's? I'd rather not move to a development version because I run a publicly hosted server that is supposed to be stable... This seems like a regression in the Minetest stable release perhaps?

I may end up scheduling a cronjob to reboot the server every 2 hours or so :\

[Walker]

I may end up scheduling a cronjob to reboot the server every 2 hours or so :\

yes, but the problem (with me) was (or is) that Minecraft itself no longer reacted to SIGTERM ... and killing Minecraft with SIGKILL every two hours is really not a good solution.
besides ... a server that restarts every two hours!? ... I prefer to stay at 5.0.1 xD

[FreeGamers]

Sigterm seems to work for me but it takes longer than expected. When this issue occurs it can take up to a minute.

I wouldn't want to use sigkill on a server either.

Debian doesn't have a repo for 5.0.1 now. I could compile and just run 5.0.1 or use a snapshot from debian, but I was hoping to figure out what is causing this anomaly. I really don't want to put people through this buggy server though. Having the world reset is enough pain for players.

[Festus1965]

better you check this first:

Code: Select all

active block modifiers took 201ms (processed 600 of 1521 active blocks)

as it look very much, like 30 gamer on. And a lot of mod have ABM included ...
So give it more time, like 400ms (by recompile) or reduce what is causing so much work.

[sfan5]

I'd rather not move to a development version because I run a publicly hosted server that is supposed to be stable...

5.2.0-dev is actually pretty stable, we try to make sure git HEAD isn't terribly broken (though waiting a few days before pulling changes is a good idea) and several public servers run it.
Additionally, you might be wasting your time debugging and dealing with this issue since #9114 might have already fixed it.

Also some general advice:
To debug freezing/stalling issues run minetestserver inside gdb. Then when the server has frozen press Ctrl+C to get to the gdb prompt, then run thread apply all bt.
You will get a backtrace of all threads, allowing you to see if they are stuck at all and in which code.

[Festus1965]

5.2.0-dev is actually pretty stable, ...

agreed, no crash since upgraded ... at my server
* no freeze as of massive same time logout situation,
* no other crash yet

[FreeGamers]

5.2.0-dev is just the version in "master" on github?

https://github.com/minetest/minetest

[Walker]

Additionally, you might be wasting your time debugging and dealing with this issue since #9114 might have already fixed it.

... as i say: freu mich schon auf 5.2 :)

[FreeGamers]

So yes, it seems master branch refers to 5.2.0-dev.

How can I output to the proper folders? make just puts everything in the directory git is used in.

Code: Select all

mkdir minetest-source-code
cd minetest-source-code
git clone --depth 1 https://github.com/minetest/minetest.git
cd minetest
git clone --depth 1 https://github.com/minetest/minetest_game.git games/minetest_game
mkdir myoutput
cd myoutput
cmake ../ -DBUILD_CLIENT=FALSE -DBUILD_SERVER=TRUE
make -j$(nproc)

then once its compiled

Code: Select all

cd ..
cd bin
sudo cp minetestserver /usr/games/minetestserver

Is that it?

[Walker]

i use minetest "JUST-IN-PLACE" ... if you want it to installed it into the system-filesystem try

Code: Select all

sudo make install

[FreeGamers]

So on debian the paths are usually...

bin = /usr/games/
share = /usr/share/games/minetest/

if using client:
user = ~/.minetest/