What's the deal with the forums lately?

[sorcerykid]

Does anyone know what's going on? I've been getting errors for the past few weeks, and pages take extraordinarily long to load. At first I thought it might've just been a glitch, but then I checked IRC and several others have mentioned the forums being inaccessible or slow.

[v-rob]

I've also had a situation where the CSS wasn't fully loaded, leading to a very basic view.

I think it might have something to do with this:

In total there are 617 users online :: 8 registered, 0 hidden and 609 guests (based on users active over the past 5 minutes)

My guess is that a majority of those are bots since this number barely changes.

[ShadMOrdre]

With most of the world at home, I'd say it's more likely do to heavy internet traffic, worldwide.

Preferential treatment to the big tech websites. And perhaps a not as good as it could / should be server hosting up the site....

[Linuxdirk]

I've also had a situation where the CSS wasn't fully loaded, leading to a very basic view.

That’s just a side effect of the super slow response times. It just loads and loads and loads and at one time the browser runs into timeout and skips loading.

During the last few weeks it feels like the forum was moved to a Raspberry Pi for home hosting with low bandwidth.

[sorcerykid]

During the last few weeks it feels like the forum was moved to a Raspberry Pi for home hosting with low bandwidth.

Interesting, as I made almost that exact same comment in chat the other day:)

Code: Select all

2020-04-01 12:04:14: ACTION[Server]: CHAT: <Nemo> i think celeron55 needs a better webhost, or else he's running on a raspberry pi

[Linuxdirk]

Interesting, as I made almost that exact same comment in chat the other day:)

Code: Select all

2020-04-01 12:04:14: ACTION[Server]: CHAT: <Nemo> i think celeron55 needs a better webhost, or else he's running on a raspberry pi

Ha! :) Nice coincidence. Seems like you also have experience with running a Pi for hosting publicly available dynamic web sites :D

[Hume2]

I have the same problem too.

[BlueTangs Rock]

This situation was happening to me as well for the last month or two, with the most recent instance happening a few days ago. I thought it was just an internet connection issue on my side or something, lol.

[Oil_boi]

It's extremely annoying :(

[rubenwardy]

There are hardware issues, possibly related to the record breaking number of views recently. celeron55 is on it

[pampogokiraly]

There are hardware issues, possibly related to the record breaking number of views recently. celeron55 is on it

Sounds good because this is getting really annoying.

[sorcerykid]

It looks like it may have been fixed. Or else it's just super good luck, because the page loads are blazingly fast now.

Edit: It was working fine for several hours, but maybe that was an anomaly. Who knows.

[Linuxdirk]

It looks like it may have been fixed. Or else it's just super good luck, because the page loads are blazingly fast now.

I load the forums in the background and come back after a few minutes to perform any action (or reload because internal server errors) and then do something else for a few minutes to wait for the page to load (or run into another internal server error)

This is far from being fixed.

[Captcha]

Does anyone know what's going on? I've been getting errors for the past few weeks, and pages take extraordinarily long to load. At first, I thought it might've just been a glitch, but then I checked IRC and several others have mentioned the forums being inaccessible or slow.

The site is really slow for me, I've spent more than 4 minutes waiting for the index to load.

He should change his web host.... this is real slow and I got several times the bad gateway error...

[Hume2]

This might help :D

[BlueTangs Rock]

This might help :D

Is the horizontal axis the number of days and the vertical axis the number of forum crashes?

[Hume2]

This might help :D

Is the horizontal axis the number of days and the vertical axis the number of forum crashes?

It's the number of tries to access the forum.

[Linuxdirk]

This might help :D

The chart looks funny but is actually pretty useless because HTTP response codes are not “measurable“ and the x axis has no legend. A timeline would be much more useful.

Assign a color to the HTTP response codes and horizontally or vertically stack by date/time.

Running this command every X minutes gives a nicely parsable CSV (time,duration,response) as result.

Code: Select all

LC_ALL=C curl -s -o /dev/null -I -w "$(date --iso-8601=minutes),%{time_total},%{http_code}\n" https://forum.minetest.net >> mt-status

Could be done in bash by simply putting this in an infinite loop followed by a sleep of the desired amount of seconds.

Currently the forum works so I can’t test but if the server properly returns the HTTP response code as header on errors then this should be logged with this command.

[runs]

Yes, the forum now is a damn hell :-D

[Linuxdirk]

I took my idea in the field and actually measured roughly 12 hours of server availability using the method I described.



Orientation

Duration: From 12:11 pm German time (UTC+2) to 11:30 pm
Date: April 9, 2020
Direction is from 12:11 pm to 11:30 pm from left to right
Resolution: 1 minute between calls (end of previous call + 60 seconds)
Top: HTTP status code
Bottom: Response time from curl call to response from server

HTTP status code

Gray: no data due to curl reason
Green: 200 (OK, response was fine)
Red: Any kind of error returned from the server

Response time

Green: below 1.5 seconds
Dark green: 1.5-3 seconds
Yellow: 3-6 seconds
Dark red: 6-10 seconds
Red: more than 10 seconds

[Hume2]

I didn't mean it that seriously but OK. Your data is way more valuable.

[BlueTangs Rock]

A couple days ago, even the MineTest block and item Wiki (I check those web pages daily when I'm bored) was having loading speed and temporary crashing issues, and those websites seemed to not have the crashing and connection issues that the forums had or at least they were much less effected by the issues for quite a while.

[paramat]

Celeron55 needs more donations to cover his MT expenses, so you can donate to improve forum speed.
As is often the case the answer is: It is a free game.

[rubenwardy]

celeron55 is currently migrating the forums to a new machine. This needs to be done with scratch, and is being done with more up to date methods that will make it easier in the future

[Linuxdirk]

Since I’m a data and statistics nerd here’s the final result. How this can be used to track down issues or help raising donations for a better server? I have no idea. I am just the messenger here :)

Meta

Monitoring duration was from April 9, 12:11pm (2020-04-09T12:11+02:00) to April 17, 12:03am (2020-04-17T00:03+02:00)

Same test setup as before, regarding orientation, HTTP status code colors and response time colors. Tests were performed from location Germany using a 100 Mbit/s VDSL2 connection (ITU G.993.5) with SNR of 10 dB (Rx and Tx) with a median connection speed to DSLAM of 110 Mbit/s Rx and 45 Mbit/s Tx.

Machine used for testing was a “mini PC” running on Arch Linux with Kernel 5.5.13 having an Intel Celeron J3455 CPU running at 1.5 GHz. Test was performed with the following command running in a tmux session.

Code: Select all

while true; do
    LC_ALL=C curl -s -o /dev/null -I \
        -w "$(date --iso-8601=minutes),%{time_total},%{http_code}\n" \
        https://forum.minetest.net \
        >> mt-status
    sleep 60
done

See resulting data source file in zip attachment.



(You might want to open the timeline in a new tab and zoom to 100% and scroll sideways)

Orientation

Previous orientation applies with following addition: each one pixel wide line represents one call and the lines horizontal pixel position directly relates to the Nth line in the data source file.

HTTP response code counter

000 (curl error): 228
200 (OK): 7947
500 (Internal Server Error): 72
502 (Bad Gateway): 46
503 (Service Unavailable): 36
504 (Gateway Timeout): 475

Response time counter

< 1.5 seconds: 3738
>= 1.5 seconds: 473
>= 3 seconds: 542
>= 6 seconds: 544
>= 10 seconds: 3507

Availability

8804 calls in total
228 curl errors resulting in no response code
8576 successful requests (got a response code)
629 various server errors

~92.7% availability based on HTTP response codes

8804 calls in total
3507 requests above 10 seconds

~60.2% availability based on bad response times

8804 calls in total
481 requests took over 60 seconds

~94.5% availability based on requests above 60 seconds