Page 1 of 2

Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 11:09
by celeron55
The forum and wikis were down from 2019-05-01 to 2019-05-03 due to a VPS crash and disk corruption.

During downtime I restored the forum and the wikis to my latest off-site backup dated 2019-03-13 and upgraded the base system to OpenBSD 6.3. Much of the data left on the server was unusable, like the forum database.

Unfortunately that means the latest 6 weeks of data was lost. I'll be looking into implementing more frequent off-site backups to other servers ran by trusted community members.

ContentDB has recorded the missing mod and game titles and the list can be found here: https://gist.github.com/rubenwardy/1cd6 ... 1ffdcbdc4d

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it. Feel free to re-post any of your lost content.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 12:44
by Joseph16
Glad to see it's been restored.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 12:47
by Wuzzy
Ouch. That hurts. But it could have been worse. Thankfully, there was a backup. Phew. :-)

Obviously, some of my forum threads for my countless mods became outdated as well. So I had to manually restore them to point to the latest mod version.
@modders: If you released a new version in the past 6 months or so, this means you must go through your mod and game forum threads and re-update them to the latest version.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 13:23
by Festus1965
I had much more time without the Forum to play ... I am not so sad.

add:
Maybe a good warning about what is sure not needed, off topic, video, music and other threads making a lot of work - bad for the forum itself.
The less the better, easier, less mod work ... also look forward to new rules of ... EU

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 14:32
by b3u
Thanks to all who helped get the forums back online.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 14:47
by RajMahal
___ Thank You for the forum recovery. I suggest FEWER backups, NOT more.

*in-process*
+ Spoiler

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 14:48
by DS-minetest
celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 15:06
by Linuxdirk
DS-minetest wrote:
celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 15:10
by sorcerykid
Thanks for all your efforts celeron55. I'm relieved that you were able to recover most of the forum posts.

However, I can't seem to get the Google cache to work. I've tried several forum URLs, and they all result in this message:

Image

Yet in the search results, it shows these pages are indexed and that cached versions are available.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 15:48
by Codesound
Thanks @Celeron55,

for ALL, ALL, ALL, ALL your works!!!

R

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 17:30
by DELTA_FORCE
Great to see that things can be fixed fast here. Good job :)

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 17:37
by DS-minetest
Linuxdirk wrote:
DS-minetest wrote:
celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.

I see, I have to use that green arrow.

However, I get the same results as sorcerykid.

the internet never forgets

Hehe, I guess that rule was broken.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 18:09
by VanessaE
Yeah, neither Wayback Machine nor Google Cache have recent enough snapshots of the Forum to be useful. Bummer, that. :-/

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 19:59
by WayBack Machine

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 21:16
by Skamiz Kazzarch
I just realized probably why I couldn't log in. I asume the passwords were also returned?
Thank you for your hard work in making the forum run again.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Fri May 03, 2019 22:42
by sorcerykid
To aid in recovery efforts, I generated snapshots of the forum index using Bing's cache (see, Microsoft isn't so bad after all ;)

Featured
General
User-contributed Content

Using a custom awk script, I redirected all topic URLs back to Bing search. I can't promise that Bing has cached copies of all such pages, but at least it should be easier to navigate than manually copying and pasting every topic URL into the search engine.

PS. If that doesn't work out and you are still desperate, then give Yandex.com a try. It seems to have archived a good number of recent discussions that haven't even been crawled by Google or Bing.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Sat May 04, 2019 10:54
by voxelproof
It's very frustrating, but luckily at least the communication platform has been restored. Thanks for your work, guys, and for bringing the situation back under control :)

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Sat May 04, 2019 14:43
by homthack
I forgot my thing, I lost my thing that I did last April I published a statement on my server that is scheduled the next day April 26 that I did for tomorrow and then I cancel it before Saturday I'm not there but I'm I went there and saw the news that happened yesterday and then I came to that program and I went down last Sunday, I did not know what else it was that made a backup copies and for that I am happy that minetest forum has been alive step on the line that put it in a backup since March.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Sat May 04, 2019 19:14
by Fixer
@celeron55
What's your avg backup size?

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Sun May 05, 2019 11:13
by uwu
^ I'd like to know this as well.
If it's not too outrageous, I'm willing to provide some storage space with sftp access on one of my VPS.

After this crash, we as a community should work towards increasing our Bus factor, so the Minetest project won't fail if one person suddenly disappears or another catastrophic event like this happens again. I've seen too many projects in complete disarray just because one guy is missing, e.g. Void Linux.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Mon May 06, 2019 00:56
by Kilarin
You don't realize how much you really needed something until it's gone.
I missed the minetest forums!

I want to add my voice to the other saying thank you for all the hard work getting the forums back up and keeping them running.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Mon May 06, 2019 06:33
by Linuxdirk
Is it me or is the server noticeably slower?

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Mon May 06, 2019 16:30
by celeron55
@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Mon May 06, 2019 16:48
by TumeniNodes
celeron55 wrote:@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.


Yeh, it's that 2d bit which scares me more than any EU issues. At least the EU can be reasoned with.
I do not envy your position of needing to make a choice eventually.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

PostPosted: Mon May 06, 2019 17:44
by sorcerykid
I'm not very comfortable with random people being handed the keys to the castle (no offense Fixer, it's nothing personal :)

It is my understanding that the forum database contains private messages, hidden subforums, hashed passwords, and IP addresses. Even if fully encrypted, that's a lot of sensitive, personal information that is just begging for a privacy breach to happen. We've all seen the news of major companies and even government agencies in the line-of-fire for not properly protecting confidential data ;) In my view, nobody else except the core developers and moderators should be trusted with access to the forum database.

Why not look into a cloud hosting solution? I've been using ADrive for automated weekly backups of my server. They support all the industry standard protocols including rsync, sftp and scp. And of course under Linux, you can remote mount the filesystem via SSHFS. The 100 GB storage plan is just 25 USD for an entire year. This way you could maintain a centralized, secured backup that is always online. And only appointed staff would be give the login credentials.