Recent forum and wiki downtime 2019-05-01...2019-05-03

User avatar
celeron55
Administrator
Posts: 462
Joined: Tue Apr 19, 2011 10:10
GitHub: celeron55
IRC: celeron55

Recent forum and wiki downtime 2019-05-01...2019-05-03

by celeron55 » Post

The forum and wikis were down from 2019-05-01 to 2019-05-03 due to a VPS crash and disk corruption.

During downtime I restored the forum and the wikis to my latest off-site backup dated 2019-03-13 and upgraded the base system to OpenBSD 6.3. Much of the data left on the server was unusable, like the forum database.

Unfortunately that means the latest 6 weeks of data was lost. I'll be looking into implementing more frequent off-site backups to other servers ran by trusted community members.

ContentDB has recorded the missing mod and game titles and the list can be found here: https://gist.github.com/rubenwardy/1cd6 ... 1ffdcbdc4d

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it. Feel free to re-post any of your lost content.

User avatar
Joseph16
Member
Posts: 308
Joined: Tue Dec 06, 2016 05:35
In-game: Joseph16

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Joseph16 » Post

Glad to see it's been restored.
Testin' mines since 1989

User avatar
Wuzzy
Member
Posts: 3962
Joined: Mon Sep 24, 2012 15:01
GitHub: Wuzzy2
IRC: Wuzzy
In-game: Wuzzy

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Wuzzy » Post

Ouch. That hurts. But it could have been worse. Thankfully, there was a backup. Phew. :-)

Obviously, some of my forum threads for my countless mods became outdated as well. So I had to manually restore them to point to the latest mod version.
@modders: If you released a new version in the past 6 months or so, this means you must go through your mod and game forum threads and re-update them to the latest version.
My creations. I gladly accept bitcoins: 17fsUywHxeMHKG41UFfu34F1rAxZcrVoqH

User avatar
Festus1965
Member
Posts: 1264
Joined: Sun Jan 03, 2016 11:58
In-game: Thomas Explorer Thailand
Location: Thailand - Chiang Mai (+5-6h to MEZ)
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Festus1965 » Post

I had much more time without the Forum to play ... I am not so sad.

add:
Maybe a good warning about what is sure not needed, off topic, video, music and other threads making a lot of work - bad for the forum itself.
The less the better, easier, less mod work ... also look forward to new rules of ... EU
Last edited by Festus1965 on Fri May 03, 2019 22:44, edited 1 time in total.

User avatar
b3u
Member
Posts: 12
Joined: Mon Jun 25, 2018 01:41
GitHub: b3u
IRC: Fairion
Location: Maryland, USA
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by b3u » Post

Thanks to all who helped get the forums back online.

User avatar
RajMahal
Member
Posts: 10
Joined: Wed Jan 16, 2019 20:56

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by RajMahal » Post

Image
Introducing the #t.backup.#t.borem.#t. recovery system
Thank You for the forum recovery. I suggest FEWER backups, NOT more.
inprocess

ingredients:
#t.backup.&.borem. recovery.system
#t.flog



sources:
#t.forum.minetest
#t.forum.member...celeron55
#t.forum.member...sorcerykid
_ _ _....more to be added
+ Spoiler
Attachments
25.1.png
ScreanShotOfSpreadsheet_revg
(167.02 KiB) Not downloaded yet
Last edited by RajMahal on Fri May 01, 2020 17:32, edited 32 times in total.

User avatar
DS-minetest
Member
Posts: 1137
Joined: Thu Jun 19, 2014 19:49
GitHub: Desour
IRC: DS-minetest
In-game: DS
Location: I'm scared that if this is too exact, I will be unable to use my keyboard.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by DS-minetest » Post

celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.
How exactly does this work?
I am not really experienced in using google.
Note that I've recently renamed myself on github and co. to "Desour". (I'm bad at naming things.)
Feel free to call me DS.

User avatar
Linuxdirk
Member
Posts: 2538
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Linuxdirk » Post

DS-minetest wrote:
celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.
How exactly does this work?
I am not really experienced in using google.
Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.

User avatar
sorcerykid
Member
Posts: 1307
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by sorcerykid » Post

Thanks for all your efforts celeron55. I'm relieved that you were able to recover most of the forum posts.

However, I can't seem to get the Google cache to work. I've tried several forum URLs, and they all result in this message:

Image

Yet in the search results, it shows these pages are indexed and that cached versions are available.

User avatar
Codesound
Member
Posts: 304
Joined: Thu Jun 09, 2016 14:56

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Codesound » Post

Thanks @Celeron55,

for ALL, ALL, ALL, ALL your works!!!

R

DELTA_FORCE
Member
Posts: 132
Joined: Tue Oct 30, 2018 01:26
IRC: DELTA_FORCE
In-game: DELTA_FORCE
Location: Somewhere in the Atlantic

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by DELTA_FORCE » Post

Great to see that things can be fixed fast here. Good job :)
MT5.3.0 for the win!

User avatar
DS-minetest
Member
Posts: 1137
Joined: Thu Jun 19, 2014 19:49
GitHub: Desour
IRC: DS-minetest
In-game: DS
Location: I'm scared that if this is too exact, I will be unable to use my keyboard.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by DS-minetest » Post

Linuxdirk wrote:
DS-minetest wrote:
celeron55 wrote:I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.
How exactly does this work?
I am not really experienced in using google.
Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.
I see, I have to use that green arrow.

However, I get the same results as sorcerykid.
the internet never forgets
Hehe, I guess that rule was broken.
Note that I've recently renamed myself on github and co. to "Desour". (I'm bad at naming things.)
Feel free to call me DS.

User avatar
VanessaE
Moderator
Posts: 4517
Joined: Sun Apr 01, 2012 12:38
GitHub: VanessaE
IRC: VanessaE
In-game: VanessaE
Location: Western NC
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by VanessaE » Post

Yeah, neither Wayback Machine nor Google Cache have recent enough snapshots of the Forum to be useful. Bummer, that. :-/
You might like some of my stuff: Plantlife ~ More Trees ~ Home Decor ~ Pipeworks ~ HDX Textures (64-512px)

WayBack Machine
New member
Posts: 6
Joined: Fri May 03, 2019 19:42

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by WayBack Machine » Post

Attachments
WaybackButtons.png
(10.29 KiB) Not downloaded yet

Skamiz Kazzarch
Member
Posts: 138
Joined: Fri Mar 09, 2018 20:34
GitHub: Skamiz
In-game: Skamiz
Location: la lojbaugag.

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Skamiz Kazzarch » Post

I just realized probably why I couldn't log in. I asume the passwords were also returned?
Thank you for your hard work in making the forum run again.

User avatar
sorcerykid
Member
Posts: 1307
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by sorcerykid » Post

To aid in recovery efforts, I generated snapshots of the forum index using Bing's cache (see, Microsoft isn't so bad after all ;)

Featured General User-contributed Content Using a custom awk script, I redirected all topic URLs back to Bing search. I can't promise that Bing has cached copies of all such pages, but at least it should be easier to navigate than manually copying and pasting every topic URL into the search engine.

PS. If that doesn't work out and you are still desperate, then give Yandex.com a try. It seems to have archived a good number of recent discussions that haven't even been crawled by Google or Bing.

User avatar
voxelproof
Member
Posts: 984
Joined: Sat Aug 05, 2017 08:13
Location: Europe

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by voxelproof » Post

It's very frustrating, but luckily at least the communication platform has been restored. Thanks for your work, guys, and for bringing the situation back under control :)
Last edited by voxelproof on Sat May 04, 2019 17:11, edited 1 time in total.
Game is the essence of life; therefore every good game makes life better.

homthack
Member
Posts: 34
Joined: Sun Apr 15, 2018 19:11
GitHub: homt8
IRC: homthack
In-game: homthack
Location: Caracas
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by homthack » Post

I forgot my thing, I lost my thing that I did last April I published a statement on my server that is scheduled the next day April 26 that I did for tomorrow and then I cancel it before Saturday I'm not there but I'm I went there and saw the news that happened yesterday and then I came to that program and I went down last Sunday, I did not know what else it was that made a backup copies and for that I am happy that minetest forum has been alive step on the line that put it in a backup since March.
[Server-Online] HM-Server [Discord] homthack#9917
[Website-Account]
Discord HM-Server MineCity MinetestHosting Trello

User avatar
Fixer
Member
Posts: 895
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Fixer » Post

@celeron55
What's your avg backup size?

User avatar
uwu
Member
Posts: 42
Joined: Sun Jan 21, 2018 15:01

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by uwu » Post

^ I'd like to know this as well.
If it's not too outrageous, I'm willing to provide some storage space with sftp access on one of my VPS.

After this crash, we as a community should work towards increasing our Bus factor, so the Minetest project won't fail if one person suddenly disappears or another catastrophic event like this happens again. I've seen too many projects in complete disarray just because one guy is missing, e.g. Void Linux.

User avatar
Kilarin
Member
Posts: 778
Joined: Mon Mar 10, 2014 00:36

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Kilarin » Post

You don't realize how much you really needed something until it's gone.
I missed the minetest forums!

I want to add my voice to the other saying thank you for all the hard work getting the forums back up and keeping them running.

User avatar
Linuxdirk
Member
Posts: 2538
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by Linuxdirk » Post

Is it me or is the server noticeably slower?

User avatar
celeron55
Administrator
Posts: 462
Joined: Tue Apr 19, 2011 10:10
GitHub: celeron55
IRC: celeron55

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by celeron55 » Post

@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by TumeniNodes » Post

celeron55 wrote:@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.
Yeh, it's that 2d bit which scares me more than any EU issues. At least the EU can be reasoned with.
I do not envy your position of needing to make a choice eventually.
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
sorcerykid
Member
Posts: 1307
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Re: Recent forum and wiki downtime 2019-05-01...2019-05-03

by sorcerykid » Post

I'm not very comfortable with random people being handed the keys to the castle (no offense Fixer, it's nothing personal :)

It is my understanding that the forum database contains private messages, hidden subforums, hashed passwords, and IP addresses. Even if fully encrypted, that's a lot of sensitive, personal information that is just begging for a privacy breach to happen. We've all seen the news of major companies and even government agencies in the line-of-fire for not properly protecting confidential data ;) In my view, nobody else except the core developers and moderators should be trusted with access to the forum database.

Why not look into a cloud hosting solution? I've been using ADrive for automated weekly backups of my server. They support all the industry standard protocols including rsync, sftp and scp. And of course under Linux, you can remote mount the filesystem via SSHFS. The 100 GB storage plan is just 25 USD for an entire year. This way you could maintain a centralized, secured backup that is always online. And only appointed staff would be give the login credentials.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests