Recent forum and wiki downtime 2019-05-01...2019-05-03

[celeron55]

The forum and wikis were down from 2019-05-01 to 2019-05-03 due to a VPS crash and disk corruption.

During downtime I restored the forum and the wikis to my latest off-site backup dated 2019-03-13 and upgraded the base system to OpenBSD 6.3. Much of the data left on the server was unusable, like the forum database.

Unfortunately that means the latest 6 weeks of data was lost. I'll be looking into implementing more frequent off-site backups to other servers ran by trusted community members.

ContentDB has recorded the missing mod and game titles and the list can be found here: https://gist.github.com/rubenwardy/1cd6 ... 1ffdcbdc4d

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it. Feel free to re-post any of your lost content.

[Joseph16]

Glad to see it's been restored.

[Wuzzy]

Ouch. That hurts. But it could have been worse. Thankfully, there was a backup. Phew. :-)

Obviously, some of my forum threads for my countless mods became outdated as well. So I had to manually restore them to point to the latest mod version.
@modders: If you released a new version in the past 6 months or so, this means you must go through your mod and game forum threads and re-update them to the latest version.

[Festus1965]

I had much more time without the Forum to play ... I am not so sad.

add:
Maybe a good warning about what is sure not needed, off topic, video, music and other threads making a lot of work - bad for the forum itself.
The less the better, easier, less mod work ... also look forward to new rules of ... EU

[b3u]

Thanks to all who helped get the forums back online.

[RajMahal]

[img]no%20image%20yet[/img]

(back em' & bore em'.recovery)

*#t.raj.inprocess*
Thank You for the forum recovery. I suggest FEWER backups, NOT more.

+ Spoiler

.................... RAJ_INDEX _____





*#t.raj.closed*
I suggest FEWER Backups, NOT More
***mt_raj_25***





*#t.raj.closed*
YES to security...NO to cloud backups.
***mt_raj_26***





*#t.raj.inprocess*
Introducing the Back em' up & Bore em' system.
***mt_raj_27***










.................... RAJ_DETAIL .....





***mt_raj_25***

Celeron55 said:

....."I'll be looking into implementing more frequent off-site backups to other servers ran by trusted community members."…

Recent forum and wiki downtime 2019-05-01...2019-05-03

I suggest FEWER Backups, NOT More

25.1.png

ScreanShotOfSpreadsheet_revg

(167.02 KiB) Not downloaded yet

___My Opinion___

Code: Select all

                                                                                    
                                                                                          
                                          _____Backups are a bitter sweet endeavor.  I do have suggestions, despite the appearance I may be taking advantage of a tragedy (but so what if I am?).                                                
                                          I design and compile my offline spreadsheet into phbb code for EVERY post.  Yes its time consuming.  But I still have records of conversations to refer to.                                                
                                          The shortcoming of “MY” system is that it fails to read chronologically.  That is, if I edit my post, it does not appear in a the order of forum conversation.  I would like to get ideas on a chronological  display of a single users single post within the context of a GROUP chat.                                                
                                          Losing 6 weeks of work in any personal sense is unnacceptable.  Multply that by 1000 people and that makes for a big loss, even if its “only” a forum.                                                
                                          I have seen complaints about “MY” system (I use “MY” to distinguish from the norm, not as ownership).  Say what you will, but if my posts are deleted, I have backups.  As it is I havent posted recently so it has minor relevance.                                                
                                          A single post by user to handle the context of group topic discussions de-clutters the forum and provides for a primitive means of user backup.  If fails chronological integration, but I hope something can remedy that.                                                
                                          I could have called it a "FLOG", the forum & blog.  But that would make this a beginners guide to advance phbb flogging!                                                
                                          Will call this system "Borum".  A forum and a BLog....and because it can bore 'em. (be boring)                                                
                                          Total system becomes the "Back_em_Up 'n' Borem" system.                                                
                                                                                          
                                                                                 

___My Conclusion___

Code: Select all

                                                                                    
                                                                                          
                                          I propose the initiative to promote user level backups of posted data.  Although large forum recoveries are obviously still necessary, the forums would not be “AS” dependant on mass backups.                                                
                                                                                          
                                          The “Back_em_up_n_Borem” system is here for any to copy/paste/utilyze in a spreadsheet format.                                                
                                                                                          
                                          I am not advocating everyone should, that would be absurd.  I am very appreciative of the work many put into the forums, it has been an indispensable resource.                                                
                                                                                 

***mt_raj_26***

SorceryKid wrote:

"It is my understanding that the forum database contains private messages, hidden subforums, hashed passwords, and IP addresses. Even if fully encrypted, that's a lot of sensitive, personal information that is just begging for a privacy breach to happen. We've all seen the news of major companies and even government agencies in the line-of-fire for not properly protecting confidential data ;) In my view, nobody else except the core developers and moderators should be trusted with access to the forum database."

Recent forum and wiki downtime 2019-05-01...2019-05-03

Am I speaking a foreign language?
It's as if solutions to this problem consist ONLY of more frequent backups. It seems clear FEWER forum backups are desirable!!!

___My Opinion___

Code: Select all

                                                                                    
                                                                                          
                                          Exactly, I couldn’t agree more.  Security MUST come first and foremost before convenience and lost time.                                                
                                          And so the solution that is suggested includes “cloud” backups?  You have to be  X!#$%**@*  kidding me.                                                
                                          Losing 6 weeks of work in any personal sense is unnacceptable.  Potentially exposing user data for exploitation….do I need to ellaborate?                                                
                                          If individuals care about the work they post, they need to back it up THEMSELVES.  That way moderators and contributors can focus on other projects, not forum backups.                                                
                                          A single post by user to handle the context of group topic discussions de-clutters the forum and provides for a primitive means of user backup.  If fails chronological integration, but I hope something can remedy that.                                                
                                          It appears I am attempting to create a forum in which each USer has ONE post per topic.  ANd then within this ONE post, conversation is Biased to the USer.  IT does not work well considering content is hidden within Spoilers.  But Spoilers allow excessive amount of information to be minimized and avoid clutter.                                                
                                          Some smart people have identified the problems with backups and security.  But they fail to identify a valid solution and instead suggest MORE backups.                                                 
                                                                                          
                                                                                 

***mt_raj_27***
Introducing the Back em' up & Bore em' system.
Or for those seeking violent alternatives, FLOG, for forum/blog.
___My Opinion___

Code: Select all

                                                                                    
                                          Its a primitive yet effective approach to create bbcode templates.                                                
                                          Simply copy N paste your custom template into Minetest forum for an organized Single post alternative.                                                
                                          I am going to attempt to QUOTE my post in a different post.  If it works as intended, that means anyone is welcome to copy and paste my post into a .xls spreadsheet.                                                
                                                                                          
                                                                                 

[DS-minetest]

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

[Linuxdirk]

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.

[sorcerykid]

Thanks for all your efforts celeron55. I'm relieved that you were able to recover most of the forum posts.

However, I can't seem to get the Google cache to work. I've tried several forum URLs, and they all result in this message:



Yet in the search results, it shows these pages are indexed and that cached versions are available.

[Codesound]

Thanks @Celeron55,

for ALL, ALL, ALL, ALL your works!!!

R

[DELTA_FORCE]

Great to see that things can be fixed fast here. Good job :)

[DS-minetest]

I recommend searching your lost forum posts by their title on Google and using Google's cache to recover the content, and to do this in time before they delete it.

How exactly does this work?
I am not really experienced in using google.

Just search for your mod’s forum post URL in your browsing history and copy the URL into Googles search field. You’re either lucky and it finds the post (and then click "from cache") or it does not find your post.

I see, I have to use that green arrow.

However, I get the same results as sorcerykid.

the internet never forgets

Hehe, I guess that rule was broken.

[VanessaE]

Yeah, neither Wayback Machine nor Google Cache have recent enough snapshots of the Forum to be useful. Bummer, that. :-/

[WayBack Machine]

Wayback Machine

FWIW, some latest indexes (and a few individual threads therein).
Don't forget the Date-Forward/Backward arrow buttons :


Maybe at least a little bit of an aide-memoire for some contributors.

News
https://web.archive.org/web/20190321143 ... m.php?f=18

General Discussion
https://web.archive.org/web/20190423102 ... um.php?f=3

Feature Discussion
https://web.archive.org/web/20190423102 ... um.php?f=5

Problems
https://web.archive.org/web/20190423103 ... um.php?f=6

Servers
https://web.archive.org/web/20190423103 ... m.php?f=10

Mod Releases
https://web.archive.org/web/20190401101 ... m.php?f=11

WIP Mods
https://web.archive.org/web/20190331095 ... um.php?f=9

Modding Discussion
https://web.archive.org/web/20190418131 ... m.php?f=47

Minetest-related Projects
https://web.archive.org/web/20190423103 ... m.php?f=14

Game Releases
https://web.archive.org/web/20190423105 ... m.php?f=15

Game Discussion
https://web.archive.org/web/20190423105 ... m.php?f=49

WIP Games
https://web.archive.org/web/20190319171 ... m.php?f=50

[Skamiz Kazzarch]

I just realized probably why I couldn't log in. I asume the passwords were also returned?
Thank you for your hard work in making the forum run again.

[sorcerykid]

To aid in recovery efforts, I generated snapshots of the forum index using Bing's cache (see, Microsoft isn't so bad after all ;)

Featured

• News: Page 1
  
• Builds: Page 1

General

• General Discussion: Page 1, Page 2
  
• Feature Discussion: Page 1
  
• Problems: Page 1, Page 2
  
• Servers: Page 1, Page 2

User-contributed Content

• Mod Releases: Page 1, Page 2, Page 3
  
• WIP Mods: Page 1, Page 2, Page 3
  
• Modding Discussion: Page 1, Page 2
  
• Client-Side Modding: Page 1
  
• Maps: Page 1
  
• Texture Packs: Page 1
  
• Minetest Related Projects: Page 1
  
• Game Releases: Page 1
  
• Game Discussion: Page 1
  
• WIP Games: Page 1

Using a custom awk script, I redirected all topic URLs back to Bing search. I can't promise that Bing has cached copies of all such pages, but at least it should be easier to navigate than manually copying and pasting every topic URL into the search engine.

PS. If that doesn't work out and you are still desperate, then give Yandex.com a try. It seems to have archived a good number of recent discussions that haven't even been crawled by Google or Bing.

[voxelproof]

It's very frustrating, but luckily at least the communication platform has been restored. Thanks for your work, guys, and for bringing the situation back under control :)

[homthack]

I forgot my thing, I lost my thing that I did last April I published a statement on my server that is scheduled the next day April 26 that I did for tomorrow and then I cancel it before Saturday I'm not there but I'm I went there and saw the news that happened yesterday and then I came to that program and I went down last Sunday, I did not know what else it was that made a backup copies and for that I am happy that minetest forum has been alive step on the line that put it in a backup since March.

[Fixer]

@celeron55
What's your avg backup size?

[uwu]

^ I'd like to know this as well.
If it's not too outrageous, I'm willing to provide some storage space with sftp access on one of my VPS.

After this crash, we as a community should work towards increasing our Bus factor, so the Minetest project won't fail if one person suddenly disappears or another catastrophic event like this happens again. I've seen too many projects in complete disarray just because one guy is missing, e.g. Void Linux.

[Kilarin]

You don't realize how much you really needed something until it's gone.
I missed the minetest forums!

I want to add my voice to the other saying thank you for all the hard work getting the forums back up and keeping them running.

[Linuxdirk]

Is it me or is the server noticeably slower?

[celeron55]

@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.

[TumeniNodes]

@Fixer @uwu

The backup size for the forum and wikis is currently at about 5GB.

If you think the community might trust you with an encrypted backup and you're up for setting up the receiving end (including limited ssh access, the necessary cron jobs to prune old backups on your end and whatever else is needed), send me an email. Once I know our resources, I can figure out what would make most sense.

What I do know is the backups do contain some personal information (which at least in EU matters a lot) so I need to be careful. Also, there's a certain someone who really would like to get his hands on these backups to... ehm... "update" his website and send very useful and not spammy at all emails to people's private addresses.

Yeh, it's that 2d bit which scares me more than any EU issues. At least the EU can be reasoned with.
I do not envy your position of needing to make a choice eventually.

[sorcerykid]

I'm not very comfortable with random people being handed the keys to the castle (no offense Fixer, it's nothing personal :)

It is my understanding that the forum database contains private messages, hidden subforums, hashed passwords, and IP addresses. Even if fully encrypted, that's a lot of sensitive, personal information that is just begging for a privacy breach to happen. We've all seen the news of major companies and even government agencies in the line-of-fire for not properly protecting confidential data ;) In my view, nobody else except the core developers and moderators should be trusted with access to the forum database.

Why not look into a cloud hosting solution? I've been using ADrive for automated weekly backups of my server. They support all the industry standard protocols including rsync, sftp and scp. And of course under Linux, you can remote mount the filesystem via SSHFS. The 100 GB storage plan is just 25 USD for an entire year. This way you could maintain a centralized, secured backup that is always online. And only appointed staff would be give the login credentials.