[Server] HOMETOWN

random_geek
Member
Posts: 55
Joined: Wed Jun 20, 2018 18:55
GitHub: random-geek
In-game: Techy5 random_geek

Re: [Server] HOMETOWN

by random_geek » Post

Miss Micah wrote:Soooo no rush or anything or pressure but uh....When do you think the server will be up?
When EDad makes all the things work, and turns it on. :D

Miss Micah
Member
Posts: 109
Joined: Fri Jul 14, 2017 23:39
In-game: micah

Re: [Server] HOMETOWN

by Miss Micah » Post

random_geek wrote:
Miss Micah wrote:Soooo no rush or anything or pressure but uh....When do you think the server will be up?
When EDad makes all the things work, and turns it on. :D
SEE THIS IS WHY I DON’T ASK QUESTIONS.........anyway ima move on
God's Not Dead!!
You are Loved!!
Hiya! I'm Miss Micah XD

random_geek
Member
Posts: 55
Joined: Wed Jun 20, 2018 18:55
GitHub: random-geek
In-game: Techy5 random_geek

Re: [Server] HOMETOWN

by random_geek » Post

We don't know much yet, that's all.

shacharr
Member
Posts: 11
Joined: Thu Jun 21, 2018 06:36
In-game: shacharr

Re: [Server] HOMETOWN

by shacharr » Post

micheal65536 wrote:<-- snip -->
Michael - how are you hashing? If you hash a block, how do you find it based on partial information? Care to post the code somewhere to chat about code?

Also, it seems you assume you can keep all blocks + indices in memory. You will need a server with ~128GB of RAM to do this (or a bunch of these to scale). This is quite a hefty beast.

Regarding eliminating used blocks - I think that the general problem is NP-complete (given partial blocks, is there a mapping for a list of full blocks such that all blocks are used exactly once and all partial blocks are matching). I believe we can show this by showing reducibility to minimal set cover (which is NP-complete), but didn't think the proof all the way through. I guess that a good way to approach that would be to give each potential block a score for matching to the partial content we have (and to the surrounding blocks), and select the option with the highest combined score.

As I said, the problem is interesting, but at this point purely theoretical :).

micheal65536
Member
Posts: 167
Joined: Mon May 22, 2017 20:27

Re: [Server] HOMETOWN

by micheal65536 » Post

shacharr wrote:Michael - how are you hashing? If you hash a block, how do you find it based on partial information?
Firstly, I'm converting all the nodes to numerical values. This speeds up both hashing and direct comparison. Unlike Minetest's node ID system, these values are globally-assigned and are shared between all databases that are being compared.

As all the nodes are numerical values, the blocks are hashed by dividing the nodes into four "bins" based on the numerical value of the node and then storing the total count of nodes in each "bin". The distance between hashes is calculated by taking the absolute value of the differences between each bin and adding them all together. It can be proven that, for any two blocks where the number of differing nodes is n, the distance between the hashes will not exceed 2n.

It's not as simple as finding the blocks with the lowest hash distance. The block with the lowest (best) actual score may well have a higher hash distance, while a block with a very high score may have a very low hash distance (depending on which nodes have changed to which other nodes and which bins all of those nodes fell into). However, on the basis that the hash distance will not exceed twice the score (see end of previous paragraph), an iterative algorithm may be used.

First, the blocks with the lowest hash distance are found. A full comparison is performed against each of these blocks and the lowest score is noted. Then, the blocks with the second-lowest hash distance are found, and the lowest score is noted again. Then, the blocks with the third-lowest hash distance are found, and the lowest score is noted again. This continues as many times as needed until the lowest score is lower than twice the hash distance currently under consideration (because the hash distance for the best block in the entire database will never be greater than twice the lowest score in the entire database, so once the hash distance exceeds twice the lowest score that's already been found, it may be concluded that that lowest score is in fact the best block in the entire database). In practice, between 1 and 3 iterations are required to find the best match.
shacharr wrote:Also, it seems you assume you can keep all blocks + indices in memory. You will need a server with ~128GB of RAM to do this (or a bunch of these to scale).
At the moment, yes, I am keeping everything in RAM, and for a full-size map (with every single block generated) this would equate to more than 128 GB of RAM. However the actual RAM usage scales linearly based on the size of the map. While 128 GB sounds frightening, the actual RAM usage even on the HOMETOWN database will be much lower.

At a minimum, a completely empty block in a map database takes 17 bytes (for version 27, assuming that I counted up correctly, and not including the headers for the two Zlib sections). Most blocks are not completely empty so the actual average size will probably be at least 256 bytes (an uncompressed map block, by comparison, is 16384 bytes, and on average a compression algorithm can usually achieve a 50% compression ratio so the actual amount is probably even higher than 256 bytes - air compresses very small but actual terrain, caves, and buildings don't, and I'm removing all the air blocks beforehand anyway). I'm storing 8 bytes for each block.

For the HOMETOWN map, assuming a size of 256 bytes for each block, the map will contain 356515840 blocks (again, it's probably even less than this because the average size of each block is probably higher). This will take 2.6 GB of RAM if I store 8 bytes for each block.

And if RAM usage does become a problem, I can write some of the data out to disk until it's needed again. It would make the code slightly more complicated but the performance tradeoff wouldn't be outrageous.
shacharr wrote:I guess that a good way to approach that would be to give each potential block a score for matching to the partial content we have (and to the surrounding blocks), and select the option with the highest combined score.
Calculating a score in both directions was one option that I thought of for resolving conflicts (although I'm still not convinced that there will always be a definite answer, but my understanding of computational theory is quite limited in this regard). But the extra work required (doubling the time taken to complete the job, if not worse) is not worth it as this kind of situation probably won't occur that often when dealing with buildings anyway.

Code: https://drive.google.com/file/d/1huqy4i ... sp=sharing The code's a bit messy and the "trim-map" binary doesn't build at the moment. You'll need CMake, SQLite3 and Zlib installed to compile it. Don't expect it to compile on anything other than Linux using GCC. The "hash-database" binary is supposed to run first for each database (make sure to pass the same name map file each time) followed by the "choose-blocks" binary and finally the "assemble-map" binary to produce the final map. You'll need an empty SQLite map file (one with the table in place but no blocks) for the output file - you can use the "output.sqlite" file in the source tarball. There's no documentation so refer to the code for the correct command-line parameters.

User avatar
pangland
Member
Posts: 14
Joined: Sun Sep 03, 2017 12:11
In-game: pangland

Re: [Server] HOMETOWN

by pangland » Post

ExeterDad wrote:
pangland wrote:Thanks ExeterDad keep trying overnight! Also a lots of players providing backup support. I cant wait to see a fresh HT! I keep searching HT every hours and wonder if I could be the first player to join new HT :p

By the way, anybody have backup my garden before, would you please tell me if you did...I used more than one year to build The Green and the White Garden. I feel comfort to build in HT, so I never backup my building before. My heart was tearing after I noticed this.
pangland,
As far as I can see all your areas that surround spawn are completely intact. Are you referring to the area near my mountain? If so... you're in great shape :) The radius around spawn is intact for quit a distance. The players that donated covered a bunch of ground, thoroughly.
Miniminaut's area is also intact. I can't tell how much for sure because when I tested, all the various "gens" were wiping things out before my eyes.
But my test's were on a disposable copy so no worries :)

Also... I'm not going to engage any more about this map recovery tool development. I've already made my case and expressed that I do not want to proceed with it.
And for what it's worth. I would never release the logs to the public. There's conversations and dark secret's that players/admin's/staff would rather not have "go viral" lol
What a inspiring news that my garden still exist! And yes...I agree log should not be disclose to public as there are many "HOMETOWN Secret" lol

Hmm please dont count me out if you need a man to rebuild the buildings/landscape/roads
etc. I get ready to contribute for HT always. Sound like the scene of "Day after tomorrow", rebuilding the city after disaster.

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Server] HOMETOWN

by ExeterDad » Post

Generating a map and placing the map of donated data has proven to be harder than expected. The merge part is actually pretty painless.
I thought I could generate a clean base map fairly quickly. Apparently the /emergeblocks command will crash our server with anything larger than 1k nodes. I can't seem to overcome this. Even using the minetest.emerge_area function the command uses will crash. So I'm doing the "old school" fly in calculated spirals that grow until the desired area is generated. It's slow, even though I have it moving as fast as I dare. In the last 12 hours I've got surface of the map generated for 1/3 of what we need to prevent map/tree/river/cave *gens from ripping up the surface builds.
We will be gone through most of the day taking the kids to summer camp. Hopefully my "flying player" doesn't run out of carrots. Or we will need to basically start over.

User avatar
h-v-smacker
Member
Posts: 115
Joined: Wed Nov 29, 2017 23:04
GitHub: h-v-smacker
In-game: Smacker

Re: [Server] HOMETOWN

by h-v-smacker » Post

We stand in solidarity. Sorry for your loss.

Image
Attachments
screenshot_20180624_184157.jpg
screenshot_20180624_184157.jpg (526.35 KiB) Viewed 856 times
Linux Forks, the Best Forks: linux-forks.de // Play @ LinuxWorks Next Generation

User avatar
harmony
Member
Posts: 410
Joined: Tue Jun 20, 2017 22:16
IRC: ynomrah
In-game: ynomrah
Location: 고향 ^-^

Re: [Server] HOMETOWN

by harmony » Post

Miss Micah wrote:You take a nap for an hour due to the 30 minutes you have been getting and you miss the people that you have missed!!! XD

Heya ynomrah and GD and GH!!!!!!!! Also happy ynomrah and Loki and Clasher!!!!!!!!!!!!!!! XD
Hiii, ty!!
Happy birthday Loki & Clasher!! :D
저는 방탄소년단 를 사랑해요!!! 아미!! 김남준,김석진,민윤기,정호석,박지민,김태형,전정국,방탄소년단!
저는 블랙핑크 를 사랑해요!!! 블링크!! 김지수,김제니,박채영,리사(ปราณ ปริ ยา มโน บาล)

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Server] HOMETOWN

by Fixer » Post

Sadly I don't have any saves :/ I've spotted my base in recovered saves, which is nice. I hope there is save for this castle at 2000x-360
Image
Attachments
2017-07-18 01_45_59-Minetest 0.4.16 Hometown Castle 2.png
2017-07-18 01_45_59-Minetest 0.4.16 Hometown Castle 2.png (413.85 KiB) Viewed 856 times

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: [Server] HOMETOWN

by ChimneySwift » Post

6722 wrote:Hi rubenwardy
I'm going to enjoy watching you get banned.
A spoon is basically a tiny bowl with a stick on it

User avatar
Stix
Member
Posts: 1385
Joined: Fri Aug 04, 2017 14:19
IRC: nil
In-game: Stix [+alts]
Location: USA

Re: [Server] HOMETOWN

by Stix » Post

ChimneySwift wrote:
6722 wrote:Hi rubenwardy
I'm going to enjoy watching you get banned.
Agreed, this spamming is getting out of hand.
Hey, what can i say? I'm the bad guy.

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Server] HOMETOWN

by ExeterDad » Post

So the fly in circles method to generate map sucked. The amount generated I reported earlier was not accurate. I didn't take into account that the starting point was at the center, and each "circle" around the map was getting progressively bigger... and taking longer. Also staying connected was a challange from my end. And hunger was a issue. Also there wasn't a resume. So all attempts would have to be from the beginning.
Yesterday morning I whipped up a script that broke the map down into 3600 grid squares. Offhand I think that made 1008x1008 node squares that break down into 16x16 mapblocks. It seems minetest.emerge_area can hand that size without crashing the server. The script has been running without issue since yesterday and has completed "Count: 1146 of 3600 (31.83%)" so far. It takes roughly 70 seconds to do each square. It also has resume in case something happens. So I can continue from where ever I want.
I did attempt the voxel manip block loading method. It was twice as fast.... but for some reason there was a memory leak that would consume all the ram eventually and even chew through massive swap space. I reverted that code right away lol.
So... we wait :) Thanks for reading lol

random_geek
Member
Posts: 55
Joined: Wed Jun 20, 2018 18:55
GitHub: random-geek
In-game: Techy5 random_geek

Re: [Server] HOMETOWN

by random_geek » Post

So, the base map has to be generated, and then the .sqlites added back in?

Miss Micah
Member
Posts: 109
Joined: Fri Jul 14, 2017 23:39
In-game: micah

Re: [Server] HOMETOWN

by Miss Micah » Post

ExeterDad wrote:So the fly in circles method to generate map sucked. The amount generated I reported earlier was not accurate. I didn't take into account that the starting point was at the center, and each "circle" around the map was getting progressively bigger... and taking longer. Also staying connected was a challange from my end. And hunger was a issue. Also there wasn't a resume. So all attempts would have to be from the beginning.
Yesterday morning I whipped up a script that broke the map down into 3600 grid squares. Offhand I think that made 1008x1008 node squares that break down into 16x16 mapblocks. It seems minetest.emerge_area can hand that size without crashing the server. The script has been running without issue since yesterday and has completed "Count: 1146 of 3600 (31.83%)" so far. It takes roughly 70 seconds to do each square. It also has resume in case something happens. So I can continue from where ever I want.
I did attempt the voxel manip block loading method. It was twice as fast.... but for some reason there was a memory leak that would consume all the ram eventually and even chew through massive swap space. I reverted that code right away lol.
So... we wait :) Thanks for reading lol
English EDad, please.
God's Not Dead!!
You are Loved!!
Hiya! I'm Miss Micah XD

User avatar
Lupercus
Member
Posts: 30
Joined: Tue Jan 03, 2017 02:08
GitHub: lupercus13
IRC: Lupercus-Server
In-game: Lupercus

Re: [Server] HOMETOWN

by Lupercus » Post

yes, that is correct micah

User avatar
Stix
Member
Posts: 1385
Joined: Fri Aug 04, 2017 14:19
IRC: nil
In-game: Stix [+alts]
Location: USA

Re: [Server] HOMETOWN

by Stix » Post

Miss Micah wrote:
ExeterDad wrote:So the fly in circles method to generate map sucked. The amount generated I reported earlier was not accurate. I didn't take into account that the starting point was at the center, and each "circle" around the map was getting progressively bigger... and taking longer. Also staying connected was a challange from my end. And hunger was a issue. Also there wasn't a resume. So all attempts would have to be from the beginning.
Yesterday morning I whipped up a script that broke the map down into 3600 grid squares. Offhand I think that made 1008x1008 node squares that break down into 16x16 mapblocks. It seems minetest.emerge_area can hand that size without crashing the server. The script has been running without issue since yesterday and has completed "Count: 1146 of 3600 (31.83%)" so far. It takes roughly 70 seconds to do each square. It also has resume in case something happens. So I can continue from where ever I want.
I did attempt the voxel manip block loading method. It was twice as fast.... but for some reason there was a memory leak that would consume all the ram eventually and even chew through massive swap space. I reverted that code right away lol.
So... we wait :) Thanks for reading lol
English EDad, please.
That statement from Edad was for programming-savy people and to let everyone know hes still working on this, i doubt you'll get much more out of him.
Hey, what can i say? I'm the bad guy.

random_geek
Member
Posts: 55
Joined: Wed Jun 20, 2018 18:55
GitHub: random-geek
In-game: Techy5 random_geek

Re: [Server] HOMETOWN

by random_geek » Post

Miss Micah wrote:
ExeterDad wrote:-snip-
English EDad, please.
Someone needs to roam around the world, or bad things will happen. Flying around the world in circles is too hard, and takes too long. Edad made a magic robot that roams around the world unassisted. That's helpful and good, but it might take some time. There might be a better robot, but it doesn't behave as well.

Simple enough? :D

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Server] HOMETOWN

by ExeterDad » Post

No magic robot, but it is automated lol.
Think of the map as a giant chess board. My script loads the top left square first. When completed it moves to the next square to the right. It loads each one to the right until the top row is completed and then begins on the top second row. Wash, rinse, repeat.
This is done without a player logged in at all.
Here's a sample message of my script so I can see where we are at.
"Generating (14352,-32,9840) (15360,112,8832)"
This is a 1008x1008 size map section. It's -32 nodes below 0 (sea level) by 112 nodes above 0 (or 112 nodes into the air).
This will create a nice even "slab" of map without any holes or irregularities. The fly method can't insure this.
Once we run out of rows, the base map will be complete. Then.... I can add all the map saves everyone donated.
I'm going to use a nice map save as a example. One of them contained some long roads. The person only traveled on the actual road, so the road was saved very nicely, and "some" map on either side of it. In some places the road was actually a bridge. It looked like a perfect save. But if you travel just to the side of the bridge... your client would try to load some of the land to the side that wasn't actually part of the map save. No worries, the server would simply generate that part of the land. Then the river generator would start to cut the riverbed into the land and fill it with water. Then the tree generator would do the same. There might be some snow needed in that area as well. The river generator doesn't stop right away when it reaches map that's already generated. It will go for a while to do a nice blend. PoOf! The bridge is gone. And then some tree's are put in there. And maybe the entire remaining road will be instantly covered in snow that will ruin the rails.
This would happen with homes or whatever. Most of the time it would happen before you ever walk up to the house. It might make it look like the house was never saved at all.
So if we have a base map that does not have any "holes" on the surface, there will be no need for any kind of generation to be triggered. So our map saves will be safe.
But keep in mind. Those underground builds could be wiped out in the same way of a cave generator comes ripping through. But we can't possibly generate the entire underground to prevent this.

Miss Micah
Member
Posts: 109
Joined: Fri Jul 14, 2017 23:39
In-game: micah

Re: [Server] HOMETOWN

by Miss Micah » Post

random_geek wrote:
Miss Micah wrote:
ExeterDad wrote:-snip-
English EDad, please.
Someone needs to roam around the world, or bad things will happen. Flying around the world in circles is too hard, and takes too long. Edad made a magic robot that roams around the world unassisted. That's helpful and good, but it might take some time. There might be a better robot, but it doesn't behave as well.

Simple enough? :D
Ooooooooh XD

EDIT: ooooh, so this also means I’m gonna have to redo my basement stuff!! :D this is actually really good because I wanted to redo all of my basement areas. XD
God's Not Dead!!
You are Loved!!
Hiya! I'm Miss Micah XD

micheal65536
Member
Posts: 167
Joined: Mon May 22, 2017 20:27

Re: [Server] HOMETOWN

by micheal65536 » Post

ExeterDad wrote:But keep in mind. Those underground builds could be wiped out in the same way of a cave generator comes ripping through. But we can't possibly generate the entire underground to prevent this.
I hope that the underground blocks are at least generated to a depth of 3 or 4 blocks. A lot of players, myself included, had underground storerooms or underground parts to their builds just under the surface.

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Server] HOMETOWN

by ExeterDad » Post

micheal65536 wrote: I hope that the underground blocks are at least generated to a depth of 3 or 4 blocks. A lot of players, myself included, had underground storerooms or underground parts to their builds just under the surface.
The underground map saves will be placed if we have them. Protecting them from getting ruined is another matter.
If the underground is no deeper than Y -32 (2 blocks) there shouldn't be any need for worry. But even so. Underground, the only concern I can think of is caves. If your stuff is in a existing cave, it will likely be wiped clean. If it's something you dug... no worries.

Edit: Current status
Count: 1333 of 3600 (37.03%)
Generating (-17904,-32,7824) (-16896,112,6816)

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: [Server] HOMETOWN

by ChimneySwift » Post

ExeterDad wrote:Underground, the only concern I can think of is caves. If your stuff is in a existing cave, it will likely be wiped clean. If it's something you dug... no worries.
6O.o well, I guess most of minecity should be good, except ofc my house which is like the only house that isn't dug out LOL.

And ecki's cave city place will be gone... I guess LW got into that bunker anyway, should probably redo that.

And the cavegen doesn't care about whether the blocks exist, it'll regen anyway? Oh well, I'm glad progress is being made and look forward to playing on the server again :D
A spoon is basically a tiny bowl with a stick on it

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Server] HOMETOWN

by ExeterDad » Post

ChimneySwift wrote: 6O.o well, I guess most of minecity should be good, except ofc my house which is like the only house that isn't dug out LOL.

And ecki's cave city place will be gone... I guess LW got into that bunker anyway, should probably redo that.

And the cavegen doesn't care about whether the blocks exist, it'll regen anyway? Oh well, I'm glad progress is being made and look forward to playing on the server again :D
It's entirely possible to "re-apply" the underground map saves after cavegen does it's nastiness. It's not trivial as it would be done at the database level and we would have to decode the cords as we would need to cherry pick but I'm sure it could be done. It would "rollback" any repairs owners may attempt once open. But I think it's completely possible.

Edit:
Count: 1378 of 3600 (38.28%)
Generating (27456,-32,7824) (28464,112,6816)

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: [Server] HOMETOWN

by ChimneySwift » Post

ExeterDad wrote:
ChimneySwift wrote: 6O.o well, I guess most of minecity should be good, except ofc my house which is like the only house that isn't dug out LOL.

And ecki's cave city place will be gone... I guess LW got into that bunker anyway, should probably redo that.

And the cavegen doesn't care about whether the blocks exist, it'll regen anyway? Oh well, I'm glad progress is being made and look forward to playing on the server again :D
It's entirely possible to "re-apply" the underground map saves after cavegen does it's nastiness. It's not trivial as it would be done at the database level and we would have to decode the cords as we would need to cherry pick but I'm sure it could be done. It would "rollback" any repairs owners may attempt once open. But I think it's completely possible.

Edit:
Count: 1378 of 3600 (38.28%)
Generating (27456,-32,7824) (28464,112,6816)
Oh cool, that might be nice. I could always go into my world and grab some WorldEdit saves of some builds if it's easier.

Also omg. This is great we have like a countdown now (or at least for the generation part). I doubt it'd be feasible but would be pretty funny if you made it write the current count and what it's generating to a little HTML file and hosted it XD (pls don't feel like you have to do that :P)
A spoon is basically a tiny bowl with a stick on it

Locked

Who is online

Users browsing this forum: Google [Bot] and 2 guests