Minetest map generation using GPU

Post Reply
User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

Minetest map generation using GPU

by asdf7389 » Post

So currently I'm working on using CUDA to parallel map generation on minetest for my senior project. I'm fairly new to parallel programming, as well as C++ so it's been quite a challenge so far. I'm looking at the mapgen files and comparing to the process of how it runs when a users clicks the create new world button. Any suggestions on a focal point in what takes the longest time, taking aside rendering speeds. My main focus is the map generation. More info to come as I progress.

User avatar
rarkenin
Member
Posts: 668
Joined: Tue Nov 20, 2012 20:48

by rarkenin » Post

asdf7389 wrote:So currently I'm working on using CUDA to parallel map generation on minetest for my senior project. I'm fairly new to parallel programming, as well as C++ so it's been quite a challenge so far. I'm looking at the mapgen files and comparing to the process of how it runs when a users clicks the create new world button. Any suggestions on a focal point in what takes the longest time, taking aside rendering speeds. My main focus is the map generation. More info to come as I progress.
I think that the GPU should be usable for map generation, which is really important, but this may be relatively difficult as Lua would need to compile down to GPU instructions.
Admin pro tempore on 0gb.us:30000. Ask me if you have a problem, or just want help.
This is a signature virus. Add me to your signature so that I can multiply.
Now working on my own clone, Mosstest.
I guess I'm back for some time.

User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

by asdf7389 » Post

I was imagining making a file outside of Minetest to generate a world then dump it back into Minetest to explore, but I still don't fully understand how mapgen outputs it's initial file. Still just pouring through source code and beating wolves back with a stick.

User avatar
Calinou
Moderator
Posts: 3166
Joined: Mon Aug 01, 2011 14:26
GitHub: Calinou
IRC: Calinou
In-game: Calinou
Location: Troyes, France
Contact:

by Calinou » Post

asdf7389 wrote:So currently I'm working on using CUDA to parallel map generation on minetest for my senior project. I'm fairly new to parallel programming, as well as C++ so it's been quite a challenge so far. I'm looking at the mapgen files and comparing to the process of how it runs when a users clicks the create new world button. Any suggestions on a focal point in what takes the longest time, taking aside rendering speeds. My main focus is the map generation. More info to come as I progress.
Why not OpenCL? CUDA is NVIDIA-only and it's a PITA to set it up on Linux.

Is there really any noticeable performance improvement by using the GPU for map generation? I bet the gains would be small since it's fast enough already. Feel free to make this just for fun, though. :)

User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

by asdf7389 » Post

Calinou wrote:
Why not OpenCL? CUDA is NVIDIA-only and it's a PITA to set it up on Linux.

Is there really any noticeable performance improvement by using the GPU for map generation? I bet the gains would be small since it's fast enough already. Feel free to make this just for fun, though. :)
Well it's in CUDA because my project sponsor suggested that its optimized for my current hardware, however the code is easily translated from CUDA to OpenCL. I'm also trying to generate a maximum mapsize file at once vs progressive generation.

User avatar
rarkenin
Member
Posts: 668
Joined: Tue Nov 20, 2012 20:48

by rarkenin » Post

asdf7389 wrote:I was imagining making a file outside of Minetest to generate a world then dump it back into Minetest to explore, but I still don't fully understand how mapgen outputs it's initial file. Still just pouring through source code and beating wolves back with a stick.
It's just a SQLite database. Take a look at the MapBlock serialization methods, and look for the way they determine block ID.
Admin pro tempore on 0gb.us:30000. Ask me if you have a problem, or just want help.
This is a signature virus. Add me to your signature so that I can multiply.
Now working on my own clone, Mosstest.
I guess I'm back for some time.

User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

by asdf7389 » Post

Well finding time to work on this, I've decided to see about generating a map that hits the max size in a square fashion. Looking through to see if initial generation can be stretched to accommodate generation of the entire map at once.

Jordach
Member
Posts: 4527
Joined: Mon Oct 03, 2011 17:58
GitHub: Jordach
IRC: Jordach
In-game: Jordach
Location: Blender Scene

by Jordach » Post

asdf7389, dont try to generate a full Minetest map at once. No really, don't.

The SQLITE map file can only go upto 4GB and the file becomes unstable, not to mention that the file size will grow to about 14TB.

Good Luck.
Last edited by Jordach on Fri Apr 05, 2013 17:09, edited 1 time in total.
viewtopic.php?f=10&t=19056 Solar Plains Dev Server

User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

by asdf7389 » Post

Jordach wrote:asdf7389, dont try to generate a full Minetest map at once. No really, don't.

The SQLITE map file can only go upto 4GB and the file becomes unstable, not to mention that the file size will grow to about 14TB.

Good Luck.
Hmm...I thought there was a built in limit for the max size of a map file that can be generate in Mintest. Wondering if I can softcap it around 4GB if not.

User avatar
asdf7389
New member
Posts: 6
Joined: Wed Mar 20, 2013 17:55

by asdf7389 » Post

So at this point, I agree that manipulating the sqlite file may be volatile, so I've shifted my focus to just modifying the map node and map generation to be ported to a GPU implementation, I just wish I could figure out the key class and method I need to focus on, because all of it seems to be relevant;.

User avatar
Likwid H-Craft
Member
Posts: 1113
Joined: Sun Jan 06, 2013 14:20
Location: Lost in Crypt

by Likwid H-Craft » Post

Jordach wrote:asdf7389, dont try to generate a full Minetest map at once. No really, don't.

The SQLITE map file can only go upto 4GB and the file becomes unstable, not to mention that the file size will grow to about 14TB.

Good Luck.
That will be needing a new pc HDD for some so I think it not that great have up to 1TB.
My Domain's/others:
http://likwidtest.hj.cx/ (Not Done)

User avatar
rarkenin
Member
Posts: 668
Joined: Tue Nov 20, 2012 20:48

by rarkenin » Post

Likwid H-Craft wrote:
Jordach wrote:asdf7389, dont try to generate a full Minetest map at once. No really, don't.

The SQLITE map file can only go upto 4GB and the file becomes unstable, not to mention that the file size will grow to about 14TB.

Good Luck.
That will be needing a new pc HDD for some so I think it not that great have up to 1TB.
No, you hit filesystem limits and instability first. Please don't bring up the obvious, especially when there is something more important than it.
Admin pro tempore on 0gb.us:30000. Ask me if you have a problem, or just want help.
This is a signature virus. Add me to your signature so that I can multiply.
Now working on my own clone, Mosstest.
I guess I'm back for some time.

User avatar
sfan5
Moderator
Posts: 3944
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

by sfan5 » Post

Jordach wrote:asdf7389, dont try to generate a full Minetest map at once. No really, don't.

The SQLITE map file can only go upto 4GB and the file becomes unstable, not to mention that the file size will grow to about 14TB.

Good Luck.
Assuming every block is one byte its 216 TB.
As every block has a light level, param1 and param2 that is 4 bytes per block.
That makes 864 TB.
But since almost 50% of the map is air (~ 0 -- +30 000) and 30% of the rest should be easily compressable, a complete minetest map would occupy 172.8 TB.
Comparisons by Wolfram Alpha:
  • ~~ ( 0.086 ~~ 1/12 ) × identifiable storage capacity of the human brain (~~ 2 PB )
  • ~~ ( 0.086 ~~ 1/12 ) × approximate data content of the Internet Archive Wayback Machine as of 2006 (~~ 2 PB )
  • ~~ estimated data content of the surface web (~~ 170 TB )
That file size also shouldn't break any system as it is expressable (in bytes) with a 64-bit integer (signedness doesn't matter)
Last edited by sfan5 on Mon Apr 08, 2013 19:55, edited 1 time in total.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests