[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

by Gael de Sailly » Post

Hello,

This mod aims to generate Minetest worlds from real topographical data (Digital Elevation Model).

I've been working for a while trying to generate maps of real places of the world, especially working with bobomb on his Real Terrain mod that was a great experiment, but faced huge stability problems because of the external libraries we needed to run in Lua to read the DEM image, or the hacky method that was implemented to read the image in pure Lua.

Here I come with a different approach. The image is not read in Lua but by a separate Python file (image_convert.py) that deals with the image and generate a database in the specified world directory, that is easily readable by Minetest natively.

This has the advantage of providing a very stable and proper code for the Lua mapgen.
This has the drawback of needing to prepare the world by running this Python file before.

This mod doesn't come with pre-loaded geographical data, it only provides the tools to use them. You need to find your data on the Internet. But I can help you :)
+ Spoiler
I've generated a worldwide map using NOAA Global DEM for elevation, and MODIS-based Global Land Cover for land cover data. You can download the database directly at this link (place it in your Minetest world directory and rename it "heightmap.dat"). The screenshots show the world with a resolution of 2km horizontally, and 200m vertically.
Image^ France and southern England
+ Spoiler
+ Spoiler
Where to find the mod:
Download (Code on GitHub)
License: LGPL v2.1

This mod is still experimental and is not ready for regular use. The database format, especially, could change in future versions without backward compatibility.
I encourage you to report me any bug, or suggestion for the code (I think it could be optimized a lot). Any contribution would be appreciated.
Last edited by Gael de Sailly on Sun Mar 22, 2020 09:06, edited 14 times in total.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Hey Gael,
I am so glad you are still working on this. It looks great.

I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?

I am also amazed at how minimal the options are in finding geotiff images of the world (very few regions)
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?
Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote: Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Ah..., I am a goof.
Got it now, thank you.
I should have everything ready now, but it is very late for me

Will attempt again tomorrow and give feedback
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Well, it works
(I had no doubts) ; )

But, maybe I should familiarize myself a bit better with the scaling (because I was way off. :P
Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

The image I used is from: https://asterweb.jpl.nasa.gov/images/GDEM-10km-BW.tif

This was my result, because I was lazy and simply used your scaling for the maps you had
Image

But, this is awesome, and I will tinker with it more. Thank you

I also uploaded the .tif, and the resulting world down there v
Attachments
geomap.zip
(376.73 KiB) Downloaded 93 times
gdem_10km_BW2.tif
(46 KiB) Not downloaded yet
First GeoMapgen.jpg
(313.07 KiB) Not downloaded yet
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I think something is wrong with my code, your map clearly don't match the image. That can't be due to the scale parameter that only sets vertical scale and son't modify shapes at all. I think this is because of the fragmentation: during converting, the image is cut into square chunks, but I've never thought about what would happen if the whole image is smaller than one chunk. That shouldn't be too hard to fix.

The other problem is with your image: the grey shade here represents slope (derivative value of elevation) and not elevation itself. You would get a map with recognizable shapes but completely wrong elevation and slopes.
TumeniNodes wrote:Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise
Yes it uses singlenode but anyway if you use a different mapgen in your world, the mod automatically set it on singlenode.
Just realize how bored we would be if the world was perfect.

User avatar
Fixer
Member
Posts: 895
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Fixer » Post


User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

Thank you Fixerol for the links :)
I've used the third one for my screenshots. It covers the whole world with one pixel every 93m.

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed). Just run image_convert.py again and delete map.sqlite.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote: The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed).
You're fast Gael, thank you : )

and yes, thank you Fixerol

another link which may be helpful for people, makes it a bit easier to get coords if you want to enter them yourself:
https://www.findlatitudeandlongitude.com/

I will refresh the mod

This is really a great start so far though... nice work
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
duane
Member
Posts: 1612
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by duane » Post

This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Believe in people and you don't need to believe anything else.

User avatar
Fixer
Member
Posts: 895
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Fixer » Post

Yeah, handling rivers and oceans would be nice and even setting your own water level.

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

duane wrote:This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Rivers are planned, I have already written the algorithm on python side: rivermapper.py, that for now transforms a SRTM tile to a river map.
The algorithm is slow (20min for a SRTM tile on my laptop), I'm looking for optimizations.
If this script works well, I think rivers could be implemented in the mapgen without major difficulties.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Would be nice to have 2 water level settings

The current/existing one for ocean
a second for lakes, which streams/rivers can run down from

It would also be nice to have water and river water the same color :P (but that is an offtopic matter in relation to this) sorry.
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes: I think I have what you were looking for.
https://www.eea.europa.eu/data-and-maps ... del-etopo5
Download in GeoTiff format.
You'll find the USA at 1100 -600.

(EDIT: coordinates were wrong)
Last edited by Gael de Sailly on Sun Jan 14, 2018 13:29, edited 1 time in total.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Nice find Gael, thank you.
Gael de Sailly wrote:image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
it's witchcraft is what it is. That is a nice performance boost.
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I'm working on rivers on a new branch.
They should work now, although the code is not very stable for now.

Download
Code

You can try by adding the parameter "-r 2000" when launching the py script (this parameter represents the surface needed for catchment area to be drawn as a river, so decrease it if you want more rivers).

River calculation is EXTREMELY slow for now. It can easily take >15 minutes for a big image, so please be patient.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Have you converted that full world map yet?

For me, I get an error stating no .dat file is created.

Code: Select all

~/.minetest/mods/geo_mapgen $ ./image_convert.py '/home/paul/dem/alwdgg.tif' '/home/paul/worlds/geomap03' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 101, in <module>
    file_output = open(fpath_output, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/worlds/geomap03/heightmap.dat'
I am using your first branch.
Also, is there a specific scale you recommend? (as you can tell, I know nothing related to -f / -s scaling)

I was tempted to run convert with no vertical scaling but, not sure if that is recommended :P

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

Are you sure that the output path is correct? There is no ".minetest".

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote:Are you sure that the output path is correct? There is no ".minetest".
Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Gael de Sailly wrote:In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Thank you, that makes sense now.
I will try it again, only this time I won't forget a major path to the directory :D
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Don't worry, that has just happened to me today! For my part, I forgot "/worlds". I was waiting until the river algorithm finishes and discovered at the end that it crashed and all was to start again. You're allowed to laugh :D

Now the output file is loaded at the beginning, to make sure you'll not wait 15 minutes before it crashes.
TumeniNodes wrote:When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
Thank you, you reminded me that I had not tested my mod on a map without rivers since I added them. The code wasn't checking whether rivers are enabled and expected to use a river map that didn't exist. Fixed now.

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2853
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

yes, it was throwing some nil error related to a layer, etc.. glad it's fixed. You're quick

found something which might work well?
http://srtm.csi.cgiar.org/
and then to this from that link http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

the extracted file for the Western map is a bit over 8gb.
I am not sure what that will equal in size when converted but, I have to be careful as I only have 11.5 gb right now after extracting it... (I need to clean out some space)

be sure to read the use terms
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
RSLRedstonier
Member
Posts: 451
Joined: Wed May 10, 2017 21:00
GitHub: RSL-Redstonier
In-game: RSLRedstonier
Location: some were in middle earth

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by RSLRedstonier » Post

WOW
this
is
amazing
"A programmer is just a tool which converts caffeine into code"

try out my skytest mod always being updated!
viewtopic.php?f=9&t=17568

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:found something which might work well?
http://srtm.csi.cgiar.org/
Yeah SRTM is the one I always use. But the resampled SRTM of the second link contains a single image with over 5GB for Europe/Asia (I think it's the same for other continents), that is not convenient to work with. If I use image_convert with this, my computer will just... die!
The one I use is http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp. You can download 6001x6001 images covering 5° lat/lon, with a resolution of 90m.
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 752
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I wrote:Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
That's mostly done.
I found a latest minor bug this evening, it has been fixed. Now I consider that the river branch is stable enough to be in the download, so it's merged.
I've updated the screenshots (that was really fun!) and (quickly) the description. If you don't understand all the parameters, that's not a problem, they have coherent default values. I'll do further work on documentation this weekend.
Just realize how bored we would be if the world was perfect.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], runs and 5 guests