Page 1 of 4

[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

Posted: Wed Jan 10, 2018 23:36
by Gael de Sailly
Hello,

This mod aims to generate Minetest worlds from real topographical data (Digital Elevation Model).

I've been working for a while trying to generate maps of real places of the world, especially working with bobomb on his Real Terrain mod that was a great experiment, but faced huge stability problems because of the external libraries we needed to run in Lua to read the DEM image, or the hacky method that was implemented to read the image in pure Lua.

Here I come with a different approach. The image is not read in Lua but by a separate Python file (image_convert.py) that deals with the image and generate a database in the specified world directory, that is easily readable by Minetest natively.

This has the advantage of providing a very stable and proper code for the Lua mapgen.
This has the drawback of needing to prepare the world by running this Python file before.

This mod doesn't come with pre-loaded geographical data, it only provides the tools to use them. You need to find your data on the Internet. But I can help you :)
+ Spoiler
I've generated a worldwide map using NOAA Global DEM for elevation, and MODIS-based Global Land Cover for land cover data. You can download the database directly at this link (place it in your Minetest world directory and rename it "heightmap.dat"). The screenshots show the world with a resolution of 2km horizontally, and 200m vertically.
Image^ France and southern England
+ Spoiler
+ Spoiler
Where to find the mod:
Download (Code on GitHub)
License: LGPL v2.1

This mod is still experimental and is not ready for regular use. The database format, especially, could change in future versions without backward compatibility.
I encourage you to report me any bug, or suggestion for the code (I think it could be optimized a lot). Any contribution would be appreciated.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 05:14
by TumeniNodes
Hey Gael,
I am so glad you are still working on this. It looks great.

I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?

I am also amazed at how minimal the options are in finding geotiff images of the world (very few regions)

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 06:54
by Gael de Sailly
TumeniNodes wrote:I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?
Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 07:26
by TumeniNodes
Gael de Sailly wrote: Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Ah..., I am a goof.
Got it now, thank you.
I should have everything ready now, but it is very late for me

Will attempt again tomorrow and give feedback

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 18:20
by TumeniNodes
Well, it works
(I had no doubts) ; )

But, maybe I should familiarize myself a bit better with the scaling (because I was way off. :P
Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

The image I used is from: https://asterweb.jpl.nasa.gov/images/GDEM-10km-BW.tif

This was my result, because I was lazy and simply used your scaling for the maps you had
Image

But, this is awesome, and I will tinker with it more. Thank you

I also uploaded the .tif, and the resulting world down there v

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 21:40
by Gael de Sailly
I think something is wrong with my code, your map clearly don't match the image. That can't be due to the scale parameter that only sets vertical scale and son't modify shapes at all. I think this is because of the fragmentation: during converting, the image is cut into square chunks, but I've never thought about what would happen if the whole image is smaller than one chunk. That shouldn't be too hard to fix.

The other problem is with your image: the grey shade here represents slope (derivative value of elevation) and not elevation itself. You would get a map with recognizable shapes but completely wrong elevation and slopes.
TumeniNodes wrote:Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise
Yes it uses singlenode but anyway if you use a different mapgen in your world, the mod automatically set it on singlenode.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 11, 2018 22:29
by Fixer

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 00:32
by Gael de Sailly
Thank you Fixerol for the links :)
I've used the third one for my screenshots. It covers the whole world with one pixel every 93m.

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed). Just run image_convert.py again and delete map.sqlite.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 00:45
by TumeniNodes
Gael de Sailly wrote: The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed).
You're fast Gael, thank you : )

and yes, thank you Fixerol

another link which may be helpful for people, makes it a bit easier to get coords if you want to enter them yourself:
https://www.findlatitudeandlongitude.com/

I will refresh the mod

This is really a great start so far though... nice work

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 07:03
by duane
This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 17:59
by Fixer
Yeah, handling rivers and oceans would be nice and even setting your own water level.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 18:00
by Gael de Sailly
duane wrote:This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Rivers are planned, I have already written the algorithm on python side: rivermapper.py, that for now transforms a SRTM tile to a river map.
The algorithm is slow (20min for a SRTM tile on my laptop), I'm looking for optimizations.
If this script works well, I think rivers could be implemented in the mapgen without major difficulties.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 18:05
by TumeniNodes
Would be nice to have 2 water level settings

The current/existing one for ocean
a second for lakes, which streams/rivers can run down from

It would also be nice to have water and river water the same color :P (but that is an offtopic matter in relation to this) sorry.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 20:32
by Gael de Sailly
image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Fri Jan 12, 2018 21:57
by Gael de Sailly
TumeniNodes: I think I have what you were looking for.
https://www.eea.europa.eu/data-and-maps ... del-etopo5
Download in GeoTiff format.
You'll find the USA at 1100 -600.

(EDIT: coordinates were wrong)

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Sat Jan 13, 2018 02:36
by TumeniNodes
Nice find Gael, thank you.
Gael de Sailly wrote:image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
it's witchcraft is what it is. That is a nice performance boost.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Sat Jan 13, 2018 15:25
by Gael de Sailly
I'm working on rivers on a new branch.
They should work now, although the code is not very stable for now.

Download
Code

You can try by adding the parameter "-r 2000" when launching the py script (this parameter represents the surface needed for catchment area to be drawn as a river, so decrease it if you want more rivers).

River calculation is EXTREMELY slow for now. It can easily take >15 minutes for a big image, so please be patient.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Tue Jan 16, 2018 03:55
by TumeniNodes
Have you converted that full world map yet?

For me, I get an error stating no .dat file is created.

Code: Select all

~/.minetest/mods/geo_mapgen $ ./image_convert.py '/home/paul/dem/alwdgg.tif' '/home/paul/worlds/geomap03' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 101, in <module>
    file_output = open(fpath_output, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/worlds/geomap03/heightmap.dat'
I am using your first branch.
Also, is there a specific scale you recommend? (as you can tell, I know nothing related to -f / -s scaling)

I was tempted to run convert with no vertical scaling but, not sure if that is recommended :P

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Tue Jan 16, 2018 07:00
by Gael de Sailly
Are you sure that the output path is correct? There is no ".minetest".

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Tue Jan 16, 2018 10:19
by TumeniNodes
Gael de Sailly wrote:Are you sure that the output path is correct? There is no ".minetest".
Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Gael de Sailly wrote:In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Thank you, that makes sense now.
I will try it again, only this time I won't forget a major path to the directory :D

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Tue Jan 16, 2018 23:33
by Gael de Sailly
TumeniNodes wrote:Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Don't worry, that has just happened to me today! For my part, I forgot "/worlds". I was waiting until the river algorithm finishes and discovered at the end that it crashed and all was to start again. You're allowed to laugh :D

Now the output file is loaded at the beginning, to make sure you'll not wait 15 minutes before it crashes.
TumeniNodes wrote:When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
Thank you, you reminded me that I had not tested my mod on a map without rivers since I added them. The code wasn't checking whether rivers are enabled and expected to use a river map that didn't exist. Fixed now.

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Wed Jan 17, 2018 00:23
by TumeniNodes
yes, it was throwing some nil error related to a layer, etc.. glad it's fixed. You're quick

found something which might work well?
http://srtm.csi.cgiar.org/
and then to this from that link http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

the extracted file for the Western map is a bit over 8gb.
I am not sure what that will equal in size when converted but, I have to be careful as I only have 11.5 gb right now after extracting it... (I need to clean out some space)

be sure to read the use terms

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Wed Jan 17, 2018 01:01
by RSLRedstonier
WOW
this
is
amazing

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Wed Jan 17, 2018 06:41
by Gael de Sailly
TumeniNodes wrote:found something which might work well?
http://srtm.csi.cgiar.org/
Yeah SRTM is the one I always use. But the resampled SRTM of the second link contains a single image with over 5GB for Europe/Asia (I think it's the same for other continents), that is not convenient to work with. If I use image_convert with this, my computer will just... die!
The one I use is http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp. You can download 6001x6001 images covering 5° lat/lon, with a resolution of 90m.

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

Posted: Thu Jan 18, 2018 00:04
by Gael de Sailly
I wrote:Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
That's mostly done.
I found a latest minor bug this evening, it has been fixed. Now I consider that the river branch is stable enough to be in the download, so it's merged.
I've updated the screenshots (that was really fun!) and (quickly) the description. If you don't understand all the parameters, that's not a problem, they have coherent default values. I'll do further work on documentation this weekend.