[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

[Gael de Sailly]

Hello,

This mod aims to generate Minetest worlds from real topographical data (Digital Elevation Model).

I've been working for a while trying to generate maps of real places of the world, especially working with bobomb on his Real Terrain mod that was a great experiment, but faced huge stability problems because of the external libraries we needed to run in Lua to read the DEM image, or the hacky method that was implemented to read the image in pure Lua.

Here I come with a different approach. The image is not read in Lua but by a separate Python file (image_convert.py) that deals with the image and generate a database in the specified world directory, that is easily readable by Minetest natively.

This has the advantage of providing a very stable and proper code for the Lua mapgen.
This has the drawback of needing to prepare the world by running this Python file before.

This mod doesn't come with pre-loaded geographical data, it only provides the tools to use them. You need to find your data on the Internet. But I can help you :)

+ Spoiler

Make sure you have a working Python 3 installation, and install numpy, gdal and tkinter libraries. Also, using recent 0.5.0-dev Minetest is recommended.

To generate your map:

1. Find DEM (Digital Elevation Model) data of the region you want (I personally use SRTM that has one pixel every 93m). Download it in image format (usually GeoTiff).
   
   You can use additional maps for land cover and rivers, this is explained in the README.
2. Launch Minetest, create a world and active this mod. Don't start the game for now!
3. Use the python script provided by this mod (image_convert.py) to generate the database, by calling it like this:

   Code: Select all

   ./image_convert.py

   This opens a configuration window with all the options you may need for your map.
   
   
   You might find some options a bit complicated, but you don't necessarily need to understand and tune everything. They are explained in the README.
   
   It generates 2 files in the world dir: heightmap.dat which is the database itself, and heightmap.dat.conf, configuration file easily readable with text editor that allow to change some parameters (currently only scale is supported).
4. Now start the game and enjoy!

You can also download pre-built databases at Geo Mapgen demos repository. Place the file in the Minetest world directory and rename it "heightmap.dat".

I hope this is clear enough. Please say it if you have any difficulty.

I've generated a worldwide map using NOAA Global DEM for elevation, and MODIS-based Global Land Cover for land cover data. You can download the database directly at this link (place it in your Minetest world directory and rename it "heightmap.dat"). The screenshots show the world with a resolution of 2km horizontally, and 200m vertically.
^ France and southern England

+ Spoiler

^ Indian Ocean with a view on Réunion at the center left, Mauritius at bottom right, and Madagascar at the horizon.

^ British Isles

^ Northern India. You're above the Ganga Plain, looking eastwards. You can make out the Ganges river bed at the right foreground, the Brahmaputra at the background, both converging in the huge Bengal Delta at the extreme right.
At the left the forests are gradually climbing up the Himalayas chain, until you reach the highest summits. The icy peak at the extreme left is the legendary Mount Everest. Beyond this mountain barrier stands the Tibet Plateau.

^ Iceland, with the famous Vatnajökull Glacier at the right background.

+ Spoiler

They have been taken using SRTM tiles with a resolution of 93m horizontally and 40m vertically.^ Grand Canyon, USA, generated using SRTM tile #14-05, rivers enabled.

^ Mont Blanc, France, generated using tile #38-03, rivers disabled.

^ Great view over the Doubs valley, (France/Switzerland), and its curious hairpinned loop. Tile #38-03

^ Ardèche gorge using tile #37-04, French canyon very popular for kayaking (I did it this autumn and that's really something I'll remember!!)
I'll obviously make a screenshot with rivers enabled as soon as possible.

^ Fuji mountain in Japan, tile #64-05

^ Saar Loop in Germany, very nice landscape on tile #38-03. Screenshot needs to be updated too with the new rivers feature.

Where to find the mod:
Download (Code on GitHub)
License: LGPL v2.1

This mod is still experimental and is not ready for regular use. The database format, especially, could change in future versions without backward compatibility.
I encourage you to report me any bug, or suggestion for the code (I think it could be optimized a lot). Any contribution would be appreciated.

[TumeniNodes]

Hey Gael,
I am so glad you are still working on this. It looks great.

I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'

I do have the imageio libs installed so, not sure where my issue is?

I am also amazed at how minimal the options are in finding geotiff images of the world (very few regions)

[Gael de Sailly]

I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'

I do have the imageio libs installed so, not sure where my issue is?

Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.

[TumeniNodes]

Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.

Ah..., I am a goof.
Got it now, thank you.
I should have everything ready now, but it is very late for me

Will attempt again tomorrow and give feedback

[TumeniNodes]

Well, it works
(I had no doubts) ; )

But, maybe I should familiarize myself a bit better with the scaling (because I was way off. :P
Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

The image I used is from: https://asterweb.jpl.nasa.gov/images/GDEM-10km-BW.tif

This was my result, because I was lazy and simply used your scaling for the maps you had


But, this is awesome, and I will tinker with it more. Thank you

I also uploaded the .tif, and the resulting world down there v

[Gael de Sailly]

I think something is wrong with my code, your map clearly don't match the image. That can't be due to the scale parameter that only sets vertical scale and son't modify shapes at all. I think this is because of the fragmentation: during converting, the image is cut into square chunks, but I've never thought about what would happen if the whole image is smaller than one chunk. That shouldn't be too hard to fix.

The other problem is with your image: the grey shade here represents slope (derivative value of elevation) and not elevation itself. You would get a map with recognizable shapes but completely wrong elevation and slopes.

Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

Yes it uses singlenode but anyway if you use a different mapgen in your world, the mod automatically set it on singlenode.

[Fixer]

I was waiting for this for so loooong :D Thank you!

Useful links:
http://www.imagico.de/map/demsearch.php
http://www.viewfinderpanoramas.org/dem3.html
http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp

[Gael de Sailly]

I was waiting for this for so loooong :D Thank you!

Useful links:
http://www.imagico.de/map/demsearch.php
http://www.viewfinderpanoramas.org/dem3.html
http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp

Thank you Fixerol for the links :)
I've used the third one for my screenshots. It covers the whole world with one pixel every 93m.

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed). Just run image_convert.py again and delete map.sqlite.

[TumeniNodes]

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed).

You're fast Gael, thank you : )

and yes, thank you Fixerol

another link which may be helpful for people, makes it a bit easier to get coords if you want to enter them yourself:
https://www.findlatitudeandlongitude.com/

I will refresh the mod

This is really a great start so far though... nice work

[duane]

This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.

[Fixer]

Yeah, handling rivers and oceans would be nice and even setting your own water level.

[Gael de Sailly]

This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.

Rivers are planned, I have already written the algorithm on python side: rivermapper.py, that for now transforms a SRTM tile to a river map.
The algorithm is slow (20min for a SRTM tile on my laptop), I'm looking for optimizations.
If this script works well, I think rivers could be implemented in the mapgen without major difficulties.

[TumeniNodes]

Would be nice to have 2 water level settings

The current/existing one for ocean
a second for lakes, which streams/rivers can run down from

It would also be nice to have water and river water the same color :P (but that is an offtopic matter in relation to this) sorry.

[Gael de Sailly]

image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!

[Gael de Sailly]

TumeniNodes: I think I have what you were looking for.
https://www.eea.europa.eu/data-and-maps ... del-etopo5
Download in GeoTiff format.
You'll find the USA at 1100 -600.

(EDIT: coordinates were wrong)

[TumeniNodes]

Nice find Gael, thank you.

image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!

it's witchcraft is what it is. That is a nice performance boost.

[Gael de Sailly]

I'm working on rivers on a new branch.
They should work now, although the code is not very stable for now.

Download
Code

You can try by adding the parameter "-r 2000" when launching the py script (this parameter represents the surface needed for catchment area to be drawn as a river, so decrease it if you want more rivers).

River calculation is EXTREMELY slow for now. It can easily take >15 minutes for a big image, so please be patient.

[TumeniNodes]

Have you converted that full world map yet?

For me, I get an error stating no .dat file is created.

Code: Select all

~/.minetest/mods/geo_mapgen $ ./image_convert.py '/home/paul/dem/alwdgg.tif' '/home/paul/worlds/geomap03' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 101, in <module>
    file_output = open(fpath_output, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/worlds/geomap03/heightmap.dat'

I am using your first branch.
Also, is there a specific scale you recommend? (as you can tell, I know nothing related to -f / -s scaling)

I was tempted to run convert with no vertical scaling but, not sure if that is recommended :P

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)

[Gael de Sailly]

Are you sure that the output path is correct? There is no ".minetest".

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.

[TumeniNodes]

Are you sure that the output path is correct? There is no ".minetest".

Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.

Thank you, that makes sense now.
I will try it again, only this time I won't forget a major path to the directory :D

[Gael de Sailly]

Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?

Don't worry, that has just happened to me today! For my part, I forgot "/worlds". I was waiting until the river algorithm finishes and discovered at the end that it crashed and all was to start again. You're allowed to laugh :D

Now the output file is loaded at the beginning, to make sure you'll not wait 15 minutes before it crashes.

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)

Thank you, you reminded me that I had not tested my mod on a map without rivers since I added them. The code wasn't checking whether rivers are enabled and expected to use a river map that didn't exist. Fixed now.

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.

[TumeniNodes]

yes, it was throwing some nil error related to a layer, etc.. glad it's fixed. You're quick

found something which might work well?
http://srtm.csi.cgiar.org/
and then to this from that link http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

the extracted file for the Western map is a bit over 8gb.
I am not sure what that will equal in size when converted but, I have to be careful as I only have 11.5 gb right now after extracting it... (I need to clean out some space)

be sure to read the use terms

[RSLRedstonier]

WOW
this
is
amazing

[Gael de Sailly]

found something which might work well?
http://srtm.csi.cgiar.org/

Yeah SRTM is the one I always use. But the resampled SRTM of the second link contains a single image with over 5GB for Europe/Asia (I think it's the same for other continents), that is not convenient to work with. If I use image_convert with this, my computer will just... die!
The one I use is http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp. You can download 6001x6001 images covering 5° lat/lon, with a resolution of 90m.

[Gael de Sailly]

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.

That's mostly done.
I found a latest minor bug this evening, it has been fixed. Now I consider that the river branch is stable enough to be in the download, so it's merged.
I've updated the screenshots (that was really fun!) and (quickly) the description. If you don't understand all the parameters, that's not a problem, they have coherent default values. I'll do further work on documentation this weekend.