[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

by Gael de Sailly » Post

Hello,

This mod aims to generate Minetest worlds from real topographical data (Digital Elevation Model).

I've been working for a while trying to generate maps of real places of the world, especially working with bobomb on his Real Terrain mod that was a great experiment, but faced huge stability problems because of the external libraries we needed to run in Lua to read the DEM image, or the hacky method that was implemented to read the image in pure Lua.

Here I come with a different approach. The image is not read in Lua but by a separate Python file (image_convert.py) that deals with the image and generate a database in the specified world directory, that is easily readable by Minetest natively.

This has the advantage of providing a very stable and proper code for the Lua mapgen.
This has the drawback of needing to prepare the world by running this Python file before.

This mod doesn't come with pre-loaded geographical data, it only provides the tools to use them. You need to find your data on the Internet. But I can help you :)
Spoiler
Make sure you have a working Python 3 installation, and install numpy, gdal and tkinter libraries. Also, using recent 0.5.0-dev Minetest is recommended.

To generate your map:
  1. Find DEM (Digital Elevation Model) data of the region you want (I personally use SRTM that has one pixel every 93m). Download it in image format (usually GeoTiff).

    You can use additional maps for land cover and rivers, this is explained in the README.
  2. Launch Minetest, create a world and active this mod. Don't start the game for now!
  3. Use the python script provided by this mod (image_convert.py) to generate the database, by calling it like this:

    Code: Select all

    ./image_convert.py
    This opens a configuration window with all the options you may need for your map.
    Image

    You might find some options a bit complicated, but you don't necessarily need to understand and tune everything. They are explained in the README.

    It generates 2 files in the world dir: heightmap.dat which is the database itself, and heightmap.dat.conf, configuration file easily readable with text editor that allow to change some parameters (currently only scale is supported).
  4. Now start the game and enjoy!
You can also download pre-built databases at Geo Mapgen demos repository. Place the file in the Minetest world directory and rename it "heightmap.dat".

I hope this is clear enough. Please say it if you have any difficulty.
I've generated a worldwide map using NOAA Global DEM for elevation, and MODIS-based Global Land Cover for land cover data. You can download the database directly at this link (place it in your Minetest world directory and rename it "heightmap.dat"). The screenshots show the world with a resolution of 2km horizontally, and 200m vertically.
Image^ France and southern England
Spoiler
Image^ Indian Ocean with a view on Réunion at the center left, Mauritius at bottom right, and Madagascar at the horizon.

Image^ British Isles

Image^ Northern India. You're above the Ganga Plain, looking eastwards. You can make out the Ganges river bed at the right foreground, the Brahmaputra at the background, both converging in the huge Bengal Delta at the extreme right.
At the left the forests are gradually climbing up the Himalayas chain, until you reach the highest summits. The icy peak at the extreme left is the legendary Mount Everest. Beyond this mountain barrier stands the Tibet Plateau.

Image^ Iceland, with the famous Vatnajökull Glacier at the right background.
Spoiler
They have been taken using SRTM tiles with a resolution of 93m horizontally and 40m vertically.Image^ Grand Canyon, USA, generated using SRTM tile #14-05, rivers enabled.

Image^ Mont Blanc, France, generated using tile #38-03, rivers disabled.

Image^ Great view over the Doubs valley, (France/Switzerland), and its curious hairpinned loop. Tile #38-03

Image^ Ardèche gorge using tile #37-04, French canyon very popular for kayaking (I did it this autumn and that's really something I'll remember!!)
I'll obviously make a screenshot with rivers enabled as soon as possible.

Image^ Fuji mountain in Japan, tile #64-05

Image^ Saar Loop in Germany, very nice landscape on tile #38-03. Screenshot needs to be updated too with the new rivers feature.
Where to find the mod:
Download (Code on GitHub)
License: LGPL v2.1

This mod is still experimental and is not ready for regular use. The database format, especially, could change in future versions without backward compatibility.
I encourage you to report me any bug, or suggestion for the code (I think it could be optimized a lot). Any contribution would be appreciated.
Last edited by Gael de Sailly on Sun Mar 22, 2020 09:06, edited 14 times in total.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Hey Gael,
I am so glad you are still working on this. It looks great.

I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?

I am also amazed at how minimal the options are in finding geotiff images of the world (very few regions)
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:I am having an issue, however.

Code: Select all

 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'
I do have the imageio libs installed so, not sure where my issue is?
Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote: Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Ah..., I am a goof.
Got it now, thank you.
I should have everything ready now, but it is very late for me

Will attempt again tomorrow and give feedback
A Wonderful World

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Well, it works
(I had no doubts) ; )

But, maybe I should familiarize myself a bit better with the scaling (because I was way off. :P
Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

The image I used is from: https://asterweb.jpl.nasa.gov/images/GDEM-10km-BW.tif

This was my result, because I was lazy and simply used your scaling for the maps you had
Image

But, this is awesome, and I will tinker with it more. Thank you

I also uploaded the .tif, and the resulting world down there v
Attachments
geomap.zip
(376.73 KiB) Downloaded 181 times
gdem_10km_BW2.tif
gdem_10km_BW2.tif (46 KiB) Viewed 3473 times
First GeoMapgen.jpg
First GeoMapgen.jpg (313.07 KiB) Viewed 3473 times
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I think something is wrong with my code, your map clearly don't match the image. That can't be due to the scale parameter that only sets vertical scale and son't modify shapes at all. I think this is because of the fragmentation: during converting, the image is cut into square chunks, but I've never thought about what would happen if the whole image is smaller than one chunk. That shouldn't be too hard to fix.

The other problem is with your image: the grey shade here represents slope (derivative value of elevation) and not elevation itself. You would get a map with recognizable shapes but completely wrong elevation and slopes.
TumeniNodes wrote:Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise
Yes it uses singlenode but anyway if you use a different mapgen in your world, the mod automatically set it on singlenode.
Just realize how bored we would be if the world was perfect.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Fixer » Post


User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

Thank you Fixerol for the links :)
I've used the third one for my screenshots. It covers the whole world with one pixel every 93m.

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed). Just run image_convert.py again and delete map.sqlite.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote: The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed).
You're fast Gael, thank you : )

and yes, thank you Fixerol

another link which may be helpful for people, makes it a bit easier to get coords if you want to enter them yourself:
https://www.findlatitudeandlongitude.com/

I will refresh the mod

This is really a great start so far though... nice work
A Wonderful World

User avatar
duane
Member
Posts: 1715
Joined: Wed Aug 19, 2015 19:11
GitHub: duane-r
Location: Oklahoma City
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by duane » Post

This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Believe in people and you don't need to believe anything else.

User avatar
Fixer
Member
Posts: 904
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Fixer » Post

Yeah, handling rivers and oceans would be nice and even setting your own water level.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

duane wrote:This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Rivers are planned, I have already written the algorithm on python side: rivermapper.py, that for now transforms a SRTM tile to a river map.
The algorithm is slow (20min for a SRTM tile on my laptop), I'm looking for optimizations.
If this script works well, I think rivers could be implemented in the mapgen without major difficulties.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Would be nice to have 2 water level settings

The current/existing one for ocean
a second for lakes, which streams/rivers can run down from

It would also be nice to have water and river water the same color :P (but that is an offtopic matter in relation to this) sorry.
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes: I think I have what you were looking for.
https://www.eea.europa.eu/data-and-maps ... del-etopo5
Download in GeoTiff format.
You'll find the USA at 1100 -600.

(EDIT: coordinates were wrong)
Last edited by Gael de Sailly on Sun Jan 14, 2018 13:29, edited 1 time in total.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Nice find Gael, thank you.
Gael de Sailly wrote:image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
it's witchcraft is what it is. That is a nice performance boost.
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I'm working on rivers on a new branch.
They should work now, although the code is not very stable for now.

Download
Code

You can try by adding the parameter "-r 2000" when launching the py script (this parameter represents the surface needed for catchment area to be drawn as a river, so decrease it if you want more rivers).

River calculation is EXTREMELY slow for now. It can easily take >15 minutes for a big image, so please be patient.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Have you converted that full world map yet?

For me, I get an error stating no .dat file is created.

Code: Select all

~/.minetest/mods/geo_mapgen $ ./image_convert.py '/home/paul/dem/alwdgg.tif' '/home/paul/worlds/geomap03' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 101, in <module>
    file_output = open(fpath_output, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/worlds/geomap03/heightmap.dat'
I am using your first branch.
Also, is there a specific scale you recommend? (as you can tell, I know nothing related to -f / -s scaling)

I was tempted to run convert with no vertical scaling but, not sure if that is recommended :P

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

Are you sure that the output path is correct? There is no ".minetest".

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Gael de Sailly wrote:Are you sure that the output path is correct? There is no ".minetest".
Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Gael de Sailly wrote:In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Thank you, that makes sense now.
I will try it again, only this time I won't forget a major path to the directory :D
A Wonderful World

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?
Don't worry, that has just happened to me today! For my part, I forgot "/worlds". I was waiting until the river algorithm finishes and discovered at the end that it crashed and all was to start again. You're allowed to laugh :D

Now the output file is loaded at the beginning, to make sure you'll not wait 15 minutes before it crashes.
TumeniNodes wrote:When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
Thank you, you reminded me that I had not tested my mod on a map without rivers since I added them. The code wasn't checking whether rivers are enabled and expected to use a river map that didn't exist. Fixed now.

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2941
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

yes, it was throwing some nil error related to a layer, etc.. glad it's fixed. You're quick

found something which might work well?
http://srtm.csi.cgiar.org/
and then to this from that link http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

the extracted file for the Western map is a bit over 8gb.
I am not sure what that will equal in size when converted but, I have to be careful as I only have 11.5 gb right now after extracting it... (I need to clean out some space)

be sure to read the use terms
A Wonderful World

User avatar
TechNolaByte
Member
Posts: 465
Joined: Wed May 10, 2017 21:00
GitHub: TechNolaByte

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TechNolaByte » Post

WOW
this
is
amazing
The great quest of alchemy neither failed nor attained gold; programmers are those great alchemists who transmute caffeine into code.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

TumeniNodes wrote:found something which might work well?
http://srtm.csi.cgiar.org/
Yeah SRTM is the one I always use. But the resampled SRTM of the second link contains a single image with over 5GB for Europe/Asia (I think it's the same for other continents), that is not convenient to work with. If I use image_convert with this, my computer will just... die!
The one I use is http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp. You can download 6001x6001 images covering 5° lat/lon, with a resolution of 90m.
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 845
Joined: Sun Jan 26, 2014 17:01
GitHub: gaelysam
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly gaelysam
Location: Voiron, France

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Post

I wrote:Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
That's mostly done.
I found a latest minor bug this evening, it has been fixed. Now I consider that the river branch is stable enough to be in the download, so it's merged.
I've updated the screenshots (that was really fun!) and (quickly) the description. If you don't understand all the parameters, that's not a problem, they have coherent default values. I'll do further work on documentation this weekend.
Just realize how bored we would be if the world was perfect.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Majestic-12 [Bot] and 7 guests