[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

by Gael de Sailly » Wed Jan 10, 2018 23:36

Hello,

This mod aims to generate Minetest worlds from real topographical data (Digital Elevation Model).

I've been working for a while trying to generate maps of real places of the world, especially working with bobomb on his Real Terrain mod that was a great experiment, but faced huge stability problems because of the external libraries we needed to run in Lua to read the DEM image, or the hacky method that was implemented to read the image in pure Lua.

Here I come with a different approach. The image is not read in Lua but by a separate Python file (image_convert.py) that deals with the image and generate a database in the specified world directory, that is easily readable by Minetest natively.

This has the advantage of providing a very stable and proper code for the Lua mapgen.
This has the drawback of needing to prepare the world by running this Python file before.

This mod doesn't come with pre-loaded geographical data, it only provides the tools to use them. You need to find your data on the Internet. But I can help you :)

+ HOW TO USE THIS MOD

I've generated a worldwide map using NOAA Global DEM for elevation, and MODIS-based Global Land Cover for land cover data. You can download the database directly at this link (place it in your Minetest world directory and rename it "heightmap.dat"). The screenshots show the world with a resolution of 2km horizontally, and 200m vertically.
Image^ France and southern England
+ More screenshots

+ Old screenshots

Where to find the mod:
Download (Code on GitHub)
License: LGPL v2.1

This mod is still experimental and is not ready for regular use. The database format, especially, could change in future versions without backward compatibility.
I encourage you to report me any bug, or suggestion for the code (I think it could be optimized a lot). Any contribution would be appreciated.
Last edited by Gael de Sailly on Fri Feb 23, 2018 23:28, edited 13 times in total.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Thu Jan 11, 2018 05:14

Hey Gael,
I am so glad you are still working on this. It looks great.

I am having an issue, however.
Code: Select all
 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'


I do have the imageio libs installed so, not sure where my issue is?

I am also amazed at how minimal the options are in finding geotiff images of the world (very few regions)
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Thu Jan 11, 2018 06:54

TumeniNodes wrote:I am having an issue, however.
Code: Select all
 ~/.minetest/mods/geo_mapgen $ ./image_convert.py 'home/paul/DEM/srtm_38_04.tif' '/home/paul/.minetest/worlds/geomap' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 5, in <module>
    import imageio
ImportError: No module named 'imageio'


I do have the imageio libs installed so, not sure where my issue is?

Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Thu Jan 11, 2018 07:26

Gael de Sailly wrote:Have you installed imageio in python 3? (Python 2 and 3 are independant). If you want to install it by pip, you should install python3-pip and then run pip3 install imageio.


Ah..., I am a goof.
Got it now, thank you.
I should have everything ready now, but it is very late for me

Will attempt again tomorrow and give feedback
I'm not your buddy, guy...
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Thu Jan 11, 2018 18:20

Well, it works
(I had no doubts) ; )

But, maybe I should familiarize myself a bit better with the scaling (because I was way off. :P
Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

The image I used is from: https://asterweb.jpl.nasa.gov/images/GDEM-10km-BW.tif

This was my result, because I was lazy and simply used your scaling for the maps you had
Image

But, this is awesome, and I will tinker with it more. Thank you

I also uploaded the .tif, and the resulting world down there v
Attachments
geomap.zip
(376.73 KiB) Downloaded 9 times
gdem_10km_BW2.tif
(46 KiB) Not downloaded yet
First GeoMapgen.jpg
(313.07 KiB) Not downloaded yet
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Thu Jan 11, 2018 21:40

I think something is wrong with my code, your map clearly don't match the image. That can't be due to the scale parameter that only sets vertical scale and son't modify shapes at all. I think this is because of the fragmentation: during converting, the image is cut into square chunks, but I've never thought about what would happen if the whole image is smaller than one chunk. That shouldn't be too hard to fix.

The other problem is with your image: the grey shade here represents slope (derivative value of elevation) and not elevation itself. You would get a map with recognizable shapes but completely wrong elevation and slopes.

TumeniNodes wrote:Also, as a note..., I created the world using single node mapgen.
I figured this would be the best option, unless you recommend otherwise

Yes it uses singlenode but anyway if you use a different mapgen in your world, the mod automatically set it on singlenode.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
Fixer
Member
 
Posts: 866
Joined: Sun Jul 31, 2011 11:23
Location: Ukraine
IRC: Fixer
In-game: Fixer
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Fri Jan 12, 2018 00:32


Thank you Fixerol for the links :)
I've used the third one for my screenshots. It covers the whole world with one pixel every 93m.

The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed). Just run image_convert.py again and delete map.sqlite.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Fri Jan 12, 2018 00:45

Gael de Sailly wrote:The problem that TumeniNodes found is fixed now, please update the mod, and the worlds (the database structure has slightly changed).


You're fast Gael, thank you : )

and yes, thank you Fixerol

another link which may be helpful for people, makes it a bit easier to get coords if you want to enter them yourself:
https://www.findlatitudeandlongitude.com/

I will refresh the mod

This is really a great start so far though... nice work
I'm not your buddy, guy...
 

User avatar
duane
Member
 
Posts: 1274
Joined: Wed Aug 19, 2015 19:11
Location: Oklahoma City
GitHub: duane-r

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by duane » Fri Jan 12, 2018 07:03

This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.
Believe in people and you don't need to believe anything else.
 

User avatar
Fixer
Member
 
Posts: 866
Joined: Sun Jul 31, 2011 11:23
Location: Ukraine
IRC: Fixer
In-game: Fixer

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Fixer » Fri Jan 12, 2018 17:59

Yeah, handling rivers and oceans would be nice and even setting your own water level.
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Fri Jan 12, 2018 18:00

duane wrote:This looks very cool. Make a good collection of real world maps and you won't need to bother with mapgens anymore. I can't wait to see how you deal with water features.

Rivers are planned, I have already written the algorithm on python side: rivermapper.py, that for now transforms a SRTM tile to a river map.
The algorithm is slow (20min for a SRTM tile on my laptop), I'm looking for optimizations.
If this script works well, I think rivers could be implemented in the mapgen without major difficulties.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Fri Jan 12, 2018 18:05

Would be nice to have 2 water level settings

The current/existing one for ocean
a second for lakes, which streams/rivers can run down from

It would also be nice to have water and river water the same color :P (but that is an offtopic matter in relation to this) sorry.
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Fri Jan 12, 2018 20:32

image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Fri Jan 12, 2018 21:57

TumeniNodes: I think I have what you were looking for.
https://www.eea.europa.eu/data-and-maps/data/world-digital-elevation-model-etopo5
Download in GeoTiff format.
You'll find the USA at 1100 -600.

(EDIT: coordinates were wrong)
Last edited by Gael de Sailly on Sun Jan 14, 2018 13:29, edited 1 time in total.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Sat Jan 13, 2018 02:36

Nice find Gael, thank you.

Gael de Sailly wrote:image_convert.py has been sped up a lot by using BytesIO instead of bytes concatenation. My computer now takes 7 seconds to convert a 6000² px image instead of 65 seconds before!


it's witchcraft is what it is. That is a nice performance boost.
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Sat Jan 13, 2018 15:25

I'm working on rivers on a new branch.
They should work now, although the code is not very stable for now.

Download
Code

You can try by adding the parameter "-r 2000" when launching the py script (this parameter represents the surface needed for catchment area to be drawn as a river, so decrease it if you want more rivers).

River calculation is EXTREMELY slow for now. It can easily take >15 minutes for a big image, so please be patient.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Tue Jan 16, 2018 03:55

Have you converted that full world map yet?

For me, I get an error stating no .dat file is created.
Code: Select all
~/.minetest/mods/geo_mapgen $ ./image_convert.py '/home/paul/dem/alwdgg.tif' '/home/paul/worlds/geomap03' -f 100 -s 90
Traceback (most recent call last):
  File "./image_convert.py", line 101, in <module>
    file_output = open(fpath_output, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/worlds/geomap03/heightmap.dat'


I am using your first branch.
Also, is there a specific scale you recommend? (as you can tell, I know nothing related to -f / -s scaling)

I was tempted to run convert with no vertical scaling but, not sure if that is recommended :P

When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Tue Jan 16, 2018 07:00

Are you sure that the output path is correct? There is no ".minetest".

In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Tue Jan 16, 2018 10:19

Gael de Sailly wrote:Are you sure that the output path is correct? There is no ".minetest".

Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?

Gael de Sailly wrote:In practice, -f does not change anything, it's just a matter of internal representation of the map.
But -s will change the height of the hills/mountains. -s is 40 by default, you don't have to specify it, unless you want a different value. This means that one node height on the map equals to 40 meters in reality.

Thank you, that makes sense now.
I will try it again, only this time I won't forget a major path to the directory :D
I'm not your buddy, guy...
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Tue Jan 16, 2018 23:33

TumeniNodes wrote:Ohhh..., "duh" haha, I am so sorry. That was embarrasing. :P How the heck did I miss that?

Don't worry, that has just happened to me today! For my part, I forgot "/worlds". I was waiting until the river algorithm finishes and discovered at the end that it crashed and all was to start again. You're allowed to laugh :D

Now the output file is loaded at the beginning, to make sure you'll not wait 15 minutes before it crashes.

TumeniNodes wrote:When I attempted using your rivers branch the game crashed in under one minute.
(though I did not use the parameter -r 2000)

Thank you, you reminded me that I had not tested my mod on a map without rivers since I added them. The code wasn't checking whether rivers are enabled and expected to use a river map that didn't exist. Fixed now.

Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
TumeniNodes
Member
 
Posts: 2536
Joined: Fri Feb 26, 2016 19:49
Location: in the dark recesses of the mind
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Wed Jan 17, 2018 00:23

yes, it was throwing some nil error related to a layer, etc.. glad it's fixed. You're quick

found something which might work well?
http://srtm.csi.cgiar.org/
and then to this from that link http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

the extracted file for the Western map is a bit over 8gb.
I am not sure what that will equal in size when converted but, I have to be careful as I only have 11.5 gb right now after extracting it... (I need to clean out some space)

be sure to read the use terms
I'm not your buddy, guy...
 

User avatar
RSLRedstonier
Member
 
Posts: 443
Joined: Wed May 10, 2017 21:00
Location: some were in middle earth
GitHub: RSL-Redstonier
In-game: RSLRedstonier

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by RSLRedstonier » Wed Jan 17, 2018 01:01

WOW
this
is
amazing
"A programmer is just a tool which converts caffeine into code"

try out my skytest mod always being updated!
https://forum.minetest.net/viewtopic.php?f=9&t=17568
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Wed Jan 17, 2018 06:41

TumeniNodes wrote:found something which might work well?
http://srtm.csi.cgiar.org/

Yeah SRTM is the one I always use. But the resampled SRTM of the second link contains a single image with over 5GB for Europe/Asia (I think it's the same for other continents), that is not convenient to work with. If I use image_convert with this, my computer will just... die!
The one I use is http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp. You can download 6001x6001 images covering 5° lat/lon, with a resolution of 90m.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

User avatar
Gael de Sailly
Member
 
Posts: 670
Joined: Sun Jan 26, 2014 17:01
Location: Lyon, France
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by Gael de Sailly » Thu Jan 18, 2018 00:04

I wrote:Tomorrow my plan is to merge this branch, update my post (take a lot of screenshots) and the README.

That's mostly done.
I found a latest minor bug this evening, it has been fixed. Now I consider that the river branch is stable enough to be in the download, so it's merged.
I've updated the screenshots (that was really fun!) and (quickly) the description. If you don't understand all the parameters, that's not a problem, they have coherent default values. I'll do further work on documentation this weekend.
Contributor of Minetest and OpenStreetMap, on Ubuntu 16.04. ~ ~ ~ Meet me on MinetestForFun, VanessaE's Basic, Capture the Flag and Inside the Box.
 

Next

Return to WIP Mods



Who is online

Users browsing this forum: KCoombes and 4 guests