Page 2 of 3

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

PostPosted: Thu Jan 18, 2018 02:23
by TumeniNodes
Great work Gael.
The new screenshots are really nice, especially the first one, it's beautiful.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sun Jan 21, 2018 21:44
by Gael de Sailly
I'm trying to learn to use QGIS and related Python libs that seems very interesting for what I'm trying to do.
QGIS is the only application I've found that has managed to open huge TIFF files without literally burning my RAM.

The second step is to know what to do with the loaded image.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sun Jan 21, 2018 22:50
by TumeniNodes
Looks promising, and like a really great project (meaning QGIS)

Do you think the mtmapprune could be incorporated somehow, eventually to scale large maps?
viewtopic.php?f=14&t=18579

I fully understand it is probably way to early to know, I'm just wondering.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sun Jan 21, 2018 23:40
by Inocudom
There are many possibilities with a mod like this. Why, I could end up with a map based on the mountainous terrain surrounding Great Cacapon, WV.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sun Jan 28, 2018 21:20
by Gael de Sailly
I found something interesting with GDAL library. It's able to crop, resample and change the projection of a raster image (through function "Warp"). It could allow using several images of different projections, and resample them to use only the part of the world we want.

Potentially good data source for biomes: Corine Land Cover (in Europe). I suppose (and I hope) there are equivalent data for other parts of the world. IIRC bobomb found one for United States when he developed RealTerrain some years ago.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sat Feb 03, 2018 23:01
by Gael de Sailly
Today I've worked on graphical interface:
Image
Code is on tkinter branch. EDIT: merged
You need tkinter library to use it.

Command to start on graphical mode:
Code: Select all
./image_convert.py -g


For now only the input is on GUI. The output is still on the console.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Sun Feb 04, 2018 19:15
by TumeniNodes
Very very nice Gael de Sailly
this makes it very much easier to work with

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Mon Feb 19, 2018 00:31
by Gael de Sailly
I finally managed to get my Python script working with the GDAL library. It was difficult. I've pushed it on a new branch: https://github.com/Gael-de-Sailly/geo-mapgen/tree/gdal.
GDAL allows to manipulate and change the projection of geographical data like the DEMs this script uses.
I did this because it's an essential feature for a more long-run project: using biome maps like Corine Land Cover, superimposed on the heightmap, that may be in a different projection.
The script as it is now could deal with several maps in different projections with very few more work.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Tue Feb 20, 2018 23:44
by Gael de Sailly
New branch with good changes: it's now able to handle land cover images!
https://github.com/Gael-de-Sailly/geo-mapgen/tree/landcover
For now only the ground nodes are changed, but decorations are planned.

See the result on the Mont Blanc (I've not taken the time to make more good screenshots for now)
Image

However, it's quite complicated to generate such maps. You need to find an heightmap like before, and also a raster land cover image covering the same region (this is already hard to find). But I faced a major problem: land cover maps are not standardized, and according to the organization who provide the map, the map legend can be different. You must write a file that associates each value with one of the biomes defined in biomes.csv. See the file "clc.lct" that matches Corine Land Cover.

I'll explain this in more details when it will be finished. I think I should even make a complete tutorial for a very special mod like this one, but that's something hard to do for me. I always have difficulties in explaining my work to others.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Wed Feb 21, 2018 15:44
by Gael de Sailly
News of the day, on landcover branch:
Trees and plants, according to the biome!

Look at the Mont Blanc now:
Image

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Mon Feb 26, 2018 07:22
by slemon
This looks great!

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Mon Mar 05, 2018 22:32
by Fixer
Mount Blanc looks nice, can you make it 1:1 scale? That will be epic for me, I will waste my time to conquer it :D

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Mar 06, 2018 00:46
by ExeterDad
This is very exciting stuff! Although I wouldn't let players dig a thing. It's too pretty.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Mar 06, 2018 18:54
by Gael de Sailly
Fixerol wrote:Mount Blanc looks nice, can you make it 1:1 scale? That will be epic for me, I will waste my time to conquer it :D

Unfortunately I don't think you can find free data that are accurate enough. The best I know is Copernicus EU-DEM 1.1 (resolution 25m). It's possible to rescale it to 1 node = 1m, but it would be completely smoothed, and you will not see far.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Mar 06, 2018 22:17
by Fixer
I'm fine with approximated version

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

PostPosted: Fri Mar 09, 2018 11:51
by azekill_DIABLO
Gael de Sailly wrote:QGIS is the only application I've found that has managed to open huge TIFF files without literally burning my RAM.

I have some French docs on Qgis because I used it a while ago! PM me if you need help!

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Fri Mar 09, 2018 19:29
by Gael de Sailly
azekill_DIABLO wrote:I have some French docs on Qgis because I used it a while ago! PM me if you need help!

Thanks, but I have found what I was looking for, with GDAL (Geospatial Data Abstraction Library). It's the library used by Qgis to open raster layers. The python script is currently using it, works well.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Sat Mar 10, 2018 09:09
by Chewy509
Excellent mod. Been able to do a few areas with data from SRTM, however encountered an area where it fails.

I've posted bug reports to github for tracking. https://github.com/Gael-de-Sailly/geo-mapgen/issues

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Mar 13, 2018 20:19
by BirgitLachner
Very nice ... I was already looking for Real Terrain but I need the data.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Mon Jul 16, 2018 04:18
by TumeniNodes
Gael de Sailly,
Thought you might be interested in this:
https://www.michaelfogleman.com/projects/terrarium/

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Sep 04, 2018 00:46
by ChimneySwift
Unfortunately I get an error on 5.0-dev Win64 https://pastebin.com/1QRa8RxZ

Any ideas?

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Sep 04, 2018 07:19
by Gael de Sailly
It fails to decompress data while reading a chunk. There is 2 reasons I can see: there may be an error in heightmap.dat file structure, or the mod is trying to read the file from the wrong position.
What map were you using?

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Tue Sep 04, 2018 07:42
by ChimneySwift
Gael de Sailly wrote:It fails to decompress data while reading a chunk. There is 2 reasons I can see: there may be an error in heightmap.dat file structure, or the mod is trying to read the file from the wrong position.
What map were you using?


I tried both of the demo heightmaps and a height map I generated from a section of Australia from http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Sun Oct 14, 2018 14:57
by bobomb
It's great to see that you have taken this approach. In fact I had abandoned realterrain because I came to realize that the better approach was indeed to do something like this where you use QGIS or python GDAL to prebuild the map using that minetest-lib python library, but that had issues too and your idea to build an intermediate database format that the lua mapgen can read directly is probably the only way this will ever work. I also realized that realterrain would have to give up on using the in-game form-based mapgen settings tool I had built, which was actually a huge part of the work, for an out-of-game interface. I tinkered with the idea of building a form in tkinter or something to simulate a "worldpaint" type of approach, but ultimately, why not just build a QGIS plugin to take any variety of geographic source files and output to your database format... the only thing remaining is how to give the user the tools to map biomes to LCLU (land cover land use) source data. Anyway I am going to get up to speed with what you have built here and try to contribute.

ps: as for huge DEM files, that is why imagemagick was the best image library: it didn't load the entire image into memory to extract pixel data... but it would be way better to use GDAL to extract this data anyway so that you can scale and distort it using map projections rather than image tools. ideally, this project should get to the point where it can handle vector source files and not just rasters. For example British Columbia has "biomes" called biogeoclimatic zones which are freely available as shapefiles.

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

PostPosted: Sun Oct 14, 2018 16:44
by bobomb
it would be nice to have real world coordinates available to the player in the game but I can't think how to do it best.