[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

User avatar
TumeniNodes
Member
Posts: 2879
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Map generation from DEM [geo-mapgen] (WIP)

by TumeniNodes » Post

Great work Gael.
The new screenshots are really nice, especially the first one, it's beautiful.
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

I'm trying to learn to use QGIS and related Python libs that seems very interesting for what I'm trying to do.
QGIS is the only application I've found that has managed to open huge TIFF files without literally burning my RAM.

The second step is to know what to do with the loaded image.
Last edited by Gael de Sailly on Sun Jan 21, 2018 23:37, edited 1 time in total.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2879
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by TumeniNodes » Post

Looks promising, and like a really great project (meaning QGIS)

Do you think the mtmapprune could be incorporated somehow, eventually to scale large maps?
viewtopic.php?f=14&t=18579

I fully understand it is probably way to early to know, I'm just wondering.
Last edited by TumeniNodes on Sun Jan 28, 2018 21:24, edited 1 time in total.
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Inocudom
Member
Posts: 3098
Joined: Sat Sep 29, 2012 01:14
IRC: Inocudom
In-game: Inocudom

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Inocudom » Post

There are many possibilities with a mod like this. Why, I could end up with a map based on the mountainous terrain surrounding Great Cacapon, WV.
You can now find my videos at BitChute: https://www.bitchute.com/channel/some_cheeky_jinuskian/

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

I found something interesting with GDAL library. It's able to crop, resample and change the projection of a raster image (through function "Warp"). It could allow using several images of different projections, and resample them to use only the part of the world we want.

Potentially good data source for biomes: Corine Land Cover (in Europe). I suppose (and I hope) there are equivalent data for other parts of the world. IIRC bobomb found one for United States when he developed RealTerrain some years ago.
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

Today I've worked on graphical interface:
Image
Code is on tkinter branch. EDIT: merged
You need tkinter library to use it.

Command to start on graphical mode:

Code: Select all

./image_convert.py -g
For now only the input is on GUI. The output is still on the console.
Just realize how bored we would be if the world was perfect.

User avatar
TumeniNodes
Member
Posts: 2879
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by TumeniNodes » Post

Very very nice Gael de Sailly
this makes it very much easier to work with
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

I finally managed to get my Python script working with the GDAL library. It was difficult. I've pushed it on a new branch: https://github.com/Gael-de-Sailly/geo-mapgen/tree/gdal.
GDAL allows to manipulate and change the projection of geographical data like the DEMs this script uses.
I did this because it's an essential feature for a more long-run project: using biome maps like Corine Land Cover, superimposed on the heightmap, that may be in a different projection.
The script as it is now could deal with several maps in different projections with very few more work.
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

New branch with good changes: it's now able to handle land cover images!
https://github.com/Gael-de-Sailly/geo-m ... /landcover
For now only the ground nodes are changed, but decorations are planned.

See the result on the Mont Blanc (I've not taken the time to make more good screenshots for now)
Image

However, it's quite complicated to generate such maps. You need to find an heightmap like before, and also a raster land cover image covering the same region (this is already hard to find). But I faced a major problem: land cover maps are not standardized, and according to the organization who provide the map, the map legend can be different. You must write a file that associates each value with one of the biomes defined in biomes.csv. See the file "clc.lct" that matches Corine Land Cover.

I'll explain this in more details when it will be finished. I think I should even make a complete tutorial for a very special mod like this one, but that's something hard to do for me. I always have difficulties in explaining my work to others.
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by Gael de Sailly » Post

News of the day, on landcover branch:
Trees and plants, according to the biome!

Look at the Mont Blanc now:
Image
Just realize how bored we would be if the world was perfect.

slemon
Member
Posts: 111
Joined: Sun Apr 27, 2014 03:56

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by slemon » Post

This looks great!

User avatar
Fixer
Member
Posts: 899
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Fixer » Post

Mount Blanc looks nice, can you make it 1:1 scale? That will be epic for me, I will waste my time to conquer it :D

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by ExeterDad » Post

This is very exciting stuff! Although I wouldn't let players dig a thing. It's too pretty.

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

Fixerol wrote:Mount Blanc looks nice, can you make it 1:1 scale? That will be epic for me, I will waste my time to conquer it :D
Unfortunately I don't think you can find free data that are accurate enough. The best I know is Copernicus EU-DEM 1.1 (resolution 25m). It's possible to rescale it to 1 node = 1m, but it would be completely smoothed, and you will not see far.
Just realize how bored we would be if the world was perfect.

User avatar
Fixer
Member
Posts: 899
Joined: Sun Jul 31, 2011 11:23
IRC: Fixer
In-game: Fixer
Location: Ukraine

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Fixer » Post

I'm fine with approximated version

User avatar
azekill_DIABLO
Member
Posts: 7497
Joined: Wed Oct 29, 2014 20:05
GitHub: azekillDIABLO
In-game: azekill_DIABLO
Location: OMICRON
Contact:

Re: [Mod] Mapgen from geographical data [geo-mapgen] (WIP)

by azekill_DIABLO » Post

Gael de Sailly wrote:QGIS is the only application I've found that has managed to open huge TIFF files without literally burning my RAM.
I have some French docs on Qgis because I used it a while ago! PM me if you need help!

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

azekill_DIABLO wrote:I have some French docs on Qgis because I used it a while ago! PM me if you need help!
Thanks, but I have found what I was looking for, with GDAL (Geospatial Data Abstraction Library). It's the library used by Qgis to open raster layers. The python script is currently using it, works well.
Just realize how bored we would be if the world was perfect.

Chewy509
New member
Posts: 5
Joined: Sun Jan 29, 2017 10:05
GitHub: dkartaschew

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Chewy509 » Post

Excellent mod. Been able to do a few areas with data from SRTM, however encountered an area where it fails.

I've posted bug reports to github for tracking. https://github.com/Gael-de-Sailly/geo-mapgen/issues

User avatar
BirgitLachner
Member
Posts: 391
Joined: Thu May 05, 2016 10:18
In-game: Bibs

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by BirgitLachner » Post

Very nice ... I was already looking for Real Terrain but I need the data.

User avatar
TumeniNodes
Member
Posts: 2879
Joined: Fri Feb 26, 2016 19:49
GitHub: TumeniNodes
IRC: tumeninodes
In-game: TumeniNodes
Location: in the dark recesses of the mind
Contact:

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by TumeniNodes » Post

Gael de Sailly,
Thought you might be interested in this:
https://www.michaelfogleman.com/projects/terrarium/
Ich mag keine grünen Eier und Schinken, ich mag sie nicht Sam I Am

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by ChimneySwift » Post

Unfortunately I get an error on 5.0-dev Win64 https://pastebin.com/1QRa8RxZ

Any ideas?
A spoon is basically a tiny bowl with a stick on it

User avatar
Gael de Sailly
Member
Posts: 794
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

It fails to decompress data while reading a chunk. There is 2 reasons I can see: there may be an error in heightmap.dat file structure, or the mod is trying to read the file from the wrong position.
What map were you using?
Just realize how bored we would be if the world was perfect.

User avatar
ChimneySwift
Member
Posts: 320
Joined: Fri Sep 22, 2017 06:46
GitHub: ChimneySwift
IRC: ChimneySwift
In-game: ChimneySwift
Location: 127.0.0.1

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by ChimneySwift » Post

Gael de Sailly wrote:It fails to decompress data while reading a chunk. There is 2 reasons I can see: there may be an error in heightmap.dat file structure, or the mod is trying to read the file from the wrong position.
What map were you using?
I tried both of the demo heightmaps and a height map I generated from a section of Australia from http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp
A spoon is basically a tiny bowl with a stick on it

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

It's great to see that you have taken this approach. In fact I had abandoned realterrain because I came to realize that the better approach was indeed to do something like this where you use QGIS or python GDAL to prebuild the map using that minetest-lib python library, but that had issues too and your idea to build an intermediate database format that the lua mapgen can read directly is probably the only way this will ever work. I also realized that realterrain would have to give up on using the in-game form-based mapgen settings tool I had built, which was actually a huge part of the work, for an out-of-game interface. I tinkered with the idea of building a form in tkinter or something to simulate a "worldpaint" type of approach, but ultimately, why not just build a QGIS plugin to take any variety of geographic source files and output to your database format... the only thing remaining is how to give the user the tools to map biomes to LCLU (land cover land use) source data. Anyway I am going to get up to speed with what you have built here and try to contribute.

ps: as for huge DEM files, that is why imagemagick was the best image library: it didn't load the entire image into memory to extract pixel data... but it would be way better to use GDAL to extract this data anyway so that you can scale and distort it using map projections rather than image tools. ideally, this project should get to the point where it can handle vector source files and not just rasters. For example British Columbia has "biomes" called biogeoclimatic zones which are freely available as shapefiles.

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

it would be nice to have real world coordinates available to the player in the game but I can't think how to do it best.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests