[Mod] Mapgen from geographical data [geo-mapgen] (Biomes!)

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

Another approach to cropping/scaling would be to define an extent using real world coordinates (manually entering long/lat or UTM coords), or selecting the extent from a shapefile that you have created for the purpose, or from the extent of one of your input files. Many QGIS tools already act this way. Then you can set your x/y/z scale and then using gdal you can sample the source files (possibly taking a neighborhood average) rather than handling them pixel by pixel. It's a different approach but it eliminates issues with large DEM files, resolution or extent mismatches, and lends itself to future features better (using vector inputs, querying map services, etc).

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

Is there any reason why this project shouldn't BE a QGIS tool? anybody interested in this kind of thing would be able to install and run QGIS. It would eliminate the need to make a custom tkInter GUI tool. it would bring people from the QGIS world to the minetest world.

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

users might find this useful (oldie but a goodie)
https://www.usna.edu/Users/oceano/pguth ... crodem.htm

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post


bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

On windows the best way to install QGIS and gdal is OSGEO4W
http://download.osgeo.org/osgeo4w/osgeo4w-setup.exe
This installs a newer version of QGIS than the default installed on most linux distros.
For linux you must install the unstable repo to get the most recent QGIS.
https://www.qgis.org/en/site/forusers/a ... html#linux
Basically, don't waste your time with QGIS 2.x

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

It would be nice to save the settings in the tkinter form. This would allow for saving presets for comparison and sharing. It would be nice if the final form of the mod shipped with some presets including some datasets so that right out of the box you can play on a world map, Canada map, France map, etc. DEMs are too big for this so the mod would have to work with smaller files (vector based files).

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

making a QGIS plugin looks easy: https://gis.ucar.edu/building-plugin-qgis

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

the final database format should include underground geoformations, soil layers, aquifers and also bathymetry.

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

Well... I was not expecting to have so many posts to read, let me answer your questions one by one.
bobomb wrote:it would be way better to use GDAL to extract this data anyway so that you can scale and distort it using map projections rather than image tools.
It is already using GDAL, btw you need to have this lib installed if you want to use the python script.
bobomb wrote:it would be nice to have real world coordinates available to the player in the game but I can't think how to do it best.
This would need to convert MT coordinates into lat/lon, using the projection. but the Lua has no access to GDAL. We could still implement a conversion function for several particular projections like Mercator.
bobomb wrote:Another approach to cropping/scaling would be to define an extent using real world coordinates (manually entering long/lat or UTM coords), or selecting the extent from a shapefile that you have created for the purpose, or from the extent of one of your input files. Many QGIS tools already act this way. Then you can set your x/y/z scale and then using gdal you can sample the source files (possibly taking a neighborhood average) rather than handling them pixel by pixel.
I'm not sure to understand everything but it seems that it's already what I do. I use GDAL to reproject and/or resample the image: https://github.com/Gael-de-Sailly/geo-m ... rm.py#L134
bobomb wrote:Is there any reason why this project shouldn't BE a QGIS tool? anybody interested in this kind of thing would be able to install and run QGIS. It would eliminate the need to make a custom tkInter GUI tool. it would bring people from the QGIS world to the minetest world.
This, is a very interesting way to explore, would make it much easier to install and allow a cleaner interface. This idea has already crossed my mind, but I had chosen to stick on plain Python until I have something working, and possibly turning it into a QGIS plugin after. But as always, once it was working, I spent my time exploring my worlds and forgot everything else xD
bobomb wrote:It would be nice to save the settings in the tkinter form. This would allow for saving presets for comparison and sharing.
Yea the content of the form should be stored somewhere, also to find again the same parameters on next use if you often use the same. Also would be nice to store the params in some metadata of the database.
bobomb wrote:It would be nice if the final form of the mod shipped with some presets including some datasets so that right out of the box you can play on a world map, Canada map, France map, etc. DEMs are too big for this so the mod would have to work with smaller files (vector based files).
I'm hesitant to pack big files in the download, that not everyone will use. This can discourage some people with very low internet from downloading the mod. I think we should always keep a light, purely functional download, and provide in addition, either easily downloadable presets, or another version including these presets.
bobomb wrote:the final database format should include underground geoformations, soil layers, aquifers and also bathymetry.
Bathymetry is already supported, just use a negative value for elevation. Current presets (Alps and High-res world) just doesn't use it because of the input data I was using. For the other features, it would need a 3D geological model, and I very highly doubt this could be found easily on the web. Maybe I can find some 2D data of geological land cover for France or US, there are already things to do with that.

Thank you very much for your interest. You have excellent ideas, and you are giving me motivation to work on this again (this project has been a bit abandoned too for the last 6 months). By being several at working on a project, we can move mountains, but literally in our case ;)
Just realize how bored we would be if the world was perfect.

bobomb
Member
Posts: 115
Joined: Sat May 23, 2015 20:28
GitHub: bobombolo
IRC: bobomb

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by bobomb » Post

thanks for replying to my many posts, i also am excited about this.
regarding the real world coordinates, if we stuck to UTM we could use simple maths to convert. if we wanted long/lat maybe we can embed the coordinates in param1 or param2 of a node every so often (the chunk corner?) and do some simple maths to at least approximate...

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

I'm now retaking this old –but not dying– project and I'm rewriting the Python side from scratch (it was a bit of a mess). The idea is to turn it into a flexible Python library that could be used either with standalone Python or with QGIS.

Also found this: https://luarocks.org/modules/geoffleyland/proj, a small Lua library for map projection. Doesn't seem actively developed but is working as is. It could be used to provide extra in-game features like teleporting to a location, or displaying real-world coordinates (would be optional anyway because I want the mod to be usable without extra Lua libs).

EDIT: new branch on github https://github.com/Gael-de-Sailly/geo-m ... ee/rewrite
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

I've been working on it quite a lot these days, and now I've got the rewritten Python side working as a library, you can find it on the rewrite branch. No user interface for now for this version but this concept will allow more simple and efficient work to develop one, that would ideally be based on QGIS.

Teleporting to real-world coordinates has been implemented, based on the proj library of my previous post. The database now contains georeferences to allow this.

And one screenshot: Sunset above Loch Awe and Ben Cruachan summit, Scottish Highlands, UK. 1 node is only 25 meters.
Image
Just realize how bored we would be if the world was perfect.

User avatar
Extex
Member
Posts: 184
Joined: Wed Mar 14, 2018 23:14
GitHub: Extex101
In-game: Extex

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Extex » Post

Will this work on android?
Player of PK, RIP. Creator of the light_tool and motorbike mods. Working on another mod which should be coming soon!

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

Extex wrote:Will this work on android?
I've never used Minetest on Android so I don't have an absolute answer, but probably the Lua part will work fine because it doesn't rely on anything external. However the pre-processing part needs a Python installation with Tkinter and GDAL libs (I don't know whether it's easy to set up on Android) and can be quite heavy (this depends on the size of your map). I would recommend to generate the file using a computer and then transfer the output file on Android, but if you think you can do it on Android, you can try too.

I plan to do a command-line only version that would be easier to use on some devices, the Tkinter backend becoming optional. IIRC I already have a start of it somewhere, I just need enough motivation to dig into it again.
Just realize how bored we would be if the world was perfect.

qu4ntumrush
New member
Posts: 2
Joined: Sat Feb 15, 2020 07:44
GitHub: qu4ntumrush
In-game: qu4ntumrush

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by qu4ntumrush » Post

Reposting from: viewtopic.php?f=47&t=15272

I want to use open-mapgen (posting here because the official thread is dead) for a school project and I have large DEMs (20 km wide) of Florida as GeoTIFFs I want to convert into Minetest maps. I get errors no matter which DEM I use or the mod's parameters including scaling down. Recently every error says this:

Code: Select all

zerr: invalid or incomplete deflate data
ModError: Failed to load and run script from C:\Games\minetest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:
decompressZlib: inflate failed
stack traceback:
	[C]: in function 'decompress'
	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:121: in main chunk
Check debug.txt for details.
I often try using a land cover and I may have had slightly more luck getting a map to load without it (I've lost track), but when it does load, it's completely flat so it must not be reading the DEM as a height map.

Other error messages I have gotten:

First error message I got when it was installed on my desktop before moving to C:\Games\Minetest\mods\:

Code: Select all

2020-02-11 02:24:06: ERROR[Main]: ModError: Failed to load and run script from C:\Users\[Me]\Desktop\minetest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:
2020-02-11 02:24:06: ERROR[Main]: ...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:34: attempt to index local 'str' (a nil value)
2020-02-11 02:24:06: ERROR[Main]: stack traceback:
2020-02-11 02:24:06: ERROR[Main]: 	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:34: in function 'parse'
2020-02-11 02:24:06: ERROR[Main]: 	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:105: in main chunk
2020-02-11 02:24:06: ERROR[Main]: Check debug.txt for details.
Once I ran the converter w/o land cover, the world opened but crashed:

Code: Select all

AsyncErr: Lua: finishGenRuntime error from mod 'geo_mapgen' in callback environment_OnGenerated(): decompressZlib: inflate failed
stack traceback:
	[C]: in function 'decompress'
	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:81: in function '__index'
	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:192: in function 'value'
	...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:231: in function <...etest-5.1.1-win64\bin\..\mods\geo-mapgen-master\init.lua:195>
	...es\minetest-5.1.1-win64\bin\..\builtin\game\register.lua:429: in function <...es\minetest-5.1.1-win64\bin\..\builtin\game\register.lua:413>
I know nothing about Lua scripts and only basic Python knowledge (pip install, QGIS, etc.), so all I can understand from "invalid or incomplete deflate data" is that something is wrong with the compression or file format. Well it works perfectly fine in QGIS (I've used it to make hillshade and slope rasters, and I've even added height maps to Worldpainter but without land cover classes), but it also won't open in MicroDEM because "not acceptable DEM." If you must know, I downloaded from the NOAA Data Access Viewer and if the proper data is chosen, it should be multiple TIFF files with multiple world files and an index sheet shapefile. I don't know why open-mapgen and MicroDEM consider it corrupted whereas every other program can use it. Someone in the first thread suggested the "format" changed and "they" (the NOAA?) might be using a different compression method - not sure how to parse that.

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

Hi qu4ntumrush,

It looks like your heightmap.dat is truncated so I suppose the converter crashed. Do you get any error while running the converter? If it exits too fast and the size of heightmap.dat is smaller that several megabytes, it's very likely that something went wrong.
qu4ntumrush wrote:If you must know, I downloaded from the NOAA Data Access Viewer and if the proper data is chosen, it should be multiple TIFF files with multiple world files and an index sheet shapefile.
I think that's what goes wrong. Geo-mapgen doesn't support this kind of multiple file input. If possible, you need to group them into one single TIFF file, that you give to the converter. I'll try to know more about this format, to understand what happens.
Geo-mapgen's converter is based on the GDAL library, so it should be able to read any file readable by GDAL (usually TIFFs, or many other raster formats, but not shapefiles).

Side note: I see you're on Windows. As a full Linux user, I didn't test geo-mapgen much on Windows so I don't really know how well it runs on it. Python and Lua are cross-platform languages but I'm unsure the way I wrote the code (filenames, etc) is fully compatible. I need to check.

I hope it helps you, and I'm sorry if my code is so difficult to use.
Just realize how bored we would be if the world was perfect.

Pablo.R
Member
Posts: 20
Joined: Thu Mar 24, 2016 12:02
Location: Chile

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Pablo.R » Post

Hi Gael, I was toying with your geo-mapgen, and I really love it.

I was using GEBCO (https://download.gebco.net/) because it contain elevation and bathymetry data and the results are beautiful despite the low resolution (5 times smaller than SRTM), because the your scale transform get some good results.

The last two days I was trying to combine SRTM elevation data with GEBCO bathymetry data, but no luck yet (the borders are a big problem). And was planing to add a simple temperature and precipitation calculations to define Holdridge zones as biomes.

But I have a big problem with the performance of rivers as I was creating maps of only islands (I want a bigger map but must stop below 8000 nodes per side. I look into the function but I was lost in it. Can you tell me where to look, and what are the parameters.

Please don't abandon it.

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

Hi Pablo,

Thank you for your kind words. Geo Mapgen is not abandoned, it's just going very slowly. I will have more time to spend on it in the upcoming weeks (confined at home because of covid-19…), provided I have enough motivation :)

Combining GEBCO and SRTM is a very good idea, and I remember doing something similar with other data sources. What did you try? Maybe I can find again the script I wrote to do it, if you are interested (it was also not perfect on the coastline).

For the climatic parameters, I'm not aware of any freely available data I could use with geo-mapgen (if you have found some, I'm very interested). The best I've found is directly biome data, I can recommand the MODIS-based Global Land Cover (500m resolution).

The rivermap code is a very old piece of code that has been uncleanly bound with geo-mapgen (it was unrelated at first). I'd need to clean it up and optimize it, but the algorithm itself is very complicated and intensive, anyway. For a 8000x8000 map you will need to be patient (30 mins - 1 hour maybe) and have enough free RAM (maybe 2G). I'm also afraid you may hit the recursion limit on the final phase.
River parameters can change the way rivers are displayed on the map, but they won't solve any performance-related issue. I should implement a way to calculate rivers on a downscaled map, to be less intensive.

I hope I answered to your questions :)
Just realize how bored we would be if the world was perfect.

User avatar
Gael de Sailly
Member
Posts: 753
Joined: Sun Jan 26, 2014 17:01
GitHub: Gael-de-Sailly
IRC: Gael-de-Sailly
In-game: Gael-de-Sailly Ginkgoo
Location: France

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Gael de Sailly » Post

BTW, qu4ntumrush, you didn't give news. Did you manage to do what you wanted?
Just realize how bored we would be if the world was perfect.

Pablo.R
Member
Posts: 20
Joined: Thu Mar 24, 2016 12:02
Location: Chile

Re: [Mod] Mapgen from geographical data [geo-mapgen] (Biomes

by Pablo.R » Post

Gael de Sailly wrote:Combining GEBCO and SRTM is a very good idea, and I remember doing something similar with other data sources. What did you try? Maybe I can find again the script I wrote to do it, if you are interested (it was also not perfect on the coastline).
I was trying to scale GEBCO by 5 horizontal factor, delete elevation data from GEBCO and then fusion SRTM with the modified GEBCO, but now I think it was a stupid me trying to do that because with python GDAL I can simply replace NoDataValues of SRTM by scaled GEBCO data. I must try that different aproach, surelly it can generate a better coastline.
Gael de Sailly wrote:For the climatic parameters, I'm not aware of any freely available data I could use with geo-mapgen (if you have found some, I'm very interested). The best I've found is directly biome data, I can recommand the MODIS-based Global Land Cover (500m resolution).
I will look into real Land Cover Data, but what I really want is experiment with different landcover biomes for the same map by manipulation of climate parameters (temperature and precipitation) in a more "real" form (cold places in far north and south, and high altitude, big precipitation in ecuator, etc.) to see what can be obtained (think of it as take an island and move it around the globe). Is a different aproach than minetest patchy of biomes.
Gael de Sailly wrote:The rivermap code is a very old piece of code that has been uncleanly bound with geo-mapgen (it was unrelated at first). I'd need to clean it up and optimize it, but the algorithm itself is very complicated and intensive, anyway. For a 8000x8000 map you will need to be patient (30 mins - 1 hour maybe) and have enough free RAM (maybe 2G). I'm also afraid you may hit the recursion limit on the final phase.
River parameters can change the way rivers are displayed on the map, but they won't solve any performance-related issue. I should implement a way to calculate rivers on a downscaled map, to be less intensive
I think is a good idea do the rivers calculus over the original dem image prior to scale it to final size, that is what I want to change but can't because I don't understand what must be do to change it and my priority is the fusion of SRTM and GEBCO. For testing, I was using the same source image (in SRTM 2400x3400) and it take only less than 5 minutes in original size, but when I scale it, at some point the rivers calculus abort (I have 8 GB of RAM). If I can succesfully fusion SRTM and GEBCO I will retake the rivers analysis.

Sorry, for my poor english.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 5 guests