How does the SQLite DB work ?

For people working on the C++ code.
Post Reply
User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

How does the SQLite DB work ?

by LMD » Post

A question to you, core developers :

After working with SQLite(for MTPC), I have learned from my mistakes, and am feeling quite safe and have learned much. Now, I'm asking myself whether it would be good to modify the SQLite DB from outside(Java) to make map ops not in LUA, but from outside program; more specific, I want to have - Java program which creates Schematics from 3d models -> writes them directly into map.
So, I know how it basically works : You got indices & blobs. Each indice(int) stores blob(chunk) position in map and is primary key. Blob is mapchunk(which size ? 80 ?). How do indices work(which order ? x+y*mapsize+z*mapsize² ?). And now, the final question : How do the binary large objects work ? Of course, every node has content ID. So, are you just storing the content IDs in giant array, indiced by x+y*chunksize+z*chunksize² ? Where's the metadata ?
My stuff: Projects - Mods - Website

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: How does the SQLite DB work ?

by ExeterDad » Post

I did a casual read of this just the other day. I believe what you are looking for is documented here:
https://github.com/minetest/minetest/bl ... format.txt

User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

Re: How does the SQLite DB work ?

by LMD » Post

Thanks ! Exactly what I was searching for !
This answers my indexing question :

Code: Select all

The key
--------
"pos" is created from the three coordinates of a MapBlock using this
algorithm, defined here in Python:

  def getBlockAsInteger(p):
      return int64(p[2]*16777216 + p[1]*4096 + p[0])

  def int64(u):
      while u >= 2**63:
          u -= 2**64
      while u <= -2**63:
          u += 2**64
      return u

It can be converted the other way by using this code:

  def getIntegerAsBlock(i):
      x = unsignedToSigned(i % 4096, 2048)
      i = int((i - x) / 4096)
      y = unsignedToSigned(i % 4096, 2048)
      i = int((i - y) / 4096)
      z = unsignedToSigned(i % 4096, 2048)
      return x,y,z

  def unsignedToSigned(i, max_positive):
      if i < max_positive:
          return i
      else:
return i - 2*max_positive
Blob/Mapchunk size : 16x16

Giant array node index :

Code: Select all

- The location of a node in each of those arrays is (z*16*16 + y*16 + x).
...I guess it's gonna be too complicated to access that from outside :( However, some ops still make sense, such as swapping mapchunks etc :|
My stuff: Projects - Mods - Website

User avatar
ExeterDad
Member
Posts: 1717
Joined: Sun Jun 01, 2014 20:00
In-game: ExeterDad
Location: New Hampshire U.S.A

Re: How does the SQLite DB work ?

by ExeterDad » Post

Take a peek at the python minetest mapper version. It might be written in a way that may help with the "understanding"

User avatar
BuckarooBanzay
Member
Posts: 435
Joined: Tue Apr 24, 2018 05:58
GitHub: BuckarooBanzay
IRC: BuckarooBanzai
In-game: BuckarooBanzai

Re: How does the SQLite DB work ?

by BuckarooBanzay » Post

I implemented a map server in java the last few weeks, maybe you find something useful there: Parsers:
¯\_(ツ)_/¯ Not active here anymore, contact me on the minetest discord, irc, lemmy or github (for programming issues)

User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

Re: How does the SQLite DB work ?

by LMD » Post

Nice !
My stuff: Projects - Mods - Website

User avatar
BuckarooBanzay
Member
Posts: 435
Joined: Tue Apr 24, 2018 05:58
GitHub: BuckarooBanzay
IRC: BuckarooBanzai
In-game: BuckarooBanzai

Re: How does the SQLite DB work ?

by BuckarooBanzay » Post

LMD wrote:Nice !
So, did you make any progress...?
¯\_(ツ)_/¯ Not active here anymore, contact me on the minetest discord, irc, lemmy or github (for programming issues)

User avatar
LMD
Member
Posts: 1386
Joined: Sat Apr 08, 2017 08:16
GitHub: appgurueu
IRC: appguru[eu]
In-game: LMD
Location: Germany
Contact:

Re: How does the SQLite DB work ?

by LMD » Post

I sort of understood it, but am working on other projects.
My stuff: Projects - Mods - Website

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest