Helper library for Minetest mods.
(Mods i used it for are "jtdb" and "names_per_ip")
Simple persistent key/value data storage.
For medium sized datasets - when simply reading/writing files is not enough, but setting up real database is overkill.
License: LGPL 2.1
Dependencies: none
Browse on:GitHub
Download:zip
[Mod] JustTest Database [0.9][jtdb]
- lag01
- Member
- Posts: 321
- Joined: Sun Mar 16, 2014 03:41
- GitHub: AndrejIT
- IRC: lag01
- In-game: lag
- Contact:
[Mod] JustTest Database [0.9][jtdb]
Last edited by lag01 on Sun Sep 16, 2018 21:19, edited 1 time in total.
- rubenwardy
- Moderator
- Posts: 6972
- Joined: Tue Jun 12, 2012 18:11
- GitHub: rubenwardy
- IRC: rubenwardy
- In-game: rubenwardy
- Location: Bristol, United Kingdom
- Contact:
Re: [MOD]jtdb[just test database][0.9]
Your topic title is badly formed, please rename it
It should beSubject of topic must be in one of these formats:
- [Mod] <Mod Name> [<modname>]
- [Mod] <Mod Name> [<version number>] [<modname>]
- [Modpack] <Name>
- [Modpack] <Name> [<version number>]
eg.- [Mod] More Blox [0.1] [moreblox]
- [Modpack] The Awesome Modpack
Code: Select all
[Mod] JustTest Database [0.9][jtdb]
- lag01
- Member
- Posts: 321
- Joined: Sun Mar 16, 2014 03:41
- GitHub: AndrejIT
- IRC: lag01
- In-game: lag
- Contact:
Re: [MOD]jtdb[just test database][0.9]
Thank you!rubenwardy wrote:Your topic title is badly formed, please rename it
...
It should be
Code: Select all
[Mod] JustTest Database [0.9][jtdb]
- sorcerykid
- Member
- Posts: 1841
- Joined: Fri Aug 26, 2016 15:36
- GitHub: sorcerykid
- In-game: Nemo
- Location: Illinois, USA
Re: [Mod] JustTest Database [0.9][jtdb]
Very impressive! Thanks for sharing these mods from your server. Any chance of also publishing the braziers?
- rnd
- Member
- Posts: 220
- Joined: Sun Dec 28, 2014 12:24
- GitHub: ac-minetest
- IRC: ac_minetest
- In-game: rnd
Re: [Mod] JustTest Database [0.9][jtdb]
Finally something worthy commenting on. Short description of how it works:
STRUCTURES:
dbfile : containing values, format: each value separate line
idfile : contains positions of value for given key in dbfile, format as key + \n + position of key in dbfile
self.id = hashtable containing all positions of keys; self.id[key] stores position in dbfile where value is stored
actual value is retrieved as:
self.dbfile:seek("set", self.id[key]);local value = self.dbfile:read("*line")
self.cache = dont read value from disk if we already read it before
OPERATION:
writing: just seek to end of dbfile and move position of self.id[key] there and simply append write new value in dbfile (at end)
-occasionaly (1 in 10000) write out idfile (if self.flush enabled actually write to file system)
since we could have many redundant new values maintain function only writes recent one
( at position specified by self.id[key]). this prunes dbfile of all duplicates.
-1 in 10 actually write out dbfile.
reading: given key look up position in dbfile at self.id[key]. Optionally cache the result as self.cache[key] = value
PROBLEMS: if you get crash before 1 in 10000 happens your recent value updates/creations are lost unless you have self.flush enabled. Positions in idfile are updated (append), causing idfile to grow ( maintain fixes this by writing only key/values from pairs(self.id)) - but this wont help since position now points to unflushed dbfile ( non existent position )
STRUCTURES:
dbfile : containing values, format: each value separate line
idfile : contains positions of value for given key in dbfile, format as key + \n + position of key in dbfile
self.id = hashtable containing all positions of keys; self.id[key] stores position in dbfile where value is stored
actual value is retrieved as:
self.dbfile:seek("set", self.id[key]);local value = self.dbfile:read("*line")
self.cache = dont read value from disk if we already read it before
OPERATION:
writing: just seek to end of dbfile and move position of self.id[key] there and simply append write new value in dbfile (at end)
-occasionaly (1 in 10000) write out idfile (if self.flush enabled actually write to file system)
since we could have many redundant new values maintain function only writes recent one
( at position specified by self.id[key]). this prunes dbfile of all duplicates.
-1 in 10 actually write out dbfile.
reading: given key look up position in dbfile at self.id[key]. Optionally cache the result as self.cache[key] = value
PROBLEMS: if you get crash before 1 in 10000 happens your recent value updates/creations are lost unless you have self.flush enabled. Positions in idfile are updated (append), causing idfile to grow ( maintain fixes this by writing only key/values from pairs(self.id)) - but this wont help since position now points to unflushed dbfile ( non existent position )
1EvCmxbzl5KDu6XAunE1K853Lq6VVOsT
- lag01
- Member
- Posts: 321
- Joined: Sun Mar 16, 2014 03:41
- GitHub: AndrejIT
- IRC: lag01
- In-game: lag
- Contact:
Re: [Mod] JustTest Database [0.9][jtdb]
Thanks sorcerykid! I definitively want to publish braziers (they are included in my mese_protector mod). After that CleanerPro incident i wanted to double-chek if protectors works as intended, before publishing...
Hi rnd, thank you, i am really flattered to receive such great feedback!
Thanks to pointing to weak point in my logic - all random operations was created only
for maintenance purpose, saving to disk was just side effect and i probably didn't gave it enough attention.
Actually id appends should be flushed the same time values are...
I will double-check, losing unflushed data on crash is only natural, but inconsistent database is not ok.
Thank you again!
Hi rnd, thank you, i am really flattered to receive such great feedback!
Thanks to pointing to weak point in my logic - all random operations was created only
for maintenance purpose, saving to disk was just side effect and i probably didn't gave it enough attention.
Actually id appends should be flushed the same time values are...
I will double-check, losing unflushed data on crash is only natural, but inconsistent database is not ok.
Thank you again!
Who is online
Users browsing this forum: No registered users and 24 guests