How can I store and search a large tabel from a mod?

User avatar
burli
Member
 
Posts: 1641
Joined: Fri Apr 10, 2015 13:18

How can I store and search a large tabel from a mod?

by burli » Sat Jun 04, 2016 12:34

I want to be able to store and search a larger amount of data on disc. Sadly there is still no database API to the default database.

Is there any efficient way for a lua mod to store and search a large tabel on disc? The only thing I found that might work because it is pure lua is this http://scilua.org/ljsqlite3.html. Just requires the LuaJIT (and clib_sqlite3, damn)

I don't want to store and parse JSON to a plain text file

Any suggestions?
 

vitalie
Member
 
Posts: 66
Joined: Wed Feb 24, 2016 07:38
GitHub: ciubotaru

Re: How can I store and search a large tabel from a mod?

by vitalie » Sat Jun 04, 2016 15:13

Code: Select all
$ sudo luarocks install lsqlite


Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod).
 

User avatar
Krock
Developer
 
Posts: 4414
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: How can I store and search a large tabel from a mod?

by Krock » Sat Jun 04, 2016 15:20

"the default database"
I think you mean map.sqlite here. There are already enough read/write actions performed when you play Minetest. So using a new database is the best way to save and manage the data. lsqlite3 might not be the fastest way to do this but it works well.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
burli
Member
 
Posts: 1641
Joined: Fri Apr 10, 2015 13:18

Re: How can I store and search a large tabel from a mod?

by burli » Sat Jun 04, 2016 16:07

vitalie wrote:
Code: Select all
$ sudo luarocks install lsqlite


Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod).

Does this work for Windows and OS X too?
 

User avatar
Krock
Developer
 
Posts: 4414
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: How can I store and search a large tabel from a mod?

by Krock » Sat Jun 04, 2016 16:14

burli wrote:Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
burli
Member
 
Posts: 1641
Joined: Fri Apr 10, 2015 13:18
 

User avatar
burli
Member
 
Posts: 1641
Joined: Fri Apr 10, 2015 13:18

Re: How can I store and search a large tabel from a mod?

by burli » Sat Jun 04, 2016 16:21

This is correct

Code: Select all
$ sudo luarocks install lsqlite3


Code: Select all
require("lsqlite3")
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 01:54

Krock wrote:
burli wrote:Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.


Can you share any pointers about how you compiled lsqlite.dll ? I am trying to install the global_exchange mod under windows. I see that I need to use luarocks to install lsqlite, but I'm not sure how to install luarocks itself under windows, in such a way that it will work with the lua built into minetest.

Do I just need to make a separate installation of lua somewhere, install luarocks with it, and then install lsqlite with that and copy it into minetest? If so, should I be using the Windows lua from https://github.com/rjpcomputing/luaforwindows or the Windows lua from http://luabinaries.sourceforge.net/ ?

I am a complete newbie to lua, but not at all a newbie to programming or to programming library package managers.

I realize this is an old thread, but I'm hoping somebody can help!
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 02:24

Okay, I finally figured out the option to get LuaRocks to install its own self-contained version of Lua is:
INSTALL /L

working from there, I'll see how this goes...
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 02:27

I had to locate the luarocks executable and run from that directory:
C:\Program Files (x86)\LuaRocks\bin

And I also had to name the library lsqlite3 in the command-line, not just lsqlite:
luarocks

Looks like that's still not enough, though; I'll have to go make a separate Windows install of sqlite. Unless I can connive it to use one of the several that I'm sure are on this machine (minetest, cygwin, and elsewhere).
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 20:58

Well, after lots of messing around I've gotten a bit further. I downloaded the Windows version of SQLite and extracted it, which came with a dll and one other file, and I downloaded the "amalgamation" source release of SQLite that comes with the sqlite3.h include file. The contents of both these went into C:\external (which is where luarocks seems to want to find sqlite), and the .h file went into C:\external\include .

This got me past all the messages looking for sqlite; now it is reporting that it's missing a command-line utility named cl. I'll be looking for that next.
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 21:11

Looks like cl.exe is a component of visual studio and the recommended way to get it in your path is to launch a command-line from visual studio. With VS installed there's a Visual Studio Tools menu that allows this. The command-line environment needs to be launched with administrator permission. If it isn't, luarocks spawns a new environment requesting administrator permission, and it won't have the right paths to find cl.

The next obstacle is the compile is looking for a file sqlite3.lib. Not sure why it's looking for a .lib instead of a .dll. It does look like it's gone through a step that used sqlite3.dll, though, so this must be something different.
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 21:57

Okay, I had to create the sqlite3.lib myself, from inside of the VS command-line environment, with the link command:

cd \external
link /lib /def:sqlite3.def

(found this described here: http://sqlite.1065341.n5.nabble.com/lib ... 52677.html)

The sqlite3.def file came from the downloaded sqlite dll for windows package.

I've been using the 32 bit sqlite because that seems to be what LuaRocks is using. But since my minetest is 64 bit I wonder if this is going to ultimately mean I need to use the other. The link command noted that I could've used /MACHINE to specify an architecture other than 32 bit.

The resulting sqlite3.lib file has to be copied into C:\external\include.

This built me an lsqlite3.dll in C:\Program Files (x86)\LuaRocks\systree\lib\lua\5.1\ (along with a lot of other related files there). I'll copy this into minetest and see how things go now... (might have to redo all this as 64 bit, if that's even possible).
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 22:06

That gets me this far, error coming from minetest finaly and not from the debug file:

Failed to load and run script from C:\Path\to\minetest-0.4.16-win64\bin\..\mods\global_exchange\init.lua:
error loading module 'lsqlite3' from file '.\lsqlite3.dll':
%1 is not a valid Win32 application.

Maybe I need to try the 64 bit route...
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 22:16

Switching to 64 I was able to build a 64 bit sqlite3.lib with VS's link command as described above, but then luarocks throws a lot of unrecognized symbol errors trying to use it. My luarocks is installed in Program Files (x86) so I assume it is 32 bit, and this seemed to be the only version available for download. I'm wondering if there's a 64 bit version of luarocks I missed.
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sat Jun 10, 2017 22:25

It sounds like luarocks can manage both 64 bit and 32 bit installs, but I guess when I used the /L option for the luarocks install I got a 32 bit version and that's it. I guess I have to go get a full 64 bit lua to make this work, or switch to 32 bit minetest
 

User avatar
Linuxdirk
Member
 
Posts: 2226
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: How can I store and search a large tabel from a mod?

by Linuxdirk » Sat Jun 10, 2017 23:02

Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439
 

skiphoppy
Member
 
Posts: 32
Joined: Sat Jun 10, 2017 01:50

Re: How can I store and search a large tabel from a mod?

by skiphoppy » Sun Jun 11, 2017 02:07

That might be a great improvement for the mod or mods that were using this (global_exchange is the one I am trying to use). Right now I'm not sure how anyone gets these to run on Windows, if possible.
 

Byakuren
Member
 
Posts: 816
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

Re: How can I store and search a large tabel from a mod?

by Byakuren » Thu Jun 15, 2017 19:18

Linuxdirk wrote:Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439

global_exchange still benefits from using a relational DBMS because it queries sets of data (for example to show buy/sell orders on the exchange). With mod storage I would have to either keep the whole table in memory all the time or else access each mod storage key to filter through all of them.
Every time a mod API is left undocumented, a koala dies.
 


Return to Modding Discussion



Who is online

Users browsing this forum: No registered users and 1 guest