How can I store and search a large tabel from a mod?

[burli]

I want to be able to store and search a larger amount of data on disc. Sadly there is still no database API to the default database.

Is there any efficient way for a lua mod to store and search a large tabel on disc? The only thing I found that might work because it is pure lua is this http://scilua.org/ljsqlite3.html. Just requires the LuaJIT (and clib_sqlite3, damn)

I don't want to store and parse JSON to a plain text file

Any suggestions?

[vitalie]

Code: Select all

$ sudo luarocks install lsqlite

Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod).

[Krock]

"the default database"
I think you mean map.sqlite here. There are already enough read/write actions performed when you play Minetest. So using a new database is the best way to save and manage the data. lsqlite3 might not be the fastest way to do this but it works well.

[burli]

Code: Select all

$ sudo luarocks install lsqlite

Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod).

Does this work for Windows and OS X too?

[Krock]

Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.

[burli]

Thx, I will try it

[burli]

This is correct

Code: Select all

$ sudo luarocks install lsqlite3

Code: Select all

require("lsqlite3")

[skiphoppy]

Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.

Can you share any pointers about how you compiled lsqlite.dll ? I am trying to install the global_exchange mod under windows. I see that I need to use luarocks to install lsqlite, but I'm not sure how to install luarocks itself under windows, in such a way that it will work with the lua built into minetest.

Do I just need to make a separate installation of lua somewhere, install luarocks with it, and then install lsqlite with that and copy it into minetest? If so, should I be using the Windows lua from https://github.com/rjpcomputing/luaforwindows or the Windows lua from http://luabinaries.sourceforge.net/ ?

I am a complete newbie to lua, but not at all a newbie to programming or to programming library package managers.

I realize this is an old thread, but I'm hoping somebody can help!

[skiphoppy]

Okay, I finally figured out the option to get LuaRocks to install its own self-contained version of Lua is:
INSTALL /L

working from there, I'll see how this goes...

[skiphoppy]

I had to locate the luarocks executable and run from that directory:
C:\Program Files (x86)\LuaRocks\bin

And I also had to name the library lsqlite3 in the command-line, not just lsqlite:
luarocks

Looks like that's still not enough, though; I'll have to go make a separate Windows install of sqlite. Unless I can connive it to use one of the several that I'm sure are on this machine (minetest, cygwin, and elsewhere).

[skiphoppy]

Well, after lots of messing around I've gotten a bit further. I downloaded the Windows version of SQLite and extracted it, which came with a dll and one other file, and I downloaded the "amalgamation" source release of SQLite that comes with the sqlite3.h include file. The contents of both these went into C:\external (which is where luarocks seems to want to find sqlite), and the .h file went into C:\external\include .

This got me past all the messages looking for sqlite; now it is reporting that it's missing a command-line utility named cl. I'll be looking for that next.

[skiphoppy]

Looks like cl.exe is a component of visual studio and the recommended way to get it in your path is to launch a command-line from visual studio. With VS installed there's a Visual Studio Tools menu that allows this. The command-line environment needs to be launched with administrator permission. If it isn't, luarocks spawns a new environment requesting administrator permission, and it won't have the right paths to find cl.

The next obstacle is the compile is looking for a file sqlite3.lib. Not sure why it's looking for a .lib instead of a .dll. It does look like it's gone through a step that used sqlite3.dll, though, so this must be something different.

[skiphoppy]

Okay, I had to create the sqlite3.lib myself, from inside of the VS command-line environment, with the link command:

cd \external
link /lib /def:sqlite3.def

(found this described here: http://sqlite.1065341.n5.nabble.com/lib ... 52677.html)

The sqlite3.def file came from the downloaded sqlite dll for windows package.

I've been using the 32 bit sqlite because that seems to be what LuaRocks is using. But since my minetest is 64 bit I wonder if this is going to ultimately mean I need to use the other. The link command noted that I could've used /MACHINE to specify an architecture other than 32 bit.

The resulting sqlite3.lib file has to be copied into C:\external\include.

This built me an lsqlite3.dll in C:\Program Files (x86)\LuaRocks\systree\lib\lua\5.1\ (along with a lot of other related files there). I'll copy this into minetest and see how things go now... (might have to redo all this as 64 bit, if that's even possible).

[skiphoppy]

That gets me this far, error coming from minetest finaly and not from the debug file:

Failed to load and run script from C:\Path\to\minetest-0.4.16-win64\bin\..\mods\global_exchange\init.lua:
error loading module 'lsqlite3' from file '.\lsqlite3.dll':
%1 is not a valid Win32 application.

Maybe I need to try the 64 bit route...

[skiphoppy]

Switching to 64 I was able to build a 64 bit sqlite3.lib with VS's link command as described above, but then luarocks throws a lot of unrecognized symbol errors trying to use it. My luarocks is installed in Program Files (x86) so I assume it is 32 bit, and this seemed to be the only version available for download. I'm wondering if there's a 64 bit version of luarocks I missed.

[skiphoppy]

It sounds like luarocks can manage both 64 bit and 32 bit installs, but I guess when I used the /L option for the luarocks install I got a 32 bit version and that's it. I guess I have to go get a full 64 bit lua to make this work, or switch to 32 bit minetest

[Linuxdirk]

Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439

[skiphoppy]

That might be a great improvement for the mod or mods that were using this (global_exchange is the one I am trying to use). Right now I'm not sure how anyone gets these to run on Windows, if possible.

[Byakuren]

Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439

global_exchange still benefits from using a relational DBMS because it queries sets of data (for example to show buy/sell orders on the exchange). With mod storage I would have to either keep the whole table in memory all the time or else access each mod storage key to filter through all of them.