## How can I store and search a large tabel from a mod?

burli
Member

Posts: 1641
Joined: Fri Apr 10, 2015 13:18

### How can I store and search a large tabel from a mod?

I want to be able to store and search a larger amount of data on disc. Sadly there is still no database API to the default database.

Is there any efficient way for a lua mod to store and search a large tabel on disc? The only thing I found that might work because it is pure lua is this http://scilua.org/ljsqlite3.html. Just requires the LuaJIT (and clib_sqlite3, damn)

I don't want to store and parse JSON to a plain text file

Any suggestions?

vitalie
Member

Posts: 66
Joined: Wed Feb 24, 2016 07:38
GitHub: ciubotaru

Code: Select all
$sudo luarocks install lsqlite Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod). Krock Developer Posts: 4414 Joined: Thu Oct 03, 2013 07:48 Location: Switzerland GitHub: SmallJoker ### Re: How can I store and search a large tabel from a mod? "the default database" I think you mean map.sqlite here. There are already enough read/write actions performed when you play Minetest. So using a new database is the best way to save and manage the data. lsqlite3 might not be the fastest way to do this but it works well. Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here> burli Member Posts: 1641 Joined: Fri Apr 10, 2015 13:18 ### Re: How can I store and search a large tabel from a mod? vitalie wrote: Code: Select all $ sudo luarocks install lsqlite

Then add "require(sqlite3)" to your lua file and work with the db transparently (send queries, get results and use them in mod).

Does this work for Windows and OS X too?

Krock
Developer

Posts: 4414
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

### Re: How can I store and search a large tabel from a mod?

burli wrote:Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

burli
Member

Posts: 1641
Joined: Fri Apr 10, 2015 13:18

### Re: How can I store and search a large tabel from a mod?

Thx, I will try it

burli
Member

Posts: 1641
Joined: Fri Apr 10, 2015 13:18

### Re: How can I store and search a large tabel from a mod?

This is correct

Code: Select all
\$ sudo luarocks install lsqlite3

Code: Select all
require("lsqlite3")

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Krock wrote:
burli wrote:Does this work for Windows and OS X too?

Works fine on Windows (using the compiled lsqlite.dll in the same directory). Not sure about OS X but it should work there too.

Can you share any pointers about how you compiled lsqlite.dll ? I am trying to install the global_exchange mod under windows. I see that I need to use luarocks to install lsqlite, but I'm not sure how to install luarocks itself under windows, in such a way that it will work with the lua built into minetest.

Do I just need to make a separate installation of lua somewhere, install luarocks with it, and then install lsqlite with that and copy it into minetest? If so, should I be using the Windows lua from https://github.com/rjpcomputing/luaforwindows or the Windows lua from http://luabinaries.sourceforge.net/ ?

I am a complete newbie to lua, but not at all a newbie to programming or to programming library package managers.

I realize this is an old thread, but I'm hoping somebody can help!

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Okay, I finally figured out the option to get LuaRocks to install its own self-contained version of Lua is:
INSTALL /L

working from there, I'll see how this goes...

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

I had to locate the luarocks executable and run from that directory:
C:\Program Files (x86)\LuaRocks\bin

And I also had to name the library lsqlite3 in the command-line, not just lsqlite:
luarocks

Looks like that's still not enough, though; I'll have to go make a separate Windows install of sqlite. Unless I can connive it to use one of the several that I'm sure are on this machine (minetest, cygwin, and elsewhere).

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Well, after lots of messing around I've gotten a bit further. I downloaded the Windows version of SQLite and extracted it, which came with a dll and one other file, and I downloaded the "amalgamation" source release of SQLite that comes with the sqlite3.h include file. The contents of both these went into C:\external (which is where luarocks seems to want to find sqlite), and the .h file went into C:\external\include .

This got me past all the messages looking for sqlite; now it is reporting that it's missing a command-line utility named cl. I'll be looking for that next.

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Looks like cl.exe is a component of visual studio and the recommended way to get it in your path is to launch a command-line from visual studio. With VS installed there's a Visual Studio Tools menu that allows this. The command-line environment needs to be launched with administrator permission. If it isn't, luarocks spawns a new environment requesting administrator permission, and it won't have the right paths to find cl.

The next obstacle is the compile is looking for a file sqlite3.lib. Not sure why it's looking for a .lib instead of a .dll. It does look like it's gone through a step that used sqlite3.dll, though, so this must be something different.

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Okay, I had to create the sqlite3.lib myself, from inside of the VS command-line environment, with the link command:

cd \external

(found this described here: http://sqlite.1065341.n5.nabble.com/lib ... 52677.html)

The sqlite3.def file came from the downloaded sqlite dll for windows package.

I've been using the 32 bit sqlite because that seems to be what LuaRocks is using. But since my minetest is 64 bit I wonder if this is going to ultimately mean I need to use the other. The link command noted that I could've used /MACHINE to specify an architecture other than 32 bit.

The resulting sqlite3.lib file has to be copied into C:\external\include.

This built me an lsqlite3.dll in C:\Program Files (x86)\LuaRocks\systree\lib\lua\5.1\ (along with a lot of other related files there). I'll copy this into minetest and see how things go now... (might have to redo all this as 64 bit, if that's even possible).

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

That gets me this far, error coming from minetest finaly and not from the debug file:

Failed to load and run script from C:\Path\to\minetest-0.4.16-win64\bin\..\mods\global_exchange\init.lua:
%1 is not a valid Win32 application.

Maybe I need to try the 64 bit route...

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

Switching to 64 I was able to build a 64 bit sqlite3.lib with VS's link command as described above, but then luarocks throws a lot of unrecognized symbol errors trying to use it. My luarocks is installed in Program Files (x86) so I assume it is 32 bit, and this seemed to be the only version available for download. I'm wondering if there's a 64 bit version of luarocks I missed.

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

It sounds like luarocks can manage both 64 bit and 32 bit installs, but I guess when I used the /L option for the luarocks install I got a 32 bit version and that's it. I guess I have to go get a full 64 bit lua to make this work, or switch to 32 bit minetest

Linuxdirk
Member

Posts: 2226
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

### Re: How can I store and search a large tabel from a mod?

Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439

skiphoppy
Member

Posts: 32
Joined: Sat Jun 10, 2017 01:50

### Re: How can I store and search a large tabel from a mod?

That might be a great improvement for the mod or mods that were using this (global_exchange is the one I am trying to use). Right now I'm not sure how anyone gets these to run on Windows, if possible.

Byakuren
Member

Posts: 816
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

### Re: How can I store and search a large tabel from a mod?

Linuxdirk wrote:Since 0.4.16 is out which provides a mod metadata storage API (which unfortunately is pretty much undocumented, but it was in the changelog) maybe you could store your data there without having to depend on 3rd-party libraries.

viewtopic.php?p=265439#p265439

global_exchange still benefits from using a relational DBMS because it queries sets of data (for example to show buy/sell orders on the exchange). With mod storage I would have to either keep the whole table in memory all the time or else access each mod storage key to filter through all of them.
Every time a mod API is left undocumented, a koala dies.