Serious limitations of minetest.deserialize

User avatar
gpcf
Member
 
Posts: 328
Joined: Fri May 27, 2016 10:48
In-game: gabriel

Serious limitations of minetest.deserialize

by gpcf » Mon Jul 22, 2019 10:31

minetest.deserialize has a serious limitation: It will not work if you try to deserialize a table with more than 2¹⁶ entries under luajit and 2¹⁸ entries under regular lua. This can be a serious issue with mods that need to store large amounts of data, such as advtrains.

Are there any known workarounds to read large tables written by minetest.serialize?
 

User avatar
Linuxdirk
Member
 
Posts: 2223
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Serious limitations of minetest.deserialize

by Linuxdirk » Mon Jul 22, 2019 12:33

This is a technical limitation that is not fixable.

If you need a table with 65536 entries you're doing it wrong - no matter what you do.
 

User avatar
Lejo
Member
 
Posts: 639
Joined: Mon Oct 19, 2015 16:32
GitHub: Lejo1
In-game: Lejo
 

sofar
Developer
 
Posts: 2086
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Serious limitations of minetest.deserialize

by sofar » Mon Jul 22, 2019 17:40

If you have that many table entries, you should look into alternative storage methods that are more efficient at storing that many entries. I would certainly consider `lsqlite3` as a viable alternative - but using lua sockets to talk to a SQL database would also work really well.

This depends heavily on your data, of course. If your data is simple, like, just 65k numbers, you could manually serialize it, but I suspect it's complex data.
 

BuckarooBanzay
Member
 
Posts: 320
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai

Re: Serious limitations of minetest.deserialize

by BuckarooBanzay » Tue Jul 23, 2019 16:21

gpcf wrote:minetest.deserialize has a serious limitation: It will not work if you try to deserialize a table with more than 2¹⁶ entries under luajit and 2¹⁸ entries under regular lua. This can be a serious issue with mods that need to store large amounts of data, such as advtrains.

Are there any known workarounds to read large tables written by minetest.serialize?


What data is it exactly that you have problems with? I'm guessing its not the node-database, rather the train-wagon data, right?
 

User avatar
gpcf
Member
 
Posts: 328
Joined: Fri May 27, 2016 10:48
In-game: gabriel

Re: Serious limitations of minetest.deserialize

by gpcf » Tue Jul 23, 2019 22:03

yes, especially the interlocking data is seriously huge. For now, I'd recommend to update advtrains, since this commit provides a temporary relief for the issue by splitting up the database into many different files.
 

Xudo
Member
 
Posts: 159
Joined: Wed Nov 09, 2016 16:43
GitHub: akryukov92
In-game: Xudo

Re: Serious limitations of minetest.deserialize

by Xudo » Wed Jul 24, 2019 05:19

Other option to improve performance is to partition data.
Make 2 or more files instead of 1 and distribute entries between them by some criteria.
 

User avatar
gpcf
Member
 
Posts: 328
Joined: Fri May 27, 2016 10:48
In-game: gabriel

Re: Serious limitations of minetest.deserialize

by gpcf » Wed Jul 24, 2019 14:55

That is what I did. Btw, xban2 might suffer from the exact same issues if the db doesn't get cleaned out every now and then.
 

BuckarooBanzay
Member
 
Posts: 320
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai

Re: Serious limitations of minetest.deserialize

by BuckarooBanzay » Wed Jul 24, 2019 17:13

gpcf wrote:That is what I did. Btw, xban2 might suffer from the exact same issues if the db doesn't get cleaned out every now and then.


Thats exactly what my latest commit does :D
https://github.com/minetest-mods/xban2/ ... 825ed9b5bd
 

rheo
Member
 
Posts: 16
Joined: Fri May 03, 2019 20:40
GitHub: fluxionary
IRC: flux fluxflux
In-game: flux rheo

Re: Serious limitations of minetest.deserialize

by rheo » Fri Aug 02, 2019 07:46

This is only tangentially related, but the 64ki limit is built in to Lua itself, not minetest.serialize itself. Lua can only process scripts w/ a maximum of 64ki constants in them, and minetest.serialize (like most Lua serializers) encodes its data as a Lua script.

The comment about a table w/ more than 64ki entries indicates "you're doing it wrong" is itself absolutely wrong, but everyone elses suggestions are great.

To further the tangent: you still hit that 64ki limit if you're using other structures (e.g. a binary tree) to store your data; Lua tables perform far better than SQLite and are more suitable many applications w/ medium-sized data; however, you should be aware that LuaJIT can't handle more than 1GiB of memory without enabling an unstable feature, and you should design your mods w/ that limit in mind.
 


Return to Problems



Who is online

Users browsing this forum: No registered users and 2 guests