Serious limitations of minetest.deserialize

[gpcf]

minetest.deserialize has a serious limitation: It will not work if you try to deserialize a table with more than 2¹⁶ entries under luajit and 2¹⁸ entries under regular lua. This can be a serious issue with mods that need to store large amounts of data, such as advtrains.

Are there any known workarounds to read large tables written by minetest.serialize?

[Linuxdirk]

This is a technical limitation that is not fixable.

If you need a table with 65536 entries you're doing it wrong - no matter what you do.

[Lejo]

Use the unsafe way:
Write the table directly to the file. Use minetest.safe_file_write

[sofar]

If you have that many table entries, you should look into alternative storage methods that are more efficient at storing that many entries. I would certainly consider `lsqlite3` as a viable alternative - but using lua sockets to talk to a SQL database would also work really well.

This depends heavily on your data, of course. If your data is simple, like, just 65k numbers, you could manually serialize it, but I suspect it's complex data.

[BuckarooBanzay]

minetest.deserialize has a serious limitation: It will not work if you try to deserialize a table with more than 2¹⁶ entries under luajit and 2¹⁸ entries under regular lua. This can be a serious issue with mods that need to store large amounts of data, such as advtrains.

Are there any known workarounds to read large tables written by minetest.serialize?

What data is it exactly that you have problems with? I'm guessing its not the node-database, rather the train-wagon data, right?

[gpcf]

yes, especially the interlocking data is seriously huge. For now, I'd recommend to update advtrains, since this commit provides a temporary relief for the issue by splitting up the database into many different files.

[Xudo]

Other option to improve performance is to partition data.
Make 2 or more files instead of 1 and distribute entries between them by some criteria.

[gpcf]

That is what I did. Btw, xban2 might suffer from the exact same issues if the db doesn't get cleaned out every now and then.

[BuckarooBanzay]

That is what I did. Btw, xban2 might suffer from the exact same issues if the db doesn't get cleaned out every now and then.

Thats exactly what my latest commit does :D
https://github.com/minetest-mods/xban2/ ... 825ed9b5bd

[rheo]

This is only tangentially related, but the 64ki limit is built in to Lua itself, not minetest.serialize itself. Lua can only process scripts w/ a maximum of 64ki constants in them, and minetest.serialize (like most Lua serializers) encodes its data as a Lua script.

The comment about a table w/ more than 64ki entries indicates "you're doing it wrong" is itself absolutely wrong, but everyone elses suggestions are great.

To further the tangent: you still hit that 64ki limit if you're using other structures (e.g. a binary tree) to store your data; Lua tables perform far better than SQLite and are more suitable many applications w/ medium-sized data; however, you should be aware that LuaJIT can't handle more than 1GiB of memory without enabling an unstable feature, and you should design your mods w/ that limit in mind.