New MTS format

For people working on the C++ code.
Post Reply
bzt
Member
Posts: 133
Joined: Tue Sep 24, 2019 14:26

New MTS format

by bzt » Post

Dear developers,

I'd like to propose some modifications to the MTS format, version 5. I'm doing so because I keep getting issues with my MTSEdit, people asking for features that are simply not possible with version 4.

I've gave it a lot of though to minimize these modifications, and using the MTS version would allow to simultaneously support both version 4 and version 5 in the Minetest engine. I'm also willing to code this and provide a PR when the format is finalized.

Proposed modifications:
1. add ground level after size Z (same as WEOffsetY in MC schematics)
2. add param3 as a string property (for chests and command blocks and probably others)
3. move layer probability into the zlib compressed block (not important, but would be nice to have)

So the header would look like this:

Code: Select all

| Offset  | Length | Description                              |
|--------:|-------:|------------------------------------------|
|       0 |      4 | magic "MTSM"                             |
|       4 |      2 | file format version, 5                   |
|       6 |      2 | size X                                   |
|       8 |      2 | size Y                                   |
|      10 |      2 | size Z                                   |
|      12 |      2 | ground Y                                 |
|      14 |      2 | number of strings in Name-ID table       |
|      16 |      x | Name-ID table                            |
|       x |      x | Block definitions (zlib compressed)      |
Ground Y is a value that's simply substracted from the Y coordinate when schematics are placed. With being 0, it would work like now. With values greater than zero, it would allow to place underground parts of the schematic correctly (same way as WEOfffsetY works in Minecraft).

And the zlib compressed block would look like:

Code: Select all

| Offset    | Length    | Description                              |
|----------:|----------:|------------------------------------------|
|         0 |         Y | layer probability values                 |
|         Y |   2*X*Y*Z | block IDs (param0)                       |
| Y+2*X*Y*Z |     X*Y*Z | probability values (param1)              |
| Y+3*X*Y*Z |     X*Y*Z | rotation info (param2)                   |
| Y+4*X*Y*Z | 2*X*Y*Z+n | property strings                         |
Here the 2nd, 3rd and 4th blocks are the same as now. 1st block is now uncompressed, this proposal would move that inside the zlib compressed block otherwise the format of layer probabilities wouldn't change. The 5th block would be an addition, that's currently missing from MTS files and would be very much needed.

When no properties defined at all (as with current MTS), the last block would be 2*X*Y*Z zeros (something that would be compressed to very small size with zlib, probably no more than 3-5 additional bytes to the file size). Otherwise for nodes without property, would be two zero bytes, and a big-endian string length and string for nodes with properties (same encoding as in Name-ID table).

This property string could be used depending on the node's type: for example chests could store their list of contents here, and command block the list of commands. The same callback could be used to parse these strings as the current Lua callback.

An alternative would be to add these property strings to the Name-ID table, and there would be no need for param3 block, but that would mean a more drastic change in the MTS specification (Name wouldn't be the node's Name only any more), an a whole different parsing method in the C++ code.

What are your thoughts?

Cheers,
bzt

User avatar
sfan5
Moderator
Posts: 4013
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

Re: New MTS format

by sfan5 » Post

A few points:

1. I don't quite get why the MTS format should have an y offset. For use with mapgen decorations there is a place_offset_y that can be defined per decoration and it's necessary there.
But if users are just manually placing their schematics, they should set the positions to match the contents of the schematic.

2. What are you describing is metadata, just in way that is unlike how the engine does it. The omission of metadata support from MTS files is intentional since it was invented for mapgen decorations.
Metadata would also break functionality like minetest.place_schematic_on_vmanip, since vmanips do not handle metadata (differently put: the focus on just the "basics" is intentional).

3. I agree. If the MTS format is changed at one point, doing this would be good.

If you'd still like to implement metadata support in MTSEdit you could consider adding the WorldEdit file format as an option. Unlike MTS it does not support probabilities but does fully support metadata.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

bzt
Member
Posts: 133
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

sfan5 wrote:
Sat Mar 13, 2021 23:17
A few points:
Thanks for your feedback!
sfan5 wrote:
Sat Mar 13, 2021 23:17
1. I don't quite get why the MTS format should have an y offset. For use with mapgen decorations there is a place_offset_y that can be defined per decoration and it's necessary there.
Because how do you know what to put in that place_offset_y if it isn't stored in the MTS file? MTS files aren't limited to mapgens any more, plus requiring coding a manual place_offset_y makes it harder to replace schematics in mapgens.
sfan5 wrote:
Sat Mar 13, 2021 23:17
But if users are just manually placing their schematics, they should set the positions to match the contents of the schematic.
You cannot expect the users to know how big the underground part is in a schem before they place them. They can find the X and Z coordinate easily, but not the Y one, they expect the schem to be placed on the ground (which works fine only if the schem doesn't have an underground part like the pyramid for example.)

It's a simple modification btw, at line 253:

Code: Select all

	if (flags & DECO_PLACE_CENTER_Y)
		p.Y -= (s.Y - 1) / 2;
 +      else
 +              p.Y -= goundlevel;
(where groundlevel is a property set up in the same class in the deserializeFromMts() method)
sfan5 wrote:
Sat Mar 13, 2021 23:17
2. What are you describing is metadata, just in way that is unlike how the engine does it. The omission of metadata support from MTS files is intentional since it was invented for mapgen decorations.
Nope, that ain't true. See my comments and links into the source below.
sfan5 wrote:
Sat Mar 13, 2021 23:17
Metadata would also break functionality like minetest.place_schematic_on_vmanip, since vmanips do not handle metadata (differently put: the focus on just the "basics" is intentional).
Nope, it definitely won't. Loading WE Lua tables with the place_schematic Lua API doesn't break vmanip, right?

So if loading a Lua table in line 168

Code: Select all

schem = load_schematic_from_def(L, index, ndef, replace_names);
does not break vmanip, then there isn't really any reason why line 177 would (which in turn calls deserializeFromMts()):

Code: Select all

schem = SchematicManager::create(SCHEMATIC_NORMAL);
schem->loadSchematicFromFile(filepath, ndef, replace_names)
I was thinking about loading param3 into exactly the same class Schematic fields where load_schematic_from_def loads them from a WE Lua table (actually into the very same *schem variable). Minimal modification, as I've said :-) I did my homework before I said I'm really willing to implement this :-)
sfan5 wrote:
Sat Mar 13, 2021 23:17
3. I agree. If the MTS format is changed at one point, doing this would be good.
Right? And it's just a simple thing.
sfan5 wrote:
Sat Mar 13, 2021 23:17
If you'd still like to implement metadata support in MTSEdit you could consider adding the WorldEdit file format as an option. Unlike MTS it does not support probabilities but does fully support metadata.
Well, both the MTS files and the WE Lua tables are loaded in the same function into the same Schematic object (actually into the very same local variable). Sadly WE format is extremely bloated and extremely difficult to parse without a Lua interpreter. I was thinking about that btw., but users and modders don't like it, all mods are using .mts files, so .we is a no go.

Cheers,
bzt

User avatar
sfan5
Moderator
Posts: 4013
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

Re: New MTS format

by sfan5 » Post

bzt wrote:
Sun Mar 14, 2021 04:07
Because how do you know what to put in that place_offset_y if it isn't stored in the MTS file? MTS files aren't limited to mapgens any more, plus requiring coding a manual place_offset_y makes it harder to replace schematics in mapgens.
That's true, it would be more convenient for usage both inside and outside of mapgen.
I see two remaining concerns regarding this:
• How would this interact with a place_offset_y set in a decoration definitions? Would either just always take priority?
• It can be confusing that if you load a schematic at a given position, it might appear with an arbitrary vertical offset. I guess WorldEdit could / should provide some convenience functions to determine the real size and offset of a schematic.
bzt wrote:
Sun Mar 14, 2021 04:07
Nope, it definitely won't. Loading WE Lua tables with the place_schematic Lua API doesn't break vmanip, right?

So if loading a Lua table in line 168

Code: Select all

schem = load_schematic_from_def(L, index, ndef, replace_names);
does not break vmanip, then there isn't really any reason why line 177 would (which in turn calls deserializeFromMts()):

Code: Select all

schem = SchematicManager::create(SCHEMATIC_NORMAL);
schem->loadSchematicFromFile(filepath, ndef, replace_names)
I was thinking about loading param3 into exactly the same class Schematic fields where load_schematic_from_def loads them from a WE Lua table (actually into the very same *schem variable). Minimal modification, as I've said :-) I did my homework before I said I'm really willing to implement this :-)
place_schematic does not accept WorldEdit schematics so the premise here is wrong.

The reason that adding metadata doesn't work is found inside Schematic::blitToVManip(), which places the schematic on a VoxelManipulator. VoxelManipulators do not contain metadata, so even if you had loaded some metadata you could not put it anywhere. Them not handling metadata is intentional since they are used heavily by mapgen and their speed depends on just doing just the basics.
bzt wrote:
Sun Mar 14, 2021 04:07
Sadly WE format is extremely bloated and extremely difficult to parse without a Lua interpreter. I was thinking about that btw., but users and modders don't like it, all mods are using .mts files, so .we is a no go.
That's unfortunately true, but this shouldn't prevent you from adding an export-only option for .we (I assume MTSEdit has its' own format for "normal" saving).
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

bzt
Member
Posts: 133
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

sfan5 wrote:
Sun Mar 14, 2021 13:22
• How would this interact with a place_offset_y set in a decoration definitions? Would either just always take priority?
I would recommend both. My suggestion is DecoSchematic::generate() in mg_decoration.cpp the line currently p.Y += place_offset_y; should be replaced by

Code: Select all

p.Y += place_offset_y - schematic->groundlevel;
So if version 4 MTS is loaded or version 5 with groundlevel 0 it would work exactly as now. With groundlevel being bigger than zero, the Y coordinate would be adjusted, that's all. Actually all I want to achieve with this is, that when the Lua place_schematic API is used, move the schem down by a value defined by the schem, exactly the same way how place_offset_y moves it down. The one and only difference is, groundlevel isn't a mapgen or whatever calculated nor a user provided value, but comes from the schem itself.
sfan5 wrote:
Sun Mar 14, 2021 13:22
• It can be confusing that if you load a schematic at a given position, it might appear with an arbitrary vertical offset. I guess WorldEdit could / should provide some convenience functions to determine the real size and offset of a schematic.
It doesn't influence the real size of the schematic just like place_offset_y doesn't influence the size either.
sfan5 wrote:
Sun Mar 14, 2021 13:22
place_schematic does not accept WorldEdit schematics so the premise here is wrong.
Oh, my apologies. I was totally convinced that the Lua table that place_schematic loads and the Lua table that WE saves are the same. My bad. Still, I don't think adding a string property to the Schematic class that vmanip doesn't reference would be a problem.
sfan5 wrote:
Sun Mar 14, 2021 13:22
The reason that adding metadata doesn't work is found inside Schematic::blitToVManip(), which places the schematic on a VoxelManipulator. VoxelManipulators do not contain metadata, so even if you had loaded some metadata you could not put it anywhere. Them not handling metadata is intentional since they are used heavily by mapgen and their speed depends on just doing just the basics.
I understand, but I don't see any reason why couldn't the place_schematic hook call another method after calling vmanip which would use exactly the same functions like WE does. It seems perfectly doable, but I admit, this needs more coding than I've originally thought.
sfan5 wrote:
Sun Mar 14, 2021 13:22
That's unfortunately true, but this shouldn't prevent you from adding an export-only option for .we
I see no point in that if modders aren't using the .we format, only .mts. (I'm planning to implement .we import though, however reading Lua tables without a Lua interpreter seems to be a nightmare, needs lot of work.)
sfan5 wrote:
Sun Mar 14, 2021 13:22
I assume MTSEdit has its' own format for "normal" saving
Nope, MTSEdit's primary format is MTS, it was specifically written to give support for that format. It can load and save in that format, and you can modify all aspects of the schematic stored in that format (including but not limited to layer probabilities, arbitrary param1 and param2 values, force placement bit, converting between MTG and MCL2 node palette etc.).

You see, it can import all voxel formats (MC NBT, Sponge NBT, VOX, GOX, Qubicle, binvox, even Tiled, you name it) and convert them into MTS with a specific node palette, that's the tool's main purpose. The one and only non-MTS format it can export to is VOX images, but just to provide two-way workflows for the modders, and that's very limited and not really useful because VOX does not store node names, only colors, let alone chests contents and such.

Cheers,
bzt

Sokomine
Member
Posts: 4202
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: New MTS format

by Sokomine » Post

Having the y-offset stored in the .mts file would be very helpful. In handle_schematics, I'm storing rotation (most builds have a front side, and that may not always be the same) and y offset in the file name - because there is no other reasonable way.

minetest.place_schematic is more or less useless for placing non-mapgen schematics because it caches the replacements and does not allow to supply a new set.

handle_schematic has an experimental feature for storing metadata. It's saved in an extra .meta file. I didn't have much time lately to work on that but it certainly deserves more love. Things in chests, signs, written books - all those are important when moving a building. And they need to be rotated to the correct position as well.
A list of my mods can be found here.

bzt
Member
Posts: 133
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

Sokomine wrote:
Thu Jul 08, 2021 23:02
Having the y-offset stored in the .mts file would be very helpful.
Thanks for backing!
Sokomine wrote:
Thu Jul 08, 2021 23:02
In handle_schematics, I'm storing rotation (most builds have a front side, and that may not always be the same) and y offset in the file name - because there is no other reasonable way.
I come around that problem with always storing the schem in the file in a specific direction (when you use rotation=0 with place schematic). I can do that, because my MTSEdit can rotate schems, so users can set the correct direction before they press the save button, however I agree this isn't always possible with all the tools.

So here's a little bit modified proposal:

Code: Select all

| Offset  | Length | Description                              |
|--------:|-------:|------------------------------------------|
|       0 |      4 | magic "MTSM"                             |
|       4 |      2 | file format version, 5                   |
|       6 |      2 | size X                                   |
|       8 |      2 | size Y                                   |
|      10 |      2 | size Z                                   |
|      12 |      2 | ground Y                                 |
|      14 |      2 | rotation (0, 90, 180, 270)               |
|      16 |      2 | number of strings in Name-ID table       |
|      18 |      x | Name-ID table                            |
|       x |      x | Block definitions (zlib compressed)      |
This additional value should be simply added to the place_schematic's rotation parameter modulo 360. (So if the field in the file is zero, it would work as now, no difference. If the field would be 90, then placing with rotation=0 would actually place as 90 degree, placing with rotation=90 as 180 degree etc.) This would only involve adjusting the rotation parameter, which means only minimal modification required in the engine.

Cheers,
bzt

Sokomine
Member
Posts: 4202
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine

Re: New MTS format

by Sokomine » Post

bzt wrote: This additional value should be simply added to the place_schematic's rotation parameter modulo 360. (So if the field in the file is zero, it would work as now, no difference. If the field would be 90, then placing with rotation=0 would actually place as 90 degree, placing with rotation=90 as 180 degree etc.) This would only involve adjusting the rotation parameter, which means only minimal modification required in the engine.
That sounds great! Having to store these two values in the file name is a really dirty hack I had to resort to. While both values may not be necessary for mapgen placed schematics, they're very helpful for all other uses of them and wouldn't really bloat the format.

Another option might be to add another layer and to contain y offset, rotation, metadata and the .mts file in a new file format. The disadvantage is that that would hide the .mts file and require support from many other mods that deal with schematics.
A list of my mods can be found here.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest