New MTS format

For people working on the C++ code.
Post Reply
bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

New MTS format

by bzt » Post

Dear developers,

I'd like to propose some modifications to the MTS format, version 5. I'm doing so because I keep getting issues with my MTSEdit, people asking for features that are simply not possible with version 4.

I've gave it a lot of though to minimize these modifications, and using the MTS version would allow to simultaneously support both version 4 and version 5 in the Minetest engine. I'm also willing to code this and provide a PR when the format is finalized.

Proposed modifications:
1. add ground level after size Z (same as WEOffsetY in MC schematics)
2. add param3 as a string property (for chests and command blocks and probably others)
3. move layer probability into the zlib compressed block (not important, but would be nice to have)

So the header would look like this:

Code: Select all

| Offset  | Length | Description                              |
|--------:|-------:|------------------------------------------|
|       0 |      4 | magic "MTSM"                             |
|       4 |      2 | file format version, 5                   |
|       6 |      2 | size X                                   |
|       8 |      2 | size Y                                   |
|      10 |      2 | size Z                                   |
|      12 |      2 | ground Y                                 |
|      14 |      2 | number of strings in Name-ID table       |
|      16 |      x | Name-ID table                            |
|       x |      x | Block definitions (zlib compressed)      |
Ground Y is a value that's simply substracted from the Y coordinate when schematics are placed. With being 0, it would work like now. With values greater than zero, it would allow to place underground parts of the schematic correctly (same way as WEOfffsetY works in Minecraft).

And the zlib compressed block would look like:

Code: Select all

| Offset    | Length    | Description                              |
|----------:|----------:|------------------------------------------|
|         0 |         Y | layer probability values                 |
|         Y |   2*X*Y*Z | block IDs (param0)                       |
| Y+2*X*Y*Z |     X*Y*Z | probability values (param1)              |
| Y+3*X*Y*Z |     X*Y*Z | rotation info (param2)                   |
| Y+4*X*Y*Z | 2*X*Y*Z+n | property strings                         |
Here the 2nd, 3rd and 4th blocks are the same as now. 1st block is now uncompressed, this proposal would move that inside the zlib compressed block otherwise the format of layer probabilities wouldn't change. The 5th block would be an addition, that's currently missing from MTS files and would be very much needed.

When no properties defined at all (as with current MTS), the last block would be 2*X*Y*Z zeros (something that would be compressed to very small size with zlib, probably no more than 3-5 additional bytes to the file size). Otherwise for nodes without property, would be two zero bytes, and a big-endian string length and string for nodes with properties (same encoding as in Name-ID table).

This property string could be used depending on the node's type: for example chests could store their list of contents here, and command block the list of commands. The same callback could be used to parse these strings as the current Lua callback.

An alternative would be to add these property strings to the Name-ID table, and there would be no need for param3 block, but that would mean a more drastic change in the MTS specification (Name wouldn't be the node's Name only any more), an a whole different parsing method in the C++ code.

What are your thoughts?

Cheers,
bzt

sfan5
Moderator
Posts: 4093
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

Re: New MTS format

by sfan5 » Post

A few points:

1. I don't quite get why the MTS format should have an y offset. For use with mapgen decorations there is a place_offset_y that can be defined per decoration and it's necessary there.
But if users are just manually placing their schematics, they should set the positions to match the contents of the schematic.

2. What are you describing is metadata, just in way that is unlike how the engine does it. The omission of metadata support from MTS files is intentional since it was invented for mapgen decorations.
Metadata would also break functionality like minetest.place_schematic_on_vmanip, since vmanips do not handle metadata (differently put: the focus on just the "basics" is intentional).

3. I agree. If the MTS format is changed at one point, doing this would be good.

If you'd still like to implement metadata support in MTSEdit you could consider adding the WorldEdit file format as an option. Unlike MTS it does not support probabilities but does fully support metadata.
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

sfan5 wrote:
Sat Mar 13, 2021 23:17
A few points:
Thanks for your feedback!
sfan5 wrote:
Sat Mar 13, 2021 23:17
1. I don't quite get why the MTS format should have an y offset. For use with mapgen decorations there is a place_offset_y that can be defined per decoration and it's necessary there.
Because how do you know what to put in that place_offset_y if it isn't stored in the MTS file? MTS files aren't limited to mapgens any more, plus requiring coding a manual place_offset_y makes it harder to replace schematics in mapgens.
sfan5 wrote:
Sat Mar 13, 2021 23:17
But if users are just manually placing their schematics, they should set the positions to match the contents of the schematic.
You cannot expect the users to know how big the underground part is in a schem before they place them. They can find the X and Z coordinate easily, but not the Y one, they expect the schem to be placed on the ground (which works fine only if the schem doesn't have an underground part like the pyramid for example.)

It's a simple modification btw, at line 253:

Code: Select all

	if (flags & DECO_PLACE_CENTER_Y)
		p.Y -= (s.Y - 1) / 2;
 +      else
 +              p.Y -= goundlevel;
(where groundlevel is a property set up in the same class in the deserializeFromMts() method)
sfan5 wrote:
Sat Mar 13, 2021 23:17
2. What are you describing is metadata, just in way that is unlike how the engine does it. The omission of metadata support from MTS files is intentional since it was invented for mapgen decorations.
Nope, that ain't true. See my comments and links into the source below.
sfan5 wrote:
Sat Mar 13, 2021 23:17
Metadata would also break functionality like minetest.place_schematic_on_vmanip, since vmanips do not handle metadata (differently put: the focus on just the "basics" is intentional).
Nope, it definitely won't. Loading WE Lua tables with the place_schematic Lua API doesn't break vmanip, right?

So if loading a Lua table in line 168

Code: Select all

schem = load_schematic_from_def(L, index, ndef, replace_names);
does not break vmanip, then there isn't really any reason why line 177 would (which in turn calls deserializeFromMts()):

Code: Select all

schem = SchematicManager::create(SCHEMATIC_NORMAL);
schem->loadSchematicFromFile(filepath, ndef, replace_names)
I was thinking about loading param3 into exactly the same class Schematic fields where load_schematic_from_def loads them from a WE Lua table (actually into the very same *schem variable). Minimal modification, as I've said :-) I did my homework before I said I'm really willing to implement this :-)
sfan5 wrote:
Sat Mar 13, 2021 23:17
3. I agree. If the MTS format is changed at one point, doing this would be good.
Right? And it's just a simple thing.
sfan5 wrote:
Sat Mar 13, 2021 23:17
If you'd still like to implement metadata support in MTSEdit you could consider adding the WorldEdit file format as an option. Unlike MTS it does not support probabilities but does fully support metadata.
Well, both the MTS files and the WE Lua tables are loaded in the same function into the same Schematic object (actually into the very same local variable). Sadly WE format is extremely bloated and extremely difficult to parse without a Lua interpreter. I was thinking about that btw., but users and modders don't like it, all mods are using .mts files, so .we is a no go.

Cheers,
bzt

sfan5
Moderator
Posts: 4093
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5
Location: Germany

Re: New MTS format

by sfan5 » Post

bzt wrote:
Sun Mar 14, 2021 04:07
Because how do you know what to put in that place_offset_y if it isn't stored in the MTS file? MTS files aren't limited to mapgens any more, plus requiring coding a manual place_offset_y makes it harder to replace schematics in mapgens.
That's true, it would be more convenient for usage both inside and outside of mapgen.
I see two remaining concerns regarding this:
• How would this interact with a place_offset_y set in a decoration definitions? Would either just always take priority?
• It can be confusing that if you load a schematic at a given position, it might appear with an arbitrary vertical offset. I guess WorldEdit could / should provide some convenience functions to determine the real size and offset of a schematic.
bzt wrote:
Sun Mar 14, 2021 04:07
Nope, it definitely won't. Loading WE Lua tables with the place_schematic Lua API doesn't break vmanip, right?

So if loading a Lua table in line 168

Code: Select all

schem = load_schematic_from_def(L, index, ndef, replace_names);
does not break vmanip, then there isn't really any reason why line 177 would (which in turn calls deserializeFromMts()):

Code: Select all

schem = SchematicManager::create(SCHEMATIC_NORMAL);
schem->loadSchematicFromFile(filepath, ndef, replace_names)
I was thinking about loading param3 into exactly the same class Schematic fields where load_schematic_from_def loads them from a WE Lua table (actually into the very same *schem variable). Minimal modification, as I've said :-) I did my homework before I said I'm really willing to implement this :-)
place_schematic does not accept WorldEdit schematics so the premise here is wrong.

The reason that adding metadata doesn't work is found inside Schematic::blitToVManip(), which places the schematic on a VoxelManipulator. VoxelManipulators do not contain metadata, so even if you had loaded some metadata you could not put it anywhere. Them not handling metadata is intentional since they are used heavily by mapgen and their speed depends on just doing just the basics.
bzt wrote:
Sun Mar 14, 2021 04:07
Sadly WE format is extremely bloated and extremely difficult to parse without a Lua interpreter. I was thinking about that btw., but users and modders don't like it, all mods are using .mts files, so .we is a no go.
That's unfortunately true, but this shouldn't prevent you from adding an export-only option for .we (I assume MTSEdit has its' own format for "normal" saving).
Mods: Mesecons | WorldEdit | Nuke & Minetest builds for Windows (32-bit & 64-bit)

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

sfan5 wrote:
Sun Mar 14, 2021 13:22
• How would this interact with a place_offset_y set in a decoration definitions? Would either just always take priority?
I would recommend both. My suggestion is DecoSchematic::generate() in mg_decoration.cpp the line currently p.Y += place_offset_y; should be replaced by

Code: Select all

p.Y += place_offset_y - schematic->groundlevel;
So if version 4 MTS is loaded or version 5 with groundlevel 0 it would work exactly as now. With groundlevel being bigger than zero, the Y coordinate would be adjusted, that's all. Actually all I want to achieve with this is, that when the Lua place_schematic API is used, move the schem down by a value defined by the schem, exactly the same way how place_offset_y moves it down. The one and only difference is, groundlevel isn't a mapgen or whatever calculated nor a user provided value, but comes from the schem itself.
sfan5 wrote:
Sun Mar 14, 2021 13:22
• It can be confusing that if you load a schematic at a given position, it might appear with an arbitrary vertical offset. I guess WorldEdit could / should provide some convenience functions to determine the real size and offset of a schematic.
It doesn't influence the real size of the schematic just like place_offset_y doesn't influence the size either.
sfan5 wrote:
Sun Mar 14, 2021 13:22
place_schematic does not accept WorldEdit schematics so the premise here is wrong.
Oh, my apologies. I was totally convinced that the Lua table that place_schematic loads and the Lua table that WE saves are the same. My bad. Still, I don't think adding a string property to the Schematic class that vmanip doesn't reference would be a problem.
sfan5 wrote:
Sun Mar 14, 2021 13:22
The reason that adding metadata doesn't work is found inside Schematic::blitToVManip(), which places the schematic on a VoxelManipulator. VoxelManipulators do not contain metadata, so even if you had loaded some metadata you could not put it anywhere. Them not handling metadata is intentional since they are used heavily by mapgen and their speed depends on just doing just the basics.
I understand, but I don't see any reason why couldn't the place_schematic hook call another method after calling vmanip which would use exactly the same functions like WE does. It seems perfectly doable, but I admit, this needs more coding than I've originally thought.
sfan5 wrote:
Sun Mar 14, 2021 13:22
That's unfortunately true, but this shouldn't prevent you from adding an export-only option for .we
I see no point in that if modders aren't using the .we format, only .mts. (I'm planning to implement .we import though, however reading Lua tables without a Lua interpreter seems to be a nightmare, needs lot of work.)
sfan5 wrote:
Sun Mar 14, 2021 13:22
I assume MTSEdit has its' own format for "normal" saving
Nope, MTSEdit's primary format is MTS, it was specifically written to give support for that format. It can load and save in that format, and you can modify all aspects of the schematic stored in that format (including but not limited to layer probabilities, arbitrary param1 and param2 values, force placement bit, converting between MTG and MCL2 node palette etc.).

You see, it can import all voxel formats (MC NBT, Sponge NBT, VOX, GOX, Qubicle, binvox, even Tiled, you name it) and convert them into MTS with a specific node palette, that's the tool's main purpose. The one and only non-MTS format it can export to is VOX images, but just to provide two-way workflows for the modders, and that's very limited and not really useful because VOX does not store node names, only colors, let alone chests contents and such.

Cheers,
bzt

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: New MTS format

by Sokomine » Post

Having the y-offset stored in the .mts file would be very helpful. In handle_schematics, I'm storing rotation (most builds have a front side, and that may not always be the same) and y offset in the file name - because there is no other reasonable way.

minetest.place_schematic is more or less useless for placing non-mapgen schematics because it caches the replacements and does not allow to supply a new set.

handle_schematic has an experimental feature for storing metadata. It's saved in an extra .meta file. I didn't have much time lately to work on that but it certainly deserves more love. Things in chests, signs, written books - all those are important when moving a building. And they need to be rotated to the correct position as well.
A list of my mods can be found here.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

Sokomine wrote:
Thu Jul 08, 2021 23:02
Having the y-offset stored in the .mts file would be very helpful.
Thanks for backing!
Sokomine wrote:
Thu Jul 08, 2021 23:02
In handle_schematics, I'm storing rotation (most builds have a front side, and that may not always be the same) and y offset in the file name - because there is no other reasonable way.
I come around that problem with always storing the schem in the file in a specific direction (when you use rotation=0 with place schematic). I can do that, because my MTSEdit can rotate schems, so users can set the correct direction before they press the save button, however I agree this isn't always possible with all the tools.

So here's a little bit modified proposal:

Code: Select all

| Offset  | Length | Description                              |
|--------:|-------:|------------------------------------------|
|       0 |      4 | magic "MTSM"                             |
|       4 |      2 | file format version, 5                   |
|       6 |      2 | size X                                   |
|       8 |      2 | size Y                                   |
|      10 |      2 | size Z                                   |
|      12 |      2 | ground Y                                 |
|      14 |      2 | rotation (0, 90, 180, 270)               |
|      16 |      2 | number of strings in Name-ID table       |
|      18 |      x | Name-ID table                            |
|       x |      x | Block definitions (zlib compressed)      |
This additional value should be simply added to the place_schematic's rotation parameter modulo 360. (So if the field in the file is zero, it would work as now, no difference. If the field would be 90, then placing with rotation=0 would actually place as 90 degree, placing with rotation=90 as 180 degree etc.) This would only involve adjusting the rotation parameter, which means only minimal modification required in the engine.

Cheers,
bzt

Sokomine
Member
Posts: 4276
Joined: Sun Sep 09, 2012 17:31
GitHub: Sokomine
IRC: Sokomine
In-game: Sokomine

Re: New MTS format

by Sokomine » Post

bzt wrote: This additional value should be simply added to the place_schematic's rotation parameter modulo 360. (So if the field in the file is zero, it would work as now, no difference. If the field would be 90, then placing with rotation=0 would actually place as 90 degree, placing with rotation=90 as 180 degree etc.) This would only involve adjusting the rotation parameter, which means only minimal modification required in the engine.
That sounds great! Having to store these two values in the file name is a really dirty hack I had to resort to. While both values may not be necessary for mapgen placed schematics, they're very helpful for all other uses of them and wouldn't really bloat the format.

Another option might be to add another layer and to contain y offset, rotation, metadata and the .mts file in a new file format. The disadvantage is that that would hide the .mts file and require support from many other mods that deal with schematics.
A list of my mods can be found here.

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

The schematics should be replaced with similar materials when the specified materials do not exist. In addition, you should be able to note the materials you use, as well as the nature, color, texture, transparency and brightness of the materials, and the mods you use.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Mon Aug 23, 2021 09:11
The schematics should be replaced with similar materials when the specified materials do not exist. In addition, you should be able to note the materials you use, as well as the nature, color, texture, transparency and brightness of the materials, and the mods you use.
I'm not entirely sure what you mean, makes no sense.

Schematics are a format which store a node palette and a 3D grid indexing that palette. That's all. It has nothing to do with materials, color, texture etc. nor with mods. And that's the best and preferred way to storing node structures. Schems do not need to know how a stone block looks like (which might be different depending on mods and texture packs anyway), they only need to store which nodes are stones in the structure. Does this make sense to you?

Nodes are defined by mods, but schematics are only referring to those by their technical names, like "default:stone" for example. Obviously you have to enable all mods to register those nodes that your schem file's palette is referring to, otherwise you'll get "unknown node" errors.

You can also use my mtsedit tool to replace the technical names of the nodes in the schematics, but this has nothing to do with the node's material, color, texture, etc. it will just replace the string "default:stone" with the string "mcl_core:stone_block" or something for example.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Mon Aug 23, 2021 18:44
Schematics are a format which store a node palette and a 3D grid indexing that palette. That's all. It has nothing to do with materials, color, texture etc. nor with mods. And that's the best and preferred way to storing node structures. Schems do not need to know how a stone block looks like (which might be different depending on mods and texture packs anyway), they only need to store which nodes are stones in the structure. Does this make sense to you?
Yes It's the original schematics format. But now we are envisioning a new format. Your `mtsedit` mod can save these additional info into the new schematics format.
bzt wrote:
Mon Aug 23, 2021 18:44
Nodes are defined by mods, but schematics are only referring to those by their technical names, like "default:stone" for example. Obviously you have to enable all mods to register those nodes that your schem file's palette is referring to, otherwise you'll get "unknown node" errors.
Yes, if you are the author of schematics, that should be no probelm. But if you are the user of schematics, you'll get into trouble. If the mods used are in the schematics then at least I know which mods to install to use the file. Even further, can download and install these mods automatically via https://content.minetest.net/.
bzt wrote:
Mon Aug 23, 2021 18:44
You can also use my mtsedit tool to replace the technical names of the nodes in the schematics, but this has nothing to do with the node's material, color, texture, etc. it will just replace the string "default:stone" with the string "mcl_core:stone_block" or something for example.
Yes, you have the `blocks.csv` file to extends.

Maybe I said it too simple, I mean, should leave some places to be expanded in the future in the new format. Then it is implemented in two steps. The easiest to implement first is to indicate which mod (manual or automatic) the schematics uses. Then it is to realize the labeling material characteristics: stone-like(default:stone), metal-like, glass-like, liquid-like...

I don’t know if I have made it clear. But no problem just say it, you are welcome.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Tue Aug 24, 2021 00:15
Yes It's the original schematics format.
Not just that. Using only a node reference is common to all schematics formats, like Minecraft Schematics and Sponge Schematics too. You can't find any texture, material, color etc. definitions nor mod list in those either. Minecraft Schematics uses a global numeric id, and Sponge Schematics uses the same node palette concept with technical names and schem local indeces as the MTS format.
snowyu wrote:
Tue Aug 24, 2021 00:15
But now we are envisioning a new format.
Nope, not an entirely new format. We are talking about a new version of the already existing format, by expanding it to store two additional properties. I'm considering the needed development effort to, and I want to keep the required modifications as minimal as possible.
snowyu wrote:
Tue Aug 24, 2021 00:15
Your `mtsedit` mod can save these additional info into the new schematics format.
No can do, because the editor does not have that information. It uses a 32x32 icon to represent a node, that's it.
snowyu wrote:
Tue Aug 24, 2021 00:15
If the mods used are in the schematics then at least I know which mods to install to use the file.
Not possible, because there's no one-to-one relation between nodes and mods. Multiple mods can provide the same node (hence those mods are incompatible). For example, the node which goes by the name "mesecons_pistons:piston_sticky_off" can be provided by the mesecons mod or MCL2 or MCL5 internal mods too. Trying to install mesecons along with MCL would end in errors, because MCL already registers the piston node for example.
snowyu wrote:
Tue Aug 24, 2021 00:15
Maybe I said it too simple, I mean, should leave some places to be expanded in the future in the new format.
I would be very happy if I could get the devs to add these two static variables, I don't think they would agree to replace the MTS format with an entirely new, chunk-based format with optional contents.
snowyu wrote:
Tue Aug 24, 2021 00:15
The easiest to implement first is to indicate which mod (manual or automatic) the schematics uses.
Not possible as I've pointed out, there's no one-to-one relation between nodes and mods.
snowyu wrote:
Tue Aug 24, 2021 00:15
Then it is to realize the labeling material characteristics: stone-like(default:stone), metal-like, glass-like, liquid-like...
Not good either. First, a node characteristics cannot and should not change across schems, so there's absolutely no point in bloating the files with that info (a stone node in schem A and in schem B must look and act the same way when loaded into the same world). Second there's already mapgen aliases (like "mapgen_stone") that you can use if you don't know the actual node's technical name (but this feature of the engine isn't as good as it should be, I agree).

FYI, If you're into a more flexible format, take a look at Model 3D, which is exactly what you're talking about: it can store not only node palette and grid, but also optionally materials, textures, colors, other characteristics etc. It is a chunk based format which can store optional contents. For schematics example, go to Models > Geometry > Notre Dame. This format encodes more than half a million voxels in just 17k.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Tue Aug 24, 2021 13:33
No can do, because the editor does not have that information. It uses a 32x32 icon to represent a node, that's it.
I mean your mtsedit mod, not the editor.
bzt wrote:
Tue Aug 24, 2021 13:33
Not possible, because there's no one-to-one relation between nodes and mods. Multiple mods can provide the same node (hence those mods are incompatible). For example, the node which goes by the name "mesecons_pistons:piston_sticky_off" can be provided by the mesecons mod or MCL2 or MCL5 internal mods too. Trying to install mesecons along with MCL would end in errors, because MCL already registers the piston node for example.
When using `mtsedit-mod` to save the schematics, you can save additional information by the way:
  • The manual way: Let the author himself note which mods are used.
  • The automatic way: collect all the mods from current running env(see chat command: `/mods`). Maybe it’s not precise, but it works.
  • The semi-automatic: Let the author choose the used mod from the mod list
bzt wrote:
Tue Aug 24, 2021 13:33
Not good either. First, a node characteristics cannot and should not change across schems, so there's absolutely no point in bloating the files with that info (a stone node in schem A and in schem B must look and act the same way when loaded into the same world). Second there's already mapgen aliases (like "mapgen_stone") that you can use if you don't know the actual node's technical name (but this feature of the engine isn't as good as it should be, I agree).
I think the schematics is a kindof blueprint. When constructing according to the blueprint, if a certain material does not exist, is it not normal to replace it with another similar material? And this makes it easier to share schematics in different game worlds.
bzt wrote:
Tue Aug 24, 2021 13:33
FYI, If you're into a more flexible format, take a look at Model 3D, which is exactly what you're talking about: it can store not only node palette and grid, but also optionally materials, textures, colors, other characteristics etc. It is a chunk based format which can store optional contents. For schematics example, go to Models > Geometry > Notre Dame. This format encodes more than half a million voxels in just 17k.
That's great. Could you use M3D instead of schematics in minetest?

IMHO, the mts current format should be called node list and not a blueprint. Mts still has a long way to go.

BTW, I am mainly engaged in software architecture work, so this is not my specialty about 3D.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Wed Aug 25, 2021 07:43
When using `mtsedit-mod` to save the schematics, you can save additional information by the way:
Nope, because the mod just calls the create_schematic Lua API. It does not save the schem per se, it delegates the job to the engine, therefore saving any additional info must be implemented in the engine and not in the mod.
snowyu wrote:
Wed Aug 25, 2021 07:43
  • The manual way: Let the author himself note which mods are used.
  • The automatic way: collect all the mods from current running env(see chat command: `/mods`). Maybe it’s not precise, but it works.
  • The semi-automatic: Let the author choose the used mod from the mod list
You're still missing the point, neither of these will solve the issue that multiple mods might implement the same node. This would add huge bloat to every schem without any real benefit.
snowyu wrote:
Wed Aug 25, 2021 07:43
I think the schematics is a kindof blueprint. When constructing according to the blueprint, if a certain material does not exist, is it not normal to replace it with another similar material?
No, it is not. I haven't seen any IRL blueprints that would explain the material. For example a blueprint for a house just refers to concrete, but it never contains a detailed description of how to create concrete or what the actual chemical composition of the concrete is.
snowyu wrote:
Wed Aug 25, 2021 07:43
And this makes it easier to share schematics in different game worlds.
Absolutely not. How would you manage the event handlers of the nodes? There's a good reason why a mod registers the nodes, because that same mod must also implement the event handlers for the node. Even if you store the entire mod with all the Lua code in the schem, that's just waste of storage space and wouldn't provide shareability (a schem with an MTG mod in it wouldn't load into an MCL world because that mod isn't compatible with the MCL game).

If you want to move schems between different worlds, use a tool like MTSEdit or Sokomine's Handle Schematics to replace the technical names and you're good to go.
snowyu wrote:
Wed Aug 25, 2021 07:43
That's great. Could you use M3D instead of schematics in minetest?
If you could, then I would just use that instead of having hard time figuring out the minimal changes required for MTS. By the way, it is ready, well-documented, well-tested, integrated into multiple FOSS projects, which is a lot more than your (as of now) fictional format. That's why I've recommended it. If you want to add such support into the MT engine, use an existing, known-to-work library, don't try to reinvent the wheel.
snowyu wrote:
Wed Aug 25, 2021 07:43
IMHO, the mts current format should be called node list and not a blueprint.
As I've noted before, IRL blueprints does not store ingredients, texture etc. information either, just a reference to the material to be used. Same with MTS, therefore MTS is a blueprint.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Thu Aug 26, 2021 14:57
You're still missing the point, neither of these will solve the issue that multiple mods might implement the same node. This would add huge bloat to every schem without any real benefit.
I understand that this does not solve the problem of multiple mods registering nodes with the same name, but at least I can know which mods are used by schematics. It's no benefit from the creator's perspective, but from the user's perspective... eg, I can not even use your example schematics for using minetest-game.

Yes, this way can only be regarded as a temporary and simple solution: you don't need to list the use of mods in an additional documentation, just download schematics and use it.

And the author just upload the schematics file to forum, no additional explanation is required at all, the forum can directly read the file and display the relevant information.
bzt wrote:
Thu Aug 26, 2021 14:57
No, it is not. I haven't seen any IRL blueprints that would explain the material. For example a blueprint for a house just refers to concrete, but it never contains a detailed description of how to create concrete or what the actual chemical composition of the concrete is.
If we want to build the metaverse("internet") of the game world, then such a blueprint is indispensable. I hope that in this way, schematic creators can focus on design and creation instead of focusing on different games.

Without such detailed description how to create object via schematics across games/applications?
bzt wrote:
Thu Aug 26, 2021 14:57
snowyu wrote:
Wed Aug 25, 2021 07:43
And this makes it easier to share schematics in different game worlds.
Absolutely not. How would you manage the event handlers of the nodes? There's a good reason why a mod registers the nodes, because that same mod must also implement the event handlers for the node. Even if you store the entire mod with all the Lua code in the schem, that's just waste of storage space and wouldn't provide shareability (a schem with an MTG mod in it wouldn't load into an MCL world because that mod isn't compatible with the MCL game).
Yes, the only thing schematics can do at most is the view, and interaction is not possible, at least for now.
If you want to achieve interaction, I think it is more appropriate to upgrade to the 3D Component standard, similar to Web Component. There is no such standard currently.

The only simple "interaction" is through the properties of matter:
  • density
  • liquefaction temperature(melting point)
  • vaporization temperature(becoming gas)
  • electrical conductivity
  • ...
Only when the blueprint has the attributes representing the nodes, the application can determine whether the building can be blown away by the wind or burned in the environment where it is placed.
Of course, these are unnecessary if they are in the same game world.

bzt wrote:
Thu Aug 26, 2021 14:57
If you want to move schems between different worlds, use a tool like MTSEdit or Sokomine's Handle Schematics to replace the technical names and you're good to go.
This is a terrible experience. I have to waste more time on this. Originally, I didn't have much free time.
I am not a 3D designer or game creator. I just wanna use MT to teach my kid(cognition of architecture,gravity, acceleration, etc.). Placing some buildings in the game world is a minimum. So I have to search for real simple schematics and find this discussion.

The roblox has done a little simplification and integration work to brag about the "metaverse", in fact, it is still very early too. I believe metaverse is not a commercial company can be made out.
bzt wrote:
Thu Aug 26, 2021 14:57
If you could, then I would just use that instead of having hard time figuring out the minimal changes required for MTS. By the way, it is ready, well-documented, well-tested, integrated into multiple FOSS projects, which is a lot more than your (as of now) fictional format. That's why I've recommended it. If you want to add such support into the MT engine, use an existing, known-to-work library, don't try to reinvent the wheel.
It's good as a low-level schematics library. It's better if can support the sem-version info in M3D format. And a little questions:
  • Could the skeleton specify the range(limits) of motion angles?
  • Could reference another schematics file as component?
  • how to specify the rigid body, fluid or soft body as component?
bzt wrote:
Thu Aug 26, 2021 14:57
As I've noted before, IRL blueprints does not store ingredients, texture etc. information either, just a reference to the material to be used. Same with MTS, therefore MTS is a blueprint.
I do not know what's the IRL blueprint. I just know the architectural blueprint. I found that what we discussed may not be on the same channel.

My care is the schematics standard how to be used in any 3d game application. How can it be simpler and make it easier to use for those who are not very proficient in 3d(include me). At least It could be extened in the future.

Code: Select all

| Offset  | Length | Description                              |
|--------:|-------:|------------------------------------------|
|       0 |      4 | magic "MTSM"                             |
|       4 |      2 | file format version, 5                   |
|       6 |      2 | size X                                   |
|       8 |      2 | size Y                                   |
|      10 |      2 | size Z                                   |
|      12 |      2 | ground Y                                 |
|      14 |      4 | The addtional info length |
|      18 |      x | The addtional info can extened for future |
The addinal info:

Code: Select all

|      0 |      2 | The Major Versoin number     |
|      2 |      2 | The Minor Versoin number      |
|      4 |      2 | The Patch Versoin number      |
|      8 |      ? | The author (pascal-str) |
|       x |     ? | The license (pascal-str) |
|        x|      2| The count of mods(Temporary solution)      |
|        x|      ?| The first mod name (pascal-str)  |
|       x |      2 | The Major Versoin number of the first mod    |
|       x |      2 | The Minor Versoin number of the first mod     |
|       x |      2 | The Patch Versoin number of the first mod     |
...
pascal-str: the first byte is the string length.

The optional node descriptor(instead of mods, but need engine do more):

* id:node id
* name: the general name, eg, copper, iron, tin ...
* characteristics: optional
* color
* refractive index
* extinction coefficient
* luminous(light) intensity
* texture
* density
* liquefaction temperature(melting point)
* vaporization temperature(becoming gas)
* electrical conductivity
* ...

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Fri Aug 27, 2021 13:23
eg, I can not even use your example schematics for using minetest-game.
Yes, you can, just load them in the editor and select "Minetest Game" node palette in the save window. Or with the Handle Schematics mod you should be able to load into the world as-is (because that mod converts the names during load).
snowyu wrote:
Fri Aug 27, 2021 13:23
And the author just upload the schematics file to forum, no additional explanation is required at all, the forum can directly read the file and display the relevant information.
Yes, that would be great, but that's not how the MT engine works. You'd need a global node-kind list for that, shared across all games and mods. Like a periodic table for virtual worlds (which is still quite a big task of its own), but no such thing exists.
snowyu wrote:
Fri Aug 27, 2021 13:23
The only simple "interaction" is through the properties of matter:
  • density
  • liquefaction temperature(melting point)
  • vaporization temperature(becoming gas)
  • electrical conductivity
  • ...
I ask again, do these change from schem to schem? And if not then why on earth would you want to bloat schems with these information? I mean, why don't you want to store value of Pi, the Euler-constant and the speed of light into schems as well?

(Furthermore, these aren't material specific only values, rather heavily depends on the world, for example the boiling point of water is different at sea-level and in the top of Himalayas or on the Moon. Properties should change depending on where you place the schem in the world.)
snowyu wrote:
Fri Aug 27, 2021 13:23
bzt wrote:
Thu Aug 26, 2021 14:57
If you want to move schems between different worlds, use a tool like MTSEdit or Sokomine's Handle Schematics to replace the technical names and you're good to go.
This is a terrible experience. I have to waste more time on this.
Like it or not, this is how things are. Different MT games use different node names, you have to convert between those.
snowyu wrote:
Fri Aug 27, 2021 13:23
I just wanna use MT to teach my kid(cognition of architecture,gravity, acceleration, etc.).
Man, this is just a voxel game, the MT engine isn't the Matrix. Don't expect reality-like omnipotent physics simulation which can reproduce the two-holes experiment as well as the Tacoma Narrows bridge accident.
snowyu wrote:
Fri Aug 27, 2021 13:23
I do not know what's the IRL blueprint. I just know the architectural blueprint.
The same. (IRL = in real life) No architectural blueprint contains the boiling point of water or the chemical composition of a cobblestone for example.
snowyu wrote:
Fri Aug 27, 2021 13:23
My care is the schematics standard how to be used in any 3d game application. How can it be simpler and make it easier to use for those who are not very proficient in 3d(include me). At least It could be extened in the future.
That's exactly what Model3D is about.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Fri Aug 27, 2021 20:20
I ask again, do these change from schem to schem? And if not then why on earth would you want to bloat schems with these information? I mean, why don't you want to store value of Pi, the Euler-constant and the speed of light into schems as well?
All are optional, if it's the RL material, of cause it's unnecessary. But if it's fantasy. Do you really think metaverse is a mirror of the real world?
bzt wrote:
Fri Aug 27, 2021 20:20
(Furthermore, these aren't material specific only values, rather heavily depends on the world, for example the boiling point of water is different at sea-level and in the top of Himalayas or on the Moon. Properties should change depending on where you place the schem in the world.)
You should read it carefully about the boiling point:
wikipedia wrote: The boiling point of a substance is the temperature at which the vapor pressure of a liquid equals the pressure surrounding the liquid and the liquid changes into a vapor.

The normal boiling point (also called the atmospheric boiling point or the atmospheric pressure boiling point) of a liquid is the special case in which the vapor pressure of the liquid equals the defined atmospheric pressure at sea level, one atmosphere. At that temperature, the vapor pressure of the liquid becomes sufficient to overcome atmospheric pressure and allow bubbles of vapor to form inside the bulk of the liquid. The standard boiling point has been defined by IUPAC since 1982 as the temperature at which boiling occurs under a pressure of one bar.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Sat Aug 28, 2021 13:11
Do you really think metaverse is a mirror of the real world?
No, I have no clue what made you think that. I've said the opposite actually, and I was proposing a virtual "periodic table" for virtual worlds (aka. global node definitions consistent across mods, games and worlds, but I have also pointed out it is unlikely to happen).
snowyu wrote:
Sat Aug 28, 2021 13:11
You should read it carefully about the boiling point:
It says exactly what I've said, the boiling point of water is different at sea level and in the Himalayas (eg. where the air pressure is considerably lower). That's what I've said: it isn't a constant, depends where you place the schem in the world, therefore boiling point can't be the property of a schem.


I still don't get it why do you want to store information in schem which does not belong there. Properties of a node must not change from schem to schem, but could change from world to world (or even within the world) therefore there's no reason to store those in a schem. Simple as that.

If you want to make cross-world compatible schems, you have to options:
  • use a dictionary to convert technical node names between worlds (eg. "default:stone" represents the same node as "mcl_core:stone")
  • make all mods register mapgen alises for their nodes, and only use mapgen aliases in schems (like "mapgen_stone" for example, which is resolved by the engine to the proper technical name depending on the world / loaded mods)
The second option is the better solution IMHO, however not all mods register aliases, and you can't force all mods to do so, that's why I've went with the dictionary approach in my editor (but also supports node aliases "palette" BTW).

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Sat Aug 28, 2021 17:54
No, I have no clue what made you think that. I've said the opposite actually, and I was proposing a virtual "periodic table" for virtual worlds (aka. global node definitions consistent across mods, games and worlds, but I have also pointed out it is unlikely to happen).
Referencing the virtual periodic table can indeed reduce the size. But where to reference it, this may not be realized until the next to next step. It may happen, but it takes more time to trial and error.
bzt wrote:
Sat Aug 28, 2021 17:54
It says exactly what I've said, the boiling point of water is different at sea level and in the Himalayas (eg. where the air pressure is considerably lower). That's what I've said: it isn't a constant, depends where you place the schem in the world, therefore boiling point can't be the property of a schem.
Image
The Clausius-Clapeyron equation based on the ideal gas assumption can easily express the relationship between saturated vapor pressure and "the normal boiling point".

Or the relationship between pressure and boiling point can even be simplified to a linear relationship.
bzt wrote:
Sat Aug 28, 2021 17:54
I still don't get it why do you want to store information in schem which does not belong there. Properties of a node must not change from schem to schem, but could change from world to world (or even within the world) therefore there's no reason to store those in a schem. Simple as that.
You seem to feel that everything is unknown, unpredictable in different game worlds. But I think there are still some things that can be done.
bzt wrote:
Sat Aug 28, 2021 17:54
If you want to make cross-world compatible schems, you have to options:
  • use a dictionary to convert technical node names between worlds (eg. "default:stone" represents the same node as "mcl_core:stone")
  • make all mods register mapgen alises for their nodes, and only use mapgen aliases in schems (like "mapgen_stone" for example, which is resolved by the engine to the proper technical name depending on the world / loaded mods)
The second option is the better solution IMHO, however not all mods register aliases, and you can't force all mods to do so, that's why I've went with the dictionary approach in my editor (but also supports node aliases "palette" BTW).
Again, I understand that this is the current obligatory workaround.

But what you are talking about is the current status quo, and I am talking about the future. This is why I said that we are not on the same channel.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Sun Aug 29, 2021 05:47
Referencing the virtual periodic table can indeed reduce the size.
No, it wouldn't. It would need exactly the same amount of space to store, the only difference is that the node technical names would be standardized and shared across mods and worlds. (Again, not going to happen soon.)
snowyu wrote:
Sun Aug 29, 2021 05:47
Or the relationship between pressure and boiling point can even be simplified to a linear relationship.
Yeah, I know, I've learned that in elementary school. You still haven't answered the question what would be the benefit (if any) of storing such schem-independent info in each and every schem. I see no point in that.
snowyu wrote:
Sun Aug 29, 2021 05:47
You seem to feel that everything is unknown, unpredictable in different game worlds.
You make no sense. You have definitely made the wrong conclusion here.
snowyu wrote:
Sun Aug 29, 2021 05:47
This is why I said that we are not on the same channel.
Definitely. I want to keep it as simple as possible (storage-wise as well as engine-modification-wise), while you want to store unnecessary bloat in schems which would require rewriting huge parts of engine (not just the MTS-related things, but the node registration and mods interface too).

Please understand that this topic is about the smallest change needed to address the most lacking features, and not about redesigning the entire format from ground up.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

bzt wrote:
Sun Aug 29, 2021 19:47
No, it wouldn't. It would need exactly the same amount of space to store, the only difference is that the node technical names would be standardized and shared across mods and worlds. (Again, not going to happen soon.)
I can not understand your words. Do you think the size of url referencing and directly embedding content into the file is the same?

URI way:
  • id: tin
  • name: tin
  • characteristics: file://./tin-characteristics.vpt
Embeded way:
bzt wrote:
Sun Aug 29, 2021 19:47
Yeah, I know, I've learned that in elementary school. You still haven't answered the question what would be the benefit (if any) of storing such schem-independent info in each and every schem. I see no point in that.
Ok, this is another question. Maybe you don't think that it is very important to be able to use schematics in other (any) virtual world applications(games). And use multiple schematics combinations to generate new ones.
bzt wrote:
Sun Aug 29, 2021 19:47
I want to keep it as simple as possible (storage-wise as well as engine-modification-wise), while you want to store unnecessary bloat in schems which would require rewriting huge parts of engine (not just the MTS-related things, but the node registration and mods interface too).

Please understand that this topic is about the smallest change needed to address the most lacking features, and not about redesigning the entire format from ground up.
Then maybe "a little improvement for MTS format" is a more correct title. Sorry, my fault. I misunderstood your intentions.

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

snowyu wrote:
Tue Aug 31, 2021 09:40
bzt wrote:
Sun Aug 29, 2021 19:47
Yeah, I know, I've learned that in elementary school. You still haven't answered the question what would be the benefit (if any) of storing such schem-independent info in each and every schem. I see no point in that.
Ok, this is another question.
No, this is exactly the same question I'm keep asking you and you refusing to answer.
snowyu wrote:
Tue Aug 31, 2021 09:40
Maybe you don't think that it is very important to be able to use schematics in other (any) virtual world applications(games). And use multiple schematics combinations to generate new ones.
From my previous posts it is perfectly clear what I think about schem compatibility and how to implement that, I won't repeat myself (again). What I think is, that you're deliberately trolling. Please don't pollute these forums, go somewhere else. I won't reply to you any more.

Thank you.
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

last words, I've answered your question. but you can not see it:
snowyu wrote:
Fri Aug 27, 2021 13:23
The only simple "interaction" is through the properties of matter...
snowyu wrote:
Tue Aug 31, 2021 09:40
Maybe you don't think that it is very important to be able to use schematics in other (any) virtual world applications(games). And use multiple schematics combinations to generate new ones.
Let me summarize here:
  • The matter and it's properties could be fantasy. And All are optional.
  • Implement the simplest "interaction" through the properties of matter
  • Achieving complete interaction requires support similar to HTML Script standards
  • Use schematics with simplest "interaction" in other (any) virtual world applications(games).
  • Use multiple schematics combinations to generate a new schematic.
  • Make schematic easier to use, even non-professionals can use it.
  • Use semantic versioning to manage the schematics.
  • Refer to another schematic file with version info via URI
  • Refer to virutal matter file with version info via URI or embeded directly
My idea is to create a metaverse standard specification similar to the Web. But this must start from bit by bit.

Maybe you always look down at the ground and forget to look up for the beauty of the sky.
bzt wrote:
Tue Aug 31, 2021 18:35
From my previous posts it is perfectly clear what I think about schem compatibility and how to implement that, I won't repeat myself (again). What I think is, that you're deliberately trolling. Please don't pollute these forums, go somewhere else. I won't reply to you any more.
There is a sentence to say: "I disapprove of what you say, but I will defend to the death your right to say it". So please calm down, this is not my intention. As you feel, I have been explaining your question from different angles, but you have been asking the same question over and over again. But in the end I understand that you want to make small enhancements, not look to the future. I'm sorry that I got off the topic, but really, if your discussion title is "Improvement for MTS" as I said, then I will understand and I won't say it here.

BTW: I absolutely agree with you about the enhancement of MTS, and take it for granted, no need to discuss.

Good lucky,
Riceball

bzt
Member
Posts: 217
Joined: Tue Sep 24, 2019 14:26

Re: New MTS format

by bzt » Post

Okay, last chance. I'll try to explain, but if you still can't understand, I'm afraid there's not much I can do :-(
snowyu wrote:
Wed Sep 01, 2021 05:02
last words, I've answered your question. but you can not see it:
snowyu wrote:
Fri Aug 27, 2021 13:23
The only simple "interaction" is through the properties of matter...
This does not answer my question: if these are properties of the matter and not of the schem, then why bloat each and every schem with these?
snowyu wrote:
Tue Aug 31, 2021 09:40
My idea is to create a metaverse standard specification similar to the Web. But this must start from bit by bit.
Yes I know, but you're doing it wrong. Just by storing irrelevant and engine-incompatible properties into a schem won't automagically create your desired metaverse. For that, you'll have to standardize material references, there's no other way. Even if two engines implement "water", "wood" or "concrete" for example differently (with different textures, effects, what interactions allowed, different event handlers etc.), they must agree on how "water", "wood" and "concrete" is referenced in the schem if you want to load the same schem in both.

Cheers,
bzt

snowyu
Member
Posts: 25
Joined: Mon Jun 07, 2021 06:42
GitHub: snowyu

Re: New MTS format

by snowyu » Post

Yes, me too.
bzt wrote:
Wed Sep 01, 2021 16:54
This does not answer my question: if these are properties of the matter and not of the schem, then why bloat each and every schem with these?
I'm trying to guess what you are thinking: if you think that schematic is only for display(3D Viewer or 3D Editor), then it is true that matter has nothing to do with schematic. But if the definition of schematic is prepared for the components of the virtual world, it is different.
bzt wrote:
Wed Sep 01, 2021 16:54
Yes I know, but you're doing it wrong. Just by storing irrelevant and engine-incompatible properties into a schem won't automagically create your desired metaverse. For that, you'll have to standardize material references, there's no other way. Even if two engines implement "water", "wood" or "concrete" for example differently (with different textures, effects, what interactions allowed, different event handlers etc.), they must agree on how "water", "wood" and "concrete" is referenced in the schem if you want to load the same schem in both.
First of all, I agree that this requires standardization of matter properties. I listed these attributes formally based on this. First, trial and error with these attributes is required to gradually standardize.

I do not agree that the engine should implement water, wood and concrete things. If I was forced to make an analogy, then the engine is similar to a web browser, and the world is similar to a skin theme. It mainly stipulates environmental parameters and some core schematics. The schematic in my mind is the core element in world interaction. It can be combined and inherited. Even the world is essentially a schematic too.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest