This is really quite neat! I have been wondering why none of the Pokemon/Pokemon-like mods ever seem to have gained steam on Minetest, when mods like Pixelmon are a big draw in the Minecraft world.
I gave it a quick go just to experiment and have some very initial thoughts/observations (I respect this mod is in its early stages so far):
Things that might need addressing:
The behaviour of the ESC key - is there a way to prevent this from just automatically closing the formspec battle window? I noticed that when you press ESC during a battle, the little 2D Zoonami (creature?) disappears from the world map altogether. I guess this works ok with Pokemon, where most monsters are not visible until encountered, but it's a bit jarring. Even in Pokemon, I think you can't 'Run away' from 100% of battles.
A way to keep this general behaviour but make it a bit less sudden might be to do a small animation with particles on the worldmap Zoonami sprite. Perhaps there could be different ones for different situations, such as defeating a monster, leaving the battle with your monster, capturing a monster etc.
When your first monster is defeated, I didn't seem to be able to continue the battle with my other monsters. Is this intentional?
Will there be a way to set certain Zoonami to be aggressive, i.e. they will approach you and prompt a formspec battle?
Is there any way to make the formspec battle backgrounds change according to the biome you are in, or does this already happen?
Is there a way to make the Zoonami a tiny bit thicker, i.e. not make them disappear from view entirely when viewed side on? I know that you can do that for wielded items I think, even if you only provide a 2D sprite. Of course you could choose to have them totally flat for artistic reasons too.
It'd be great to have your own little monsters more 'present' in the world somehow. Maybe even just as a wield item, or maybe you have a little Pokedex type computer as a wield item that you can log into to check on them (maybe this could be another formspec). Alternatively, maybe you could pick one to follow you around in the worldmap, just like in games like Pokemon Heartgold, etc.
Capture mechanics/items would be a big thing to integrate, but I suspect you might be able to get some more contributors to help you with this.
I really like your art style, it's very simple but the monsters are fun and unique looking. It doesn't feel like a 1 for 1 Pokemon ripoff. The music is actually pretty great too.
For the initial phase of the mod, I would recommend you personally churn out a few more of these Zoonami, just to get more people interested and involved, but I wonder if after this phase it would be possible to build a little utility to quickly create/edit your own Zoonami in a way that does not require you to be adept at programming. This would really help the longevity of this mod.
I have heard some negative opinions about previous attempts to have 2D mobs in the 3D voxel world of Minetest, but I actually thing it could really work for this type of mod. I'm a bit of a sucker for a story though, so I think it might be great to have an explicit "in universe" explanation for why these little Zoonami things are 2D. My idea: they are interdimensional interlopers or invaders. They come from the "Second Dimension" or something like that.
From there you could take that idea in all kinds of directions - the player is a kind of 'wildlife ranger' to control their spread, etc.
On the topic of adding more Zoonami - what about making ones of the 'non-official' but widely accepted Minetest creatures, like the dirt monster, stone monster, etc. But maybe more interestingly, the Orekki (spelling?) and Dungeon Master, since they're more distinct. That would make this mod firmly a part of the 'Minetest' community. It might even be fun to encourage other mod makers to port some of their monsters over into this quirky 2D format.
I think there was other mods that did something similar to this already, which you might be able to draw from:
(A game, but has 2D monsters I think).
On a related note: maybe reach out to the Tuxemon
community and project? It's a really neat little project, but I don't think it really got far and seems to have stalled somewhat. They do however already seem to have quite a few little Tuxemon that might be really easy to port over into this game if licensing allows (it is FOSS as far as I known). That would also help bring new people into the Minetest community maybe, and breathe some life into the project. Of course, maybe engaging that community first before porting over anything would probably be better received.
I believe they also have a Reddit, maybe they have some sort of Discord/IRC presence?
Theoretically, you could have dozens of extra Zoonami pretty quickly and maybe even get some extra eyes in on your code to help you out.
A little diversion directly from this project and more into an untapped niche for FOSS games, but still applicable:
I'm not a programmer (though I am really trying to learn) or a pixel artist, but I'd be interested in seeing if I could help out somehow maybe once this project gains a little bit of steam and has a set plan of action. Think this idea really has a lot of potential. Best of luck with it!