Minetest Forums

MTPC - Minetest Package Control

Introduction Video :



(image is link)

Why MTPC ?

MTPC stands for Minetest Package Control.

You may know the popular (Sub)Game Voxellar. Version 2.0, I was told by Voxellar main developer azekill_DIABLO, is going to have at least 200 Mods ! Also, Minetest Game, Minimal Development Test, Wastelands, Dungeontest, LOTH and Regnum got at least 160 Mods together ! Impossible to manage for one human.
Now MTPC becomes useful : Manage installed mods, update mods, deinstall subgames, do what you want !
For developers :
Add your (Sub)Games as entries, or your Mods, so that a DB is created which consists of thousands of browsable items.
It should unite a Mod/(Sub)-Game/TP/Forks...-browse option, plus the opportunity of installing Mods or (Sub)Games !

In the new version, there are also Crawlers !

MTPC aims to be a simple tool, with a SQLite DataBase in background. You can deinstall/install/update mods or (Sub)Games, and all dependencies are going to be installed to. Furthermore, you can browse the DB for available Mods/(Sub)Games/Texture Packs/MT Forks and Authors.
There are also some configurations which allow you to customize GUI just as you like it !

Brief listing :

"Content" stands for Minetest Mods, (Sub)Games, TPs, Forks and Authors
sometimes the features listed only apply to mods and (sub)games, such as installation/deinstallation

Features :
- self-made nice gui, color changeable
- MT gamepath + table row height can be specified
- 3 possible types of crawlers to update your DB with the newest content from (a) HTML Tables (b) MT Forum (c) MT Forum Memberlist
- Well-organized DB; multiple features : simple search clause system + simple sorting system, in GUI-table cell editing, rating, downloading, (de)installing, downloading & unpacking, reading information in new window (with screenshots downloaded from internet when needed)
- Show installed content
- Show content of which the version number listed in your DB is higher than the one on your installation(to update tab) and update em
- Adding DB entries

(extracted from MTPC Source repo readme)

Logo :

https://github.com/appgurueu/appgurueu. ... ILI0si.gif

Written in Java, based on my AGE - Appguru General Engine.

Starting tutorial :
1. Download zip from GitHub
2. Unpack it
3. Execute jar(Double-click on some systems, you can also use terminal)
If you start it from Terminal, here's how it should look like :

And that's the GUI when its started :

(image outdated, crawler button not visible)

Download :

https://github.com/appgurueu/MTPC-Release
On GitHub now !

Sourcecode/GitHub :

https://github.com/appgurueu/MTPC-Source/tree/master

Licensing :

WTFPL

MTPC reworked !

Changelog :
- made much much much more efficient
- added Crawlers option
- added more detailed help
- made release file much smaller -> Release isn't hosted on Google anymore !

It may appear very very small, but many buttons make new windows pop up - and on these windows, there are also some buttons....

Enjoy !

Interesting project - you may be over-complicating the requirements. Have you looked at Minetest Bower already?
https://minetest-bower.herokuapp.com/

You might be able to accomplish your goal by using the Bower API to handle the centralized mod data. That would make synchronization a much simpler problem, and free you up to focus on the installation and local management side.

However, I need a server, storing the main repo - a SQLite DB, which is always synced with all clients.
The server must be able to run Java and Socket on certain ports 24/7 with acceptable upload/download speed.
The client will upload a changelog which is then applied to the server-side DB.
Afterwards, the client can obtain an up-to-date DB from the server.
But all this stuff is to-do right now...
So, help needed : Where to host server ?

You didn't explain this in the other thread. Why do you need such a monstrous server design? What data is stored on the server that you need a java server? You really need to explain your data model better and honestly I don't think you're properly designing your data flow correctly.

Clients only need some master list of some sorts. I don't see why you couldn't even manually maintain one as a JSON file, literally with a text editor if needed. All you'd need would be a HTTP server, and you could even host it straight out of github. You could even use github to take PRs so that people could submit changes. Literally you don't need your own server and you definitely do not need to build a Java websocket enabled monster.

Think about this some more and try and explain it, I think you're making something interesting but it won't succeed because your choices force you to solve problems by throwing money at it in the form of expensive server hardware, which is totally not needed.

"so that people could commit changes"
nope.
Let's say Person A downloads the most recent Package List.
While he is working on it(adding his mods), Person B adds it own mods to the list.
In SQLite, all this operations are kinda "addrow" commands, so they want interfere in a bad manner.
But how to deal with this in JSON ?
People would have to manually insert their mods in there - AAAAAAAAAAAAAAAAAAARRRRRRRRRRRRRRGHHH !

About the server, that's why I opened this thread, basically.
I am not sure I'll do in Java, PHP also offers SQLite libs.
And there are some free host platforms which allow you to host PHP !
(000webhost etc)
However, I am not sure they allow you to fire up a Socket easily...

BTW, @sofar, maybe we should talk about this using PMs and in German...

LMD wrote:BTW, @sofar, maybe we should talk about this using PMs and in German...

My german isn't the best, I'm not german.

I think you are correct that the clients use sqlite for storing their list of mods, but this data doesn't need to be on the server, does it? Even if you need some sort of "cloud" storage for clients, sqlite isn't a good storage for that and you should use an SQL database instead, as sqlite may have problems scaling in the end.

Again this boils down to "what data is stored on the client, what data is stored on the server" - please describe this in detail so I can understand what you are trying to do and maybe help you get a more robust and efficient design that doesn't require you to sink money into an expensive server unless it's really needed.

okay, sorry, i somehow thought you were...
Basically, the server needs to store whole Package List.
So the client can always have an up-to-date version, but also, it's a shared DB : Anybody can contribute.

LMD wrote: Let's say Person A downloads the most recent Package List.
While he is working on it(adding his mods), Person B adds it own mods to the list.

This seems to be the crux right here and I think you're failing to understand how to build simple and stable API's.

First, allowing random people to add things to a central list without any form of review is going to be disastrous. You need to have some sort of moderation, otherwise it's going to be a crapshoot really fast.

Just doing this in sqlite doesn't solve locking, concurrency, conflicts, moderation, and it definitely doesn't create a good API for clients to sync, so it's about the worst choice you can make.

LMD wrote:okay, sorry, i somehow thought you were...
Basically, the server needs to store whole Package List.
So the client can always have an up-to-date version, but also, it's a shared DB : Anybody can contribute.

How about, people just contribute records that are separate files, one per package, in some simple JSON format. Every hour the server combines all of them into a single JSON blob, and that is what clients download so they can synchronize their package list?

All you'd need is a git tree and a shell script, and run the script. Clients could get the output from github. No server needed....

I always thought it would be cool to have some kind of web-crawler that crawls the forums looking for GitHub links. You could even provide a format for people to post their mods in so that it is easy to find images, Github downloads, descriptions, etc. Although that puts more stress on the forum servers, it will not work every time, and a web crawler requires a lot of resources of the computer and its network bandwidth but you could have a single computer do all this work, put it in JSON format and put that on Github, will still have the same issues as before.

The only reason I suggest this stuff is because I do not think using a real person to moderate it is sustainable, and there shouldn't be a reason for anyone to moderate it since people will be downloading mods they would usually download from the forums anyway.

This would basically be something that browses the forums for us.

I like the idea of having some kind of mod manager/package manager but not if people will always have to be involved in moderating and organizing everything.

BTW, a sqlite file that isn't read-only isn't accessible by non-sqlite applications and can only be read safely by the sqlite library, so your design to send a copy of the sqlite db to clients would cause your clients to download a broken DB file if they would download during an update of any of the tables.

In exactly the same way, you can NOT make a backup of an sqlite DB by making a copy if some process is currently attached to the database.

So, again, I don't think you understand what you're doing.

Here's what I would design:

1) some php/JS website where people can submit content
2) that content goes on the server into a SQL db (mariadb)
3) periodically, some server code outputs the SQL db as a JSON file onto the static download pages of the server
4) clients fetch this file over HTTP, and then
5) these clients read the JSON file and update their own SQLite DB

This is a safe and efficient approach and you can test it easily by creating a text file on a web server, so you can start working on the client code without having difficult server code to develop. You could just manually update the JSON file until you're ready to work on the server code.

Things to think about: how to handle deleted or changed items, how to do moderation, SSL to make sure clients don't download malicious code, OSS license, etc..

If only someone was working on such a project

sofar wrote:BTW, a sqlite file that isn't read-only isn't accessible by non-sqlite applications and can only be read safely by the sqlite library, so your design to send a copy of the sqlite db to clients would cause your clients to download a broken DB file if they would download during an update of any of the tables.

In exactly the same way, you can NOT make a backup of an sqlite DB by making a copy if some process is currently attached to the database.

So, again, I don't think you understand what you're doing.

Here's what I would design:

1) some php/JS website where people can submit content
2) that content goes on the server into a SQL db (mariadb)
3) periodically, some server code outputs the SQL db as a JSON file onto the static download pages of the server
4) clients fetch this file over HTTP, and then
5) these clients read the JSON file and update their own SQLite DB

This is a safe and efficient approach and you can test it easily by creating a text file on a web server, so you can start working on the client code without having difficult server code to develop. You could just manually update the JSON file until you're ready to work on the server code.

Things to think about: how to handle deleted or changed items, how to do moderation, SSL to make sure clients don't download malicious code, OSS license, etc..

Thanks for discussing this with me !

at first - why ssl ? How can "malicious code" be downloaded ? Anybody who will download it has to trust me...

Your design has various disadvantages only to avoid a custom server, I hoped to find some forum member who's already got a server runnin'(Apache 2 I'd prefer) and could let my small(very tiny) server written in java run at the same time. But I'm not sure if there's such one...
But if there was - my approach is easier, and handles changes much better.
It's also way more efficient.

small(very tiny) server written in java

This doesn't exist. Java is a monster. Java can't be a "very tiny" server.

at first - why ssl ? How can "malicious code" be downloaded ? Anybody who will download it has to trust me...

If you don't understand this, you shouldn't be working on this project, I think.

Your design has various disadvantages only to avoid a custom server,

You don't state which disadvantages, so your arguments are non-existant. I've written arguments why your solution doesn't work, and you haven't refuted them or showed that they are wrong, so you're just not listening.

But if there was - my approach is easier, and handles changes much better.
It's also way more efficient.

It is short-sighted, just like your entire approach, and now that I think of it, your attitude as well.

I'll spend my time reviewing rubenwardy's project, since he seems to understand the problem space, has a clear grasp of solid internet technologies and more importantly, his professor will fail his grade if he produced something that looks like what you are designing.

Yes, I am sure not as experienced as rubenwardy. Is he working on a Package Control ? But I don't like :
- conversion sql -> json -> sqlite. Why ? Not necessary I assume.
Why can't I directly have a SQL/SQLite DB at serverside so that I can directly send it...
Also, why does it have to be encrypted ? However, I believe that wouldn't be too hard using an Apache 2.(set up Apache 2 http , never set up https website myself before) but it appears java can directly download from a https://.... link so I would just store my SQL/SQLite DB(which is always updated) there and download it when needed.
- json blob youre talking about -> makes editing(rating for instance) harder
- "uploading by visiting the website" - nope, you edit the entries in my application, also to make sure you don't enter a mod twice or such problems. In your json blob, we would have to browse all entries - sqlite offers faster search. The user should be able to contribute through my application, to make it all centred at one place... You don't want to visit a webpage each time you want to - let's say - rate a mod, edit a description... And then, why even having a standalone application ? The only advantage is the privileges - it can install and deinstall, determine installed mods etc... I thought : If you make it all serverside, the server can't be strong enough, and you can't install etc..., so I made it clientside. You know popular package controls, let's take Ubuntu's one - they have also got a DB in the background, although I'm pretty sure it isn't SQLite...
However, I'm doing the client part first, and as we both agreed to pick SQLite/SQL for the client, there should be no problem. If I get to the server part, I'll still discuss with you and consider your idea, which has some security advantages, in my opinion : I would have to check whether somebody used the "upload changelog" way to execute malicious code on the DB - for example, destroying all data

sofar wrote:

small(very tiny) server written in java

This doesn't exist. Java is a monster. Java can't be a "very tiny" server.

I meant it wont take much code : Only a socket listener, waiting for changelogs as SQLite statements to apply them to the server-side DB.
Java isn't a monster ! In comparison (mainly security) Java is a thousand times better than LUA, and with 1.8, even nice Lambda and parts of functional programming got added - Java is efficient and great, and a well-established server-side language. However, you're right, I should not do all of it over socket. The client downloads his files using https, but uploads his changes over socket. BTW, does Minetest do any encryption when passing data between client and server ?

Yes. This is the direction minetest must take.

<Post unnecessary as it has been merged with first post>

LMD wrote:Here's the MTPC (Client-only currently) Beta Release : https://github.com/appgurueu/MTPC-Release
GIVE IT A TRY !

what is MTPC?

umm - Minetest Package Control. Written in Java.

I tried, I failed. What do I need to install to get this running properly?

Sorry, I forgave to include the necessary libs.
The new release(link above) has them.

Stix wrote:what is MTPC?

we want to understand what's going on here, but it all seems complicated. Someone should explain it to us as mothers explain complicated things to children.