MTPC - MT Package Control-Crawlers added, more efficient

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

MTPC - MT Package Control-Crawlers added, more efficient

by LMD » Sun Mar 18, 2018 19:13

MTPC - Minetest Package Control

Introduction Video :

Image

(image is link)

Why MTPC ?

MTPC stands for Minetest Package Control.

You may know the popular (Sub)Game Voxellar. Version 2.0, I was told by Voxellar main developer azekill_DIABLO, is going to have at least 200 Mods ! Also, Minetest Game, Minimal Development Test, Wastelands, Dungeontest, LOTH and Regnum got at least 160 Mods together ! Impossible to manage for one human.
Now MTPC becomes useful : Manage installed mods, update mods, deinstall subgames, do what you want !
For developers :
Add your (Sub)Games as entries, or your Mods, so that a DB is created which consists of thousands of browsable items.
It should unite a Mod/(Sub)-Game/TP/Forks...-browse option, plus the opportunity of installing Mods or (Sub)Games !

In the new version, there are also Crawlers !

MTPC aims to be a simple tool, with a SQLite DataBase in background. You can deinstall/install/update mods or (Sub)Games, and all dependencies are going to be installed to. Furthermore, you can browse the DB for available Mods/(Sub)Games/Texture Packs/MT Forks and Authors.
There are also some configurations which allow you to customize GUI just as you like it !

Brief listing :

"Content" stands for Minetest Mods, (Sub)Games, TPs, Forks and Authors
sometimes the features listed only apply to mods and (sub)games, such as installation/deinstallation

Features :
- self-made nice gui, color changeable
- MT gamepath + table row height can be specified
- 3 possible types of crawlers to update your DB with the newest content from (a) HTML Tables (b) MT Forum (c) MT Forum Memberlist
- Well-organized DB; multiple features : simple search clause system + simple sorting system, in GUI-table cell editing, rating, downloading, (de)installing, downloading & unpacking, reading information in new window (with screenshots downloaded from internet when needed)
- Show installed content
- Show content of which the version number listed in your DB is higher than the one on your installation(to update tab) and update em
- Adding DB entries

(extracted from MTPC Source repo readme)

Logo :

https://github.com/appgurueu/appgurueu. ... ILI0si.gif

Written in Java, based on my AGE - Appguru General Engine.

Starting tutorial :
1. Download zip from GitHub
2. Unpack it
3. Execute jar(Double-click on some systems, you can also use terminal)
If you start it from Terminal, here's how it should look like :
Image
And that's the GUI when its started :
Image
(image outdated, crawler button not visible)

Download :

https://github.com/appgurueu/MTPC-Release
On GitHub now !

Sourcecode/GitHub :

https://github.com/appgurueu/MTPC-Source/tree/master

Licensing :

WTFPL

MTPC reworked !

Changelog :
- made much much much more efficient
- added Crawlers option
- added more detailed help
- made release file much smaller -> Release isn't hosted on Google anymore !

It may appear very very small, but many buttons make new windows pop up - and on these windows, there are also some buttons....

Enjoy !
Last edited by LMD on Thu Jun 07, 2018 18:11, edited 15 times in total.
 

ptvirgo
Member
 
Posts: 45
Joined: Thu May 26, 2016 22:18
GitHub: ptvirgo
In-game: Mox

Re: MTPC - Minetest Package Control - Development : Help nee

by ptvirgo » Sun Mar 18, 2018 20:40

Interesting project - you may be over-complicating the requirements. Have you looked at Minetest Bower already?
https://minetest-bower.herokuapp.com/

You might be able to accomplish your goal by using the Bower API to handle the centralized mod data. That would make synchronization a much simpler problem, and free you up to focus on the installation and local management side.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Sun Mar 18, 2018 21:04

However, I need a server, storing the main repo - a SQLite DB, which is always synced with all clients.
The server must be able to run Java and Socket on certain ports 24/7 with acceptable upload/download speed.
The client will upload a changelog which is then applied to the server-side DB.
Afterwards, the client can obtain an up-to-date DB from the server.
But all this stuff is to-do right now...
So, help needed : Where to host server ?


You didn't explain this in the other thread. Why do you need such a monstrous server design? What data is stored on the server that you need a java server? You really need to explain your data model better and honestly I don't think you're properly designing your data flow correctly.

Clients only need some master list of some sorts. I don't see why you couldn't even manually maintain one as a JSON file, literally with a text editor if needed. All you'd need would be a HTTP server, and you could even host it straight out of github. You could even use github to take PRs so that people could submit changes. Literally you don't need your own server and you definitely do not need to build a Java websocket enabled monster.

Think about this some more and try and explain it, I think you're making something interesting but it won't succeed because your choices force you to solve problems by throwing money at it in the form of expensive server hardware, which is totally not needed.
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Sun Mar 18, 2018 21:12

"so that people could commit changes"
nope.
Let's say Person A downloads the most recent Package List.
While he is working on it(adding his mods), Person B adds it own mods to the list.
In SQLite, all this operations are kinda "addrow" commands, so they want interfere in a bad manner.
But how to deal with this in JSON ?
People would have to manually insert their mods in there - AAAAAAAAAAAAAAAAAAARRRRRRRRRRRRRRGHHH !
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Sun Mar 18, 2018 21:14

About the server, that's why I opened this thread, basically.
I am not sure I'll do in Java, PHP also offers SQLite libs.
And there are some free host platforms which allow you to host PHP !
(000webhost etc)
However, I am not sure they allow you to fire up a Socket easily...
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Sun Mar 18, 2018 21:15

BTW, @sofar, maybe we should talk about this using PMs and in German...
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Sun Mar 18, 2018 21:27

LMD wrote:BTW, @sofar, maybe we should talk about this using PMs and in German...


My german isn't the best, I'm not german.

I think you are correct that the clients use sqlite for storing their list of mods, but this data doesn't need to be on the server, does it? Even if you need some sort of "cloud" storage for clients, sqlite isn't a good storage for that and you should use an SQL database instead, as sqlite may have problems scaling in the end.

Again this boils down to "what data is stored on the client, what data is stored on the server" - please describe this in detail so I can understand what you are trying to do and maybe help you get a more robust and efficient design that doesn't require you to sink money into an expensive server unless it's really needed.
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Sun Mar 18, 2018 21:30

okay, sorry, i somehow thought you were...
Basically, the server needs to store whole Package List.
So the client can always have an up-to-date version, but also, it's a shared DB : Anybody can contribute.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Sun Mar 18, 2018 21:32

LMD wrote:Let's say Person A downloads the most recent Package List.
While he is working on it(adding his mods), Person B adds it own mods to the list.


This seems to be the crux right here and I think you're failing to understand how to build simple and stable API's.

First, allowing random people to add things to a central list without any form of review is going to be disastrous. You need to have some sort of moderation, otherwise it's going to be a crapshoot really fast.

Just doing this in sqlite doesn't solve locking, concurrency, conflicts, moderation, and it definitely doesn't create a good API for clients to sync, so it's about the worst choice you can make.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Sun Mar 18, 2018 21:34

LMD wrote:okay, sorry, i somehow thought you were...
Basically, the server needs to store whole Package List.
So the client can always have an up-to-date version, but also, it's a shared DB : Anybody can contribute.


How about, people just contribute records that are separate files, one per package, in some simple JSON format. Every hour the server combines all of them into a single JSON blob, and that is what clients download so they can synchronize their package list?

All you'd need is a git tree and a shell script, and run the script. Clients could get the output from github. No server needed....
 

User avatar
Devy
Member
 
Posts: 120
Joined: Sat Jan 21, 2017 02:31
GitHub: CoderForTheBetter
In-game: devy

Re: MTPC - Minetest Package Control - Development : Help nee

by Devy » Sun Mar 18, 2018 21:45

I always thought it would be cool to have some kind of web-crawler that crawls the forums looking for GitHub links. You could even provide a format for people to post their mods in so that it is easy to find images, Github downloads, descriptions, etc. Although that puts more stress on the forum servers, it will not work every time, and a web crawler requires a lot of resources of the computer and its network bandwidth but you could have a single computer do all this work, put it in JSON format and put that on Github, will still have the same issues as before.

The only reason I suggest this stuff is because I do not think using a real person to moderate it is sustainable, and there shouldn't be a reason for anyone to moderate it since people will be downloading mods they would usually download from the forums anyway.

This would basically be something that browses the forums for us.

I like the idea of having some kind of mod manager/package manager but not if people will always have to be involved in moderating and organizing everything.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Mon Mar 19, 2018 06:26

BTW, a sqlite file that isn't read-only isn't accessible by non-sqlite applications and can only be read safely by the sqlite library, so your design to send a copy of the sqlite db to clients would cause your clients to download a broken DB file if they would download during an update of any of the tables.

In exactly the same way, you can NOT make a backup of an sqlite DB by making a copy if some process is currently attached to the database.

So, again, I don't think you understand what you're doing.

Here's what I would design:

1) some php/JS website where people can submit content
2) that content goes on the server into a SQL db (mariadb)
3) periodically, some server code outputs the SQL db as a JSON file onto the static download pages of the server
4) clients fetch this file over HTTP, and then
5) these clients read the JSON file and update their own SQLite DB

This is a safe and efficient approach and you can test it easily by creating a text file on a web server, so you can start working on the client code without having difficult server code to develop. You could just manually update the JSON file until you're ready to work on the server code.

Things to think about: how to handle deleted or changed items, how to do moderation, SSL to make sure clients don't download malicious code, OSS license, etc..
 

User avatar
rubenwardy
Moderator
 
Posts: 5363
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: MTPC - Minetest Package Control - Development : Help nee

by rubenwardy » Mon Mar 19, 2018 14:15

If only someone was working on such a project
Core Developer | Donate | My Twitter | Mods | Mods 4 Android | Node Box Editor | Minetest Modding Book

Hello profile reader

LgiOxMFYXOqtqVqMPhbw1Bn3oNRvEC7j
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Mon Mar 19, 2018 16:22

sofar wrote:BTW, a sqlite file that isn't read-only isn't accessible by non-sqlite applications and can only be read safely by the sqlite library, so your design to send a copy of the sqlite db to clients would cause your clients to download a broken DB file if they would download during an update of any of the tables.

In exactly the same way, you can NOT make a backup of an sqlite DB by making a copy if some process is currently attached to the database.

So, again, I don't think you understand what you're doing.

Here's what I would design:

1) some php/JS website where people can submit content
2) that content goes on the server into a SQL db (mariadb)
3) periodically, some server code outputs the SQL db as a JSON file onto the static download pages of the server
4) clients fetch this file over HTTP, and then
5) these clients read the JSON file and update their own SQLite DB

This is a safe and efficient approach and you can test it easily by creating a text file on a web server, so you can start working on the client code without having difficult server code to develop. You could just manually update the JSON file until you're ready to work on the server code.

Things to think about: how to handle deleted or changed items, how to do moderation, SSL to make sure clients don't download malicious code, OSS license, etc..

Thanks for discussing this with me !

at first - why ssl ? How can "malicious code" be downloaded ? Anybody who will download it has to trust me...

Your design has various disadvantages only to avoid a custom server, I hoped to find some forum member who's already got a server runnin'(Apache 2 I'd prefer) and could let my small(very tiny) server written in java run at the same time. But I'm not sure if there's such one...
But if there was - my approach is easier, and handles changes much better.
It's also way more efficient.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Mon Mar 19, 2018 18:10

small(very tiny) server written in java


This doesn't exist. Java is a monster. Java can't be a "very tiny" server.
 

sofar
Developer
 
Posts: 1640
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: MTPC - Minetest Package Control - Development : Help nee

by sofar » Mon Mar 19, 2018 18:22

at first - why ssl ? How can "malicious code" be downloaded ? Anybody who will download it has to trust me...


If you don't understand this, you shouldn't be working on this project, I think.

Your design has various disadvantages only to avoid a custom server,


You don't state which disadvantages, so your arguments are non-existant. I've written arguments why your solution doesn't work, and you haven't refuted them or showed that they are wrong, so you're just not listening.

But if there was - my approach is easier, and handles changes much better.
It's also way more efficient.


It is short-sighted, just like your entire approach, and now that I think of it, your attitude as well.

I'll spend my time reviewing rubenwardy's project, since he seems to understand the problem space, has a clear grasp of solid internet technologies and more importantly, his professor will fail his grade if he produced something that looks like what you are designing.
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Mon Mar 19, 2018 18:46

Yes, I am sure not as experienced as rubenwardy. Is he working on a Package Control ? But I don't like :
- conversion sql -> json -> sqlite. Why ? Not necessary I assume.
Why can't I directly have a SQL/SQLite DB at serverside so that I can directly send it...
Also, why does it have to be encrypted ? However, I believe that wouldn't be too hard using an Apache 2.(set up Apache 2 http , never set up https website myself before) but it appears java can directly download from a https://.... link so I would just store my SQL/SQLite DB(which is always updated) there and download it when needed.
- json blob youre talking about -> makes editing(rating for instance) harder
- "uploading by visiting the website" - nope, you edit the entries in my application, also to make sure you don't enter a mod twice or such problems. In your json blob, we would have to browse all entries - sqlite offers faster search. The user should be able to contribute through my application, to make it all centred at one place... You don't want to visit a webpage each time you want to - let's say - rate a mod, edit a description... And then, why even having a standalone application ? The only advantage is the privileges - it can install and deinstall, determine installed mods etc... I thought : If you make it all serverside, the server can't be strong enough, and you can't install etc..., so I made it clientside. You know popular package controls, let's take Ubuntu's one - they have also got a DB in the background, although I'm pretty sure it isn't SQLite...
However, I'm doing the client part first, and as we both agreed to pick SQLite/SQL for the client, there should be no problem. If I get to the server part, I'll still discuss with you and consider your idea, which has some security advantages, in my opinion : I would have to check whether somebody used the "upload changelog" way to execute malicious code on the DB - for example, destroying all data
Last edited by LMD on Mon May 21, 2018 19:59, edited 1 time in total.
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - Development : Help nee

by LMD » Mon Mar 19, 2018 18:52

sofar wrote:
small(very tiny) server written in java


This doesn't exist. Java is a monster. Java can't be a "very tiny" server.


I meant it wont take much code : Only a socket listener, waiting for changelogs as SQLite statements to apply them to the server-side DB.
Java isn't a monster ! In comparison (mainly security) Java is a thousand times better than LUA, and with 1.8, even nice Lambda and parts of functional programming got added - Java is efficient and great, and a well-established server-side language. However, you're right, I should not do all of it over socket. The client downloads his files using https, but uploads his changes over socket. BTW, does Minetest do any encryption when passing data between client and server ?
 

User avatar
azekill_DIABLO
Member
 
Posts: 7381
Joined: Wed Oct 29, 2014 20:05
Location: Under my desk (but I can't see my monitor now X'[ )
GitHub: azekillDIABLO
In-game: azekill_DIABLO
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

MTPC - Minetest Package Control - BETA RELEASE !

by LMD » Sat May 05, 2018 20:39

<Post unnecessary as it has been merged with first post>
Last edited by LMD on Mon May 21, 2018 20:01, edited 19 times in total.
 

User avatar
Stix
Member
 
Posts: 1049
Joined: Fri Aug 04, 2017 14:19
Location: USA
IRC: Stix + Stix_
In-game: Stix

Re: MTPC - Minetest Package Control - BETA RELEASE !

by Stix » Sat May 05, 2018 20:42

LMD wrote:Here's the MTPC (Client-only currently) Beta Release : https://github.com/appgurueu/MTPC-Release
GIVE IT A TRY !

what is MTPC?
The harder you think the more you'll achieve --Caleb #Hi-five for no good reason --Caleb
Need to know something? "Google it" --GreenDimond
Next release of your favorite game? "Bugfixes" --Wuzzy
[OLD]The new Minetest-offtopic can be found here: https://minetestoftopic.createaforum.com/index.php
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - BETA RELEASE !

by LMD » Sat May 05, 2018 20:44

umm - Minetest Package Control. Written in Java.
Last edited by LMD on Tue May 22, 2018 09:31, edited 1 time in total.
 

User avatar
Krock
Developer
 
Posts: 3989
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: MTPC - Minetest Package Control - BETA RELEASE !

by Krock » Sat May 05, 2018 21:05

I tried, I failed.
Code: Select all
$ java -jar AGE.jar
SUCCESS : APPGURU : TIMER : TIMER : INITIALISATION
ERROR : APPGURU : DB : DATABASE : NO SQLITE
Exception in thread "main" java.lang.NullPointerException
   at appguru.AGE.main(AGE.java:263)


What do I need to install to get this running properly?
Mod Search Engine - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

User avatar
LMD
Member
 
Posts: 399
Joined: Sat Apr 08, 2017 08:16
Location: Bonn, Germany
GitHub: appgurueu
In-game: LMD + PRO_LMD + Limo

Re: MTPC - Minetest Package Control - BETA RELEASE !

by LMD » Sun May 06, 2018 08:30

Sorry, I forgave to include the necessary libs.
The new release(link above) has them.
 

parasite
Member
 
Posts: 58
Joined: Sat May 06, 2017 17:45

Re: MTPC - Minetest Package Control - BETA RELEASE !

by parasite » Sun May 06, 2018 10:25

Stix wrote:what is MTPC?


we want to understand what's going on here, but it all seems complicated. Someone should explain it to us as mothers explain complicated things to children.
Last edited by parasite on Sat Jun 02, 2018 18:27, edited 1 time in total.
 

Next

Return to Minetest-related projects



Who is online

Users browsing this forum: No registered users and 1 guest