update-minetest-mods Linux bash scripts

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Just spent part of the day creating some bash scripts. They aren't perfect and I would consider them currently alpha, but feel free to look them over and/or offer suggestions.

The goal was to allow me to keep my mods updated in a fairly automated manner but I went on a major tangent lol.

Anywho, here it is...

https://github.com/AmyMoriyama/minetest-mod-update

EDIT: New repo, github was a pain, was easier to create new repo.

UPDATE: Big cleanup, mods file now contains 857 mods that can be installed/updated using these scripts. Some may not work, there may still be bugs and issues, and you might want to shutdown minetest before updating anything.
Last edited by AmyMoriyama on Wed Jul 07, 2021 21:37, edited 1 time in total.

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

This is my server mods update script.

It's intended to be used interactively. It warns if there are any systemd units for Minetest are running and stops with a warning if this is the case. If not, all mods are pulled and afterwards a reminder is shown to start the available systemd units for Minetest.

Code: Select all

#!/bin/bash

# Set variables as needed
bold=$(tput bold)
normal=$(tput sgr0)
line=$(printf '_%.0s' {1..80})

# Check if there are any Minetest services still running and exit if this is
# the case to prevent any issues with changed mod files on a running server.
if [[ "$(systemctl is-active minetest@*)" == "active" ]]; then
        echo -ne "\n${line}\n\n\n"
        echo -e "${bold}Stop all Minetest services first${normal}\n"
        systemctl list-units --type=service minetest* --no-legend
        echo -ne "\n${line}\n\n\n"
        exit
fi

# Update the mods using a simple Git pull
for mod in $(ls -d /var/lib/minetest/.minetest/mods/*/); do
        echo -ne "${bold}Updating $(basename ${mod})${normal}\n"
        echo -ne "${bold}Location $(dirname ${mod})${normal}\n\n"
        git -C "$mod" pull
        echo -ne "\n${line}\n\n\n"
done

# Print out a note as reminder to start the Minetest services
echo -e "${bold}Don't forget to start the Minetest services again${normal}\n"
systemctl list-units --type=service minetest* --all --no-legend
echo -ne "\n${line}\n\n\n"
The path to where the the mods are located (/var/lib/minetest/.minetest/mods in my case) maybe needs to be adjusted.

Known issues: Basically the script simply runs git pull on the mod's individual paths. So everything that can't be git pulled will fail (i.e. no repo, force pushed repos, locally changed files, etc.).

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Linuxdirk wrote:
Wed Jul 07, 2021 08:01
Known issues: Basically the script simply runs git pull on the mod's individual paths. So everything that can't be git pulled will fail (i.e. no repo, force pushed repos, locally changed files, etc.).
I noticed this limitation when I was pondering the idea of some sort of automated update script. Some mods aren't in an area that is easy to fetch from and some mods are on websites that don't seem to like people using wget/curl on them. The former would be custom scripted in many cases, but the latter is typically a no-go. It seems one major mod maker here (VanessaE) uses such a place for their mods (GitLab). This may not be a limitation with git though, but it is a limitation nonetheless.

Ideally, there would be some sort of standard for where to host mods. The content database kind of fills this but also doesn't since I'm sure there are mods on there that are outdated/broken but maybe are fixed on a git repo somewhere.

There is also the issue with fetching from git repos, the mod might be more unstable or may have a new bug that you get to find out about.

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Alright, huge cleanup and now the mods file contains 857 mods that can be installed using the scripts.

More updates to come, but this update is a big one overall, hence the reply.

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Why don't you interface with the CDB API? With this you could install all mods without the hassle of manual parsing and manually adding the links to your script.

https://content.minetest.net/help/api/

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Linuxdirk wrote:
Fri Jul 09, 2021 06:28
Why don't you interface with the CDB API? With this you could install all mods without the hassle of manual parsing and manually adding the links to your script.

https://content.minetest.net/help/api/
Wasn't aware of it and the initial grab was to get an initial list of git repo addresses. Now that there is a list, the plan is to keep it updated manually.

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Just a heads up for the Windows users out there, I am beginning to learn C++ with the first project goal being to convert these bash scripts into usable C++ programs that can be run on Windows and Linux. There is no estimated time of completion. Greater than a few months expected.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: update-minetest-mods Linux bash scripts

by Blockhead » Post

AmyMoriyama wrote:
Fri Jul 16, 2021 01:33
Just a heads up for the Windows users out there, I am beginning to learn C++ with the first project goal being to convert these bash scripts into usable C++ programs that can be run on Windows and Linux. There is no estimated time of completion. Greater than a few months expected.
That's a good goal and should help adoption. But in most cases git bash suffices to run bash scripts on Windows.

Also, your mods file is (and to some extent, probably has to be) opinionated. You use mt-mods technic instead of minetest-mods.. there's also forks like Linuxforks or Tunneler's Abyss technic (at least I think that one might be a fork I don't play there). It's not an easy problem though.

Really it would be nice to have a central database of all forks and mirrors of a mod and what's considered mainline (usualy github.com/minetest-mods/*), but inevitably I think it would probably have to be made out of user-submitted data.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Blockhead wrote:
Mon Jul 26, 2021 02:25
Really it would be nice to have a central database of all forks and mirrors of a mod and what's considered mainline (usualy github.com/minetest-mods/*), but inevitably I think it would probably have to be made out of user-submitted data.
I don't think having the main mods repository hosted at Microsoft is a good idea in general.

Also, why should I - as a mod author - add myself under an organization at MS GitHub instead of hosting it wherever I want without someone else controlling the main repository of it?

AmyMoriyama
Member
Posts: 107
Joined: Wed Jun 30, 2021 14:53
GitHub: AmyMoriyama

Re: update-minetest-mods Linux bash scripts

by AmyMoriyama » Post

Right now the mods file is a work in progress and open to suggestions of improvements as well as mod additions/removals.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: update-minetest-mods Linux bash scripts

by Blockhead » Post

Linuxdirk wrote:
Mon Jul 26, 2021 06:51
Blockhead wrote:
Mon Jul 26, 2021 02:25
Really it would be nice to have a central database of all forks and mirrors of a mod and what's considered mainline (usualy github.com/minetest-mods/*), but inevitably I think it would probably have to be made out of user-submitted data.
I don't think having the main mods repository hosted at Microsoft is a good idea in general.

Also, why should I - as a mod author - add myself under an organization at MS GitHub instead of hosting it wherever I want without someone else controlling the main repository of it?
The problem you have is not with me, it's with the Minetest-mods team. Yes, the main repository for a lot of things is going to be elsewhere. For instance, it's git.bananach.space for advtrains or it's Notabug for Tenplus1's mods.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Blockhead wrote:
Tue Jul 27, 2021 02:48
Yes, the main repository for a lot of things is going to be elsewhere. For instance, it's git.bananach.space for advtrains or it's Notabug for Tenplus1's mods.
I wish the CDB would be Git-based (and not just fetching master/main from other repos) and mod authors could mirror/host their mods directly there. But this likely won't ever happen (same with a potential git.minetest.net once Microsoft fucks up GitHub enough for C55 to leave it) :(

But since the CDB is an official and integral part of Minetest I guess seeing in as "central repository" is the best we have. One could build an command line management tool around the API, including version-aware updates and search. This could even be a part of the Minetest client since the Minetest client already has all the logic for using the CDB. It would just be another UI.

Using the CDB as source would also prevent updating to broken/incompatible mod versions.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: update-minetest-mods Linux bash scripts

by Blockhead » Post

Linuxdirk wrote:
Tue Jul 27, 2021 07:18
Blockhead wrote:
Tue Jul 27, 2021 02:48
Yes, the main repository for a lot of things is going to be elsewhere. For instance, it's git.bananach.space for advtrains or it's Notabug for Tenplus1's mods.
I wish the CDB would be Git-based (and not just fetching master/main from other repos) and mod authors could mirror/host their mods directly there. But this likely won't ever happen (same with a potential git.minetest.net once Microsoft fucks up GitHub enough for C55 to leave it) :(

But since the CDB is an official and integral part of Minetest I guess seeing in as "central repository" is the best we have. One could build an command line management tool around the API, including version-aware updates and search. This could even be a part of the Minetest client since the Minetest client already has all the logic for using the CDB. It would just be another UI.

Using the CDB as source would also prevent updating to broken/incompatible mod versions.
Thankfully ContentDB has source code links. These could be scraped to find the underlying git repositories. The other problem presented by using CDB as your central source however, would be the fact that all packages are namespaced. You can view the list of metapackages here.

Process to resolve mods from ContentDB
  1. Remove all modpacks that provide a mod from the metapackage listing
  2. If there is only one metapackage remaining, install it; else you will need to prefix the mod with the publisher
  3. Scrape the ContentDB source link for the resolved mod and clone it through git.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Blockhead wrote:
Wed Jul 28, 2021 10:44
Process to resolve mods from ContentDB
  1. Remove all modpacks that provide a mod from the metapackage listing
  2. If there is only one metapackage remaining, install it; else you will need to prefix the mod with the publisher
  3. Scrape the ContentDB source link for the resolved mod and clone it through git.
Or ignore the already parsed website results and use the raw JSON provided by the API.

https://content.minetest.net/api/packages/

This lists all available packages like so:

Code: Select all

...,
...,
{
	"author": "Linuxdirk",
	"name": "hunger_ng",
	"release": 7469,
	"short_description": "Hunger NG is a mod for Minetest adding a very customizable and easy to extend hunger system.",
	"thumbnail": "https://content.minetest.net/thumbnails/1/3IztsL4nPp.png",
	"title": "Hunger NG",
	"type": "mod"
},
...,
...
You can then use the author and name to get the most recent 30 releases of that mod.

https://content.minetest.net/api/packag ... /releases/

Code: Select all

[...,
...,
{
	"commit": "ccd96fbf304cf6774107c656e7c2ddb98298bc35",
	"downloads": 230,
	"id": 7469,
	"max_minetest_version": null,
	"min_minetest_version": {
		"is_dev": false,
		"name": "5.4",
		"protocol_version": 39
	},
	"release_date": "2021-04-13T17:59:35.460300",
	"title": "2021-04-13",
	"url": "/uploads/09K6aHliCU.zip"
},
...,
...]
There you have all information you need for downloading and maintaining the mod locally based on the CDB data.

Ping :)
rubenwardy wrote:.
Can we get the Git repo of a specific mod or even release using the CDB API? By just skimming through the API documentation I only saw how to put the Git repo and not how to get it.

User avatar
rubenwardy
Moderator
Posts: 6969
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: update-minetest-mods Linux bash scripts

by rubenwardy » Post

You need to get that package specifically, eg: https://content.minetest.net/api/packag ... y/conquer/

Also, see this doc to learn how the Minetest ContentDB dialog installs and manages mods: https://github.com/minetest/contentdb/b ... _client.md
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Ah, great, thanks.

Totally ignored the /releases in my call. :)

So with this information a command-line interface for the CDB is absolutely doable.

Maybe pushing myself into Python again for fun and learning.

User avatar
rubenwardy
Moderator
Posts: 6969
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy
Location: Bristol, United Kingdom
Contact:

Re: update-minetest-mods Linux bash scripts

by rubenwardy » Post

Linuxdirk wrote:
Wed Jul 28, 2021 11:20

You can then use the author and name to get the most recent 30 releases of that mod.

https://content.minetest.net/api/packag ... /releases/
You can get all the releases for the mod using that API. It's only the API for all packages that's limited to 30, as there's quite a few thousand releases on the website
Linuxdirk wrote:
Wed Jul 28, 2021 12:13
So with this information a command-line interface for the CDB is absolutely doable
There have been two attempts so far, minetools (Go) and another one in python (I forget the name). There's room for such a tool to be made official
Renewed Tab (my browser add-on) | Donate | Mods | Minetest Modding Book

Hello profile reader

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

rubenwardy wrote:
Wed Jul 28, 2021 14:30
There have been two attempts so far, minetools (Go) and another one in python (I forget the name). There's room for such a tool to be made official
it wouldn't be a real attempt, just something fun to do on the weekend :)

But maybe this could be in the client? Since the client already has all the functionality that is needed it would just be another UI.

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: update-minetest-mods Linux bash scripts

by Blockhead » Post

Linuxdirk wrote:
Wed Jul 28, 2021 14:47
rubenwardy wrote:
Wed Jul 28, 2021 14:30
There have been two attempts so far, minetools (Go) and another one in python (I forget the name). There's room for such a tool to be made official
it wouldn't be a real attempt, just something fun to do on the weekend :)

But maybe this could be in the client? Since the client already has all the functionality that is needed it would just be another UI.
Something like this? Not sure how to handle that minetest mods might be installed in a few different places and what would happen if one is installed in multiple places.

Code: Select all

$ minetest cdb --search mesecons #List everything under metapackages/mesecons
$ minetest cdb --search --mods mesecons #List only mods with the name mesecons
$ minetest cdb --search --modpacks mesecons #List modpacks that provide mesecons
$ minetest cdb --install --global jeija/mesecons #Install to the global mod directory (I think /var/ somewhere?)
$ minetest cdb --install --local moreblocks #Install to ../mods/
$ minetest cdb --install --user advtrains #Install to ~/.minetest/mods
$ minetest cdb --install --world greggs --mod sausage_rolls@2021-07-29 #Install sausage_rolls version 2021-07-29 to the world greggs
$ minetest cdb --update --all-places #global, local and user installed mods, modpacks and texturepacks
$ minetest cdb --update --mods --all-places #Update just the mods
$ minetest cdb --update --local --mod advtrains #Update the locally-installed copy of advtrains. Fail if it isn't installed.
$ minetest cdb --remove --global moreblocks
# Being a bit daring..
$ minetest cdb --install --git advtrains #Install advtrains through git - does CDB link commits to releases?
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Blockhead wrote:
Thu Jul 29, 2021 02:26
Something like this?
Since I hate positional arguments, more like minetest --cdb ....., but yes. :)
Blockhead wrote:
Thu Jul 29, 2021 02:26
Not sure how to handle that minetest mods might be installed in a few different places.
Just provide a location path like minetest --cdb --location /var/srv/minetest/mods ...... If not provided, the current location is used. This would get rid of --user, --global, --world, and --local entirely - as well as --all-places, --local and --global for update and remove actions.

What would --user, --global and --local even mean? A assume with --global you would mess with upstream data that's handled by your package manager (an absolute no-no), but what's the difference between --user and --local? It's all basically just paths. And path differ depending on installation and configuration.

Default configuration of Arch Linux systemd package for servers is worlds in /var/lib/minetest/worldname, mods and games in /var/lib/minetest/.minetest/mods (and .../games) and config in /etc/minetest/worldname.conf and the Minetest binary is /usr/bin/minetestserver.

So what are the local, global, user, and world paths? It just screams "I will mess up anything" to assume paths :)

User avatar
Blockhead
Member
Posts: 1602
Joined: Wed Jul 17, 2019 10:14
GitHub: Montandalar
IRC: Blockhead256
In-game: Blockhead Blockhead256
Location: Land Down Under
Contact:

Re: update-minetest-mods Linux bash scripts

by Blockhead » Post

Linuxdirk wrote:
Thu Jul 29, 2021 07:07
What would --user, --global and --local even mean?
User: user directory, ~/.minetest/mods. Does minetest compiled for Windows read from %userprofile%\.minetest\mods?

Global: /var as you say. Does seem like a bad idea, yes. It just seemed (in my mind) to follow the UNIX way of working with global and then per-user settings. Not that I think many computers will have such a use case.

Local: For portable Minetests, ../mods/. This is how you would install on Windows or if you don't install through the package manager but instead build from source on Linux. I suppose this one would be disabled if you compiled with -DRUN_IN_PLACE=FALSE.

As far as I see it though, why stop people shooting themselves in the foot? It's the command line after all.
/˳˳_˳˳]_[˳˳_˳˳]_[˳˳_˳˳\ Advtrains enthusiast | My map: Noah's Railyard | My Content on ContentDB ✝️♂

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Blockhead wrote:
Thu Jul 29, 2021 07:37
User: user directory, ~/.minetest/mods.
What if people for whatever reason (which is actually a good reason because this is where locally shared packages belong to!) use ~/.local/share/minetest/mods/ for their mods?
Blockhead wrote:
Thu Jul 29, 2021 07:37
Global: /var as you say.
Again: This is how it's set up on Arch by default. Other distributions might use /var/minetest/ for whatever reason, or do not facilitate /var/ at all and use /usr/share/.
Blockhead wrote:
Thu Jul 29, 2021 07:37
As far as I see it though, why stop people shooting themselves in the foot?
It's not about the users, it's about not having to code a dozen of different features and guessing half a dozen of paths but instead simply use a mandatory parameter giving the path to use.

Pablo.R
Member
Posts: 42
Joined: Thu Mar 24, 2016 12:02
Location: Chile

Re: update-minetest-mods Linux bash scripts

by Pablo.R » Post

This is the little script that I use to update my git repositories (including minetest repos) maybe is useful for you.

#!/bin/bash
# Update All Git Repositories Under Current Folder
# TODO: Pass Folder to check as parameter (optional)

# store the current dir
CUR_DIR=$(pwd)

# Let the person running the script know what's going on.
echo "Pulling in latest changes for all Git repositories active branch under folder: $CUR_DIR"

# Find all git repositories and update it to the master latest revision
for i in $(find . -type d -name ".git"); do
echo "";
echo "Found .git folder "$i;

# We have to go to the .git parent directory to call the pull command
cd "$CUR_DIR/$i";
cd ..;

# finally pull
echo "Processing Git Repository "$(pwd);
git pull;
git submodule update --init --recursive;

# lets get back to the CUR_DIR
done
cd "$CUR_DIR"
echo "";
echo "End!";

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: update-minetest-mods Linux bash scripts

by Linuxdirk » Post

Pablo.R wrote:
Wed Aug 04, 2021 00:43
cd "$CUR_DIR/$i";
cd ..;
cd "$CUR_DIR"
You can provide the working directory to Git using the -C parameter. By using this parameter you can eliminate all directory changes in your script.

Pablo.R
Member
Posts: 42
Joined: Thu Mar 24, 2016 12:02
Location: Chile

Re: update-minetest-mods Linux bash scripts

by Pablo.R » Post

Linuxdirk wrote:
Wed Aug 04, 2021 06:30
You can provide the working directory to Git using the -C parameter. By using this parameter you can eliminate all directory changes in your script.
Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests