Severe networking issues with Minetest 5.3.0

Post Reply
Nicu
Member
Posts: 98
Joined: Sun Feb 16, 2020 15:21
GitHub: kneekoo

Severe networking issues with Minetest 5.3.0

by Nicu » Post

Disclaimer: This is not just about Minetest, but it's a more complex situation, still not fully clear to me. However, it involves Minetest. It's also a big headache, so be prepared. :P

Two days ago I started having internet connectivity issues. I suspected ISP maintenance, but it kept going on and on. My Minetest server runs on a Raspberry Pi 4 model B with 2GB RAM, which is enough for a reasonable experience with a friend of mine playing MineClone 2.

My router kept restarting while I was in the game, so I suspected maintenance, then I suspected the router being a few years old, then I suspected my USB WiFi adapter running a community-maintained driver, and then I noticed an update to the wpasupplicant package was applied just before I started having issues. That's the WiFi software client, so it's a crucial piece of the puzzle. For the record, my OS is Linux Mint 20.1 (based on Ubuntu 20.04).

Now I'm pretty sure that wpasupplicant not only fixed to CVEs, but it looks like a regression is at play, giving me a really hard time playing. And that's the thing, I only have connectivity issues (router restarts) while I'm in Minetest, playing MineClone 2. Along with this, the log has plenty of "quota" issues:

Code: Select all

2021-02-15 00:07:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 4 pkts
2021-02-15 00:07:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 1024 pkts
2021-02-15 00:07:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:07:50: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:07:52: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:07:55: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:07:56: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:08:02: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 184 pkts
2021-02-15 00:08:03: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:08:03: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 192 pkts
2021-02-15 00:08:03: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 113 pkts
2021-02-15 00:08:04: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 105 pkts
2021-02-15 00:08:04: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 175 pkts
2021-02-15 00:08:07: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 155 pkts
2021-02-15 00:08:07: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 29 pkts
2021-02-15 00:08:07: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 18 pkts
2021-02-15 00:08:09: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 147 pkts
2021-02-15 00:08:09: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 87 pkts
2021-02-15 00:08:10: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 9 pkts
2021-02-15 00:08:12: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 8 pkts
2021-02-15 00:08:12: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 150 pkts
2021-02-15 00:08:13: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 30 pkts
2021-02-15 00:08:14: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 83 pkts
2021-02-15 00:08:15: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:08:16: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:08:17: WARNING[ConnectionSend]: con(5/1) Packet quota used up after re-sending packets, max=1024
2021-02-15 00:08:18: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 79 pkts
2021-02-15 00:08:27: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 196 pkts
2021-02-15 00:08:29: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 98 pkts
2021-02-15 00:08:31: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 6 pkts
2021-02-15 00:08:31: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 157 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 3 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 177 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 103 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 22 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 92 pkts
2021-02-15 00:08:32: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 201 pkts
2021-02-15 00:08:34: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 14 pkts
2021-02-15 00:08:34: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 45 pkts
2021-02-15 00:08:34: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 4 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 27 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 117 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 113 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 32 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 113 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 25 pkts
2021-02-15 00:08:35: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 64 pkts
2021-02-15 00:08:36: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 203 pkts
2021-02-15 01:55:22: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=82, was 420 pkts
2021-02-15 02:08:45: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 393 pkts
2021-02-15 02:08:46: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 253 pkts
2021-02-15 02:08:46: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 237 pkts
2021-02-15 02:08:46: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 225 pkts
2021-02-15 02:08:47: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 215 pkts
2021-02-15 02:08:47: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 223 pkts
2021-02-15 02:08:47: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 199 pkts
2021-02-15 02:08:48: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 195 pkts
2021-02-15 02:08:48: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 199 pkts
2021-02-15 02:08:48: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 162 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 179 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 183 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 229 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 207 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 224 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 245 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 243 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 247 pkts
2021-02-15 02:08:49: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 256 pkts
2021-02-15 02:08:50: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 256 pkts
2021-02-15 02:08:50: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 256 pkts
2021-02-15 02:08:50: WARNING[ConnectionSend]: con(5/1) Packet quota used up for peer_id=80, was 256 pkts
Those happened before wpasupplicant's update which lead to router restarts, and I ran /clearobjects to "unclog" the server, which worked well enough to keep playing - either alone or with my friend (2 players max). But while these warnings didn't cause my router to restart, I wonder if now it bumps heads with the latest wpasupplicant release, making it do something so weird that the router goes nuts as a result.

Fun fact, I can leave the game running from a virtual machine, and that has already worked for 5 hours because that OS doesn't allow the game to interact more or less directly with wpasupplicant on my host OS.

For now, I have 2 questions:
- What's up with that packet quota and how can it be addressed for a smoother experience?
- What do you think could happen that only Minetest (+MineClone 2) can cause my router to crash?
Last edited by Nicu on Mon Feb 15, 2021 23:11, edited 2 times in total.

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: Networking issues with Minetest 5.3.0

by Festus1965 » Post

oh, that is the second post about this phenomena ...

"I think there was a change (unknown, unwanted) in some part of minetest protocol"

edit: the problem of gamer Thailand, Bonat, Cooking is solved ... was a old, not good net-cable

also on server jjk1 2 gamer issued just yesterday that they have unusual timeouts or cant even connect on that server with protocol failure.

So in my opinion, ask the devs ... there went something wrong ?
Last edited by Festus1965 on Tue Feb 16, 2021 03:34, edited 2 times in total.
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Networking issues with Minetest 5.3.0

by Minix » Post

You could try to exclude wifi/wpasupplicant from the problem by trying with ethernet (I think your raspberry comes with ethernet right ?). Try running your Pi with ethernet, it is always more stable and reliable than wifi. As for the packet quota, it means that the clients connected to your server are using up too much bandwidth (more than 1024 packets per server nertwork step), it could be due to them downloading the media from your server or some weird situation.

Nicu
Member
Posts: 98
Joined: Sun Feb 16, 2020 15:21
GitHub: kneekoo

Re: Networking issues with Minetest 5.3.0

by Nicu » Post

I can now confirm this issue on 3 PCs with different operating systems and different wifi adapters, all 3 running wpasupplicant:amd64 2:2.9-1ubuntu4.2 and Minetest 5.3.0.

PC 1: Ubuntu 18.04.5, WiFi adapter Intel 8265 (kernel module iwlwifi)
PC 2: Linux Mint 19.3 (based on Ubuntu 18.04), WiFi adapter Intel 7260 (kernel module iwlwifi)
PC 3: Linux Mint 20.1 (based on Ubuntu 20.04), WiFi adapter Realtek 8812AU (out-of-tree kernel module rtl8812au)

Assuming only Minetest does something wrong with its network code, wpasupplicant is dumb enough to send weird packages to the router, causing it to crash. Assuming only wpasupplicant is at fault, I don't see how only Minetest can bring my connection down, considering I use plenty of other networking software with wpasupplicant. So it looks to me like both wpasupplicant and Minetest 5.3.0 do something wrong.

As I mentioned earlier, I already bypassed wpasupplicant by leaving Minetest open for hours in a VM that doesn't have wpasupplicant running. No problem with that, other than the game being slow. This is a fun server for me and a friend of mine, so nothing's urgent enough to warrant bypassing wpasupplicant by getting off of wifi or replacing it with wicd at this point. But regardless of what I end up doing, this is worth investigating, as it's easily reproducible (for me at least). The router might as well be dumb enough to crash when this odd duo do their thing, but it would be nice for others to try this out and share their experience.

@Minix: To me, this networking issue doesn't seem to be on the Pi side. The problem only manifests from an OS running wpasupplicant, and I tried both connecting Minetest through my dynamic DNS hostname, and internal IP address, both with the same result.

I don't know about the quota. Hopefully someone familiar with the Minetest network code can shed some light here. But those warnings also happened before the bad wpasupplicant update, so maybe that's entirely a different issue.

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Severe networking issues with Minetest 5.3.0

by Minix » Post

If you think this issue deserves more investigation consider opening an issue on minetest's github repository at https://github.com/minetest

As I understand your main problem is your router restarting by itself, try resetting your router's configuration to factory defaults and see if it keeps restarting. You could also try to downgrade wpasupplicant to an older version just for the sake of testing.

User avatar
Minix
Member
Posts: 144
Joined: Thu Nov 12, 2020 13:51
In-game: Minix

Re: Severe networking issues with Minetest 5.3.0

by Minix » Post

Also, it is really weird that you see so many warnings about the quota taking into account only two players play and it is a private server. Do you/your friend play on android by a chance ? because the android version of minetest for some reason likes to download the media everytime you join a server, at least it happens to me.

You could try to run iftop on your server when the warnings come out, if you see high network traffic (more than 320 kbit/s total for each player) when the players are already in the server (not the loading screen) then somehow your clients are sucking up your bandwidth.

Nicu
Member
Posts: 98
Joined: Sun Feb 16, 2020 15:21
GitHub: kneekoo

Re: Severe networking issues with Minetest 5.3.0

by Nicu » Post

I wanted to explore this issue a little here and get more insight from the team before opening an issue there. They're anyway both in here as well as on GitHub, for the most part, so there's that. :)

I might downgrade wpasupplicant on one of my PCs, for a short test, but that's clearly not *the* solution, considering the latest update was a security one and I want it installed. But I have a high degree of confidence that the downgrade test won't reveal stuff that I didn't know. Everything was fine before the update, then it went sideways.

The router, I'd rather not touch. I have reserved IPs, port forwarding and stuff that would take more time that it's worth. While I can't rule out the router having part of the blame for its connection loss/restart, wpasupplicant and Minetest are the two that intensely stare in my face as the culprits. :))

I'm on Linux (the 3 PCs that I mentioned), and my friend uses Windows 8.1. The quota warnings happen for both of us. Here's a log analysis for the whole 2021-02-15 day:
Image

The ABM issue is insane. It might be related to the Pi's slower storage[1] I/O speed compared to that of a traditional storage device, but that excessive logging in itself generates so much I/O that it probably stands in its own way. :P

The bandwidth shouldn't be an issue. The Pi can handle a reasonable amount of data traffic, and the game has no reason to constantly download/upload stuff between the server and clients. Quite an unscientific statement here, but we need something better than iotop for this one. :)

[1] SanDisk Ultra microSDXC UHS-I 128GB

User avatar
Festus1965
Member
Posts: 4181
Joined: Sun Jan 03, 2016 11:58
GitHub: Festus1965
In-game: Festus1965 Thomas Thailand Explorer
Location: Thailand ChiangMai
Contact:

Re: Severe networking issues with Minetest 5.3.0

by Festus1965 » Post

no, it is more like active Block modifiers,
is as of been activated MapBlock as of a gamer or forceloaded,
the wider (activation range 1, 2, 3 , or more) the more MapBlocks have to be updated again and again
1 = 1 MapBlock, 2 = 9, 3 = 25, 4 = ...
and it depends on the Biome the gamer is located, as some of them (mainly I remember jungle, pipeworks) the gamer is activating

so for this, better check the tool inside minetest to find out what mods are working there so much, and if you might reduce the needed Mapblocks reducing the range ...
Human has no future (climate change)
If urgend, you find me in Roblox (as CNXThomas)

Nicu
Member
Posts: 98
Joined: Sun Feb 16, 2020 15:21
GitHub: kneekoo

Re: Severe networking issues with Minetest 5.3.0

by Nicu » Post

I just tried the game on a 4th PC that didn't have the updated version of wpasupplicant and, as expected, it didn't crash my router. Of course, using outdated/insecure software isn't a solution, but it was worth testing and reporting. I'll try my luck with this topic on GitHub. :)

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests