Server Perfomance

[frbitten]

Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks

[sofar]

Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks

it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:

- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server

etc..

You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.

[frbitten]

Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks

it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:

- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server

etc..

You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.

I understand these differences and factors that affect the player's experience.

I figured the developers would have done some perfomance testing in a controllable environment. I know that even though it can not be applied exactly in the end use it is a good analytical metric.

But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?

Thanks

[sofar]

But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?
Thanks

I don't think anyone attempted more than ~ 50 players at this point. Unless you can find 1000 actual human beings, this would be impossible to simulate.

[rubenwardy]

I've seen servers with up to 80 player on them, but they weren't snappy

Minetest isn't intended or designed for thousands of users, that's the scale when you need to start doing clever multithreading

Unless you can find 1000 actual human beings, this would be impossible to simulate

You can use bots to stress test the connection, freeminer did this

[sofar]

You can use bots to stress test the connection, freeminer did this

yes, and it's certainly a valid test case (and great data), but it won't actually say much about real world usage, like, every player is building in their own little plot, or, 250 players are attempting to fight one large monster. Both these scenarios are likely for massive servers, but you'd have to make some really smart AI to simulate it. (by all means, I'd love to see an approximation though).

[frbitten]

Thank you all.
What he wanted to know was the scale that the development of the game was thought of. If we are talking about the house of tens. Surely it would not reach hundreds or thousands.

If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.

Thank you

[sofar]

If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.

not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.

[frbitten]

not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.

For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?

I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.

Thanks

[sofar]

For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?

I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.

If I give you an estimate, and you go and pay e.g. 1000$ to procure the hosting based on that estimate, and the estimate was completely wrong, wouldn't you be very upset?

Look, most of us run a server for 10-30 or so people max, and we've had good success with hosting services that are in the 5$-20$ price per month range(I'm not even sure what most others are paying). Note that's not even an estimate, it's just a statement that it works for us, and might not work for you. If you want better estimates for numbers so large that we've never seen them, there is no way that you can or should rely on the few data points right now that exist, because they are entirely invalid for a scale that is off by a factor 10+.

So, if you want to know if 1000 players on a single server is possible, the only way is to try and properly document it. I'm game, of course, I'd love to see something like this.

[lag01]

By my observations, 0.4.17 minetestserver consumes 20-80KiB/s per player. (But possible that number increases when more players are online)
So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum.

[rnd]

"So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum."

The problem is not linear in number of players:
1. sending player data: If you have n players walking around spawn area, server needs to send player position/view angles/what did player do?... of the rest of n-1 players to each player, thats n*(n-1) = O(n^2) packets to send. This part is nonlinear, and scales very badly (2x more players, 4x more data, 5x more -> 25x more data, 10x more -> 100x more data )

2. sending map data: for each player you have to send some constant of data (on average). This part is linear.

So even 100 players walking all near large spawn area might be way over head of current networking implementation. I remember there was 10+s lag with 70-80 players on just test. On minecraft they must have some really optimized programming to be able to get : https://hypixel.net/ - "Join 19,680 other online Players"

[frbitten]

1000 simultaneous players is a very complex task to be performed. Games that achieve this perfoamance are very rare.

The only thing that I participated in something like this is EVE Online where I participated in a fight with 2500 players at the same time. But the lag was absurd.

I am a developer and I understand the difficulty associated with doing something on this scale. My post was more out of curiosity what would be the known limits of the Minetest server.

For the comments is between 50 and 80 players simultaneously. Remembering that online players are not the same as simultaneous players. For they may be in places far from the servant that do not quarrel with one another.

I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Thanks for the feedback.

[Linuxdirk]

I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.

[gpcf]

The problem with minetest is lack of multithreading. This way, it does not scale well to the strengths of modern CPUs.

[bhree]

And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.

[BBmine]

Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!

[sofar]

And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.

Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.

True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.

I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.

Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.

[bhree]

For single threaded system, the app itself is already awesome, simplest way just push the best hardwares and connection. Push the clockspeed, aquire RAM as much as possible, best internet connection. SSD over HDD. And because we are working with large database then enterprise grade DBMS should be applied. Each process is allowed to access its own core, thanks to the OS. Something like remote media server using http on another core will boost the performance. CPU of 4 cores is enough of the fastest clockspeed and if someone can afford half TB of RAM. Nothing beats that.
MPI is not the only multiprocess solution but simpler and cheaper to deploy, albeit the complexity in software design.

Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!

That will be the problem of every client, usually solved with better hardware and connection. I'm not in a capacity of questioning the software.

[sorcerykid]

I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.

What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.

I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.

All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)

[sofar]

At the risk of going offtopic, but we're still talking about an open source project here:

I'm not going to give away all the optimization tricks I used.

That's unfortunate, I wish you would. Everyone could benefit from that.

All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)

Please consider contributing some of the knowledge and methods back to the community instead.

The player community on JT2 isn't large enough to sustain all of the Minetest development. That needs many more good quality servers. You'd be helping yourself by helping other server operators run a higher quality shop.

[frbitten]

What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.

I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.

All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)

can you give details of the hardware used for that number of players?

[frbitten]

Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.

True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.

I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.

Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.

Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?

[sofar]

Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?

s/colors/threads/, I assume

The server engine does use threads but it's limited to a few easy to split up things. The only significant thing that is running in a separate thread is mapgen, essentially.

This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet. If you process packets (ipv4, UDP, socket level etc) entirely on the same core, you'd avoid that problem. This is why benchmark professionals often "pin" related software tasks all on a single core so that those cross-core overheads are eliminated.

A lot of menial tasks also don't make sense to split up. Your hot path is going to be world events. Even if you handle a login event per second, it's still peanuts compared to hundreds of server ticks with player actions and block updates. It just has to be not a burden on overall performance. You have 50ms - 100ms per server tick, and that is a lot of time to handle the occasional login event.

[rubenwardy]

This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet.

Minetest uses threads for sending and receiving on sockets.

Here are a list of threads on a minetestserver:

• Server - Main thread
• Emerge-0 - Map generation
• ... Emerge-1 ... Emerge-2 ... etc
• ConnectionSend - sending
• ConnectionRecei - receive
• CurlFetch - not sure. Announcing to server list?

You can find this out yourself by running: ps -TC minetestserver