Server Perfomance

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Server Perfomance

by frbitten » Fri Aug 24, 2018 18:47

Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Fri Aug 24, 2018 21:21

frbitten wrote:Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks


it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:

- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server

etc..

You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Fri Aug 24, 2018 21:37

sofar wrote:
frbitten wrote:Hi

Is there any test or performance study of the minetest server?

I would like to know the required requirements based on the number of players online and what the maximum levels the server supports.

Anyone know any posts on this? I did research and I did not find it.

Thanks


it varies wildly on several factors, and therefore it's really difficult to give specific answers. For instance, all of the following are huge factors in how well a minetest server performs:

- used storage backend for the map
- number of players
- lag that players may have
- CPU model (core speed, essentially, but cache size matters)
- RAM size, but also speed
- Storage speed
- network congestion factors
- mods used on the server

etc..

You can have a great server on 30$ of hardware, or you could have a terrible server on a beefcake machine, it all depends.



I understand these differences and factors that affect the player's experience.

I figured the developers would have done some perfomance testing in a controllable environment. I know that even though it can not be applied exactly in the end use it is a good analytical metric.

But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?

Thanks
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Sun Aug 26, 2018 04:54

frbitten wrote:But giving a more specific example then. On a server with only "minetest_game" would it work with 1000 simultaneous online players? What would be the hardare needed for this task?
Thanks


I don't think anyone attempted more than ~ 50 players at this point. Unless you can find 1000 actual human beings, this would be impossible to simulate.
 

User avatar
rubenwardy
Moderator
 
Posts: 5539
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Server Perfomance

by rubenwardy » Sun Aug 26, 2018 10:28

I've seen servers with up to 80 player on them, but they weren't snappy

Minetest isn't intended or designed for thousands of users, that's the scale when you need to start doing clever multithreading

so far wrote:Unless you can find 1000 actual human beings, this would be impossible to simulate


You can use bots to stress test the connection, freeminer did this
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Mon Aug 27, 2018 17:22

rubenwardy wrote:You can use bots to stress test the connection, freeminer did this


yes, and it's certainly a valid test case (and great data), but it won't actually say much about real world usage, like, every player is building in their own little plot, or, 250 players are attempting to fight one large monster. Both these scenarios are likely for massive servers, but you'd have to make some really smart AI to simulate it. (by all means, I'd love to see an approximation though).
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Sat Sep 01, 2018 14:03

Thank you all.
What he wanted to know was the scale that the development of the game was thought of. If we are talking about the house of tens. Surely it would not reach hundreds or thousands.

If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.

Thank you
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Sun Sep 02, 2018 04:10

frbitten wrote:If someone decides to do the automated test post the results here. Even not being precise for a real environment. It is a valid test for a first moment. Somanete 500 customers walked in a random direction would already be enough to see the level of refactoring that would be required to support real players on that scale.


not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Fri Sep 28, 2018 23:48

sofar wrote:not many people run on high end Xeon CPU's with high core frequencies and 10gig ethernet backbone connections. It's entirely possible that you can have 500+ people logged on to a server, and moving about and even doing things. Most people just shell out the money for low end hosting.


For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?

I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.

Thanks
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Sat Sep 29, 2018 02:57

frbitten wrote:For what you wrote wrote that the software supports this number of players, it would only be a matter of hardware needed to accomplish the task. How does developer know how to give hardare estimates for the server to support more than 500 players simultaneously interacting simultaneously in the same world?

I wanted to set up a game server but without an even shallow estimate I can not analyze the costs.


If I give you an estimate, and you go and pay e.g. 1000$ to procure the hosting based on that estimate, and the estimate was completely wrong, wouldn't you be very upset?

Look, most of us run a server for 10-30 or so people max, and we've had good success with hosting services that are in the 5$-20$ price per month range(I'm not even sure what most others are paying). Note that's not even an estimate, it's just a statement that it works for us, and might not work for you. If you want better estimates for numbers so large that we've never seen them, there is no way that you can or should rely on the few data points right now that exist, because they are entirely invalid for a scale that is off by a factor 10+.

So, if you want to know if 1000 players on a single server is possible, the only way is to try and properly document it. I'm game, of course, I'd love to see something like this.
 

User avatar
lag01
Member
 
Posts: 293
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag

Re: Server Perfomance

by lag01 » Sat Sep 29, 2018 10:07

By my observations, 0.4.17 minetestserver consumes 20-80KiB/s per player. (But possible that number increases when more players are online)
So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum.
 

User avatar
rnd
Member
 
Posts: 207
Joined: Sun Dec 28, 2014 12:24
IRC: ac_minetest
In-game: rnd

Re: Server Perfomance

by rnd » Sat Sep 29, 2018 10:40

"So 1000 players would require 20 MiB/s (170 Mbit/s) of outgoing connection as bare minimum."

The problem is not linear in number of players:
1. sending player data: If you have n players walking around spawn area, server needs to send player position/view angles/what did player do?... of the rest of n-1 players to each player, thats n*(n-1) = O(n^2) packets to send. This part is nonlinear, and scales very badly (2x more players, 4x more data, 5x more -> 25x more data, 10x more -> 100x more data )

2. sending map data: for each player you have to send some constant of data (on average). This part is linear.

So even 100 players walking all near large spawn area might be way over head of current networking implementation. I remember there was 10+s lag with 70-80 players on just test. On minecraft they must have some really optimized programming to be able to get : https://hypixel.net/ - "Join 19,680 other online Players"
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Sat Sep 29, 2018 18:58

1000 simultaneous players is a very complex task to be performed. Games that achieve this perfoamance are very rare.

The only thing that I participated in something like this is EVE Online where I participated in a fight with 2500 players at the same time. But the lag was absurd.

I am a developer and I understand the difficulty associated with doing something on this scale. My post was more out of curiosity what would be the known limits of the Minetest server.

For the comments is between 50 and 80 players simultaneously. Remembering that online players are not the same as simultaneous players. For they may be in places far from the servant that do not quarrel with one another.

I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Thanks for the feedback.
 

User avatar
Linuxdirk
Member
 
Posts: 1651
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: Server Perfomance

by Linuxdirk » Sat Sep 29, 2018 23:11

frbitten wrote:I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.
 

User avatar
gpcf
Member
 
Posts: 271
Joined: Fri May 27, 2016 10:48
In-game: gabriel

Re: Server Perfomance

by gpcf » Sat Sep 29, 2018 23:18

The problem with minetest is lack of multithreading. This way, it does not scale well to the strengths of modern CPUs.
 

User avatar
bhree
Member
 
Posts: 66
Joined: Tue Jun 19, 2018 07:45
GitHub: bhree
In-game: bhree

Re: Server Perfomance

by bhree » Sun Sep 30, 2018 14:09

And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.
 

User avatar
BBmine
Member
 
Posts: 3458
Joined: Sun Jul 12, 2015 22:51
Location: USA
GitHub: BBmine
IRC: BBmine
In-game: Baggins

Re: Server Perfomance

by BBmine » Mon Oct 01, 2018 01:00

Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Mon Oct 01, 2018 04:38

bhree wrote:And if minetest support something like mpirun then scaling across multiple servers allow players to almost unlimited number. But that may need radical design and another development team.


Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.

True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.

I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.

Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.
 

User avatar
bhree
Member
 
Posts: 66
Joined: Tue Jun 19, 2018 07:45
GitHub: bhree
In-game: bhree

Re: Server Perfomance

by bhree » Mon Oct 01, 2018 12:24

For single threaded system, the app itself is already awesome, simplest way just push the best hardwares and connection. Push the clockspeed, aquire RAM as much as possible, best internet connection. SSD over HDD. And because we are working with large database then enterprise grade DBMS should be applied. Each process is allowed to access its own core, thanks to the OS. Something like remote media server using http on another core will boost the performance. CPU of 4 cores is enough of the fastest clockspeed and if someone can afford half TB of RAM. Nothing beats that.
MPI is not the only multiprocess solution but simpler and cheaper to deploy, albeit the complexity in software design.
Server hardware isn't everything. If you have 100 players at spawn, each individual player will have player information transmitted to them about the other 99 players. That's a lot of bandwidth for the clients!

That will be the problem of every client, usually solved with better hardware and connection. I'm not in a capacity of questioning the software.
 

User avatar
sorcerykid
Member
 
Posts: 860
Joined: Fri Aug 26, 2016 15:36
Location: Illinois, USA
GitHub: sorcerykid
In-game: Nemo

Re: Server Perfomance

by sorcerykid » Mon Oct 01, 2018 18:03

Linuxdirk wrote:
frbitten wrote:I was thinking of using minitest for youtuber to be able to play with their live streaming. A minecraft live is common to pass 500 viewers.

Forget that. It is impossible for Minetest to handle 500 players. Not even 50 without horrible lag.


What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.

I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.

All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Mon Oct 01, 2018 21:38

At the risk of going offtopic, but we're still talking about an open source project here:

sorcerykid wrote:I'm not going to give away all the optimization tricks I used.


That's unfortunate, I wish you would. Everyone could benefit from that.

sorcerykid wrote:All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)


Please consider contributing some of the knowledge and methods back to the community instead.

The player community on JT2 isn't large enough to sustain all of the Minetest development. That needs many more good quality servers. You'd be helping yourself by helping other server operators run a higher quality shop.
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Mon Oct 01, 2018 21:57

sorcerykid wrote:What? JT2 frequently had 50 or more players with a max_log of just 0.12 seconds. Fixer has even commented on several occasions about the unusual performance compared to other survival servers.

I'm not going to give away all the optimization tricks I used. But off the top of my head, I'll just say that I spent a great deal of time rewriting various portions of the engine to address scalability and efficiency concerns (two of the major bottlenecks being the ABM Handler and authentication handler, but some of the other builtin Lua routines and mods like "tnt" were vastly reworked as well). I also implemented smart-caching mechanisms for node digging and placement to streamline protection and privilege checks. And the world directory for the production server is configured to use tmpfs.

All in all, it is entirely possible for a Minetest server to have 50 players with almost no perceptible lag, as long as you have sufficient bandwidth and a decently powerful CPU with enough RAM.

Edit: In fairness, by forking and changing the engine, I suppose wouldn't really be accurate of me to say that I achieved many of the performance gains with Minetest itself. Oh well :)


can you give details of the hardware used for that number of players?
 

frbitten
Member
 
Posts: 11
Joined: Thu Jul 12, 2018 00:16
GitHub: frbitten
IRC: frbitten
In-game: frbitten

Re: Server Perfomance

by frbitten » Mon Oct 01, 2018 22:00

sofar wrote:Making minetest scale across multiple cores or even across nodes isn't as simple as bolting on something like MPI. That would likely accomplish the opposite, and make performce drastically degrade.

True multi node scaling of an event driven environment like minetest where nearby players interact would have to be designed from scratch. Even then there are limits as to how many players close together you could handle. And then, if you truly manage to scale up, your lag would also increase due to the extra care that is needed.

I'm not convinced by the discussion that any of this type of work is needed. I'd rather see some actual experiments with data first to see how good/bad it actually is. And given that most responses on this thread are of the "I used a lump of clay to run a server and it sucked" type, I'm entirely pessimistic that people understand that performance work requires solid data to begin with, and ends with solid data.

Note: The game code really matters a lot. If you program your game code well (the Lua part) you can remove most of the heavy bottlenecks if you're careful. But if you grab a handful of random mods that look cool and don't look at the internals, you're going to have a bad time. Running a high quality server requires significant coding skills.


Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?
 

sofar
Developer
 
Posts: 1762
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Server Perfomance

by sofar » Mon Oct 01, 2018 22:24

frbitten wrote:Does the server code not use multiple colors in any case? Control of connections (socket), login, etc. Is it all running on the same thread?


s/colors/threads/, I assume

The server engine does use threads but it's limited to a few easy to split up things. The only significant thing that is running in a separate thread is mapgen, essentially.

This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet. If you process packets (ipv4, UDP, socket level etc) entirely on the same core, you'd avoid that problem. This is why benchmark professionals often "pin" related software tasks all on a single core so that those cross-core overheads are eliminated.

A lot of menial tasks also don't make sense to split up. Your hot path is going to be world events. Even if you handle a login event per second, it's still peanuts compared to hundreds of server ticks with player actions and block updates. It just has to be not a burden on overall performance. You have 50ms - 100ms per server tick, and that is a lot of time to handle the occasional login event.
 

User avatar
rubenwardy
Moderator
 
Posts: 5539
Joined: Tue Jun 12, 2012 18:11
Location: United Kingdom
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

Re: Server Perfomance

by rubenwardy » Mon Oct 01, 2018 22:32

sofar wrote:This isn't necessarily bad. For instance, putting socket handling on a different core isn't a way to make socket handling faster. In fact, if you move low level packet handling to a different CPU, you actually decrease the network performance, because now the data may need to be copied from core to core and there will be a significant amount of cache misses for every single packet.


Minetest uses threads for sending and receiving on sockets.

Here are a list of threads on a minetestserver:

  • Server - Main thread
  • Emerge-0 - Map generation
  • ... Emerge-1 ... Emerge-2 ... etc
  • ConnectionSend - sending
  • ConnectionRecei - receive
  • CurlFetch - not sure. Announcing to server list?

You can find this out yourself by running: ps -TC minetestserver
 

Next

Return to Servers



Who is online

Users browsing this forum: No registered users and 6 guests