Better support for multicore CPUs
Better support for multicore CPUs
As far as I know games like minetest are more limited to the CPU than the GPU. Thats why I am a little bit disappointed, because just 50% of my quadcore CPU is used. I know that multithreading is not that easy, but it could improve the game.
I tryed num_emerge_threads = 2/3/4, but it has no effect to the performance or the CPU load.
Are there any plans for that?
I tryed num_emerge_threads = 2/3/4, but it has no effect to the performance or the CPU load.
Are there any plans for that?
- Calinou
- Moderator
- Posts: 3169
- Joined: Mon Aug 01, 2011 14:26
- GitHub: Calinou
- IRC: Calinou
- In-game: Calinou
- Location: Troyes, France
- Contact:
Re: Better support for multicore CPUs
num_emerge_threads sets the number of threads to use for C++ map generation only, it won't affect mods and other areas of the engine.
-
- New member
- Posts: 3
- Joined: Wed Jan 28, 2015 12:52
- GitHub: gregorycu
- IRC: gregorycu
- In-game: gregorycu
Re: Better support for multicore CPUs
I'm always looking for ways to improve performance in ways that has the most positive impact on end users.
A question, are you noticing poor performance? Or are you just noticing minetest is not using 100% CPU?
A question, are you noticing poor performance? Or are you just noticing minetest is not using 100% CPU?
Re: Better support for multicore CPUs
Well, if the game get's more content, like more plants, the performance drops. If you are just walking and a few blocks ahead is nothing because the mapgen lacks, thats annoying.gregorycu wrote:A question, are you noticing poor performance? Or are you just noticing minetest is not using 100% CPU?
I don't have high end hardware, but a quad core with 16GB of RAM and a Nvidia GTX750 should be enough to run MT acceptable
Re: Better support for multicore CPUs
You're saying that minetest is taking 50% of your CPU. That's likely 50% of one core. If you'd split that workload to multiple cores, then that would still consume (in total) the exact same amount of CPU cycles, but it would require extra locks, and those are expensive, and generally make performance worse.
So, making things more parallel isn't always better for performance.
So, making things more parallel isn't always better for performance.
Re: Better support for multicore CPUs
I say, the CPU runs at 200% (or less) of 400%. Highest value I ever saw was 240%, so I guess there are more than 2 tasks right now.
I know that it could be tricky to deal with threads. But one suggestion might be to use LevelDB instead of SQLite. As far as I know all operations are asynchronous. So LevelDB could run as task, if it is not already doing.
And I think, the Mapgen and Lua are the bottlenecks. I don't know exactly how Mapgens work, but maybe each mapchunk can be generated in a separate task. I don't know how much nodes are connected to the neighbors
I know that it could be tricky to deal with threads. But one suggestion might be to use LevelDB instead of SQLite. As far as I know all operations are asynchronous. So LevelDB could run as task, if it is not already doing.
And I think, the Mapgen and Lua are the bottlenecks. I don't know exactly how Mapgens work, but maybe each mapchunk can be generated in a separate task. I don't know how much nodes are connected to the neighbors
Re: Better support for multicore CPUs
Mapgen is already a separate thread. There are two mapgen threads to start by default.
- rubenwardy
- Moderator
- Posts: 6978
- Joined: Tue Jun 12, 2012 18:11
- GitHub: rubenwardy
- IRC: rubenwardy
- In-game: rubenwardy
- Location: Bristol, United Kingdom
- Contact:
Re: Better support for multicore CPUs
[citation needed]sofar wrote:Mapgen is already a separate thread. There are two mapgen threads to start by default.
As far as I know, mapgen is done on the server thread as we need to support lua mapgens.
The emerge thread is for other things, I think it is for sending blocks to clients, not sure.
Re: Better support for multicore CPUs
ah, yes, you're correct. emerge threads are for loading, compressing and sending mapblocks to clients.rubenwardy wrote:[citation needed]sofar wrote:Mapgen is already a separate thread. There are two mapgen threads to start by default.
As far as I know, mapgen is done on the server thread as we need to support lua mapgens.
The emerge thread is for other things, I think it is for sending blocks to clients, not sure.
Re: Better support for multicore CPUs
Thread doesn't necessarily mean, that this thread runs on another core. Didn't look at the code
Re: Better support for multicore CPUs
IIRC, minetest does not have true multicore support, it mostly uses 1 CPU, for example my CPU load is 25% (on 4 cpu logical cores).A question, are you noticing poor performance? Or are you just noticing minetest is not using 100% CPU?
Re: Better support for multicore CPUs
If you play in singleplayer 2 cores are used. One for server, one for client
Re: Better support for multicore CPUs
I checked the Threads of a minetestserver and get the number 6.
One for Map Save.
Two for Mapgen.
But what is the reset, is there more then one for lua mods?
My Problem is that one of my Cores is often at 100%, but the other at 20%.
Can I split one Thread into two?
One for Map Save.
Two for Mapgen.
But what is the reset, is there more then one for lua mods?
My Problem is that one of my Cores is often at 100%, but the other at 20%.
Can I split one Thread into two?
Re: Better support for multicore CPUs
Yes, and, No.Lejo wrote:I checked the Threads of a minetestserver and get the number 6.
One for Map Save.
Two for Mapgen.
But what is the reset, is there more then one for lua mods?
My Problem is that one of my Cores is often at 100%, but the other at 20%.
Can I split one Thread into two?
Workload division is often not easy to do and balance, and it's often not actually faster.
All lua workloads run in the same (main) thread.
-
- Member
- Posts: 199
- Joined: Fri Oct 31, 2014 22:56
- Location: Top End, Australia
Re: Better support for multicore CPUs
Just my own observation here, through several different versions of Minetest over several generations of computer. For the most part I've had very little performance troubles running in Linux but I did notice a definite drop in performance whenever there's a large area of jungle weeds in front of me, turn around and face away from them and performance picks up again. Use worldedit to replace the weeds with air and there's no more performance drop.Well, if the game get's more content, like more plants, the performance drops. If you are just walking and a few blocks ahead is nothing because the mapgen lacks, thats annoying.
It sound like I currently have a similar computer to Burli:I don't have high end hardware, but a quad core with 16GB of RAM and a Nvidia GTX750 should be enough to run MT acceptable
3.2 Ghz Kaby Lake Core i5
16 Gb ram at 2133 Mhz
I only have the built in intel graphics but it seems pretty good considering what it is.
I have noticed a strange thing since I upgraded to Fedora 28 however. Every few minutes the game freezes for half a second, just long enough to make you put a block in the wrong place. Very annoying. I believe this must be a fedora problem however because it's the same version of minetest I was running before I upgraded and it wasn't doing it then.
Cheers,
Andrew.
Re: Better support for multicore CPUs
Is it possible to make to group of mods and each group runs in a extra thread??
As lua can not handle multiply threads to the same time, this would be a solution.
You can put mods for blocks in one and the mobs in the other thread.
As lua can not handle multiply threads to the same time, this would be a solution.
You can put mods for blocks in one and the mobs in the other thread.
Re: Better support for multicore CPUs
Try deleting the config file. When I updated from win7 to win10 the game lagged terribly and deleting the file fixed it.slopsbucket wrote:
I have noticed a strange thing since I upgraded to Fedora 28 however. Every few minutes the game freezes for half a second, just long enough to make you put a block in the wrong place. Very annoying. I believe this must be a fedora problem however because it's the same version of minetest I was running before I upgraded and it wasn't doing it then.
Cheers,
Andrew.
Testin' mines since 1989
Re: Better support for multicore CPUs
While technically something like this is possible, there is a big cost to this. Either you'd have to implement locking to prevent race issues, or you'd simply lose functionality and flexibility.Lejo wrote:Is it possible to make to group of mods and each group runs in a extra thread??
As lua can not handle multiply threads to the same time, this would be a solution.
You can put mods for blocks in one and the mobs in the other thread.
There are other solutions you can do that may help, though. For instance, you can use IPC to offload tasks to different processes if they're heavy computational, and use asynchronous methods to defer work that way.
This is a method I use on Inside The Box, for instance. The score/rank code is much too complex to run in Lua, it would stall the server for minutes, so I offload the task to a completely separate program that calculates the results and passes those results back to the server when it's updated. This does not require any modification to the server, and there are several ways of doing this (luasockets, HTTP).
- TalkLounge
- Member
- Posts: 324
- Joined: Sun Mar 26, 2017 12:42
- GitHub: TalkLounge
- In-game: TalkLounge
- Location: Germany
Re: Better support for multicore CPUs
That works, yes. But its not that effective as it would be done in Lua. The processor will run in both ways. But in the direct comparison I think Lua would have a much better performance, because you dont need to deliver it back to Lua.completely separate program that calculates the results and passes those results back to the server
I found something interesting: https://dev.minetest.net/Engine_structure#Threads
Subgames Server: Sky World Subgames German Survival Server: Wildes Land 2 E-Mail: talklounge@yahoo.de
Re: Better support for multicore CPUs
Doing it in lua would be cumbersome, as I'd have to manipulate large data sets (millions of records) in Lua and it's not necessarily well adopted to do that. Other languages properly grow arrays and stacks optimistically which is far more efficient, and the language I am using (golang) to process the data is well adapted at handling the amount of data efficiently. You don't want Lua to go out of memory either, since that crashes the server context.TalkLounge wrote: That works, yes. But its not that effective as it would be done in Lua. The processor will run in both ways. But in the direct comparison I think Lua would have a much better performance, because you dont need to deliver it back to Lua.
- TalkLounge
- Member
- Posts: 324
- Joined: Sun Mar 26, 2017 12:42
- GitHub: TalkLounge
- In-game: TalkLounge
- Location: Germany
Re: Better support for multicore CPUs
Yes, then its useful to use Java(I have programmed already in this language) etc.manipulate large data sets (millions of records) in Lua
My message was pointed at Lejos "make to group of mods and each group runs in a extra thread". That arent big tasks, but many little tasks. If every mod would run in an extra thread, you have a better performance. And you can check, which thread uses the most processor time. I know that there is a profiler, but it doesnt show me real values. The profiler show me, how long a function needs, until its finish. That arent real values. I made the experience, that I can´t look into the profiler and trust him. When I have a permanent lag from 1 but count together all values from the profiler, I am far away from one second. In theory, when I have maximal 99% processor load, there isnt a max_lag higher than 0.1, when every mod have its own thread. But there is. Currently I have a max_lag from 1.4 but on each of my two processors not more than 15% processor load(670 MB from 3 GB RAM is used). There is more capacity, but minetest doesnt use it. Minetest produces lag himself. And much servers have those lag problems.
Subgames Server: Sky World Subgames German Survival Server: Wildes Land 2 E-Mail: talklounge@yahoo.de
Re: Better support for multicore CPUs
Oversimplification, and, likely false.TalkLounge wrote:If every mod would run in an extra thread, you have a better performance.
Mods all manipulate the same basic environment. Therefore, if they run in threads, they'd be manipulating the same data and would have to be modified to do this in a safe way. This is done using locks, atomics, semaphores etc, and, those are not cheap. You'd make the overall performance cost of running Lua code much heavier, and you'd get little to no benefit out of it. In fact, it would likely be really hard to do this properly and you'd end up with a very unstable engine for a very long time. Plus, concurrency is really, really difficult to get right. Debugging locking issues is incredibly difficult.
So "just" putting mods in a thread isn't something any dev in the team even wants to do. Just write better lua code.
-
- Member
- Posts: 361
- Joined: Tue Jan 19, 2016 16:46
- GitHub: jdc843
- IRC: jdc843
- In-game: 843jdc
- Location: USA
Re: Better support for multicore CPUs
Minetest is slow.
Anyone that says otherwise is talking trash.
I'm reading a lot of forum messages discussing how things CAN"T be done, but relatively few on possible ways in which things CAN be done
I have a Intel i7-3820 cpu, 16 GB RAM, Nvidia GTX560 (now a GTX 660), HDD & a SSD. A now dated system but still plenty fast for running Windows 10 and Linux applications.
When I was running Cash's World, the big bottleneck was with the HDD. I switched to the SSD, installed Redis, and the bottleneck moved to the network. I upgraded my network speed and tinkered with Linux's networking code. Except for initial client connections which maxed out my sending speed, that moved the bottleneck to the CPU. The i7-3820 is quad-core. One core was working at 100% running the server.
Even logging in to the server on the same machine was laggy! I started tinkering with the LUA code drastically changing how the mods operate in an attempt to speed things up. (My changes were never completed. Admittedly a butcher job.) That didn't work to speed things up.
With my computer mostly idling except for the server process and occasional Redis writes to the map file, I can only conclude that the Minetest server code is slow as a bug in molasses.
Here's one thought:
1 - Have the server concentrate on what is in the world. Nothing else. It doesn't care how something works or reacts but only that something is there. A chest is only a chest. the server don't care what's in it. Et cetera
2 - Player connections get offloaded to a separate process to log players in and out. Network priority needs to be given to current players so they don't experience lag.
3 - Player movements, machine operations, tree growth, etc are handled by separate processes that inform the server of any needed changes to the map.
4 - Conflicts. Hmm, that's a toughie. I'll leave this for someone else to solve.
5 - Requests for multi-threaded and GPU coding changes go into the TODO file. Maybe someone will step up and work on something. Maybe not haha.
Baby steps. Start small. Have only a map server, one process to log players in and out, and one process to handle everything else. Then players can log in fast and not cause the server to stall while it sends map data and images to the new player. Later, that one big process that handles "everything else" can then be split up as needed.
I haven't played Minetest much in the past 2? years much less studied the server's code. So I don't know what the current state is. I just know reading posts from the developers complaining "We can't do ____ because of ____." is a total cop-out.
The developers have done a marvelous job writing this FREE game for anyone to enjoy. You have my thanks and gratitude for your hard and un-paid work. I agree to not make changes when someone asks for a fundamental change to how the game plays. But complaints about how the game currently runs should be worked on. Quit answering complaints with a complaint of your own.
Gripe Mode off. I'm outa here.
Anyone that says otherwise is talking trash.
I'm reading a lot of forum messages discussing how things CAN"T be done, but relatively few on possible ways in which things CAN be done
I have a Intel i7-3820 cpu, 16 GB RAM, Nvidia GTX560 (now a GTX 660), HDD & a SSD. A now dated system but still plenty fast for running Windows 10 and Linux applications.
When I was running Cash's World, the big bottleneck was with the HDD. I switched to the SSD, installed Redis, and the bottleneck moved to the network. I upgraded my network speed and tinkered with Linux's networking code. Except for initial client connections which maxed out my sending speed, that moved the bottleneck to the CPU. The i7-3820 is quad-core. One core was working at 100% running the server.
Even logging in to the server on the same machine was laggy! I started tinkering with the LUA code drastically changing how the mods operate in an attempt to speed things up. (My changes were never completed. Admittedly a butcher job.) That didn't work to speed things up.
With my computer mostly idling except for the server process and occasional Redis writes to the map file, I can only conclude that the Minetest server code is slow as a bug in molasses.
Here's one thought:
1 - Have the server concentrate on what is in the world. Nothing else. It doesn't care how something works or reacts but only that something is there. A chest is only a chest. the server don't care what's in it. Et cetera
2 - Player connections get offloaded to a separate process to log players in and out. Network priority needs to be given to current players so they don't experience lag.
3 - Player movements, machine operations, tree growth, etc are handled by separate processes that inform the server of any needed changes to the map.
4 - Conflicts. Hmm, that's a toughie. I'll leave this for someone else to solve.
5 - Requests for multi-threaded and GPU coding changes go into the TODO file. Maybe someone will step up and work on something. Maybe not haha.
Baby steps. Start small. Have only a map server, one process to log players in and out, and one process to handle everything else. Then players can log in fast and not cause the server to stall while it sends map data and images to the new player. Later, that one big process that handles "everything else" can then be split up as needed.
I haven't played Minetest much in the past 2? years much less studied the server's code. So I don't know what the current state is. I just know reading posts from the developers complaining "We can't do ____ because of ____." is a total cop-out.
The developers have done a marvelous job writing this FREE game for anyone to enjoy. You have my thanks and gratitude for your hard and un-paid work. I agree to not make changes when someone asks for a fundamental change to how the game plays. But complaints about how the game currently runs should be worked on. Quit answering complaints with a complaint of your own.
Gripe Mode off. I'm outa here.
Re: Better support for multicore CPUs
I can't agree with this assessment. My server (Inside the Box) barely uses 3-4% CPU for the first player. I've never seen it above 40% (10 players or so). It runs on a KVM server, so not even real hardware. Only one core, too.843jdc wrote:Minetest is slow.
Anyone that says otherwise is talking trash.
I'm reading a lot of forum messages discussing how things CAN"T be done, but relatively few on possible ways in which things CAN be done
I have a Intel i7-3820 cpu, 16 GB RAM, Nvidia GTX560 (now a GTX 660), HDD & a SSD. A now dated system but still plenty fast for running Windows 10 and Linux applications.
When I was running Cash's World, the big bottleneck was with the HDD. I switched to the SSD, installed Redis, and the bottleneck moved to the network. I upgraded my network speed and tinkered with Linux's networking code. Except for initial client connections which maxed out my sending speed, that moved the bottleneck to the CPU. The i7-3820 is quad-core. One core was working at 100% running the server.
Whatever mods you put on the server are ruining it. The engine, with well written mods, can do much better. I'm not saying it couldn't be better, but I disagree with your assessment that because your server is slow that multicore processing will solve your problem.
because that won't fix bad mods. They will just have more cores to eat all the CPU power from. nom nom nom.
-
- Member
- Posts: 361
- Joined: Tue Jan 19, 2016 16:46
- GitHub: jdc843
- IRC: jdc843
- In-game: 843jdc
- Location: USA
Re: Better support for multicore CPUs
And the rebuttals proving me as talking trash begins haha :) I almost deleted my post before sending it. I decided to wait and see what it invoked.
I got a reply half typed out. I'm stuck on deciding something and am thinking in circles. Not good.
GIST: What one player does should not affect others and vice versa. Define 'quality gameplay' I'm stuck on that one. I think I'll sleep on it. GN :)
I got a reply half typed out. I'm stuck on deciding something and am thinking in circles. Not good.
GIST: What one player does should not affect others and vice versa. Define 'quality gameplay' I'm stuck on that one. I think I'll sleep on it. GN :)
Who is online
Users browsing this forum: No registered users and 19 guests