[Mod] Farming Redo [1.48] [farming]

User avatar
TenPlus1
Member
Posts: 3715
Joined: Mon Jul 29, 2013 13:38
In-game: TenPlus1
Contact:

Re: [Mod] Farming Redo [1.12] [farming]

by TenPlus1 » Post

Ethereal mod is already compatible with Farming Redo and will spawn new plants into newly generated areas of map...

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Submitted a pull request for enhancing plant growth to occur even when the map block is unloaded and ABMs aren't running. This works by marking the last time an update was evaluated in the node's metadata, and using a statistical model (Poisson distribution) and day/night light levels to determine how many stages to grow the plant since the last ABM execution.

Should be good for single-player and for large servers where players have to house themselves in remote corners and nobody is around most of the time. They won't have to sit around watching the plants grow, and can either log out (so long as the server is still running; uses game time not wallclock time) or be off doing other interesting things like mining and exploring.

User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: [Mod] Farming Redo [1.14] [farming]

by Kilarin » Post

prestidigitator wrote: This works by marking the last time an update was evaluated in the node's metadata, and using a statistical model (Poisson distribution) and day/night light levels to determine how many stages to grow the plant since the last ABM execution.
Oooo! I've been thinking about doing something like this. I really hope it gets pulled!

User avatar
TenPlus1
Member
Posts: 3715
Joined: Mon Jul 29, 2013 13:38
In-game: TenPlus1
Contact:

Re: [Mod] Farming Redo [1.14] [farming]

by TenPlus1 » Post

Prestidigitator, thank you so much for the routine to grow plants while the player is out and about doing other things, normally it would be a great idea and I'd include it in farming redo but I have a few problems that bother me... First is that players farming indoors or underground wouldn't have their plants grow until it's daylight outside, and secondly it seems to lag my test server quite a bit when you have a decent sized farm (and times that by how many players are on the server)... So a big thank you for coding this feature but I will not be including it YET until it's viable for most players and servers...

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

TenPlus1 wrote:...players farming indoors or underground wouldn't have their plants grow until it's daylight outside...
Not quite true. What the logic does is test:
  1. Whether there is a day and night cycle at all (whether or not time is frozen by setting time_speed to 0). If there isn't, it just uses the current light level as returned by minetest.get_node_light(pos) and, if this falls in the min, max light range, the total elapsed time since the last run.
  2. The light level at midnight as reported by minetest.get_node_light(pos, 0) and at noon as reported by minetest.get_node_light(0.5). These DO factor in the light generated by other nodes (e.g. torches and stuff), and then add the light the sun would contribute at those times (if any). I tested this pretty thoroughly using the luacmd mod.
  3. Whether the nighttime (midnight) light falls within the min, max range given, and whether the daytime (noon) light does. If both fall within the given limits, it just uses the straight elapsed time. If neither do, there's no growth at all. If one or the other does, it calculates the amount of night or day that has elapsed since the last run, as appropriate.
The only difference between that and true dynamic lighting measured by minetest.get_node_light(pos) every time an ABM runs is in the last hour or so before sunup and the first hour or so after sundown. Otherwise it's the same, whether or not you are in a cave or exposed to the sky or have torches or whatever.
TenPlus1 wrote:it seems to lag my test server quite a bit when you have a decent sized farm (and times that by how many players are on the server)...
Hmm. Okay. I'll do a bit of performance testing and see about optimizing and enhancing a bit. It might work just to decrease the frequency of the ABM some more, since truly it doesn't matter how much it runs (except that plant nodes will only actually be replaced when it does run).

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Okay, so I found a defect, which accessed a nil value and potentially even caused crashes. It's now been fixed.

I also added profiling code. All you have to do is uncomment one line of code and it will periodically report timing statistics for the ABM (every 400 runs or the run that occurs at least 100 seconds after the previous report). I also tested by using this same profiling logic in the existing ABM logic on the master branch.

I used this in a 16x16 (256 node) field of wheat under various conditions (the whole lifecycle during daylight, night time with just planted, partly grown, and mostly grown plants, and daytime after leaving and returning back to the area after a long enough time to cause full or almost full growth). The results I got were interesting. The new (poisson-based) code is pretty steady under almost all conditions (average 23µs per ABM, with a standard deviation of 3µs). The standard code is indeed faster (average 10µs, with a standard deviation of 1.6µs) at night, under all nighttime conditions. During the day, throughout the normal growth cycle of the plants, the standard code is about the same (average 26µs with a standard deviation of 9µs) as the poisson-based code.

The interesting case is when returning to an unloaded region of the map. At that point the new poisson-based code slows down a bit (average 40µs with a standard deviation of 11µs) as it catches up, but then it basically quits since all the plants have matured to where they don't run ABMs anymore. So it might cause a little lag generated as players enter regions that have a lot of plants that were growing and are now mature, and then it'll cause less lag than the existing ABM (since the existing one will still be running to mature the plants).

In all, it seems my original guess of running the ABM on 2/3 as many nodes seems (coincidentally) about right. It'll be reasonably close at night, actually save time during the day, and only produce a small slowdown for a brief moment when visiting a region that had a lot of pending growth. I also believe I can optimize the calculation of the poisson distribution for exactly that condition of many pending updates, since when lambda is large (say, greater than 2) it's probably reasonable to round it to the nearest 0.5 or 1. That should lend itself nicely to a table-based lookup. I'll give that a try and see how much it helps the slow case.

EDIT: Wanted to add that I'm NOT running a build with LuaJIT enabled, but I suspect that LuaJIT would make the poisson calculation faster, if anything....

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

So I find this interesting. I optimized the heck out of the Poisson distribution calculation, and found that it didn't really change anything in terms of how stable the ABM execution time was. What's more, the circumstances under which it was taking a long time didn't quite make sense either (e.g. it often took longer the more mature the plants were, and calculation of the Poisson distribution should take LESS time in that case since the maximum number of potential growth stages is less).

So I did some more detailed profiling, and it looks like it is the time it takes to access a node's metadata that can vary like crazy. For example, the ABM execution time can go from 20µs to 45µs while the Poisson calculation stays under 2µs and the entire growth stages calculation remains under 5µs. That's pretty bad news for using metadata in ABMs in general, unfortunately, so I guess it had better only be done for nodes that are relatively rare (since I find it unlikely the engine will be optimized in this regard). I'm pondering a better in-memory caching approach that might, for example, use metadata ONCE to allow elapsed-time updates to a whole area (e.g. a 80x80x80 map block).

User avatar
TenPlus1
Member
Posts: 3715
Joined: Mon Jul 29, 2013 13:38
In-game: TenPlus1
Contact:

Re: [Mod] Farming Redo [1.14] [farming]

by TenPlus1 » Post

That's something I found also, reading metadata and overall object information can slow down mods quite substancially, even checking if an available mod is installed causes slowdown... I try to cater for this when optimizing all of my mods...

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

So I've done some research and experimentation, and I think plant growth might actually be much better handled by node timers than ABMs! This might still require some simple analysis of how much day/night has passed, but it actually means you could just pick a random lifespan (or length of growth stage) and then calculate the number of stages very easily based on the number of timer periods that have elapsed, and grow plants at exactly the right times! This is even persisted automatically when the map block is stored, and even large numbers of timers are handled quite efficiently in the C++ engine code. It also means no slow or clumsy metadata fields; the engine deals with timer values "natively." Forget the Poisson distribution; I'm thinking a normal distribution (bell curve) for lifespan, calculated when the node is first created, and then a simple timer! (EDIT: Actually, I may have to take that back slightly for the very stupid reason that there seems to be no access to the actual timeout value from a timer callback; ONLY the elapsed time! Need an engine pull request for that one, I think; it's about a 5-line change, not counting documentation.)

The only downside is that a timer might have to be added to the node definition(s), rather than just a group. Can possibly do this automatically with an override though. (In fact, a rare ABM might still be useful just to make absolutely sure that "growing" node types are properly detected.)

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Well, I'm coming to the conclusion that it's just impossible to profile things in Minetest. I'm actually computing less in the timer than was being done in the ABM (e.g. using a table to cache stages instead of parsing the node name each time), and it's 50% slower (the actual code being executed within the callback, that is). Every once in a while the amount of time just soars through the roof and doesn't come down again, even with an incredibly consistent test scenario. I'll think I've optimized enough because I'll get a streak of good times, but then suddenly it'll just crawl again and not recover.

Short story: timers behave exactly like metadata access did, though that makes absolutely no sense. Maybe the best option is just to provide a simple boolean configuration option, which uses the simple behavior by default. If a little more execution time every now and then is acceptable for a given server, it might be worth the extra functionality. If a server is already dragging its feet, it might not be worth it.

User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: [Mod] Farming Redo [1.14] [farming]

by Kilarin » Post

Possible suggestion:
Getting a nodes meta data is slow. SO, when you enter a new region, all of the nodes grabbing meta data at once creates lag.
WHAT IF, each time the ABM runs, it checks random number, and if that number is, say, 1 in 5 (or whatever works out in testing) THEN it grabs the meta data to see if its been out for a long time and updates for missing growth. If not, updates the last time field in the meta data.

That way, when the player walks into a new area, you wont' have every single plant pulling meta data at once, but, after a few minutes, they should have all discovered that they are overdue and updated their growth?

Nubm
Member
Posts: 20
Joined: Thu Nov 20, 2014 09:33

Re: [Mod] Farming Redo [1.14] [farming]

by Nubm » Post

I'd love to see a mod pack of farming and mobs redo with a diffculty setting. :p
IMO the plants give so much food, they should grow much slower. My small fields support me forever and an army of pets as well. Great mod nonetheless!

Btw, what did you use for the signs in the image? Are they just textures or is it a mod to put in items (i think MC has those)?

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Kilarin wrote:Possible suggestion:
Getting a nodes meta data is slow. SO, when you enter a new region, all of the nodes grabbing meta data at once creates lag.
WHAT IF, each time the ABM runs, it checks random number, and if that number is, say, 1 in 5 (or whatever works out in testing) THEN it grabs the meta data to see if its been out for a long time and updates for missing growth. If not, updates the last time field in the meta data.

That way, when the player walks into a new area, you wont' have every single plant pulling meta data at once, but, after a few minutes, they should have all discovered that they are overdue and updated their growth?
Well, that's just it. I'm not using metadata anymore. I'm using node timers, which tell us how much time has elapsed since the time they were set (or last executed if repeated). And the logic they run is very, very close to the logic that the original code did in the ABM. However, that same logic takes on average 50% longer (per execution) to execute for some reason, and—the bigger problem—is erratic, unlike the original ABM. And the amount of time it takes per run of the timer shows about the same characteristics as when I used the method of accessing metadata within the ABM. Same slowdown; same lack of predictability.

Yes, the original idea was that if the use of metadata was slower, it could simply be done less often. However, the slowdown doesn't seem to be bounded. If a normal ABM took a steady 20µs per run and the metadata access resulted in a steady 30µs per run, I'd just say run it 2/3 as often and everything's good. But what happens is, for example, the old code takes 20µs per run and the new code takes 30µs per run sometimes, and 70µs to 100µs per run at other times, without any good explanation as to why (e.g. if I profile the Poisson calculation it takes a steady 5µs, so the extra 30µs isn't coming from there....). I thought the metadata access was a good "why." Turns out not. Heck, I thought the fact that I had to call minetest.get_node() twice from a timer because it isn't passed the node table as a parameter like an ABM action is was a good "why." But then I used a closure to store the node name so I wouldn't have to call minetest.get_node() again just to get the name, and it didn't solve anything. If the "why" would just show up, I'd be happy; I could design a solution to avoid it. But the damned thing keeps playing hide-and-seek.

ronnietucker
New member
Posts: 7
Joined: Mon May 18, 2015 20:37

Re: [Mod] Farming Redo [1.14] [farming]

by ronnietucker » Post

Is it possible to have this mod spawn items on a map created before the mod was installed?

Thanks!

User avatar
Don
Member
Posts: 1643
Joined: Sat May 17, 2014 18:40
GitHub: DonBatman
IRC: Batman
In-game: Batman

Re: [Mod] Farming Redo [1.14] [farming]

by Don » Post

If its a map chunk that you want to regent do
/deleteblocks here.

This will regenerate the chunk and plants should grow.
Do not do it in areas that have thing built.
Many of my mods are now a part of Minetest-mods. A place where you know they are maintained!

A list of my mods can be found here

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

(Oops. Nevermind. For some reason Don's post showed at first, but the post he was responding to didn't.)

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Okay, so I have a partial and slightly more satisfying answer. I did some very careful testing, and to do so I separated the logic of the update code from the final call to minetest.set_node(), and I did this for both the new timer method and the standard ABM. I decided to test both throughout the lifetime of my 16x16 test wheat field (even though theoretically the standard ABM should perform the same throughout), and test while being present throughout the entire lifetime or leaving for the first few stages of it. Here is what I found:
  • Both sets of logic perform about the same. The standard logic and the timer-based logic typically execute in the approximate range 8-12µs per call.
  • The standard ABM call to minetest.set_node() takes about 10µs in the beginning.
  • The timer-based call to minetest.set_node() takes about 20µs in the beginning. Switching to minetest.swap_node() which doesn't reset metadata or check for construct/destruct callbacks reduces this by about 5µs (didn't try this with the standard ABM).
  • Leaving for part of the lifecycle and coming back does not significantly affect either implementation. In fact, the timer-based logic tends to run at its maximum performance of about 7µs per call when you return (it does execute more frequently for a short period though, as the timers of all the nodes catch up; this can then be made up for later because there are that many fewer growth stages left to do). It also doesn't seem to adversely affect minetest.set_node(); the impact on that is somewhat independent of player presence or absence, and is instead affected by...
  • The performance of all code, but of minetest.set_node() in particular, drops over the lifetime of the plants. This is true for both the standard ABM and for the timer-based implementation. When most plants are fully grown (say, 1400+ growth stages out of 1792 for my 16x16 test wheat field), the standard ABM can easily take 40µs or more per call including both the logic and minetest.set_node(), and the timer-based implementation can easily take 50µs or more.
Why the minetest.set_node() call take different times from the ABM vs. the timer, why it slows down over time/number of calls or the reduction of the frequency of calls due to fewer plants still growing, and whether it's the active time of the map node or the number of node replacements or what, are still mysteries. But these results are promising for the viability of updates while absent, at least compared to simple ABM updates, even though they are not very flattering to the game logic of the Minetest engine.

I may go back and use the same testing methodology to characterize the metadata-based ABM, now that I understand at least roughly what is causing the slowdown, and that it is not necessarily related to the plant update logic.

User avatar
Kilarin
Member
Posts: 894
Joined: Mon Mar 10, 2014 00:36
GitHub: Kilarin

Re: [Mod] Farming Redo [1.14] [farming]

by Kilarin » Post

prestidigitator wrote:Why the minetest.set_node() call take different times from the ABM vs. the timer, why it slows down over time/number of calls or the reduction of the frequency of calls due to fewer plants still growing, and whether it's the active time of the map node or the number of node replacements or what, are still mysteries.
Does this need to be opened as a bug report for minetest?

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

Kilarin wrote:Does this need to be opened as a bug report for minetest?
Don't know. I suppose it could be due to some kind of memory leak or something, which would qualify as a bug. But it could also be an inefficient datastructure or some property of the Lua interpreter. While it's possible performance could be enhanced if that's the case, it probably doesn't really count as a bug. Still, I suppose it wouldn't hurt to try, if someone wants to report it. It's possible a dev might feel like investigating the behavior....

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.14] [farming]

by prestidigitator » Post

I've committed the timer-based implementation and done a pretty good amount of testing. I believe it might be ready to pull/integrate. TenPlus1, if you could take another look at it and possibly test again on your server, I'd appreciate it. Below is a description of the final test I ran to compare this to the existing code.

Test Procedure
  1. Setup: Clear the test map block, its immediate neighbors, and all blocks in a vertical column from those upward of plants. Create a 16x16 floor of wet dirt within the test block. Remove all "artificial" lighting so only the sun affects plants in the area. In init.lua, turn on (uncomment) "farming.DEBUG = {}".
  2. Test Start: Set the time of day to the current trial's "start time" using the "/time" command. Instantaneously clear and then set all nodes above the wet dirt floor to "farming:wheat_1" (I did this using the luacmd mod). Also clear statistics using farming.DEBUG.reset_times().
  3. Wait for the current trial's listed "present" percent of a plant's average lifespan.
  4. Using fast move, fly straight up until at least two map blocks higher than the plants (first time, confirm you are high enough by verifying that farming.DEBUG.report_times() doesn't change).
  5. Wait until twice a plant's average lifespan has passed since the start of the test. (Almost) all growth should theoretically be complete at this point (for the timer-based solution).
  6. Test End: Descend to the level of the plants, and repeatedly call farming.DEBUG.report_times() until it doesn't change over the period of at least 3 average growth stage lengths. This indicates the plants are done growing, the test is complete, and the reported times are correct for the trial.
I also set the average growth stage length to 20 seconds for this test (by setting the ABM interval to 10 for the ABM-based solution and STAGE_LENGTH_AVG to 20 for the timer-based solution). This fit the maximum lifespan of the plants into a single daylight period and made the length of each trial tolerable (around 4-6 minutes).

Data

Code: Select all

------------------------------------------------------------------------------------------------ 
|                      ||           Standard ABM           ||            Timers                | 
| Present | Start Time || Time (ms) | Runs | Time/run (µs) || Time (ms) | Runs | Time/run (µs) | 
+---------+------------++-----------+------+---------------++-----------+------+---------------+ 
|  100%   |   10000    ||   43.9    | 1792 |     24.5      ||   64.5    | 1792 |     36.0      | 
|   75%   |   10000    ||   45.0    | 1792 |     25.1      ||   40.1    | 1407 |     28.5      | 
|   50%   |   10000    ||   44.7    | 1792 |     24.9      ||   30.8    | 1023 |     30.1      | 
|   25%   |   10000    ||   46.3    | 1792 |     25.8      ||   17.9    |  570 |     31.4      | 
|    0%   |   10000    ||   47.2    | 1792 |     26.3      ||    9.6    |  264 |     36.3      | 
|  100%   |    4000    ||   53.1    | 2950 |     18.0      ||   64.0    | 2922 |     21.9      | 
|   75%   |    4000    ||   54.4    | 3091 |     17.6      ||   28.4    | 1596 |     17.8      | 
|   50%   |    4000    ||   47.9    | 2676 |     17.9      ||   12.9    |  977 |     13.2      | 
|   25%   |    4000    ||   47.0    | 2293 |     20.5      ||    9.4    |  580 |     16.2      | 
|    0%   |    4000    ||   43.1    | 1792 |     24.1      ||    8.6    |  256 |     33.6      | 
------------------------------------------------------------------------------------------------
  • Present: The percent of average plant lifetime the player remains present until departing the area.
  • Start Time: The time-of-day set using the "/time" command at the start of the test.
  • Standard ABM: Data collected while running the standard, existing Farming Redo ABM-based updates (from commit e1df7aa).
  • Timers: Data collected while running the new, timer-based updates.
  • Time (ms): Total execution time of the (ABM or timer) update method call, added across all runs from test start to test end.
  • Runs: The number of times the (ABM or timer) update method was called, from test start to test end.
  • Time/run (µs): The time one (ABM or timer) update method took to execute, averaged over all runs from test start to test end.
Analysis/Conclusion

The timer-based solution is about 50% slower than the ABM-based solution while plants are growing (in sufficient light) and while the player is present. For 256 growing plants this comes to about 20 extra milliseconds of total server time spread across their entire lifespans (at least on the machine I used to test it). For all other test scenarios, the timer-based solution is either roughly equal to or faster when considering server execution time as a resource.

The timer-based solution also allows plants to grow while the player is away from the area. In fact, it's more efficient that way. This should encourage players to engage in other activities while the plants grow.

Finally, the timer-based solution spreads growth out in a continuous pattern. Instead of half of the plants in an area suddenly all advancing in unison, a plant or two will grow by a single stage every once in a while. This not only provides a more "organic" feel to watching the grass grow, but spreads the execution of the update method out so it frequently takes a small amount of time instead of infrequently taking lots of time. This should, in general, be good for server lag. There is one case when all timers will execute at once in a batch: when a map block is activated and timers have to catch up to the amount of time that has elapsed (ABMs seem to do a similar thing, ignoring their normal statistical chance of execution). If the elapsed time includes periods of sufficient light for plant growth, the implementation should make up for this extra time with fewer growth stages later.

ronnietucker
New member
Posts: 7
Joined: Mon May 18, 2015 20:37

Re: [Mod] Farming Redo [1.14] [farming]

by ronnietucker » Post

Don wrote:If its a map chunk that you want to regent do
/deleteblocks here.

This will regenerate the chunk and plants should grow.
Do not do it in areas that have thing built.
Thanks for the reply.

Roughly how far from built stuff should I be before doing the /deleteblocks ?

I've only really built in one area, but I'd like to keep that area if possible.

User avatar
Don
Member
Posts: 1643
Joined: Sat May 17, 2014 18:40
GitHub: DonBatman
IRC: Batman
In-game: Batman

Re: [Mod] Farming Redo [1.14] [farming]

by Don » Post

I am pretty sure a map chunk is 80 nodes each way. If you test it in an area far enough away from buildings then you should be able to see.
Many of my mods are now a part of Minetest-mods. A place where you know they are maintained!

A list of my mods can be found here

ronnietucker
New member
Posts: 7
Joined: Mon May 18, 2015 20:37

Re: [Mod] Farming Redo [1.14] [farming]

by ronnietucker » Post

Don wrote:I am pretty sure a map chunk is 80 nodes each way. If you test it in an area far enough away from buildings then you should be able to see.
Turns out you can choose the area by doing something like:

Code: Select all

/deleteblocks (-83,15,-51) (-150,14,-113)
Thanks!

User avatar
TenPlus1
Member
Posts: 3715
Joined: Mon Jul 29, 2013 13:38
In-game: TenPlus1
Contact:

Re: [Mod] Farming Redo [1.20] [farming]

by TenPlus1 » Post

A brand new farming routine has been added with the hard work from prestidigitator that allows crops to grow even when the player is far away from his/her farm...
Last edited by TenPlus1 on Wed Mar 09, 2016 13:22, edited 1 time in total.

prestidigitator
Member
Posts: 647
Joined: Thu Feb 21, 2013 23:54

Re: [Mod] Farming Redo [1.20] [farming]

by prestidigitator » Post

TenPlus1 wrote:A brand new farming routine has been added with the hard word from prestidigitator which allows crops to grow even when the player is far away from his/her farm...
Cool. It's a great mod. Thanks for giving me the opportunity to contribute.

Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 17 guests