Memory Leak

rheo
Member
 
Posts: 16
Joined: Fri May 03, 2019 20:40
GitHub: fluxionary
IRC: flux fluxflux
In-game: flux rheo

Memory Leak

by rheo » Sat Jul 20, 2019 01:59

Memory Leak

The Blocky Survival server has been having issues with a memory leak since upgrading to Minetest 5.0, and it seems to be getting worse. So far, we have been unable to diagnose the issue, but here is a list of things that it is NOT:

  1. It is (probably?) not an issue w/ one of the mods chewing up memory in Lua. Lua memory usage, as reported by collectgarbage('count'), is around 244MB, whereas the whole process is consuming about 20GB. This is after running about 22 hours. The memory usage seems to grow at a more-or-less constant rate.
  2. It was not fixed by upgrading from 5.0.1 to a 5.1-dev version, or a later 5.1 dev version. Currently we are running 5.1.0-dev-3a8ac9b0.
  3. It was not fixed by updating the auth file from a flat file to SQLite, per https://github.com/minetest/minetest/issues/6783.
  4. It was not affected whatsoever by updating our build to use LuaJIT instead of "regular" Lua.

We are soliciting any advice anyone might have in tracking this down and fixing it. I'm particularly interested in hearing from other people who have resolved memory leaks, or who know if similar things have been reported by other people before. I've had little luck searching the forums and Github.

Running e.g. Valgrind is not an option, because this is a popular public server.

I'll include some configuration here:

Output of minetestserver --version:
Code: Select all
BUILD_TYPE=Release
RUN_IN_PLACE=0
USE_GETTEXT=1
USE_SOUND=0
USE_CURL=1
USE_FREETYPE=1
USE_LUAJIT=1
STATIC_SHAREDIR="/usr/local/share/minetest"


From minetest.config:
Code: Select all
...
# player settings
max_users = 50
...
## security settings
disallow_empty_password = true
enable_rollback_recording = true
...
## debug settings
debug_log_level = action
profiler.load = false
...
## map settings
mg_name = valleys
mg_flags = caves, dungeons, light, decorations
...
## performance settings
active_block_range = 1
active_object_send_range_blocks = 4
block_send_optimize_distance = 2
max_block_generate_distance = 3
max_block_send_distance = 5
max_forceloaded_blocks = 128
max_objects_per_block = 256
max_simultaneous_block_sends_server_total = 200
num_emerge_threads = 4
player_transfer_distance = 30
server_side_occlusion_culling = true
server_unload_unused_data_timeout = 29


From world.mt:
Code: Select all
auth_backend = sqlite3
backend = sqlite3
player_backend = sqlite3


A couple of "errors" are reported repeatedly (hundreds or thousands of times) in our logs, though I'm not sure they're relevant.
Code: Select all
ERROR[Server]: suspiciously large amount of objects detected: 274 in (6,1,-36); removing all of them.
WARNING[Server]: ServerEnv: Trying to store id = 0 statically but block (6,3,-36) already contains 256 objects.


Currently installed mods (Note that these might be different forks that you're familiar with):
Code: Select all
3d_armor, 3d_armor_ip, 3d_armor_sfinv, 3d_armor_stand, 3d_armor_ui, ambience, arctic_life, areas, areasprotector, bakedclay, basic_materials, bbq, beds, binoculars, biome_lib, bls, boards, boats, bonemeal, bones, boost_cart, bowlingball, bridger, bucket, building_blocks, butterflies, carts, caverealms, cblocks, chesttools, cloaking, computer, cottages, cozy, creative, currency, default, digilines, digistuff, digiterms, display_api, doors, dungeon_loot, dye, extra, extrahorns, facade, fake_fire, farming, fire, fireflies, flowers, font_api, font_metro, game_commands, give_initial_stuff, glass_stained, goldtools, gravelsieve, hangglider, holidays, homedecor_3d_extras, homedecor_bathroom, homedecor_bedroom, homedecor_books, homedecor_climate_control, homedecor_clocks, homedecor_cobweb, homedecor_common, homedecor_doors_and_gates, homedecor_electrical, homedecor_electronics, homedecor_exterior, homedecor_fences, homedecor_foyer, homedecor_furniture, homedecor_furniture_medieval, homedecor_gastronomy, homedecor_kitchen, homedecor_laundry, homedecor_lighting, homedecor_misc, homedecor_office, homedecor_pictures_and_paintings, homedecor_roofing, homedecor_seating, homedecor_tables, homedecor_trash_cans, homedecor_wardrobe, homedecor_windows_and_treatments, hot_air_balloons, inbox, intllib, invhack, irc, irc2, irc_commands, irc_commands2, itemframes, jails, lavalamp, lazerstrike, led_marquee, letters, lib_chatcmdbuilder, locked_travelnet, locks, lurkcoin, mailbox, map, mapfix, maptools, markers, matrix, memorandum, mesecons, mesecons_alias, mesecons_blinkyplant, mesecons_button, mesecons_commandblock, mesecons_delayer, mesecons_detector, mesecons_doors, mesecons_extrawires, mesecons_fpga, mesecons_gates, mesecons_hydroturbine, mesecons_insulated, mesecons_lamp, mesecons_lightstone, mesecons_luacontroller, mesecons_materials, mesecons_microcontroller, mesecons_movestones, mesecons_mvps, mesecons_noteblock, mesecons_pistons, mesecons_powerplant, mesecons_pressureplates, mesecons_random, mesecons_receiver, mesecons_solarpanel, mesecons_stickyblocks, mesecons_switch, mesecons_torch, mesecons_walllever, mesecons_wires, mob_horse, mobs, mobs_animal, mobs_bat, mobs_birds, mobs_fish, moreblocks, morelights, morelights_extras, morelights_modern, morelights_vintage, moreores, moreplants, moretrees, my_castle_doors, my_cottage_doors, my_default_doors, my_door_wood, my_fancy_doors, my_future_doors, my_garage_door, my_hidden_doors, my_misc_doors, my_old_doors, my_saloon_doors, my_sliding_doors, nether, no_guests, nsspf, ontime_clocks, other_worlds, plasmascreen, plasticbox, player_api, playeranim, pmview, quartz, replacer, rhotator, ropes, sban, scifi_nodes, screwdriver, sethome, sfinv, shields, signs, signs_api, signs_road, skinsdb, smartrenting, smartshop, soundblocks, spawn, spawn_command, sponge, stairs, stamina, steles, streetlamps, streetsconcrete, streetsmod, technic_chests, techpack_stairway, techpack_warehouse, teleport_request, terumet, titanium, tnt, travelnet, tubelib, tubelib2, tubelib_addons1, tubelib_addons2, tubelib_addons3, unified_inventory, unifieddyes, verbana, verification, vessels, walls, waterfalls, wielded_light, wieldview, wool, worldedit, worldedit_commands, worldedit_gui, worldedit_shortcommands, xdecor, xpanes


The server is running Linux, and has multiple other instances of Minetest running on it, none of which have this issue.

More information will be happily supplied upon request, but this is about the only relevant stuff I can think of at the moment.

Thanks for any time or help.
 

User avatar
Krock
Developer
 
Posts: 4412
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: Memory Leak

by Krock » Sat Jul 20, 2019 12:49

1) What results do you get without any additional mods, and only plain MTG?
2) What was the last good Minetest version? You could try to go back as far as a8311ad (anything earlier is not 5.0.1 compatible).
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

BuckarooBanzay
Member
 
Posts: 321
Joined: Tue Apr 24, 2018 05:58
Location: Switzerland
GitHub: thomasrudin-mt
In-game: BuckarooBanzai

Re: Memory Leak

by BuckarooBanzay » Sat Jul 20, 2019 16:54

Are you able to "copy" the minetest server on a test-instance (with the minetest-binary, world and all the mods/data)?
You could use valgrind there if you don't want to use that in "production" :P

Maybe just the minetest-binary and a test-world, maybe the memory-leak can be reproduced with minimal effort...
 


Return to Problems



Who is online

Users browsing this forum: MSN [Bot] and 2 guests