Memory Leak

[rheo]

Memory Leak

The Blocky Survival server has been having issues with a memory leak since upgrading to Minetest 5.0, and it seems to be getting worse. So far, we have been unable to diagnose the issue, but here is a list of things that it is NOT:

1. It is (probably?) not an issue w/ one of the mods chewing up memory in Lua. Lua memory usage, as reported by collectgarbage('count'), is around 244MB, whereas the whole process is consuming about 20GB. This is after running about 22 hours. The memory usage seems to grow at a more-or-less constant rate.
2. It was not fixed by upgrading from 5.0.1 to a 5.1-dev version, or a later 5.1 dev version. Currently we are running 5.1.0-dev-3a8ac9b0.
3. It was not fixed by updating the auth file from a flat file to SQLite, per https://github.com/minetest/minetest/issues/6783.
4. It was not affected whatsoever by updating our build to use LuaJIT instead of "regular" Lua.

We are soliciting any advice anyone might have in tracking this down and fixing it. I'm particularly interested in hearing from other people who have resolved memory leaks, or who know if similar things have been reported by other people before. I've had little luck searching the forums and Github.

Running e.g. Valgrind is not an option, because this is a popular public server.

I'll include some configuration here:

Output of minetestserver --version:

Code: Select all

BUILD_TYPE=Release
RUN_IN_PLACE=0
USE_GETTEXT=1
USE_SOUND=0
USE_CURL=1
USE_FREETYPE=1
USE_LUAJIT=1
STATIC_SHAREDIR="/usr/local/share/minetest"

From minetest.config:

Code: Select all

...
# player settings
max_users = 50
...
## security settings
disallow_empty_password = true
enable_rollback_recording = true
...
## debug settings
debug_log_level = action
profiler.load = false
...
## map settings
mg_name = valleys
mg_flags = caves, dungeons, light, decorations
...
## performance settings
active_block_range = 1
active_object_send_range_blocks = 4
block_send_optimize_distance = 2
max_block_generate_distance = 3
max_block_send_distance = 5
max_forceloaded_blocks = 128
max_objects_per_block = 256
max_simultaneous_block_sends_server_total = 200
num_emerge_threads = 4
player_transfer_distance = 30
server_side_occlusion_culling = true
server_unload_unused_data_timeout = 29

From world.mt:

Code: Select all

auth_backend = sqlite3
backend = sqlite3
player_backend = sqlite3

A couple of "errors" are reported repeatedly (hundreds or thousands of times) in our logs, though I'm not sure they're relevant.

Code: Select all

ERROR[Server]: suspiciously large amount of objects detected: 274 in (6,1,-36); removing all of them.
WARNING[Server]: ServerEnv: Trying to store id = 0 statically but block (6,3,-36) already contains 256 objects.

Currently installed mods (Note that these might be different forks that you're familiar with):

Code: Select all

3d_armor, 3d_armor_ip, 3d_armor_sfinv, 3d_armor_stand, 3d_armor_ui, ambience, arctic_life, areas, areasprotector, bakedclay, basic_materials, bbq, beds, binoculars, biome_lib, bls, boards, boats, bonemeal, bones, boost_cart, bowlingball, bridger, bucket, building_blocks, butterflies, carts, caverealms, cblocks, chesttools, cloaking, computer, cottages, cozy, creative, currency, default, digilines, digistuff, digiterms, display_api, doors, dungeon_loot, dye, extra, extrahorns, facade, fake_fire, farming, fire, fireflies, flowers, font_api, font_metro, game_commands, give_initial_stuff, glass_stained, goldtools, gravelsieve, hangglider, holidays, homedecor_3d_extras, homedecor_bathroom, homedecor_bedroom, homedecor_books, homedecor_climate_control, homedecor_clocks, homedecor_cobweb, homedecor_common, homedecor_doors_and_gates, homedecor_electrical, homedecor_electronics, homedecor_exterior, homedecor_fences, homedecor_foyer, homedecor_furniture, homedecor_furniture_medieval, homedecor_gastronomy, homedecor_kitchen, homedecor_laundry, homedecor_lighting, homedecor_misc, homedecor_office, homedecor_pictures_and_paintings, homedecor_roofing, homedecor_seating, homedecor_tables, homedecor_trash_cans, homedecor_wardrobe, homedecor_windows_and_treatments, hot_air_balloons, inbox, intllib, invhack, irc, irc2, irc_commands, irc_commands2, itemframes, jails, lavalamp, lazerstrike, led_marquee, letters, lib_chatcmdbuilder, locked_travelnet, locks, lurkcoin, mailbox, map, mapfix, maptools, markers, matrix, memorandum, mesecons, mesecons_alias, mesecons_blinkyplant, mesecons_button, mesecons_commandblock, mesecons_delayer, mesecons_detector, mesecons_doors, mesecons_extrawires, mesecons_fpga, mesecons_gates, mesecons_hydroturbine, mesecons_insulated, mesecons_lamp, mesecons_lightstone, mesecons_luacontroller, mesecons_materials, mesecons_microcontroller, mesecons_movestones, mesecons_mvps, mesecons_noteblock, mesecons_pistons, mesecons_powerplant, mesecons_pressureplates, mesecons_random, mesecons_receiver, mesecons_solarpanel, mesecons_stickyblocks, mesecons_switch, mesecons_torch, mesecons_walllever, mesecons_wires, mob_horse, mobs, mobs_animal, mobs_bat, mobs_birds, mobs_fish, moreblocks, morelights, morelights_extras, morelights_modern, morelights_vintage, moreores, moreplants, moretrees, my_castle_doors, my_cottage_doors, my_default_doors, my_door_wood, my_fancy_doors, my_future_doors, my_garage_door, my_hidden_doors, my_misc_doors, my_old_doors, my_saloon_doors, my_sliding_doors, nether, no_guests, nsspf, ontime_clocks, other_worlds, plasmascreen, plasticbox, player_api, playeranim, pmview, quartz, replacer, rhotator, ropes, sban, scifi_nodes, screwdriver, sethome, sfinv, shields, signs, signs_api, signs_road, skinsdb, smartrenting, smartshop, soundblocks, spawn, spawn_command, sponge, stairs, stamina, steles, streetlamps, streetsconcrete, streetsmod, technic_chests, techpack_stairway, techpack_warehouse, teleport_request, terumet, titanium, tnt, travelnet, tubelib, tubelib2, tubelib_addons1, tubelib_addons2, tubelib_addons3, unified_inventory, unifieddyes, verbana, verification, vessels, walls, waterfalls, wielded_light, wieldview, wool, worldedit, worldedit_commands, worldedit_gui, worldedit_shortcommands, xdecor, xpanes

The server is running Linux, and has multiple other instances of Minetest running on it, none of which have this issue.

More information will be happily supplied upon request, but this is about the only relevant stuff I can think of at the moment.

Thanks for any time or help.

[Krock]

1) What results do you get without any additional mods, and only plain MTG?
2) What was the last good Minetest version? You could try to go back as far as a8311ad (anything earlier is not 5.0.1 compatible).

[BuckarooBanzay]

Are you able to "copy" the minetest server on a test-instance (with the minetest-binary, world and all the mods/data)?
You could use valgrind there if you don't want to use that in "production" :P

Maybe just the minetest-binary and a test-world, maybe the memory-leak can be reproduced with minimal effort...