I got something to work with a recent 5.0dev version by hacking the source code. I'm pretty happy with the results, but it has some major limitations and the overlay mask lags a bit behind the zoom.
This exercise did generate a couple of modding questions.
1) Is there any way for a mod to get the player's current screen resolution or aspect ratio? I need this to put an actually circular window on the screen.
For this hack I used minetest.setting_get("screen_w") and "screen_h", which means the server uses a fixed aspect ratio when it starts, independent of the clients' aspect ratios.
2) The 'Z' key is not available from player:get_player_control(). Is there any other way for a mod to detect when this key is pressed and released?
For this hack I added zoom to player:get_player_control(), by doing the following:
- added KeyType::ZOOM to the keypress_bits data structure in src/game.cpp,
- added player->control.zoom in src/network/serverpackethandler.cpp, and
- added control.zoom to PlayerControl &control in src/script/l_object.cpp.