Lua Threads for Computer Mod

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

I really think we should redo the whole computer environment. All privileged APIs should be loaded into the computer's environment before bootstrap.lua starts, so bootstrap.lua never has access to anything it potentially shouldn't. That way, it's actually completely sandboxed and still will be sandboxed no matter how many bugs I introduce. Should I make a branch and do this in there or what?
http://lua-users.org/wiki/SandBoxes wrote: You should start by trusting nothing and only permit things that you are absolutely sure are safe (i.e. whitelist rather than blacklist approach).
Also if you were wondering about this in dronetest/init.lua:

Code: Select all

local ret = {coroutine.resume(co)}
--	dronetest.log("Computer "..id.." result:"..dump(ret))
Computers won't run for me and I can't figure out why:

Code: Select all

09:53:07: ACTION[ServerThread]: dronetest: STATUS: suspended
09:53:07: ACTION[ServerThread]: dronetest: TYPE: table
09:53:07: ACTION[ServerThread]: dronetest: System #3 has been activated, now 1 systems active.
09:53:07: ACTION[ServerThread]: dronetest: system 3 generated output: System #3 has been activated.
09:53:07: ACTION[ServerThread]: dronetest: STATUS: suspended
09:53:07: ACTION[ServerThread]: dronetest: TYPE: table
09:53:07: ACTION[ServerThread]: dronetest: System #2 has been activated, now 2 systems active.
09:53:07: ACTION[ServerThread]: dronetest: system 2 generated output: System #2 has been activated.
09:53:07: ACTION[ServerThread]: dronetest: System #2's main process has ended! Restarting soon...
09:53:07: ACTION[ServerThread]: dronetest: System #3's main process has ended! Restarting soon...

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

COMMENCE EPIC QUADPOST!


#1
the resume-line in init.lua is what keeps computers running.
that resume executes the systems' coroutines until they yield again (or quit).
each computer has it's own coroutine.

#2
the server-log you posted looks like the systems crash while booting and try to restart then. which except for the crashing part would be correct. it should look more like this:

Code: Select all

16:56:04: ACTION[ServerThread]: dronetest: on_receive_fields received ''
16:56:04: ACTION[ServerThread]: dronetest: STATUS: suspended
16:56:04: ACTION[ServerThread]: dronetest: TYPE: table
16:56:04: ACTION[ServerThread]: dronetest: System #1 has been activated, now 1 systems active.
16:56:04: ACTION[ServerThread]: dronetest: system 1 generated output: System #1 has been activated.
16:56:04: ACTION[ServerThread]: dronetest: system 1 generated output: System #1 is booting!
16:56:04: ACTION[ServerThread]: dronetest: system 1 generated output: Finished booting #1, dropping to shell.
SEND
this probably caused by an error in your bootstrap (as it did not get to the "is booting" message).
however, in such cases errors should be thrown - but i too had some cases which would just fail silently in lua.
take out api you last added to bootstrap to check if error lies there. also try if it's same error if you just use one computer in a clean world (if that works there are ID-problems somewhere)

#3
i think you're misunderstanding the way the environments work now - bootstrap does not have access to anything it shouldn't. also the approach is whitelisting, in dronetest/init.lua userspace_environment gets prepared, which is a clean environment to which safe functions are supposed to be added. there are some unsafe ones in there i added for debugging.
bootstrap runs in that userspace-environment.
it is sys.loadApi that brings in stuff from the unsafe global environment, and it can only be used to load apis - which in turn must be save to be accessed from userspace.
so bootstrap is actually very much sandboxed - like all other userspace code it just has the ability to loadApi other stuff.
i added a few lines to bootstrap to print out its current environment, and it looks like this:

Code: Select all

17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: System #1 is booting!
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: pairs (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: table (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: mod_dir (string)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: fs (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: sys (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: peripheral (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: processInput (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: dump (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: ipairs (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: mod_name (string)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: term (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: print (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: math (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: getId (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: os (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: _G (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: pcall (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: string (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: type (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: count (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: shell (table)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: bootstrap env: xpcall (function)
17:08:17: ACTION[ServerThread]: dronetest: system 1 generated output: Finished booting #1, dropping to shell.
as you can see, this is the whitelisted sandbox environment specified earlier.
if you introduce bugs in apis there will be crashes - its as simple as that - does not matter when you add the apis to userspace.
the fact that there are functions whitelisted to userspace that should not be there (like getfenv) is only for trying out stuff more easily. they can just be removed from dronetest/init.lua userspace_environment definition to have them be gone.
i think that link to the lua-wiki is even in the code somewhere, or at least was.

maybe the name bootstrap is confusing, i also removed the =nil s from bootstrap to make it more obvious this is userspace.
it could also be called init or bios or loader or this-is-the-function-that-runs-first-in-userspace.
it really, really does not have global access.
i don't even really get where you got the idea it might have.

the only way to expose globalspace to userspace are apis and the sys-object.
this can only be breached by using a bad function from an api, that somehow returns its own globalspace.

the system-apis must be accessible to userspace, obviously, but they *must* have access to globalspace to have any use - otherwise it would not be possible to actually interact with the gameworld from userspace in any way.
(you could basically just run 1+1 and not even see the output).
if they do not need access to the minetest.* object and all the other stuff,
they are not system-apis but userspace-apis,
and shall be included from userspace to userspace using your os.loadlib for example.
that is a *very* important distinction.
nothing stops us from having as many userspace-only-apis as we want, but the basic system-apis that need to have access to the global env are simply a must, not by preference, but by cold logic - these simply must be callable from userspace but have access to globalspace.

however, if you don't believe me (which you should seeing the output of that print), branch the project and try a different way.
branch for large changes to test them out without confusing the rest of the team, well, me ;)
also that way we can compare branches to see how functionality actually differs.

in this case we would probably see that there aren't many ways to sandbox a coroutine, and doing it differently would most likely end up doing it the same way but just calling functions differently.
it basically always is this:

local f = function() print("hello from userspace") end
setfenv(f,{print=print}) -- nothing can be done in that env but print stuff
local co = coroutine.create(f)
coroutine.resume(co)

and this is what we do right now too.

excuse that wall-of-text, but i do think this is an important matter.

#4
by the way - to get access to the current userspace from an api's globalspace, you can do getfenv(2) if the apifunction was called from userspace (like in shell.run i think) to get the safe environment. thats usefull to have an api load functions and have them run in userspace, instead of an empty environment for example.

also shadowninja has pushed the fix for issue #1709, and if it works, a lot of stuff can move on.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

ninnghazad wrote: the resume-line in init.lua is what keeps computers running.
that resume executes the systems' coroutines until they yield again (or quit).
each computer has it's own coroutine.
Yeah, I know. I changed it to what I quoted it as, and you put it back. I was just saying I changed it to figure out why computers weren't running and accidentally committed it.

Computers still don't run for me. bootstrap.lua makes it past the compiler, but the first line, a print statement, never gets run.
<rest of your post>
You're right, bootstrap.lua only gets whitelisted functions. However, they have dangerous environments. You should do this to all of the functions:

Code: Select all

function wrapfunc(func, env)
	return setfenv(function(...) return func(...) end, env)
end
That way, computers can't run getfenv(sys.receiveDigilineMessage).minetest and get the global minetest table.
I already mostly did this and will push it soon.

About computers not running, I added this to the beginning of bootstrap.lua:

Code: Select all

print("Booting...")

coroutine.yield()

coroutine.yield()
That didn't even work. However, I tried replacing the loadstring(bootstrap) with function() print("Booting!") end and it worked.

EDIT:

Code: Select all

...
-- Load bootstrap code once
local bootstrap = readFile(mod_dir.."/bootstrap.lua")
dronetest.log("bootstrap:"..dump(bootstrap))
...
That prints 'boostrap:""'.
I made readFile print out error messages and return nil instead of silently fail. Here's the result and culprit:
dronetest: readFile: "/home/<username>/src/minetest/bin/../mods/dronetest/dronetest/bootstrap.lua" is not a valid filename.
I'm now fixing readFile to allow .. but still enforce sandboxing.

UPDATE:
Computers now boot for me!

The root of the problem was that my minetest is installed in my home folder, not /usr/..., so it used a relative path and not an absolute path. That means mod_dir had a .., meaning that all calls to readFile failed!

I replaced is_filename(filename) with is_filename_in_sandbox(filename, sandbox), which allows .. but doesn't allow filename to be outside of sandbox. It knows about the . and .. files. I made readFile and writeFile take another parameter, sandbox, but am still updating stuff to use it. Passing nil for a sandbox means no sandbox.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

That way, computers can't run getfenv(sys.receiveDigilineMessage).minetest and get the global minetest table.
I already mostly did this and will push it soon.
i know and was planning on doing that checking in userspace-version of getfenv (or disable it). the reason for this plan was me assuming that taking away the env from those functions may stop them from working. for example functions that use dronetest.*. but give it a try and we'll see, if your way works we'll just use it.
the other way could be wrapping getfenv and check all envs for a minetest.* object, and just return userspace if found.

but more urgent, your systems not booting:
firstoff it confuses me a little, because i don't have any problems using clean mt, clean dt and clean world.
how the system fails is usually caused by a syntax-error, in this case bootstrap. error catching is not complete yet ;). runtime errors should be caught, but syntax-errors that cause compile-fails aren't in many places.
so it might actually work to comment out or remove all of bootstrap and leave only

Code: Select all

print("HEHO!")
return true
or try a fresh bootstrap from git. i'm commiting quite often, so it might be you pulled some rancid version between commits or so. the syntax-error may be anywhere in the file, so thats usually the case when line-1-print won't do.
I'm now fixing readFile to allow .. but still enforce sandboxing.
sounds good, readFile actually isn't supposed to have this check, it's meant for userspace-filereading, and only as a quick n dirty measure. so it should be used in fs.* os.* and so on sys-apis, there is _makePath but thats all ugly, and not usuable in a release - so fix away there too if that ugliness hinders your stuff.
as you may have noticed, there's lfs included, which should give all the tools to make cross-platform (as in win/linux/android) fs.* os.* and so on for userspace. readFile in main-init is meant as blunt tool to read configs or stuff that ofc should never reach userspace.

i've been at it at multiple spots, mainly drones, userspace-version of coroutines and general stuff. the last version i pushed enabled me to setup a computer, multiple drones, multiple displays and a transceiver. i then used test_drone command on the computer, and have 2 drones moving a bit. pretty cool. computer runs the command and connects to the drones, that traffic is routed through the transceiver, and the stdout from computer is send over computers channel. displays may receive if listening on that channel.
transceiver forwards all traffic to all drones that does not come on it's own channel, that channel is to be used to controll the transceiver later on to restrict sending or so.
drones answer with trick over transceiver to computer.
oh i also just added forceloading for computers and drones.

EDIT:
just read update. that sounds even better. that function also nice for os.* fs.* in userspace.

EDIT:
when stuff works try build like above and try the test_drone with 2 turtles, u may have to change the ids/channels in test_drone

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

Are you using "userspace" to refer to sandboxed or normal code? You mean sandboxed code (code the end user writes), right?

Don't bother to take away the env from any functions. It will lead to nothing but headaches and subtle security flaws. Instead, wrap the function inside another function with a safe env. The sandbox function I wrote but haven't committed yet goes through a table you pass it and does this to all of the functions inside. I also modified your table.copy to have a third parameter, safeenv, that it will use for wrapper functions' environments. It won't wrap functions or mess with their environments if you don't pass it a safeenv.

I've already made a whitelist for setfenv/getfenv and setmetatable/getmetatable, but I've disabled it, via "if false and illegal then return end", because I haven't tested it yet.

This won't happen today, but I plan to make a function that creates a userspace environment for each system, instead of copying one table. This will make system-specific closures possible and not a headache. It will also fix the the sys.id bug.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

Are you using "userspace" to refer to sandboxed or normal code? You mean sandboxed code (code the end user writes), right?
yes, i refer to the environment and coroutine the player writes and executes code in as userspace.
The sandbox function I wrote but haven't committed yet goes through a table you pass it and does this to all of the functions inside.
that sounds proper. however, we must make sure that whatever that function returns does not contain a loophole via a returned function or object. so we need to inspect lua apis we want in userspace for unsafe return values, and wrap these in apis (where we do something with the return-value and return a userspace-version or interpretation of it)
- the other we add to the table-o-userspace-functions and walk through with your function.
coroutine for example needed an api - math.* or string.* may not need one, as they return rather predictable stuff.
This won't happen today, but I plan to make a function that creates a userspace environment for each system, instead of copying one table. This will make system-specific closures possible and not a headache. It will also fix the the sys.id bug.
Ye to that - that is how it is supposed to be, but right now it's more a mess of copies and links. system-specific stuff at that point may be usefull later on, to have different access-permissions per player/system or fractions of systems and so on.

EDIT: in case you didn't notice in the readme-change, theres a texture-bug somewhere, so updating displays leaks a bit of mem everytime - switch displays to empty channel to stop updating if that becomes a problem.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

I think we should separate the computer innards from init.lua into their own file, for manageability. A 1200 line file is too big, and there's no excuse not to split it because the computer innards seem like it would be fairly easy to separate them from node definitions and such. I would make it modular if we were to do that, so it will be easier for us and other developers (by making their mod depend on ours) to do things like attach computers to items or blocks other than the standard computer block or have non-physical computers floating around in lualand doing magical (probably server admin) stuff.

If you don't mind telling me, which timezone do you live in, so we can coordinate not making big changes like splitting the main file in two at the same time and having to deal with huge merges conflicts? I live in GMT-4:00 (EDT).
ninnghazad wrote: yes, i refer to the environment and coroutine the player writes and executes code in as userspace.
OK, good.
that sounds proper. however, we must make sure that whatever that function returns does not contain a loophole via a returned function or object. so we need to inspect lua apis we want in userspace for unsafe return values, and wrap these in apis (where we do something with the return-value and return a userspace-version or interpretation of it)
- the other we add to the table-o-userspace-functions and walk through with your function.
coroutine for example needed an api - math.* or string.* may not need one, as they return rather predictable stuff.
For functions returning functions, we can either have the wrapper sandbox things itself (slower but foolproof) or expect API functions to do it themselves (faster but more dangerous).
Ye to that - that is how it is supposed to be, but right now it's more a mess of copies and links. system-specific stuff at that point may be usefull later on, to have different access-permissions per player/system or fractions of systems and so on.
Should you or I make environments have constructor functions and not just table copies? You seem to be making good progress on peripherals right now, which I haven't even looked at yet, while I'm only really familiar with what goes on inside the actual computer block.

Whoever does this should also split it into two files at the same time.
EDIT: in case you didn't notice in the readme-change, theres a texture-bug somewhere, so updating displays leaks a bit of mem everytime - switch displays to empty channel to stop updating if that becomes a problem.
Wonderful! Do digilines LCDs have this problem too? Does breaking and re-placing a display free the lost memory?

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

I think we should separate the computer innards from init.lua into their own file, for manageability.
Ye, that init.lua has grown quickly. Lets split it up.
which timezone do you live in
GMT +1:00 (CET)
we can either have the wrapper sandbox things itself (slower but foolproof) or expect API functions to do it themselves (faster but more dangerous).
apis in that regard not only have the purpose of safely getting data to userspace, but also to provide it in a form the player may use, or combine several global-calls to form some userspace-information. if it is just about a function to be used more or less directly, theres no need. but if it is about producing complex output, a specific wrapper is needed, what i consider apis for. also we have to remember that being able not load parts of the userspace may be good when number of functions grows, to reduce server-time needed to boot systems. but i would consider those apis that exist in normal lua also to be part of a fixed minimal set.
Whoever does this should also split it into two files at the same time.
Well, you said you had stuff uncommited, so you do it i'd say right after commiting (even if not completely done). As it's weekend however that would have to happen today, so i can code myself silly over the weekend. well if i feel like it. i can split tonight too if you can't find the time.
thought of leaving init.lua with actual init and includes, and then have computer/drone/util and probably more as we go on.
Do digilines LCDs have this problem too? Does breaking and re-placing a display free the lost memory?
Probably yes, but due to their tiny texture, it will not be nearly as noticable. And nope, not even exiting to menu does. Only quitting game. I assume this will piss of other people too, so it will sooner than later be fixed.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

ninnghazad wrote:Ye, that init.lua has grown quickly. Lets split it up.
I think I can do it tonight.
apis in that regard not only have the purpose of safely getting data to userspace, but also to provide it in a form the player may use, or combine several global-calls to form some userspace-information. if it is just about a function to be used more or less directly, theres no need. but if it is about producing complex output, a specific wrapper is needed, what i consider apis for. also we have to remember that being able not load parts of the userspace may be good when number of functions grows, to reduce server-time needed to boot systems. but i would consider those apis that exist in normal lua also to be part of a fixed minimal set.
Maybe we could put a metatable.__index on _G to automatically load apis when they're first needed?
Well, you said you had stuff uncommited, so you do it i'd say right after commiting (even if not completely done). As it's weekend however that would have to happen today, so i can code myself silly over the weekend. well if i feel like it. i can split tonight too if you can't find the time.
thought of leaving init.lua with actual init and includes, and then have computer/drone/util and probably more as we go on.
I've committed everything. If you decide to split init.lua, please tell me as soon as you decide to, so I don't also do it and make a mess. I might be able to do it tonight, so please commit and push all of your changes when you're done working on stuff tonight.
Probably yes, but due to their tiny texture, it will not be nearly as noticable. And nope, not even exiting to menu does. Only quitting game. I assume this will piss of other people too, so it will sooner than later be fixed.
This leak is only on the client, right? Restarting the client will fix it?

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

Maybe we could put a metatable.__index on _G to automatically load apis when they're first needed?
sure, that's a good idea - either way we should add an easy way to list available apis. a command and function maybe. if you add an __index to userspace's _G, just add some viewer-function for the table it uses also. maybe read list of /rom/apis/ on modload and store name=>filename in table or something.
If you decide to split init.lua, please tell me as soon as you decide to, so I don't also do it and make a mess.
no prob, i think i shall do this today, at least for some stuff. but not immediately, have to make food for my woman first. also wanted to add more drone stuff.
This leak is only on the client, right? Restarting the client will fix it?
well, i actually haven't checked as i use singleplayer for testing - but i assume so, cause i see no reason for the server to load textures.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

in case you are wondering why nothing is happening, i'm waiting for https://github.com/minetest/minetest/pull/1737.
and to be true, i don't think anybody actually feels responsible for pull requests from outside, on the contrary - there are >100 pull requests and most of these will be wasted time.
and i'm not bored enough to care, and i don't care enough to beg, so until stuff moves forward imma do something else.

User avatar
Evergreen
Member
Posts: 2135
Joined: Sun Jan 06, 2013 01:22
GitHub: 4Evergreen4
IRC: EvergreenTree
In-game: Evergreen
Location: A forest in the midwest
Contact:

Re: Lua Threads for Computer Mod

by Evergreen » Post

ninnghazad wrote:in case you are wondering why nothing is happening, i'm waiting for https://github.com/minetest/minetest/pull/1737.
and to be true, i don't think anybody actually feels responsible for pull requests from outside, on the contrary - there are >100 pull requests and most of these will be wasted time.
and i'm not bored enough to care, and i don't care enough to beg, so until stuff moves forward imma do something else.
I'll see if I can annoy some of the devs about this once I get home. It seems like an extremely useful feature.
Back from the dead!

User avatar
Neuromancer
Member
Posts: 797
Joined: Tue Jun 12, 2012 22:28
GitHub: Neuromancer56

Re: Lua Threads for Computer Mod

by Neuromancer » Post

ShadowNinja gave the reason on the pull request that this would make the engine too laggy. I commented on the thread that why not then make this an optional feature that could be turned on and off like shaders which is really laggy. I really hope this pull request gets accepted because I really think that this mod and the feature it needs (keypress) would open up really cool functionality to Minetest.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

It didn't make Minecraft laggy (and I have a five year old computer), why would it make Minetest laggy?

I know barely anything about the C internals of Minetest, so I wouldn't be surprised if there is some good reason that this would cause significant lag.


In the meantime, after Nov 1 when I have free time again, I'm going to start reworking the internals of dronetest to be more secure, efficient, and readable, and might possibly even add some peripherals, regardless of what happens with this pull request.

Ninnghazad, in case you're wondering, here's a very incomplete list of plans I have, roughly in priority order:
  • make a function that returns a new sandboxed computer environment, instead of copying a table
  • sandbox stuff more properly - have two tables for each computer - one the computer can wreck as much as it wants, the other that it has no access to (except maybe through setter and getter functions)
  • make sys not have security holes (you might have already fixed this)
  • plug-and-play drivers - standard protocol for a computer to read a string from a peripheral, compile it as a function sandboxed inside another sandbox inside the computer's sandbox so it can't wreck the computer, and use whatever table that function returns after calling it as the method table for the peripheral. This allows anyone to add their own peripherals without having to put stuff in /rom/ or needing any real standards, and also makes protocols between computers trivial.
  • I/O expander peripheral, should have bundled cable support if there are any mods that add bundled cables
  • external floppy drive
  • external hard drive - will only do if you will agree with me that all hard drives should be external and computers should have no internal non-volatile storage except channel ids - there's no internal terminal , why should there be internal anything? It should just be a CPU block.
  • wireless modem peripheral
I made a TODO tag for github's issue tracker a while ago, so we can also abuse the issue tracker as a todo list. We should probably use it, but I don't have time to decide on this or add stuff to it now.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

laggy
i have commented about this on the pull request - it does not cause lag, it causes additional traffic. we don't need to discuss this here - if we want that feature, it has it's price, simple as that. PRs need 2 coredevs approving, we have like 1,5 - so getting another dev's approval is what we need on the matter.

the textures not unloading is a bigger issue performance-wise. i looked into this and it's not a bug, it's a missing feature - textures just do not get unloaded in minetest.
could add something for that, but i'll wait until the other PR is through.

@electrodude
good to see you're still around. and good to see you have plans.
you may have noticed that i'm rather picky when it comes to coding =), but most stuff i would note, you'll come along anyway.

about the external/internal drives: it is not that i am fixed on having internal hdds - i just do not want to over-complicate the mod for anti-features. to the user, the programs/tools that come with each system need to reside somewhere. of course they are not really on each computer. but to the user, the system will have a root-filesystem even without drives, like on a linux box, where the programs are.
maybe it squenches your thirst for modularity if we say that the root-filesystem of a computer is always read-only. we explain that to the user by saying the computer have a ROM-flash-bios. that way we still have it simple (easily accesible root-fs), but if the user wants to write own files and read them, they need a external hdd. which would then be practically every case where a user wants to run a custom program. so that external-hdds would be needed for practical use, but one would not have to bother about it when developing apis or ROM-programs.
thats good.

the pnp-drivers sound pretty much like how it currently is, only in a different wording. call peripheral and in the end get table with functions to use it.

about the sandboxing, that's also how it is now, with a different wording. accessible per-comp environment user can wreck, and global environment that is only accessible through functions (apis).
with a more ... controlled ... creation (your first point) of the wreckable-table, or userspace-environment, i think we will have a solid system. so we can be sure that there are only deep copies/new tables, no references in each userspace-env.

about the wireless-modem, well, the transceiver somewhat is a special wireless-modem for dronetest-peripherals. for a "real" digilines-wireless-modem i'd say thats digiline's job. and we just use it like other digiline-devices.
the transceiver however needs more love - it needs bulletproofing for cases like feedback-loops, multiple transceivers on the same net and and and.

and finally something new:
how about energy-consumption? ideas/wishes about that? i wanted to make computers use energy each coroutine.resume of their main routine. so intense tasks cost more energy (and laggy stuff with that also). computers would just not resume and wait until energy is there. so i don't want computers that crash everytime there's a powerfluke. they should just get slow, or halt. not sure about drones yet, maybe like some akku or something.
i didn't like CC's turtles burning coal, i want them to be fully electric - mass to power conversion needs to be done with other stuff.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

@ninnghazad:
I don't see any reason we can't move forward with the core while we're waiting for our PR. Even though probably no one else would, I would still use this mod even if our PR is never accepted and all code editing had to be done in an external editor.

I'm working on some of the innards now. I'm making a sandboxed environment factory function, instead of using table.copy. I already did env and am now working on env.sys.

I can't understand why this piece of code is so complicated: (sys.lua, line 49)

Code: Select all

dronetest.sys.loadApi = function(name)
	local api = getApi(name, dronetest.sys.sandbox)
	local env_save = table.copy(dronetest.userspace_environment)
	local env_global = getfenv(api)

	-- [[ debug code removed for post brevity ]]

	-- Make drone/computer's env available to APIs, with id and stuff
	--env_save.dronetest = getfenv(0).dronetest
	env_save.sys = getfenv(2).sys
	env_save.print = function(msg) dronetest.print(env_save.sys.id,msg) end
	setfenv(api,env_save)
	return api()
end
What's wrong with something like this?

Code: Select all

sys.loadApi = function(name)
	local apifunc = getApi(name, sys.sandbox)

	--following line now unnecessary, replaced by closure from containing function
	--local env = table.copy(dronetest.userspace_environment)
	
	local apienv = setmetatable({}, {__index=env})

	setfenv(apifunc, apienv)
	
	return apifunc()
end
I <3 metatables.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

@electrodude512
You are right, i guess i just got a little frustrated with not being able to move on my own pace.

That code by no means has to be that ugly, plx fix if you like. Just one of those functions that grew ugly. Do make sure you test out stuff after you change that however, maybe try to run the quarry-demo-prog or so, i guess that uses most stuff we have right now.

If you have stuff written already, please commit changes - i don't mind lots of (partly useless) commit-messages.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

ShadowNinja added loops and functions to luacontrollers. I'll investigate his changes soon. We should either get to work or just write an OS and lots of peripherals (including an external HDD for fs support) for luacontrollers.

User avatar
Jeija
Member
Posts: 686
Joined: Fri Dec 23, 2011 21:46
Location: Nürtingen, Germany

Re: Lua Threads for Computer Mod

by Jeija » Post

Just wanted to tell you that if you want to add HDDs, more sensors, peripherals etc. to digilines, I'm open for it!
Also, if my pull request for minetest (https://github.com/minetest/minetest/pull/1869) gets merged, the luacontroller can have an internet database for code snippets and such and it would even allow for stuff like real internet access via digilines.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

@Jeija:

I'm going to wait to see what ninnghazad thinks before I abandon the project I told him I would help him with. Regardless of that, I have two ideas for features I think luacontrollers should have, whether or not ninnghazad and I decide to use it. I made a fork and will PR it soon, hopefully tonight or tomorrow. I'm adding two features - do you want two separate PRs? I've never done a PR on github before, so we'll see what happens.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

Heho.

@electrodude512
I do this for fun, i don't and don't want to feel in competition about this stuff. i think it's great the luacontroller grows.
however, i also do not think digilines and dronetest are redundant.
digilines is to me more of a lower-level foundation - i think of the luacontroller as a programmable microcontroller,
whereas dronetest is in it's usage more like an old pc.
and i do imagine the existence of both, and the connectivity between both, to be a combination of epic amounts of possibilities and strange contraptions.
i would use the luacontroller to build fast, tight circuits,
and the dronetest-computer to access a bunch of these and controll em with a gui or analyze data.
or, well, to control a drone =).

@Jeija
appreciate the offer.
dronetest is currently requiring digilines, and uses it to connect computers/drones with each other
and the only existing "peripherals" (display, transceiver - a modem to talk to drones).
the only thing different to plain digilines peripherals about these is that they have a tiny protocol to present a safe ingame-api to the player for use in ingame-code based on a simple table of functions in each peripheral.
the player uses a wrapper-table generated from that info to call and use the peripheral over digilines in the ingame-code.

that means that i don't plan to reimplement sensors or other stuff that digilines offers,
but i think i do not get around having some special dronetest-peripherals, like the transceiver, or the display.
these would be accessible through digilines, but would talk protocol.
writing specific wrappers for digilines-peripherals i would generally like to avoid, but to use the luxury of that automatic wrapping i might have to in some cases.

i am not informed about the current development of digilines,
and maybe there is already some minimal protocol in the talks (diginet maybe ;) ?), maybe it's out of scope for digilines.
i do want people using dronetest to be able to hook up to some peripheral and start writing ingame-code using it, without having to visit some api-webpage just to get a list of possible functionality - but i would really like to be able to do that with any digilines-peripheral (without having to build own versions of everything or have like a parallel world of peripherals).

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

@ninnghazad:
I am not giving up on dronetest having its own CPU yet. I'm going to do this PR and see what happens. Both the dronetest CPU and the luacontroller CPU have their own advantages and disadvantages based on how they work internally. However, I don't think the deciding factor for which one to use is that one is best for fast/simple stuff (luacontrollers) and one best for slow/complex stuff (dronetest cpu). They should both be fast and powerful enough to do either.

If we switch to using the luacontroller for our CPU, dronetest won't be obsolete/redundant because all of the peripherals will still be extremely useful.

If I end up not convincing you that we should use the luacontroller CPU for dronetest, I'll most likely stick with the dronetest CPU.

Finally, I think we should develop a standard peripheral protocol on top of digilines that can easily be used by both dronetest and luacontrollers.

@Jeija:
I have semi-persistent states working in mesecons_luacontroller- you can now put anything, including functions, coroutines, self-referential tables, and whatever else you want, in the sandbox's global environment and it will stay until the computer reboots. I think mem still works as usual (i.e. completely persistent but you can only put certain things in it), but I haven't tested it yet.

User avatar
ninnghazad
Member
Posts: 36
Joined: Fri Sep 26, 2014 12:33
GitHub: ninnghazad
IRC: ninnghazad
In-game: ninnghazad

Re: Lua Threads for Computer Mod

by ninnghazad » Post

@electrodude512
heho, i thought about it and i wouldn't want to drop the dronetest cpu/core. although that is in the end just based on the fact that i like writing and thinking about it, and i wouldn't want to have to compromise about some stuff that may or may not come up.

still, yay to a common protocol for digiline-devices to be used by dronetest and others.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

I have committed a draft of the dronetest protocol to github/ninnghazad/dronetest/dronetest_protocol.md. Feel free to make any modifications you like, but if you want to seriously change things, please consider talking to me about it first or doing it on another branch.

I'm committing my changes to the userspace environment on another branch and will merge them back into master once I test them more and fix anything that got broken along the way.

electrodude512
Member
Posts: 39
Joined: Wed Sep 17, 2014 21:34
GitHub: electrodude
IRC: electrodude512
In-game: electrodude512
Location: USA

Re: Lua Threads for Computer Mod

by electrodude512 » Post

I committed my changes to the userspace environment on a new branch "newenv" and fixed a ton of security holes. Your use of getfenv(number) led to all sorts of fun, but I removed it all.

The only reason I didn't commit it to master is because you said not to push stuff without testing drones first, but I don't think we've really reached the point where we should be worrying about drones yet. We should get the basic computer working, with no security flaws and a somewhat stable OS and peripheral interface, and then worry about drones. If you like my changes, feel free to merge them into master.

Can we also get a real event system? This will make things like digiline message receiving, timers, input, etc. much easier, and will make a multitasking kernel practical. My immediate reason for needing a multitasking kernel is to allow registering packet acknowledgement callbacks and packet send retries for my digilines connection protocol. It will also be useful for computers that are supposed to simultaneously have a shell and do stuff over digilines. I can write the kernel - I wrote one for ComputerCraft that had tons of features and worked very well.

We should talk on IRC about stuff soon. If it's not too late today, can you get on #minetest-mods today?

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests