Lua Threads for Computer Mod

[aldobr]

I can help you make char based input/output. Its tedious, its slow, but its possible. But using a virtual keyboard laid out on the formspec itself. Something like this :

Is something like that usefull ?



Text would be input like some old cellphones where each number commands a group of digits/chars. Pressing the same button repeatedly changes across groups.

Its ugly, but it should work.

It dont see the point of having a text editor. You can always edit text outside minetest and feed it via chat commands.

Suppose you want to upload a large program into the computer. You base64 encode it on the pc computer, copy the text, paste it using a chat commad, like

/uploadfile <filename> <base64encodedfile>

and then the file becomes accessible on the computercraft computer.

[ninnghazad]

@aldobr
i appreciate the effort, but it would not solve that problem. which i might have failed to explain properly:
by charbased input i mean the mode of input in the program. there's charbased and there's linebased, which differ in when data is sent.
charbased means that every keystroke is immediatly sent.
linebased means that when ie enter is pressed whatever was pressed upto then gets sent.
the formspec's inputfield use the linebased approach, which is sensible for the type of ui-element they are.
it just won't work with what i want. and it's not just about the texteditor, thats just a good example.
there would be no autocomplete possible, no interactive games, no-use-arrow-keys-for-whatever and on and on...
got the idea of patching minetest to have a field type that works as a proxy for charbased input,
but for now there are more pressing issues. (about which i have posted on the bugs-forum)
ideally i want just one big black field which works like normal terminals do, i can place the cursor where i want,
switch input-modes, redraw parts of the screen and so on. plus a few buttons for power on/off maybe.

apart from that i think we may have derailed this thread somewhat. maybe i should make a new post,
but until i can clearly see if this is possible to get to a usable state (usable on a public server) i don't quite feel like it.

the threads-issue for me seems to be resolved for now, i did not want external threads,
and now it seems i can get away without any extra one at all just by using coroutines - at least if that other issue gets resolved.

ps: but like the "paste base64 to file"-idea in general

[Neuromancer]

btw: of course i wanted to call it computertest... hmpf, well its called dronetest now ;)

Why don't you call it ComputerTest? I remember that I was going to use that name too, but you all are much farther along than I would ever get with this Mod. Or do you have some other reason for not calling it ComputerTest?

[ninnghazad]

@neuromancer
well, i did not want to be rude. having written huge scripts for computercraft,
i thought about it and thought that probably anybody would call a mod like this computertest,
so i searched for it, came to some posts of yours and a github (i think) -
and so thats basically where i started, thinking maybe it can be done in pure lua,
or at least mostly lua. thanks for the offer however, maybe i'll rename it. lets see how goes first.
reading through this thread again reminded me that you where thinking about char/keystroke-based input too.
maybe you (and aldobr maybe also?) would like to join forces,
you could just keep on thinking about the matter right away =).
while the current problems with calling some minetest.* functions through coroutines stalls the development of the drone's functions,
the computer-nodes could be used to test and continue development on other parts (at least where no interaction with entities is needed).

there is much to do, more then i can handle on my own in my free time within a reasonable timescale.
good input would be part of that. or a lot of other stuff. like thinking of crafting recipes (need depend on technic-mod?) and making the crafting hooks.
i would like to reach a somewhat compatibility with computercraft, mainly so i can continue with my giant turtle-script i wrote for computercraft.
if that runs, probably 95% of the other scripts out there can run too.
so if you'd like i can give you a rundown on how to use stuff right now, where-is-what and so on.

[electrodude512]

I've finally found the time to reply to this thread.

I don't want to have to compete with someone over two very similar mods, but I was really looking forward to writing this when you stole it. Instead of ending up with two separate competing mods, can we work on this together? I'll go with your very clever debug.sethook trick, unless it proves too slow, then we can consider adding threads as an per-server optional feature. My github is https://github.com/electrodude. It has a repo called computertest, but I never bothered to push anything to it because nothing is anywhere near finished because I was focusing on threads.

I see that your mod doesn't have any mention of digilines. Wouldn't it make sense to use them as peripheral cables instead of making your own? You can send tables over digilines, btw. Then it would be trivial to do things like a network mount - just have a computer pretend to be a floppy/hard drive - or a remote login - just have a computer pretend to be a terminal. When I have time, I'll fork your mod, add digilines support to computers, and start implementing and adding digilines support to peripherals like a hard drive, floppy drive, terminal, printer, OpenCCSensors-ish scanner, digilines to mesecons I/O expander, etc.

I must admit I haven't actually had the time to try out your mod. I'll probably have time tomorrow.

[aldobr]

If you trully want a sandboxed environment, use minetest.deserialize()

Its was created for other uses. Parse your lua sourcecode and feed it to minetest.deserialize()

You will be only able to talk to it via its return value. But works.

[ninnghazad]

@electrodude512
thanks for taking the time. i don't want to compete, or steal.
i just like coding, i code for a living, i code in my free time, i code in my head when showering.
the main reason i want a mod like this is because i wrote large scripts for computercraft,
and just cannot continue them after what happened to mc. so i want to port it all to mt.
those scripts are kind of hobby, it's relaxing to watch virtual botters and systems do stuff for you.
(https://github.com/ninnghazad/SWARM)

read up upon digilines/mesecons and i like the idea. it would be great if it was kind of a modem-item/node that one could insert or attach to computers and possibly drones too. maybe put an item in a slot of the computer/drone, or have an attachable node-version for computers and a wieldable version for drones (cause they are entities). if there was an extra modem-item/node/whatever, we could plan the mod as a modpack and users with and without digilines could use the computers. same would go for other peripherals.

a computer with a digiline-adapter (digiplug?) with io-expander to mesecons would be a sweet thing. i tried using mesecons to open a door using a computer and it works. but i somehow didn't want to jam the mesecons-stuff directly into the computer, i wanted it more like something you have to craft to get. something like a hub that can be plugged in.

first peripheral i wanted to do was a wireless-modem, because it would be simple. it would have just written data to the target-system's event-queue, and read from it's own.
it is only supposed to work between computers and drones. but same method should work for connecting a digilines-modem to the mod's code.

oh, a harddrive is the only thing i can think of that isn't really needed, as the drones and computers come with an internal harddrive. just because they are rather useless without one anyway,it makes the code simpler, and there has to be a place to store system-specific configs or startup-files. but thats the only thing basic computers should have. the good stuff needs to be extra.

so many possibilities =). anyways, i'd be happy to work with you, but i'm not pissed if you gonna say "screw that ninnghazad and his weak codes! imma make my own!".
in the end it's just a some text for a game.
and it wouldn't be a catastrophe to have multiple computermods - after all mc has CC and OC, which are similar and yet very different.

hm, i will probably make a trunk/dev branch on github, and start to setup the modpack-structure in there.

@aldobr
hm, i don't quite get the idea of deserializing usercode, maybe as a sanitizing measure - but by sandboxing i mean restricting environmental access to functions, filesystem and variables,
while providing enough working features for the sandbox to be meaningfull,
and be able to monitor and control the sandboxed environment from the outside, mainly to kill processes within.
that practically means that _G has to be replaced with a protected version of itself, which does not allow any access to the outside through functions or objects.
but that works quite well right now - i checked what mesecons did for the lua-controller's sandbox,
and it's the same approach, but with more restrictions. they forbid for/while-loops in the lua-controller's sandbox for example,
which i allow - because a computer without loops... meh. when a computer wants for example to access a node next to it,
it cannot just have access to the minetest.* object or the get_node function - the sandbox has to provide a function which calls minetest.get_node in a safe manner,
so the computer can not look up nodes on the other end of the map or stuff like that, or even ban players and so on.
but i'm positive about that part, right now i'm more concerned about that coroutine/stack-bug in minetest.

[aldobr]

Has nothing to do with deserializing.

minetest.deserialize runs inside a trully sandboxed lua. This means that it EXECUTES code that is provided as parameter. But it does so inside a sandbox environment, and returns the value. If you check minetest.serialize, you will see that the serialized format is just a plain string in lua syntax.

If we could hack it in any way, we could very well have a sandbox without all the trouble.

[ninnghazad]

the issue about the coroutine-bug has been assigned to 0.4.11, so thats a good sign.

@aldobr
i have read up upon minetest.serialize/minetest.deserialize and it is not sufficient. as i have written before, this is not about having an empty sandbox, but a sandboxed environment.

[aldobr]

I downloaded minetest source code to search how does minetest.deserialize works.

The idea was to create an alternative minetest.sandbox call that would return a sandboxed lua from C++ code, cutting the complexity of this project.

I couldnt find minetest.deserialize source yet.

[ninnghazad]

that would not make sense. the sandboxed environment has to have access to a secured environment - whatever method you use you will have to write those functions. and it already is working - the work that is left to do would have to be done any way you create a secured environment. here is the part of the source of minetest.deserialize:

Code: Select all

local function stringtotable(sdata, safe)
        if sdata:byte(1) == 27 then return nil, "binary bytecode prohibited" end
        local f, message = assert(loadstring(sdata))
        if not f then return nil, message end
        if safe then
                setfenv(f, safe_env)
        else
                setfenv(f, env)
        end
        return f()
end

function core.deserialize(sdata, safe)
        local table = {}
        local okay, results = pcall(stringtotable, sdata, safe)
        if okay then
                return results
        end
        core.log('error', 'core.deserialize(): '.. results)
        return nil
end

and again the code that is there already does that, but that is not enough. you're trying to solve a problem you do not seem to understand.
sandbox = easy, safe api for user to use within that sandbox = not easy
but still, it's just work to do and not impossibly hard or anything. just takes some time.

EDIT: that does however remind me that i need to add the bytecode-check to the code.

EDIT2: btw, ShadowNinja said he is to push a fix for the coroutine-issue, which would rock - cause then i can continue coding the drone-stuff.

[aldobr]

"that would not make sense."

That would make a lot of sense. You are still thinking about minetest.deserialize as a deserializing tool. You must understand how that function works. Its a completely sandboxed environment that happen to be used to deserialize things.

If you take that function C++ source code and expand it, you can very well have a general porpuse lua sandbox from C++ code where you can run arbitrary code (not just serialized tables/data).

Thats the whole point.

[ninnghazad]

look at the code. its loadstring and pcall with an empty environment... it's not magic, the mod already does that and more.

[aldobr]

What prevents this from be changed ?

[ninnghazad]

@aldobr
feel free to submit patches or fork the project, nothing is stopping anybody from changing everything.
maybe if you actually tried what you are talking about, you would understand what i am talking about.
i'd think we're talking past each other, and time would be better spend coding, but discussing thin air.
so less talk, more code?

[aldobr]

You are saying that as the sandbox used in deserialize is empty, this would prevent you from having a pre-loaded environment. But i already know that.

You might very well append the whole environment at the header of the code sent to deserialize. The only problem is the lack of multithreading ability (by that i mean the ability to stop execution midway and restart later).

Just patch minetest, and create a persistent sandbox system. Its not exactly the hardest thing to do (if i find the place where minetest.serialize is defined i will do that).

[ninnghazad]

builtin/common/serialize.lua line 17

[electrodude512]

How about we bug the MT devs about adding a minetest.register_on_key and add some way to capture keystrokes before they make it to the game engine, so you can have the normal mode where w is go forward and then another mode where w is type w on the keyboard you're currently pointing at? Then we could drop formspecs entirely and use something like digiline's lcds (but bigger, like ComputerCraft's monitors) for output and a keyboard block for input?

The keyboard would work like this: http://wiki.theoriginalbit.com/moarperi ... e=Keyboard

[ninnghazad]

@electrodude512
heho, sounds good.
it would have to be a minetest.register_on_player_receive_key, similar to register_on_player_receive_fields.
we need to be aware that this means more traffic for the server, as every client's keystroke is send immediately,
instead of in a bunch like how it is with the formspecs now.
this may be a reason why core-devs chose not to have it in the first place.
we need that nevermind how exactly it is done however.

like the lcd-display-thingy, but i fear there may just be too little screenspace this way.
i would like to see a 80x24 characters minimum resolution a single basic screen.
guess if the texture used for the lcd can be large that might work and still be readable.
we would need like 640 pixels wide to somewhat nicely print 80 characters per line.
when i imagine using the mod, i like the real-display-way much more then the formspec-stuff.
the formspecs for a start saves us the work of actually having to render characters and stuff,
and they have the benefit of easily includable inventories.
say the drones's inventory or the computer's (possible) upgrade-slots.
i think coolest would be formspecs for kind of an inventory/upgrades/config-gui for the systems,
and the screen itself on the block as a texture like with digilines-lcd.
we could use rightclick for the formspec-menu, and crawl-punch to start/stop using a keyboard,
and punch just to dig up the systems.

we could drop a feature request for the input-thing, and just see how that goes.
we can still try patch it when none of the core's find the time, or make c-plugin if they don't want it in the mt-code.
also, lets check how digiline's does the lcds and how much pixels/characters we could squeeze on a block that way.

EDIT: see https://github.com/Jeija/minetest-mod-d ... d/init.lua - they use a sprite-node with label on it or something - seems rather easy, but not sure if it actually allows for change of fontsize to something usable.

EDIT: changing fontsize in the digilines sourcecode worked, still need to check if something like this can be done with entities (drones)

EDIT: digilines does it by using the "combine" texture-type http://dev.minetest.net/texture, so it should be possible for entities too

[ninnghazad]

using combined textures on entities works, updating the screen works too.
screenshot of drone with a test-screen on its back:

[electrodude512]

When this was still my project, I wanted terminals and hard drives to be separate from computers. This would let you have computers share harddrives (easy persistent data) and make you not need terminals (cheaper) or be able to use alternate terminals (remote login, HUD, etc.). Another reason to separate terminals is so they can be as modular as possible, so they can be easily replaced with something better when possible because we're having problems with them right now. Drones would then be just dumb peripherals that you talk to over some kind of wireless digilines, and you would need a separate computer to control them. A computer's formspec will just ask for the boot drive's channel and the main terminal's channel, and the bios will connect to them and make fake fs and terminal functions that really send stuff to the drives and terminals. What do you think about this idea?

Computers should have a digilines.wrap("channel"), like ComputerCraft's peripheral.wrap("channel"), that returns a table full of functions that send what you pass them to the peripheral on the specified channel and (optionally) waits for a response. We could come up with a standard way to ask a peripheral what methods it supports and how they work, whether they return stuff, etc.

I'm eventually going to fork dronetest on github and do some of this, as well as add some of my peripherals. By the way, mesecons luacontrollers have been very useful for testing digilines, since I never got around to getting my computers to even boot.

EDIT/double post that isn't: (are double posts allowed on this forum?)

Can I have write permission (or whatever github calls it) to your repository, or should I give you some pull requests first? I'm about to send a pull request adding userspace (sandboxed) libraries (os.loadlib) and what I think the digilines peripheral wrapper should look like.

I'm warning you now, I'm going to want to make lots of security changes (none in this coming PR). Why does bootstrap.lua have to worry about any security at all? Should privileged APIs really be in /rom? A computer loading them manually can't do anything, but then they're useless, so why are they even there? Shouldn't activate_by_id load the privileged APIs instead of bootstrap.lua? Nothing a computer shouldn't be able to see should ever be in its environment, even if it gets deleted. It's a security hole just waiting to happen.

[ninnghazad]

@electrodude512
heho. i like the modular approach, and while i have to think about how to implement drones exactly in a approach like this, i changed the computers/drones to not have displays, and added dronetest_display peripheral, which is used per digilines. just a quick shot, but it works well.
only thing i'd like to keep fixed to the computer would be internal storage - the hdd-peripheral could be more of a NAS-peripheral, used for its sharing aspect.

i was planning on implementing something like peripheral.*, with similar functionality. peripherals would accept tables as digilines-msg, those tables contain actions, and each peripheral can act upon a set of actions (defined in the peripheral's code, started from userspace, running in globalspace). furthermore each peripheral has something like a get_capabilities action which returns a table with all information to use the peripheral's actions. the peripheral-api would then use the get_capabilities action to retrieve said table and build functions for a wrapper, which just consist of a wrapped digilines-send to the peripheral.


###

now about the security, it's not where it's supposed to be, obviously. so one-by-one:

Why does bootstrap.lua have to worry about any security at all?
bootstrap.lua is the entrypoint to userspace, it is always what gets called first. however the current contents are more like a placeholder. it is later on supposed to either (file_exists) run a autostart script, or load all needed apis for a shell and drop stdin/stdout (however that will work) to the shell for cli-usage. it's the systems bootloader.
so those bla = nil lines do not belong there, they do belong in activate_by_id (or whereever i setup the userspace_env), and it does not have to worry about security - but read on.

Should privileged APIs really be in /rom?
/rom is the place for privileged stuff - it must always be read-only to userspace. everything in there may be used from userspace but runs in a privileged environment.

A computer loading them manually can't do anything, but then they're useless, so why are they even there?
they are supposed to be loadable from userspace using sys.loadApi. and there will be more apis. those loaded in this monstrosity of bootstrap are just to run the "shell" currently used (that kickass while-loop down there).

Shouldn't activate_by_id load the privileged APIs instead of bootstrap.lua?
Nope see above and below.

Nothing a computer shouldn't be able to see should ever be in its environment, even if it gets deleted. It's a security hole just waiting to happen.
i agree completely. i assume you raised this because you assumed APIs are something from globalspace that is by some mistake accessible from userspace. but this ain't so. however, APIs need to be safe to be run from userspace.
APIs do not need to worry about infinite-loops/syntax-errors-type of security (in general), but need to provide the secure fs-access, secure loading-functions and secure peripheral-wrapps and and and...
APIs are in a parallel-dimension where global/mod-space and userspace meet, they use cool global stuff and wrap it in a way safe for userspace.
APIs are like libraries.


###


I hope i did get the idea across. i know that what is there now does not quite follow those rules, but, hey, project is like a week old. theres some stuff that may seem strange, but is just in place cause of that coroutine-bug, and will be changed once that is fixed (like updating displays in a globalstep instead of when printed to). the bootstrap is just like this cause it has to wait for proper input-solution to create a shell.

what i am planning to do next would be turning the drones into a peripheral, and maybe create a modem or whatever is needed to connect them to the network. in that turn i would also like to add the peripheral-ness to the computers themselfes.
for that a peripheral-over-digilines/dronetest-over-digilines(?) protocol is needed indeed.
what i want to be able to do is for example:

local drone = peripheral.wrap("dronetest:drone:ID") -- some channel leading to a drone
drone.turnLeft()
print("print to stdout")
local secondScreen = peripheral.wrap("dronetest:display:ID") -- some secondary screen on the network
secondScreen.print("print to some other display")

and so on.

im looking forward to seeing your ideas about the peripheral-wrapper. it would be cool if it was loadable as an API.
but do elaborate about the sandboxed-libraries, as i think this might be what sys.loadApi does (loadApi loads privileged code from userspace). if it is to load unpriviliged code from userspace it could be usefull, could however overlap with what shell.run does.

i will set you up as a contributor on github when i get home.
you can add stuff as you want for now, but if you change existing stuff in way that may break stuff please discuss,
and document.

btw,drop by #minetest on irc if you can

[ninnghazad]

thought about peripheral.* some more - would be great if you could directly put it in /rom/apis/peripheral.lua as part of the main mod.
i placed something to start from there with a basic function for wrapping and a test_peripheral example that wraps a display.
to test, use /giveme dronetest:computer and /giveme dronetest_display:display and place computer and display,
connect with digilines.
set display's channel to "dronetest:computer:YOURCOMPUTERIDHERE".
then boot computer and type test_peripheral.

look in dronetest_display/init.lua, rom/bin/test_peripheral and rom/apis/peripheral.lua.
its just a tech-test to see how this may work, so it doesn't really do anything usefull besides the example.
but should be enough to serve as an example of how to put the peripheral-thing as an API.

computers that receive digiline-msgs put them in their event-queue, so they can be easily retrieved in userspace using
sys:receiveDigilineMessage(channel,message_id)

[electrodude512]

Thanks for making me a contributor. I was going to send you a pull request, but Git was being stupid (note to self: don't amend commits after pushing them). I'll talk to you about changing things already in place, but I'm going to go ahead and add stuff that doesn't change stuff you already have. I'm adding a mechanism for userspace (sandboxed) libraries (like for getting a table of functions interface for digilines peripherals).

[ninnghazad]

i see the github term is "collaborator" - be that as it may, i set you up.
hm, that loadlib thingy may be handy as a basis for a userspace-version of loadstring dofile and so on.
want to put it in /rom/apis/os.lua as os.whatever?
remember that the loadstring you use when putting this as an api is the global one, so functions it returns come with privileges - don't forget to put some empty env even if nothing else is specified.
i wanted to have userspace-versions of all those functions (except debug maybe, thats a tricky one) like loadstring and other unsafe functions. that'll be a good start.
it'll be userspace-sandboxes within the main-sandbox, sweet.

about the purpose of loading userspace-apis using that stuff to get functions for peripherals:
take a look at /dronetest_display/init.lua, the on_digilines function and stuff around it. that actions table contains possible peripheral-functions for the display. computers (or whatever is speaking the protocol) can send a GET_CAP over digilines to the display and the display responds in its digiline-callback with a call to that function ('action'), and returns its output.
this all outside of userspace, peripheral api would make that accessible to userspace.
this has a few upside versus extra apis in userspace.
- no need to secure each peripheral's networking api, rather secure peripheral api
- actions of a peripheral are in code, like digilines or mesecons.
- actions can share the object's env they belong to more easily.
- there is something stylish about the method of actually calling the peripheral to have it execute an action.
however, the format of those messages passed is not good yet, don't remember but something about them was missing. works nevertheless.

oh, i changed the formspecs to have a channel field. and removed buttons and stuff from the drone. will try to make dronetest_transceiver, some kind of node that will give a first way of connecting drones to the digilines network, and make them accessible to computers through that.

btw, i gave default digiline-channels like this: dronetest:TYPE:ID
TYPE is computer,drone,display,printer,whatever
ID is either real id or just some running number.

have fun coding