GPT-3/Codex: An AI that can write Minetest mods... kinda

[rubenwardy]

An exploration of GPT-3's knowledge of Minetest modding, with node registration, formspec manipulation, mod creation, and Minecraft mod to Minetest conversion.

Read it on blog.rubenwardy.com

[ronoaldo]

This is was an amazing reading. Thanks for sharing and doing that experiments!

[Blockhead]

We live in an era where we may soon be having people with no Lua knowledge who asked GPT-3 to write a Minetest mod for them, and then coming to the forums to ask us why their code doesn't work. Astounding :)

[sorcerykid]

I actually found this to be both intriguing and amusing, simultaneously. Very insightful!

[v-rob]

Oh no, AIs making brittle formspec code for Minetest. We're doomed! Please, nobody teach them how to make forum accounts... But actually really interesting.

Gosh, though, IDEs are getting more gross all the time. Now they're flat out suggesting code? I get that boilerplate code is annoying and prone to mistakes and all, but it's worse to have barely competent programmers just using brittle and possibly wrong code written by an AI.

[rubenwardy]

Thanks for the comments

Just an update - in the article, I mention that the fruit list formspec it makes has an error due to a missing variable storing the list.

I passed GPT-3/Codex the error message and it fixed it. This system could be very powerful if combined into some sort of feedback loop, where it can receive error messages / run tests

[rubenwardy]

...

I'll repost what I replied to someone else:

It's about as good as a beginner modder.

If you look in the formspec section, it generates complete code for a problem that almost works - just has one missing variable. If you feed it the error message, then it is able to fix the issue - just like a real programmer does

So if you could combine this tool with a way of testing its results then it will be very capable

Remember that this is an early generation of the tool as well. I fully expect subsequent tools to be much more accurate.

Also, this tool was never trained especially for lua and Minetest. Codex only really officially supports python, js, and a few other languages. So having it trained especially will make the results better

[Desour]

Interesting!
Can we throw this on feature requests?

GPT-3 (and Codex) learned how to write Minetest code by reading code on the Internet. This code may or may not be open source, and may or may not be permissively licensed. These models tend to regurgitate code, which leads to license laundering - open source code being turned into proprietary code, without credit. These products benefit from the unpaid labour of the open-source community. So whilst this technology is interesting, I’m not sure how much I agree with it ethically.

This sounds quite problematic.
If the AI outputs non-trivial material (let it be code, text, or any other media), you can't use it (especially not without giving credit). It's like you'd have found the material somewhere in the web. But you can also barely find out where the material even came from, and who to give credit.

[Blockhead]

I was thinking about the licence laundering issue. While we can require anybody to do this, perhaps we could expect that people who write code with GPT-3/codex and want to be ethical about it to also ask it "Is the above code copyrighted?". Sadly nothing can likely be done about people who don't want to be ethical. Also this is more of an issue for copyleft than permissive licences.

I think/hope that in the future, the fact that only source-available/open source code will be usable in improving AI models, coupled with increased use of AI programming techniques, may mean more companies will all agree to share their source in order to improve the overall models for everyone. At least that's something to be hopeful about, if not really guaranteed.

[Mantar]

The laundering problem is best solved by training an AI solely on public domain/permissive licensed material. If you include GPL code, then any code it generates should be treated as GPLed, which would not be a problem for us here.
Attribution licenses would have to be excluded, there's no way to figure out what the AI borrowed from where and include the necessary copyright notices.

[ShadMOrdre]

You can bet that an AI that scrubs the web for code is only gathering openly published code, regardless of license, but of course including most if not all "official" open source, and no proprietary code whatsoever.

You can bet that this AI does NOT have access to Windows or iOS code, but does have access to Linux code.

You can bet that if a corporation can make money, it will.

And you can bet that their attorneys can most likely outspend you.


This is not good. On any level. Attribution and licensing are only the superficial issues involved here. Minetest mods notwithstanding, can it generated backdoor/worm/virus code, in assembly? I'd bet it can, given all the open source code available on github alone, much less sourceforge, gitlab, and any other public repository.

Rubenwardy only asked very tailored and specific questions of this AI, and the AI generated relevant working code. What happens when the questions become less ethical, or outright criminal?

There goes any career in computer programming...

Don't say we weren't warned of the dangers.



Shad

[Blockhead]

You can bet that an AI that scrubs the web for code is only gathering openly published code, regardless of license, but of course including most if not all "official" open source, and no proprietary code whatsoever.

You can bet that this AI does NOT have access to Windows or iOS code, but does have access to Linux code.

You can bet that if a corporation can make money, it will.

And you can bet that their attorneys can most likely outspend you.

Correct it won't have the Windows kernel space code and most the other parts of their OS, but it will have most of the .NET Framework, since most of that is now open-sourced especially with .NET 5.

I also pointed out above, but let me make it clearer: it may have read proprietary source-available code like Unreal Engine.

Also it's a bit hard to sue when you can't establish someone took a small part of your code (remember GPT-3 has a very small input window), and then made it proprietary. It's kind of moot point about attorneys when we can't expect much to end up in court at all. Let me quote the OpenAI Codex paper:

We find that, to the extent the generated code appears identical to the training data, it is due to the predictive weightings in the model rather than retention and copying of specific code.

This is not good. On any level. Attribution and licensing are only the superficial issues involved here. Minetest mods notwithstanding, can it generated backdoor/worm/virus code, in assembly? I'd bet it can, given all the open source code available on github alone, much less sourceforge, gitlab, and any other public repository.

Rubenwardy only asked very tailored and specific questions of this AI, and the AI generated relevant working code. What happens when the questions become less ethical, or outright criminal?

There goes any career in computer programming...

Don't say we weren't warned of the dangers.

This is mostly quite alarmist.

GPT-3 isn't even the best AI for code. OpenAI Codex (available as GitHub Copilot) is, and it's still only best for widely-used high level languages. Also, if you want the code to actually work without bugs, an actual programmer usually has to edit it. It's just a starting point for making code, something more advanced than IDE boilerplating helpers.

Assembly is a bad choice of language to worry about here. The sample set of GPT-3 will naturally include much less assembly than higher level languages because that is where most software is being developed at. Thus, it will be worse at assembly. Yet there are plenty of ways to write bugs and exploits in higher level languages like C++, or even Java, C# or Python. Assembly is unlikely though. If you're writing an exploit in assembly you probably want fine-grained control over the exact instructions in order to target the specifics of the Instruction Set and platform generation, for instance in order to do something like row hammering or the newer Pacman Attack on Apple Silicon (arm). Assembly also is split into Intel and AT&T syntaxes and something as simple as GPT-3 might get them confused and produce mixed-syntax results. Assembly is also operating-system specific in most cases, with the system and dynamically linked library calls being very different on different operating systems (Windows, macOS, Linux, BSDs all being quite different). As for using Codex to generate assembly - yes, it may get the syntax correct unlike GPT-3, but it's unlikely to be any easier for it to write an exploit in Assembly than it is for it to come up with one in C, which will probably have better training data too.

Anything in the public sphere that can be used by blackhats can be used by white and grey hats as well, and GPT-3 is very much public. This just means we have entered a new era where the security race to find and fix 0-days before they are exploited will involve the use of machine learning from both sides. This race already involves simpler 'AI' techniques such as fuzzing.

Yes. Machine learning may help synthesize new 0-days. On the flip side, it may help analyse old code for vulnerabilities. One key feature GitLab is trying to sell in their paid plans is a suite of security scanning features for your code. Also, Machine learning is unlikely compared to a human to synthesize new novel types of attacks, unless we targeted it at finding new kinds of exploits with automated CVE analyses of the exploits it finds and using reinforcement learning - even then it will usually find the more prevalent types of exploits because reinforcement learning will ensure if it succeeds with those it's going to repeat them.

A security researcher's career isn't going away, it's just getting more complex as ever with a new tool. Nor are the careers of people who accidentally write bugs into their software. Mostly because there are often no consequences to them directly (just pay attention to how little punishment there is to programmers or corporations other than fiscally from lost business in case of a cybersecurity breach).