GPT-3/Codex: An AI that can write Minetest mods... kinda

[rubenwardy]

An exploration of GPT-3's knowledge of Minetest modding, with node registration, formspec manipulation, mod creation, and Minecraft mod to Minetest conversion.

Read it on blog.rubenwardy.com

[ronoaldo]

This is was an amazing reading. Thanks for sharing and doing that experiments!

[Blockhead]

We live in an era where we may soon be having people with no Lua knowledge who asked GPT-3 to write a Minetest mod for them, and then coming to the forums to ask us why their code doesn't work. Astounding :)

[sorcerykid]

I actually found this to be both intriguing and amusing, simultaneously. Very insightful!

[v-rob]

Oh no, AIs making brittle formspec code for Minetest. We're doomed! Please, nobody teach them how to make forum accounts... But actually really interesting.

Gosh, though, IDEs are getting more gross all the time. Now they're flat out suggesting code? I get that boilerplate code is annoying and prone to mistakes and all, but it's worse to have barely competent programmers just using brittle and possibly wrong code written by an AI.

[rubenwardy]

Thanks for the comments

Just an update - in the article, I mention that the fruit list formspec it makes has an error due to a missing variable storing the list.

I passed GPT-3/Codex the error message and it fixed it. This system could be very powerful if combined into some sort of feedback loop, where it can receive error messages / run tests

[rubenwardy]

...

I'll repost what I replied to someone else:

It's about as good as a beginner modder.

If you look in the formspec section, it generates complete code for a problem that almost works - just has one missing variable. If you feed it the error message, then it is able to fix the issue - just like a real programmer does

So if you could combine this tool with a way of testing its results then it will be very capable

Remember that this is an early generation of the tool as well. I fully expect subsequent tools to be much more accurate.

Also, this tool was never trained especially for lua and Minetest. Codex only really officially supports python, js, and a few other languages. So having it trained especially will make the results better

[DS-minetest]

Interesting!
Can we throw this on feature requests?

GPT-3 (and Codex) learned how to write Minetest code by reading code on the Internet. This code may or may not be open source, and may or may not be permissively licensed. These models tend to regurgitate code, which leads to license laundering - open source code being turned into proprietary code, without credit. These products benefit from the unpaid labour of the open-source community. So whilst this technology is interesting, I’m not sure how much I agree with it ethically.

This sounds quite problematic.
If the AI outputs non-trivial material (let it be code, text, or any other media), you can't use it (especially not without giving credit). It's like you'd have found the material somewhere in the web. But you can also barely find out where the material even came from, and who to give credit.

[Blockhead]

I was thinking about the licence laundering issue. While we can require anybody to do this, perhaps we could expect that people who write code with GPT-3/codex and want to be ethical about it to also ask it "Is the above code copyrighted?". Sadly nothing can likely be done about people who don't want to be ethical. Also this is more of an issue for copyleft than permissive licences.

I think/hope that in the future, the fact that only source-available/open source code will be usable in improving AI models, coupled with increased use of AI programming techniques, may mean more companies will all agree to share their source in order to improve the overall models for everyone. At least that's something to be hopeful about, if not really guaranteed.

[Mantar]

The laundering problem is best solved by training an AI solely on public domain/permissive licensed material. If you include GPL code, then any code it generates should be treated as GPLed, which would not be a problem for us here.
Attribution licenses would have to be excluded, there's no way to figure out what the AI borrowed from where and include the necessary copyright notices.