Finally got Chrome to work inside Minetest ...

For people working on the C++ code.
User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Not sure whether to place this here or in features, but since it is quite low level dev stuff I thought I'd post it here anyways... After about 3 weeks of messing around with CEF3, lots of dead ends, ending up using Awesomium and more sigsegvs than I bargained for, I finally managed to open a web browser embedded inside the Minetest game and have it render to an ITexture buffer inside a formspec.

Why? Because I can I guess ... The good (skip to 2:15 for the actual demo, first part is small overview of code changes). The sound actually does work, it's just that I did not record any sound in the video.

https://www.youtube.com/watch?v=kx3RPqmhJrE

Image
vlcsnap-2017-03-16-14h16m26s498.jpg
vlcsnap-2017-03-16-14h16m26s498.jpg (204.22 KiB) Viewed 1845 times
Bit of background: The proof of concept basically uses the Chrome code base (CEF3) underneath. As per the Chrome processing model, the main application (in this case, Minetest) is separate from the browser process so browser crashes will not crash the main application. Awesomium is a wrapper around the Chromium Embedded Framework (CEF3) and takes care of managing the browser sub-processes and the complexities of passing messages, handling browser events in the main program and providing access to screen buffers for off-screen rendering.

I managed to get the off-screen rendering to work with formspecs by providing a new browser formspec element that, when parsed, will open a so called Awesomium web view. The web view provides it's buffer for copying into other buffers. All that remained was to create a GUI image and copy the web view buffer into the image's texture. This will of course work for arbitrary textures, so the next logical step is to copy the web view buffers to node textures, allowing players to place YouTube video blocks, HTML documentation blocks, etc.

Awesome, right? Well, now for the bad .....

CEF3 is an absolute beast. When I started I found that the Chromium source consisted of 27000 files, it took me hours and hours to compile. There are binary packages for download of 200 - 300 MB. The core libcef.so library is 60+ MB (uncompressed). It is under continuous development so there are no real stable branches. One version works, the next one crashes for no apparent reason. There are samples online showcasing off-screen rendering e.g. the Chrome browser in 3D applications but these don't take into consideration on how to embed CEF3 in an existing large code base such as Minetest. The CEF samples are no real help either. It looks like I would need to manually write code to share the browser buffers with the main Minetest program. Fun! -_-

TL;DR, CEF3 is a pain to use. I've been hacking around for about 2 weeks now, and would probably need another 2 weeks to get any off-screen rendering to work.

In comes Awesomium. It takes care of all the nasty things one does not want to take care of. I was actually able to put the above proof of concept together in 2 days. It's awesome! But: *Awesomium is closed source*... It is free to use in Open Source not for profit projects, but still, closed source. With 65MB, the libawesomium shared library is still large as it contains libcef of course.

Since the browser runs in a separate process, all sound handling (e.g. YouTube videos, audio played in the browser) is done by that process. That is great for formspecs, but if you'd have a node you would expect the volume to lower the further away you are from the node. To redirect the audio processing somehow, no clue yet how that could be done.

There are alternatives such as embedding Gecko. That will introduce a dependency on the Gecko code base and the Gecko engine would then have to run inside the Minetest process unless I would custom roll a sub-process model for Gecko. Also, the Gecko API doesn't look that simple so lot's of coding would probably be required to get that to work.

So that's the bad... But then of course, what do think about this kind of functionality? Besides this looking cool (and even cooler when I implement casting pages to nodes), is it worth pursuing. I can come up with a few use cases, and there is the possibility of abuse (e.g. a web browser node with configurable URL could link to inappropriate content) beyond what is already possible (e.g. insulting signs).

Also, what about the approach I took? Are these kinds of code base changes worth it? It seems to be minor so far as Awesomium provides a singleton, but still this adds a dependency on (yet) another (closed source) library. Would it be better to keep this outside of the main Minetest code base, package this as a mod (with Lua -> native bridge)? But in that case, is there currently a way to access to the texture buffers? As in, how to go from Awesomium/ CEF3 -> Lua -> formspec/ node ITexture buffers?

I'm really interested in what you think: Any feedback, whether it's good or bad, much appreciated!! :)

And then there are a bunch of TODO's I will still need to look into:
  • More robust code, e.g. it crashes when I exit a game and I need to put some safety mechanisms in place when the browser sub-process crashes
  • Passing events (e.g. mouse clicks) to the embedded browser
  • Implement copy to node texture buffer for web browser nodes
  • See if I can modify the Lua browser mod to allow for configurable URLs
  • The IGUIImage is not properly created, it is not registered in the UI element children
  • Make a fork of Minetest and upload my changes to the fork
Last edited by Beerholder on Fri Mar 17, 2017 11:40, edited 1 time in total.

User avatar
maikerumine
Member
Posts: 1420
Joined: Mon Aug 04, 2014 14:27
GitHub: maikerumine
In-game: maikerumine

Re: Finally got Chrome to work inside Minetest ...

by maikerumine » Post

Just saw your video... WOW!!!
You changed minetest immensely!!!!
Talamh Survival Minetest-->viewtopic.php?f=10&t=12959

Byakuren
Member
Posts: 818
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

Re: Finally got Chrome to work inside Minetest ...

by Byakuren » Post

I'm against the inclusion of any proprietary dependencies or libraries.
Every time a mod API is left undocumented, a koala dies.

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Byakuren wrote:I'm against the inclusion of any proprietary dependencies or libraries.
Yeah I guess it is indeed still very worthwhile to put in the extra effort and try to get CEF3 to work instead of using the Awesomium wrapper. As far as I can tell, CEF3 and Chromium are both (revised/ new) BSD, so that should not be a problem with LGPL right?

Byakuren
Member
Posts: 818
Joined: Tue Apr 14, 2015 01:59
GitHub: raymoo
IRC: Hijiri
In-game: Raymoo + Clownpiece

Re: Finally got Chrome to work inside Minetest ...

by Byakuren » Post

I don't think it would be an issue.
Every time a mod API is left undocumented, a koala dies.

BBmine
Member
Posts: 3476
Joined: Sun Jul 12, 2015 22:51
GitHub: BBmine
IRC: BBmine
In-game: Baggins
Location: USA

Re: Finally got Chrome to work inside Minetest ...

by BBmine » Post

WOW! I could see problems with having this on servers tho. People could (maybe) download viruses onto the server's computer.

So that's where you've been all these weeks away from me XD

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Well the server is safe as the embedded Chrome is only started/ used on the client. My other thinking is that I should not provide a generic browser mod myself but leave that to the modders. They can hook into the new formspec element or node type and then if they want they could hard code in their mod that only certain pages can be shown. E.g. a form with buttons for tutorial 1, 2, etc. only linking to tutorial pages or hard coding the youtube.com part and only allowing the video hash to be configured on a block. Worst that could happen then is getting rickrolled XD

On a more serious note, I am going to check out the CEF3 IPC mechanism. There is already some message passing mechanism to communicate UI events to the browser sub-processes (e.g. clicking the back button), there should be something then to pass raw bytes of the screen update rects back to the main process which I can then poll in the Minetest game loop.... I am already able to launch the sub-processes right now using CEF3 only (so without Awesomium), it's just the IPC part that I will have to implement in the sub-process and the polling in the game loop. Easier said than done tho ;)

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Success!!! I managed to get it to work with purely CEF3! So no more closed source Awesomium, and I actually did not have to do anything to get the IPC to work either! I am quite surprised, and happy I got it to work so quickly! And only about 5 segmentation faults today XD

I created a mt_cef wrapper following the same sort of principles as Awesomium to make it easier to manage browser lifecycle, providing a MinetestBrowser singleton for easy access across the code, allowing it to create "WebPages" and replaced the Awesomium calls with my own custom rolled one. I'll continue hacking around when I get home tonight :-D

I can't wait to implement this on nodes ;-)

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Forked 0.4.15, merged the code and pushed my changes my repo here: https://github.com/evrooije/minetest-cef3

Some very minimal documentation on how to build, the changes I made, simple API description and the example mod [MOD] Browser [browser] can be found here: https://github.com/evrooije/minetest-cef3/wiki

This is still very alpha quality, e.g. closing web pages in this CEF3 version is not implemented yet so every punch creates a new browser, I don't keep track of opened web pages in my singleton, at this moment there is nothing whatsoever to pass events to the browser and I am getting ABM timeout warnings for some reason -_- Also, resizing the Minetest window leads to spectacular failure (in the sense that for some reason it spawns multiple browsers ... are FormSpec elements parsed anew on resize?????), but then again maybe if you try it will just core dump (or start playing Rick Astley XD) when you resize. Anyways, handle with care for now and use at your own risk ;-)

I'll definitely want to make it way more robust before venturing into the node web browser textures, and might want to change the code style a bit as well while I am at it, as not in line with the Minetest code. But that's for tomorrow... However sharing is caring so if you're interested feel free to have a look and most def let me know if you have questions or concerns!

nrz
Developer
Posts: 131
Joined: Sat Feb 07, 2015 17:16
GitHub: nerzhul
IRC: nrzkt
In-game: nrz
Location: France
Contact:

Re: Finally got Chrome to work inside Minetest ...

by nrz » Post

It's interesting

nrz
Developer
Posts: 131
Joined: Sat Feb 07, 2015 17:16
GitHub: nerzhul
IRC: nrzkt
In-game: nrz
Location: France
Contact:

Re: Finally got Chrome to work inside Minetest ...

by nrz » Post

We should have an external library, not compile it with minetest itself as it's a very huge library maybe you can provide a PR to minetest with and external library

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Hi nrz thanks, I am most definitely looking into the build and linking against libcef.so aspect. In the end libcef should obviously not be packaged with or be part of Minetest as it is indeed too big. I currently have it in the minetest/bin directory in my dev env but they do not need to be there (my current setup is still very much a dirty hack). I need to properly put the -DCEF3_ENABLED flag in place as well. Possibly I could use an existing install of Chrome to link against so I need a proper find for the library in the makefile, as well as load runtime. I will look into all of this and once there is a better way to build and link and when more stable, create PRs for devs to review as you suggested.

In the meantime, after a lot of digging through the Minetest code, nodes now also have very basic preliminary browser support:

https://www.youtube.com/watch?v=TrCkl58b9PI

User avatar
MineYoshi
Member
Posts: 5373
Joined: Wed Jul 08, 2015 13:20
Contact:

Re: Finally got Chrome to work inside Minetest ...

by MineYoshi » Post

This is awesome... +10
Have a nice day! :D

User avatar
Inocudom
Member
Posts: 3121
Joined: Sat Sep 29, 2012 01:14
IRC: Inocudom
In-game: Inocudom

Re: Finally got Chrome to work inside Minetest ...

by Inocudom » Post

What we have here is Minetest adopting a useful feature that Garry's Mod is well known for. Personally, I feel that Minetest deserves this feature way more than Garry's Mod ever did, so please keep up the good work.

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Finally got Chrome to work inside Minetest ...

by sofar » Post

Why not adding some generic layer that can act as e.g. an X11 display or supports the VNC protocol? Then you could interface with almost anything, not just a single program.

Wayland comes to mind. Have minetest be a wayland compositor.

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

Actually dummy X server (formerly known as xvfb) did cross my mind at some point :-) but obviously X11 only. To make it generic would be a good idea and should be perfectly feasible tbh (all it needs to do is copy buffers to textures and pass events to the underlying application, anything supporting that like xvfb, its Windows equivalent or an embedded VNC client should have support for that)

I'll see if I can refactor it into a generic interface at some point, it would most certainly make it a lot more powerful. Can't wait to run my bash shell or running my Xfce desktop in Minetest ;-)

User avatar
GreenXenith
Member
Posts: 1356
Joined: Wed Oct 28, 2015 01:26
GitHub: GreenXenith
Location: UTC-8:00
Contact:

Re: Finally got Chrome to work inside Minetest ...

by GreenXenith » Post

Beerholder wrote:Can't wait to run my bash shell or running my Xfce desktop in Minetest ;-)
1) Install Linux to desktop in Minetest
2) Install Minetest on it
3) Get this thing for Minetest.
4) Install Linux to that desktop
5) Get Minetest for that desktop
6) Get this thing for that Minetest
7) etc....
MinetestDesktopCeption!!!....
No but seriously, would that be possible? o-0
YouTube | Mods | Patreon | Minetest Discord @greenxenith

You should not be able to read this message.

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

GreenDimond wrote:MinetestDesktopCeption!!!....
No but seriously, would that be possible? o-0
Possibly ... O_o I cannot get Minetest to run under Xvfb though, doesn't pick up the GL extensions for some reason, I sort of expected that. For Wayland support, I'd have to make Minetest "Wayland ready", as right now it just spawns a window outside of the Weston desktop. The only other thing I haven't tried yet is Xdummy, might give that a shot some other time.

But yeah, if I get Minetest running in either Xdummy or Wayland, then yes, I could probably try to run Minetest inside a node in Minetest XD

User avatar
Linuxdirk
Member
Posts: 3216
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: Finally got Chrome to work inside Minetest ...

by Linuxdirk » Post

GreenDimond wrote:MinetestDesktopCeption!!!....
Yo dawg, I heard you like Minetest ...

User avatar
GreenXenith
Member
Posts: 1356
Joined: Wed Oct 28, 2015 01:26
GitHub: GreenXenith
Location: UTC-8:00
Contact:

Re: Finally got Chrome to work inside Minetest ...

by GreenXenith » Post

Beerholder wrote:Possibly ... O_o
But yeah, if I get Minetest running in either Xdummy or Wayland, then yes, I could probably try to run Minetest inside a node in Minetest XD
:O That would be so cool! lol the window would get smaller and smaller until you have a microscopic Minetest application xD
Linuxdirk wrote:
GreenDimond wrote:MinetestDesktopCeption!!!....
Yo dawg, I heard you like Minetest ...
Yep. Been playin some years now. Love it.
YouTube | Mods | Patreon | Minetest Discord @greenxenith

You should not be able to read this message.

wilkgr76
Member
Posts: 832
Joined: Wed Feb 18, 2015 02:44
GitHub: wilkgr76

Re: Finally got Chrome to work inside Minetest ...

by wilkgr76 » Post

GreenDimond wrote:
Beerholder wrote:Can't wait to run my bash shell or running my Xfce desktop in Minetest ;-)
1) Install Linux to desktop in Minetest
2) Install Minetest on it
3) Get this thing for Minetest.
4) Install Linux to that desktop
5) Get Minetest for that desktop
6) Get this thing for that Minetest
7) etc....
MinetestDesktopCeption!!!....
No but seriously, would that be possible? o-0
Here, have an OOM :)
N/A

sofar
Developer
Posts: 2146
Joined: Fri Jan 16, 2015 07:31
GitHub: sofar
IRC: sofar
In-game: sofar

Re: Finally got Chrome to work inside Minetest ...

by sofar » Post

@beerholder, now that we know you can code C++, why not work on some of Minetest's open issues? =)

User avatar
Beerholder
Member
Posts: 199
Joined: Wed Aug 03, 2016 20:23
GitHub: evrooije
In-game: Beerholder

Re: Finally got Chrome to work inside Minetest ...

by Beerholder » Post

sofar wrote:@beerholder, now that we know you can code C++, why not work on some of Minetest's open issues? =)
Once I am done with this little pet project I was indeed thinking about that sofar :-D I am starting to get to know parts of the engine better bit by bit :)

User avatar
azekill_DIABLO
Member
Posts: 7507
Joined: Wed Oct 29, 2014 20:05
GitHub: azekillDIABLO
In-game: azekill_DIABLO
Location: OMICRON
Contact:

Re: Finally got Chrome to work inside Minetest ...

by azekill_DIABLO » Post

amazing! i hope i could go play diep.io on minetest!
Gone, but not dead. Contact me on discord: azekill_DIABLO#6565
DMs are always open if you want to get in touch!

u34

Re: Finally got Chrome to work inside Minetest ...

by u34 » Post

azekill_DIABLO wrote:amazing! i hope i could go play diep.io on minetest!
awesome and +1 i want this feature ... i like it... when its done...

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests