A good idea to improve the sub game loading time.

Post Reply
patrick55
Member
Posts: 26
Joined: Sat Jul 30, 2016 03:16

A good idea to improve the sub game loading time.

by patrick55 » Post

Now an MT client must get lots of subgame images files from the server or a HTTP server, and load them one by one(lots of file open operation). This will cost a lot of time.
Here is my idea:
Server: Merge all the images to some big images. When you finish your subgame, run a script to merge all the images and put them in a directory.
Client: Only get those merge big images and each image's coordinate info.
Why this can save the loading time a lot?
1. Transfer time. Only transfer the merged big images, instead of hundreds or thousands of small images through the internet.
2. Loading files time. Only open the big merge files, save a lot of the files open time.

Leave your comment :)

User avatar
Linuxdirk
Member
Posts: 2555
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Post

Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.

User avatar
Krock
Developer
Posts: 4538
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker
Location: Switzerland
Contact:

Re: A good idea to improve the sub game loading time.

by Krock » Post

So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.

I proposed the same idea a few year ago, but can't find the log entry. However, sfan5's point was (roughly): "Nothing beats the filesystem. How should a database be faster than that?".
Look, I programmed a bug for you. >> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>

patrick55
Member
Posts: 26
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Post

Maybe one simple way is just zip the subgame assets and transfer one zip file to the client.
Linuxdirk wrote:Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.

patrick55
Member
Posts: 26
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Post

Save the internet transfer time will save a lot, I think. Zip the subgame assets and send one zip file to the client :>.
The assets loading time in the client side may not save a lot(all files in a database VS all files in the filesystem), because we just load a few hundreds or thousands of small file. That’s a small job for a model CPU, I think. [Need some stats]
Krock wrote:So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.

I proposed the same idea a few year ago, but can't find the log entry. However, sfan5's point was (roughly): "Nothing beats the filesystem. How should a database be faster than that?".

User avatar
Linuxdirk
Member
Posts: 2555
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Post

Krock wrote:So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.
Why not archiving all media on server startup and send the archive to the client when it connects? This would save individual files being read and sent every time a client connects. I assume due to the overhead of reading the files and establishing a connection for each of the files sending individual files is slower than sending only one file.

User avatar
Pyrollo
Member
Posts: 367
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: A good idea to improve the sub game loading time.

by Pyrollo » Post

Linuxdirk wrote:Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.
Media consisting in images and sounds (PNG, OGG) are already compressed. No need to compress then once again. Archiving should be enough.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

User avatar
Linuxdirk
Member
Posts: 2555
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Post

Pyrollo wrote:Media consisting in images and sounds (PNG, OGG) are already compressed.
By the server? I've seen dozen of mods where neither the PNGs nor the OGGs were optimized.

User avatar
Pyrollo
Member
Posts: 367
Joined: Mon Jan 08, 2018 15:14
GitHub: pyrollo
In-game: Naj
Location: Paris

Re: A good idea to improve the sub game loading time.

by Pyrollo » Post

Linuxdirk wrote:By the server? I've seen dozen of mods where neither the PNGs nor the OGGs were optimized.
That's another problem. In my opinion re-compressing (even badly) compressed data does not make sense. I'm not even sure of the gain.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]

User avatar
Linuxdirk
Member
Posts: 2555
Joined: Wed Sep 17, 2014 11:21
In-game: Linuxdirk
Location: Germany
Contact:

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Post

Yeah, then leave it uncompressed. Would still be better than reading and sending all media one-by-one on every client connection.

patrick55
Member
Posts: 26
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Post

Will this be support in the future? or some other way to improve the first time loading?

User avatar
Festus1965
Member
Posts: 1329
Joined: Sun Jan 03, 2016 11:58
In-game: Thomas Explorer Thailand
Location: Thailand - Chiang Mai (+5-6h to MEZ)
Contact:

Re: A good idea to improve the sub game loading time.

by Festus1965 » Post

Make sure that the client itself is able to check weather this mod is local available, and load then from there local.
So gamer who want to force speed and save time, just download that mods themself into client and ... happy ?

Astrobe
Member
Posts: 321
Joined: Sun Apr 01, 2018 10:46

Re: A good idea to improve the sub game loading time.

by Astrobe » Post

It looks like MT already manages a cache. I guess the client receives some manifest, checks the assets it already has in its cache, and then emits requests for the missing assets.

Is there any way to profile this part of the game? How much time is spent on connecting, sending/downloading, and processing the payload?

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest