A good idea to improve the sub game loading time.

patrick55
Member
 
Posts: 25
Joined: Sat Jul 30, 2016 03:16

A good idea to improve the sub game loading time.

by patrick55 » Mon Jun 24, 2019 15:39

Now an MT client must get lots of subgame images files from the server or a HTTP server, and load them one by one(lots of file open operation). This will cost a lot of time.
Here is my idea:
Server: Merge all the images to some big images. When you finish your subgame, run a script to merge all the images and put them in a directory.
Client: Only get those merge big images and each image's coordinate info.
Why this can save the loading time a lot?
1. Transfer time. Only transfer the merged big images, instead of hundreds or thousands of small images through the internet.
2. Loading files time. Only open the big merge files, save a lot of the files open time.

Leave your comment :)
 

User avatar
Linuxdirk
Member
 
Posts: 2106
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Mon Jun 24, 2019 16:25

Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.
 

User avatar
Krock
Developer
 
Posts: 4341
Joined: Thu Oct 03, 2013 07:48
Location: Switzerland
GitHub: SmallJoker

Re: A good idea to improve the sub game loading time.

by Krock » Mon Jun 24, 2019 17:01

So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.

I proposed the same idea a few year ago, but can't find the log entry. However, sfan5's point was (roughly): "Nothing beats the filesystem. How should a database be faster than that?".
>> Mod Search Engine << - Mods by Krock - DuckDuckGo mod search bang: !mtmod <keyword here>
 

patrick55
Member
 
Posts: 25
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Tue Jun 25, 2019 01:37

Maybe one simple way is just zip the subgame assets and transfer one zip file to the client.
Linuxdirk wrote:Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.
 

patrick55
Member
 
Posts: 25
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Tue Jun 25, 2019 02:06

Save the internet transfer time will save a lot, I think. Zip the subgame assets and send one zip file to the client :>.
The assets loading time in the client side may not save a lot(all files in a database VS all files in the filesystem), because we just load a few hundreds or thousands of small file. That’s a small job for a model CPU, I think. [Need some stats]

Krock wrote:So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.

I proposed the same idea a few year ago, but can't find the log entry. However, sfan5's point was (roughly): "Nothing beats the filesystem. How should a database be faster than that?".
 

User avatar
Linuxdirk
Member
 
Posts: 2106
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Tue Jun 25, 2019 08:41

Krock wrote:So we need some stats about how long a file opening operation takes (Windows with path hooks, and Linux). An option would be to save them into an sqlite3 database, but there you'll have to deal with the row lookup time.

Why not archiving all media on server startup and send the archive to the client when it connects? This would save individual files being read and sent every time a client connects. I assume due to the overhead of reading the files and establishing a connection for each of the files sending individual files is slower than sending only one file.
 

User avatar
Pyrollo
Member
 
Posts: 313
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: A good idea to improve the sub game loading time.

by Pyrollo » Tue Jun 25, 2019 08:47

Linuxdirk wrote:Yes, all media should be archived and send as one compressed single file/stream instead of loading every media individually.


Media consisting in images and sounds (PNG, OGG) are already compressed. No need to compress then once again. Archiving should be enough.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
Linuxdirk
Member
 
Posts: 2106
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Tue Jun 25, 2019 08:58

Pyrollo wrote:Media consisting in images and sounds (PNG, OGG) are already compressed.

By the server? I've seen dozen of mods where neither the PNGs nor the OGGs were optimized.
 

User avatar
Pyrollo
Member
 
Posts: 313
Joined: Mon Jan 08, 2018 15:14
Location: Paris
GitHub: pyrollo
In-game: Naj

Re: A good idea to improve the sub game loading time.

by Pyrollo » Tue Jun 25, 2019 12:43

Linuxdirk wrote:By the server? I've seen dozen of mods where neither the PNGs nor the OGGs were optimized.


That's another problem. In my opinion re-compressing (even badly) compressed data does not make sense. I'm not even sure of the gain.
[ Display Modpack ] - [ Digiterms ] - [ Crater MG ] - [ LATE ]
 

User avatar
Linuxdirk
Member
 
Posts: 2106
Joined: Wed Sep 17, 2014 11:21
Location: Germany
In-game: Linuxdirk

Re: A good idea to improve the sub game loading time.

by Linuxdirk » Tue Jun 25, 2019 12:50

Yeah, then leave it uncompressed. Would still be better than reading and sending all media one-by-one on every client connection.
 

patrick55
Member
 
Posts: 25
Joined: Sat Jul 30, 2016 03:16

Re: A good idea to improve the sub game loading time.

by patrick55 » Sat Jul 27, 2019 03:21

Will this be support in the future? or some other way to improve the first time loading?
 

User avatar
Festus1965
Member
 
Posts: 843
Joined: Sun Jan 03, 2016 11:58
Location: Thailand - Chiang Mai (+5-6h to MEZ)
In-game: Thomas Explorer

Re: A good idea to improve the sub game loading time.

by Festus1965 » Sat Jul 27, 2019 04:10

Make sure that the client itself is able to check weather this mod is local available, and load then from there local.
So gamer who want to force speed and save time, just download that mods themself into client and ... happy ?
Server: still up, see some gamer - but there is a new 1st important thread "climate change" for me. Something I know since 1992 - had seen the melting glacier in Switzerland.
 

Astrobe
Member
 
Posts: 231
Joined: Sun Apr 01, 2018 10:46

Re: A good idea to improve the sub game loading time.

by Astrobe » Sun Jul 28, 2019 09:32

It looks like MT already manages a cache. I guess the client receives some manifest, checks the assets it already has in its cache, and then emits requests for the missing assets.

Is there any way to profile this part of the game? How much time is spent on connecting, sending/downloading, and processing the payload?
 


Return to Feature Discussion



Who is online

Users browsing this forum: No registered users and 2 guests