Chunk loading improvement
-
- New member
- Posts: 9
- Joined: Fri Nov 25, 2011 20:37
Chunk loading improvement
A guy says that zlibing them before sending might make it faster. Now don't ask me how to do this, this is just a suggestion I read on a website and I decided to post it for the individual I heard it from here.
-
- Member
- Posts: 214
- Joined: Thu Nov 17, 2011 06:31
- Location: California, USA
I did a little testing and it seems that there is a problems with this idea. Using the compression code from http://www.firstobject.com/easy-zlib-c+ ... ession.htm (it is a wrapper around zlib 1.2.3) I was able to quickly setup a test. Using this really slowed down the server. The main reason is that the traffic between the server and clients is in mostly small packets. Most less than 20 bytes and the largest are around 512 bytes. These small packets were actually not compressing and, in fact, being made larger. Look at some sample data below (this is using Z_BEST_COMPRESSION):
Code: Select all
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 20 | To: 26 | Ratio: 130%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 94 | To: 66 | Ratio: 70.2128%
Packet Compression - From: 283 | To: 294 | Ratio: 103.887%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 512 | To: 523 | Ratio: 102.148%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 157 | To: 168 | Ratio: 107.006%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 173 | To: 177 | Ratio: 102.312%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 42 | To: 40 | Ratio: 95.2381%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 48 | To: 43 | Ratio: 89.5833%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 427 | To: 438 | Ratio: 102.576%
Packet Compression - From: 108 | To: 97 | Ratio: 89.8148%
Packet Compression - From: 512 | To: 523 | Ratio: 102.148%
Packet Compression - From: 390 | To: 401 | Ratio: 102.821%
Packet Compression - From: 108 | To: 99 | Ratio: 91.6667%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 512 | To: 523 | Ratio: 102.148%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 494 | To: 505 | Ratio: 102.227%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 20 | To: 28 | Ratio: 140%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
Packet Compression - From: 108 | To: 99 | Ratio: 91.6667%
Packet Compression - From: 11 | To: 19 | Ratio: 172.727%
-
- Member
- Posts: 214
- Joined: Thu Nov 17, 2011 06:31
- Location: California, USA
I should say that I edited socket.cpp to compress individual packets. It might be possible to compress and decompress higher up in the code and compress large clumps of map data. Here is my code. Look for COMPRESS_NETWORK to find changes.
Code: Select all
/*
Minetest-c55
Copyright (C) 2010 celeron55, Perttu Ahola <celeron55@gmail.com>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*/
#include "socket.h"
#include "debug.h"
#include <stdio.h>
#include <iostream>
#include <stdlib.h>
#include <errno.h>
#include "utility.h"
#define COMPRESS_NETWORK
#ifdef COMPRESS_NETWORK
// from http://www.firstobject.com/easy-zlib-c++-xml-compression.htm
#include "easyzlib.h"
#endif
// Debug printing options
// Set to 1 for debug output
#define DP 0
// This is prepended to everything printed here
#define DPS ""
bool g_sockets_initialized = false;
void sockets_init()
{
#ifdef _WIN32
WSADATA WsaData;
if(WSAStartup( MAKEWORD(2,2), &WsaData ) != NO_ERROR)
throw SocketException("WSAStartup failed");
#else
#endif
g_sockets_initialized = true;
}
void sockets_cleanup()
{
#ifdef _WIN32
WSACleanup();
#endif
}
Address::Address()
{
m_address = 0;
m_port = 0;
}
Address::Address(unsigned int address, unsigned short port)
{
m_address = address;
m_port = port;
}
Address::Address(unsigned int a, unsigned int b,
unsigned int c, unsigned int d,
unsigned short port)
{
m_address = (a<<24) | (b<<16) | ( c<<8) | d;
m_port = port;
}
bool Address::operator==(Address &address)
{
return (m_address == address.m_address
&& m_port == address.m_port);
}
bool Address::operator!=(Address &address)
{
return !(*this == address);
}
void Address::Resolve(const char *name)
{
struct addrinfo *resolved;
int e = getaddrinfo(name, NULL, NULL, &resolved);
if(e != 0)
throw ResolveError("");
/*
FIXME: This is an ugly hack; change the whole class
to store the address as sockaddr
*/
struct sockaddr_in *t = (struct sockaddr_in*)resolved->ai_addr;
m_address = ntohl(t->sin_addr.s_addr);
freeaddrinfo(resolved);
}
std::string Address::serializeString() const
{
unsigned int a, b, c, d;
a = (m_address & 0xFF000000)>>24;
b = (m_address & 0x00FF0000)>>16;
c = (m_address & 0x0000FF00)>>8;
d = (m_address & 0x000000FF);
return itos(a)+"."+itos(b)+"."+itos(c)+"."+itos(d);
}
unsigned int Address::getAddress() const
{
return m_address;
}
unsigned short Address::getPort() const
{
return m_port;
}
void Address::setAddress(unsigned int address)
{
m_address = address;
}
void Address::setAddress(unsigned int a, unsigned int b,
unsigned int c, unsigned int d)
{
m_address = (a<<24) | (b<<16) | ( c<<8) | d;
}
void Address::setPort(unsigned short port)
{
m_port = port;
}
void Address::print(std::ostream *s) const
{
(*s)<<((m_address>>24)&0xff)<<"."
<<((m_address>>16)&0xff)<<"."
<<((m_address>>8)&0xff)<<"."
<<((m_address>>0)&0xff)<<":"
<<m_port;
}
void Address::print() const
{
print(&dstream);
}
UDPSocket::UDPSocket()
{
if(g_sockets_initialized == false)
throw SocketException("Sockets not initialized");
m_handle = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
if(DP)
dstream<<DPS<<"UDPSocket("<<(int)m_handle<<")::UDPSocket()"<<std::endl;
if(m_handle <= 0)
{
throw SocketException("Failed to create socket");
}
/*#ifdef _WIN32
DWORD nonblocking = 0;
if(ioctlsocket(m_handle, FIONBIO, &nonblocking) != 0)
{
throw SocketException("Failed set non-blocking mode");
}
#else
int nonblocking = 0;
if(fcntl(m_handle, F_SETFL, O_NONBLOCK, nonblocking) == -1)
{
throw SocketException("Failed set non-blocking mode");
}
#endif*/
setTimeoutMs(0);
}
UDPSocket::~UDPSocket()
{
if(DP)
dstream<<DPS<<"UDPSocket("<<(int)m_handle<<")::~UDPSocket()"<<std::endl;
#ifdef _WIN32
closesocket(m_handle);
#else
close(m_handle);
#endif
}
void UDPSocket::Bind(unsigned short port)
{
if(DP)
dstream<<DPS<<"UDPSocket("<<(int)m_handle
<<")::Bind(): port="<<port<<std::endl;
sockaddr_in address;
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(port);
if(bind(m_handle, (const sockaddr*)&address, sizeof(sockaddr_in)) < 0)
{
#ifndef DISABLE_ERRNO
dstream<<(int)m_handle<<": Bind failed: "<<strerror(errno)<<std::endl;
#endif
throw SocketException("Failed to bind socket");
}
}
void UDPSocket::Send(const Address & destination, const void * data, int size)
{
bool dumping_packet = false;
if(INTERNET_SIMULATOR)
dumping_packet = (myrand()%10==0); //easy
//dumping_packet = (myrand()%4==0); // hard
if(DP){
/*dstream<<DPS<<"UDPSocket("<<(int)m_handle
<<")::Send(): destination=";*/
dstream<<DPS;
dstream<<(int)m_handle<<" -> ";
destination.print();
dstream<<", size="<<size<<", data=";
for(int i=0; i<size && i<20; i++){
if(i%2==0) DEBUGPRINT(" ");
DEBUGPRINT("%.2X", ((int)((const char*)data)[i])&0xff);
}
if(size>20)
dstream<<"...";
if(dumping_packet)
dstream<<" (DUMPED BY INTERNET_SIMULATOR)";
dstream<<std::endl;
}
else if(dumping_packet)
{
// Lol let's forget it
dstream<<"UDPSocket::Send(): "
"INTERNET_SIMULATOR: dumping packet."
<<std::endl;
}
if(dumping_packet)
return;
sockaddr_in address;
address.sin_family = AF_INET;
address.sin_addr.s_addr = htonl(destination.getAddress());
address.sin_port = htons(destination.getPort());
#ifdef COMPRESS_NETWORK
ezbuffer bufSrc( size );
memcpy( bufSrc.pBuf, data, size);
ezbuffer bufDest;
ezcompress( bufDest, bufSrc );
int sent = sendto(m_handle, (const char*)bufDest.pBuf, bufDest.nLen,
0, (sockaddr*)&address, sizeof(sockaddr_in));
if(sent != bufDest.nLen)
{
throw SendFailedException("Failed to send packet");
}
float compresssionRatio = 100.0 * (bufDest.nLen / (float)size);
dstream<<"Packet Compression - From: "<<size<<" | To: "<<bufDest.nLen<<" | Ratio: "<< compresssionRatio <<"%"<<std::endl;
#else
int sent = sendto(m_handle, (const char*)data, size,
0, (sockaddr*)&address, sizeof(sockaddr_in));
if(sent != size)
{
throw SendFailedException("Failed to send packet");
}
#endif
}
int UDPSocket::Receive(Address & sender, void * data, int size)
{
if(WaitData(m_timeout_ms) == false)
{
return -1;
}
sockaddr_in address;
socklen_t address_len = sizeof(address);
#ifdef COMPRESS_NETWORK
ezbuffer bufSrc( EZ_COMPRESSMAXDESTLENGTH(size) );
int received = recvfrom(m_handle, (char*)bufSrc.pBuf,
bufSrc.nLen, 0, (sockaddr*)&address, &address_len);
if(received < 0)
return -1;
ezbuffer bufDest;
ezuncompress( bufDest, bufSrc );
memcpy(data, bufDest.pBuf, bufDest.nLen);
received = bufDest.nLen;
#else
int received = recvfrom(m_handle, (char*)data,
size, 0, (sockaddr*)&address, &address_len);
if(received < 0)
return -1;
#endif
unsigned int address_ip = ntohl(address.sin_addr.s_addr);
unsigned int address_port = ntohs(address.sin_port);
sender = Address(address_ip, address_port);
if(DP){
//dstream<<DPS<<"UDPSocket("<<(int)m_handle<<")::Receive(): sender=";
dstream<<DPS<<(int)m_handle<<" <- ";
sender.print();
//dstream<<", received="<<received<<std::endl;
dstream<<", size="<<received<<", data=";
for(int i=0; i<received && i<20; i++){
if(i%2==0) DEBUGPRINT(" ");
DEBUGPRINT("%.2X", ((int)((const char*)data)[i])&0xff);
}
if(received>20)
dstream<<"...";
dstream<<std::endl;
}
return received;
}
int UDPSocket::GetHandle()
{
return m_handle;
}
void UDPSocket::setTimeoutMs(int timeout_ms)
{
m_timeout_ms = timeout_ms;
}
bool UDPSocket::WaitData(int timeout_ms)
{
fd_set readset;
int result;
// Initialize the set
FD_ZERO(&readset);
FD_SET(m_handle, &readset);
// Initialize time out struct
struct timeval tv;
tv.tv_sec = 0;
tv.tv_usec = timeout_ms * 1000;
// select()
result = select(m_handle+1, &readset, NULL, NULL, &tv);
if(result == 0){
// Timeout
/*dstream<<"Select timed out (timeout_ms="
<<timeout_ms<<")"<<std::endl;*/
return false;
}
else if(result < 0){
// Error
#ifndef DISABLE_ERRNO
dstream<<(int)m_handle<<": Select failed: "<<strerror(errno)<<std::endl;
#endif
#ifdef _WIN32
int e = WSAGetLastError();
dstream<<(int)m_handle<<": WSAGetLastError()="<<e<<std::endl;
if(e == 10004 /*=WSAEINTR*/)
{
dstream<<"WARNING: Ignoring WSAEINTR."<<std::endl;
return false;
}
#endif
throw SocketException("Select failed");
}
else if(FD_ISSET(m_handle, &readset) == false){
// No data
//dstream<<"Select reported no data in m_handle"<<std::endl;
return false;
}
// There is data
//dstream<<"Select reported data in m_handle"<<std::endl;
return true;
}
-
- New member
- Posts: 9
- Joined: Fri Nov 25, 2011 20:37
Another suggestion
>Try to optimize the chunk mesh with a lib like : http://www.cs.cmu.edu/~quake/triangle.html
This way you could cut down on the size of the vertexbuffers used.
>This could help with the chunk loading times and overal performance.
Bare in mind this is not my suggestion
>Try to optimize the chunk mesh with a lib like : http://www.cs.cmu.edu/~quake/triangle.html
This way you could cut down on the size of the vertexbuffers used.
>This could help with the chunk loading times and overal performance.
Bare in mind this is not my suggestion
-
- New member
- Posts: 9
- Joined: Fri Nov 25, 2011 20:37
Who is online
Users browsing this forum: No registered users and 13 guests