Finally, MYSQL + code::blocks!

Hello everybody, today I'll make a "postmortem analysis" and will install and set up MySql server

     <Been out for a long time <again>, had many things to do and my project got stucked again, but it didn't get forgotten! I have been reading all the code again and had time to think about it, the things I have done right and which things should I change / improve, and made an small list:>

GOOD THINGS

1. The server actually works!
2. Client program able to create players, connect, and "move around"
3. Worked quite fast in a desktop computer using small amounts of RAM

BAD THINGS

1. Ugly code, I feel really ashamed about that, but it is the result of writing code in a "fast 15 min way"
2. Server is unstable, yeah, after fixing many, many bugs it did worked, but still had things to fix.
3. I was not using a real scenario , data was not being loaded from server, so player creation, maps, etc was everything client-side, obviously that is not good, for many reasons (security, for example)
4. World didn't had persistent data, players, objects, etc were not stored  after closing client program

In other words I just did a network demo, I was so anxious to make the server work with thousands of clients that I didn't realized I was starting to loose the focus in my objective: MAKE THE GAME!

So I took a hard decision, dropped the code and to make a second server with all the good things the first server had + all the things it SHOULD HAVE like persistent data, and a true server-client model.

Having persistent data meant to use some kind of static storage, so I thought in using just plain files plus standard c++ io functions to write data, but I dropped the idea fast because there are already libraries to solve the "writing another text parser" like TinyXml , but after playing with that again I found that if I used that library I will be making two times the work because:

1. First I will have to write all the data in .xml files
2. To later rewrite the code again to switch to a real database..

I went back to my very first objectives and the name MySql, came to my mind...

<NOTE: I have completely dropped support for windows in the server part, that means I wont be posting how to install and make it run under Windows OS family, basically I made that because that's the OS I will be using to make and run the server and I cant loose more time in something I wont be using, although anybody could check how to download and install the required libraries for the OS you choose..>

So I installed the mysql server:

sudo apt-get install mysql-server

It just asked me for the "root" database password, it is very important because it will be used to connect to MySql and use the database, NOTE: it is not the linux root account so you can / should have different passwords .

I installed too MySql tuner:

sudo apt-get install mysqltuner

Used to get advice to tune the databases

And the client libraries:

sudo apt-get install libmysqlclient-dev

Used to connect from the client app

To check that the MySql server is running:

sudo netstat -tap | grep mysql

  
You can restart the service with:

sudo service mysql restart

 
<You can see installing MySql server is quite simple, in the next post I will write about comand-line database management, and finally start using it with code::blocks>

Rpg Client update: Rotate players

Welcome back!


Player's point of view is quite clear now...

Since I decided to continue the project I started to think again in all the things I could make to improve it, and the first thing was to  make it clear where players / foes were looking at, it's basic to make you know:

1. Where are you going (the red dot did not helped a lot)
2. Where are going all the other players / enemies
3. Can I ambush somebody?, am I being ambushed?

To make client rotate images I just needed a function to rotate a tile certain angle, I didn't had to modify the server code because I already included angle in the players data, I thought how to rotate that and finally I decided to use a library from a good friend that was open source and based in the standard library SGE:

SPRIG: The SDL Primitive Generator

A great job from Jhonny D

To use the library you will need the subversion client, (in linux "sudo apt-get install subversion", in Windows you can use Tortoise SVN client, for example)

To download / install it under linux:
svn checkout http://sprig.googlecode.com/svn/trunk/ sprig
cd sprig
make
sudo make install

(Haven't tested it under windows yet)

I created a new function: RotateBlitTileset( x,y,image,angle), that functions draws the rotated tile (image) at x,y.

void Engine::RotateBlitTileset(int x, int y, int tile, float Angle) //Rotate and Blits a tile from our tileset to the screen
{
    SDL_Surface * tmpBlit = SDL_CreateRGBSurface(SDL_SWSURFACE|SDL_SRCCOLORKEY |SDL_RLEACCEL,TileSize,TileSize,32,0,0,0,0); 
    SDL_Surface * tmpBlit2 =  SDL_CreateRGBSurface(SDL_SWSURFACE|SDL_SRCCOLORKEY |SDL_RLEACCEL,TileSize,TileSize,32,0,0,0,0);
    //1-fill temporary surface with pink
    SDL_FillRect(tmpBlit, 0, SDL_MapRGB(tmpBlit->format, 255, 0, 255));

    start.x= (tile%10)*TileSize; //Column = tile % 8 (0-7)
    start.y= (tile/10)*TileSize; //Row = tile / 8 (0-7)
    start.w=TileSize;
    start.h=TileSize;

    end.x =0; // X
    end.y =0; // Y
    end.w = TileSize;
    end.h = TileSize;

    //2-Blit the desired tile to the temporary surface
    SDL_BlitSurface(tileset, &start, tmpBlit, &end); 
    //3-I call sprig function SPG_Rotate() to rotate the temporary surface and get the resulting surface into a 2nd temporary surface
    tmpBlit2 = SPG_Rotate(tmpBlit, Angle -90,  SDL_MapRGB(tmpBlit-> format, 255, 0, 255)); 
    SDL_SetColorKey(tmpBlit2,SDL_SRCCOLORKEY |SDL_RLEACCEL,SDL_MapRGB(tmpBlit2-> format, 255, 0, 255));//set the color key
    start.x=0;
    start.y=0;
    start.w=TileSize;
    start.h=TileSize;

    //set the cliping for the destination surface (the screen)
    end.x =x; // X
    end.y =y; // Y
    end.w = TileSize;
    end.h = TileSize;

    SDL_BlitSurface(tmpBlit2, NULL, screen, &end); //Blits the rotated tile to the screen
    SDL_FreeSurface( tmpBlit);
    SDL_FreeSurface( tmpBlit2);
}

Notice I can't just call Sprig rotate function because I would get a rotated tile with pink background, so I have to use the tmp surface to set the color-key, that way is not optimized at all so I will think in a way to make it faster later, but at the moment is ok.

In the next post: Let's chat!

I'm alive!!

I could tell you I had a very bad car accident, maybe that I got into the drugs , got kidnapped or abducted but I really can't say other thing than the truth:

I have not made anything in the project for about two months :(

Got quite frustrated trying to make the project to work the non-blocking way, and in fact I made it to "work" quite easy, I was very happy because I thought I could make the server work faster and use less resources than I was using...

WRONG!, server worked fine with a low load (say less than 50 users), but when I started to add more users it just collapsed with nasty segmentation fault errors, sometimes when a client disconnected it just blocked the server, I didn't had all those problems with the "old" server and I although I tried to debug the code, errors never went away..

But that's not the worst thing, worst think is performance was poor.. I never felt the non-blocking approach was faster, basically because my blocking approach used many, many threads, a good approach if you use a multi-core processor, but the non blocking used only 3-4 threads for everything, with separate functions like accepting new clients, updating player states or sending data, so some threads have little load while other threads couldn't handle the load at all.

What happens when you have problems? usually it starts to appear more like mushrooms (Damn Murphy's law ), and I was just trying to solve other problems that were not related to the blog...
   
And then I let the blog die slowly, but even without posting more blogs many people entered and read it, and today I arrived to 3k visits and I knew I had to do something, like trying to reanimate the zombie-blog.

Then I remembered a comment I got from wise Amit Patel : How many users do I plan to have? maybe the problem is not being able to handle 10k users·

Now I can say I was distracting myself from my biggest objective, actually making the game!!! so I put my hands on the blocking sockets code again and re-started again, it has been hard, but in this week I expect to launch two updates for the game that will make it a little more user friendly and usable..

Thanks for reading and see you soon (promise)

Blocking-sockets Windows binaries released!

        I have compiled the projects under MS Windows, surprisingly almost everything worked as expected and the binaries are available in  Downloads(-RPGServer / Client / Bot Tutorial ->Windows Binaries), feel free to download and test it.

      There were only two problems:

1. Server include paths were wrong, I had <SDL/SDL.h> so I replaced them with <SDL.h> and compiled fine
2. The bot was giving me errors in the line srand(time(NULL)), solved with a #include <time.h>, curious it worked fine under linux.

When running there is only an issue: no cout output under windows so server starts, and actually works! but there is no output at all, anyway it's ok to test.

        The server has certain limits like not being able to accept more than 1018 clients and to use too much resources, but if you plan to work with a limited amount of users (200-300) it will make the job.

        I would be glad to hear (constructive) comments about the code, the bugs you may found and well, basically your opinion.

Thanks to everybody!

Blocking-Sockets server code released

Hello again!

         After I decided to stop the development for a while I tought it would be nice to share the code with everybody interested, so after a huge cleanup (probably not enought) I have uploaded the code for the server, the client and a silly bot I did to test multiple client connections, they are now in Downloads (-RPGServer / Client / Bot Tutorial -> Source code), feel free to download and test it.

        The server has certain limits like not being able to accept more than 1018 clients and to use too much resources, but if you plan to work with a limited amount of users (200-300) it will make the job.

     Server is untested under windows (it works!), client and bot wont care too much for the OS (will work in linux / windows), I don't have a machine with Mac OS X to test so I couldn't test it (any volunteer to try with an apple computer?)

        I would be glad to hear (constructive) comments about the code, the bugs you may found and well, basically your opinion.

Thanks to everybody!

STOP & GO

Blocking socket limits

After a long time trying to fix errors / improve performance I have arrived to a sad point: My server with blocking sockets wont support more than 1.018 users, I arrived that limit making an stress test trying to arrive to the 2.000 users barrier, no more clients were accepted after client 1.018.

The problem comes from the design I choose to make the server, I was using blocking sockets, that means for every client we need a thread to wait for data, 1.018 users = 1.018 threads + threads used by the server reached the default limit in Linux (1.024), I have been reading about that , and I know that limit can be modified but it's not a real solution for the problem because my test computer used 90% processor time to handle "only" 1.000 users, so I have to switch to another technology..

That technology is non-blocking sockets, because basically with a single thread you could theoretically handle 1.000 maybe 2.000 connected users, I have been speaking with people that has worked in the game industry and for linux servers the best / fastest way is to use epoll  , epoll is basically used to handle I/O from many file descriptors (connections) in real time, but using epoll to me would have meant throwing away all what I have been working on in the last 3 months.... (Thanks Neils for showing me the right way!)

Finally I decided to use socket sets  , that is the SDL_net non-blocking approach  for that problem , so basically I have to re-write all the networking code for my server...

I have decided to take a rest in the development and  I have started a collaboration with a friend to make a multiplayer platformer, it is very good because I have time to think in other things and learn, I'm just finding that I was making many errors and many things could be improved.

See you soon

So I want make an MMORPG, Where do I start from? PART 4

Messaging and optimizations:

<Messaging:>

Networking is the core of an mmorpg, and just in the top of the network we have the messaging system. What we need is to create a way to communicate between server and clients.

Clients will send commands to the server for example: move forward, turn right, atack!... In the server we could have comands to send the map, the clients data like player / npc position,etc...

Most of the people (should) use serialization to send data, for example an item could be packed in an object / serialize it and send over the net, in the destination it would be deserialized and here we have our object/item again, but I have used another way to transfer messages that is hard to code but easier to understand: Plain-text messages.

Probably now you are wondering about how does it works, suppose we want our client to send to the server we are moving forward, then we will just send a command telling the server we have pressed "forward", suppose we decide the text command is going to be "/acce", when the server receives a string "/acce" from our client it will understand the client is moving forward so it will move the client forward in the map and return the new updated position to the client; The server will send "/ping" to the clients every x seconds ,when the clients receive "/ping" it will return "/pong" to measure delay in the communication between server/clients.

What are the advantages when using this method? basically it simplifies debugging , it's easier to understand an incoming packet when you can actually see the command and not a binary-format hex value, basically it is good for didactic purposes.

The disadvantages? performance and scalability , "/acce" uses 5 bytes to send "forward", too much for a simple commands in the other side for every new command you have to modify both client and server, that can be overwhelming!

I resume here all the commands I used for the demo:

Client side:

• /nick > Set players nickname
• /imag > Set players image
• /acce  > Move forward
• /dece > Move backward
• /righ > Turn right
• /left > Turn left
• /idle > Stay idle
• /pong > Answer a /ping request
• /quit > Exit server

Server side:

• /SUID > Send Unique identifier to the player
• /upda > Send non-player updates (X,Y,Nicknames...)
• /updb > Send player updates (X,Y...)
• /ping > Send a /ping request to calculate client latency
• /kill > Sent when a player logoffs

So we have our server running, in our client we create a player and then connect to the server, then the client will wait until our server sends "/SUID" with the UID for our client, the UID wont change until we logoff, after receiving the UID our client will send our nick and image, that is how our "enemies" will see us, then we are ready to explore.

We will start moving around, when we move we will be sending "/acce", "/dece", "/righ" and "/left" to the server that will update our position in the map, if we don't move a "/idle" message will be sent, when the server updates our position it will mark if the client has changed state so it will send the command "/upda" with our updated x and y to the clients in a range (500px), so server wont send updates to clients that can't see our moves, that will limit too network traffic.

When a clients exits the game it sends a "/quit" to the server, server will clean the players data and then it will send a "/kill" command to the clients so they will clean the exiting client...

<9 messages for the client + 5 messages for the server = total 14 diferent messages! , Probably the simplest messaging system you will have ever seen>


<Optimizations:>

After I made the server I saw two simple ways to improve performance: first one in the client second one in the server:

1. In the client I was sending 25 messages/ second to the server: I was sending data no mather if it was moving or if it was idle, but I thought that instead sending messages all the time I could only send data when a change was detected, for example: the client is idle, so the first time, it will send "/idle", but if it keeps standing still it wont keep sending "/idle", server will save the command until a new one arrives, so after a while I start to move forward, I send a "/acce" to the server first time, but again if I keep on moving forward it wont send again "/acce".. that divided outgoing messages from 25 to 2 or 3 messages when moving and 0! standing still.
2. In the server I checked the messages and found that "/upda" >> (client updates) was about 99% of the outgoing traffic, that generated a huge problem when a client was in a crowded area, suppose there are 50 clients near, all them are moving so every client is generating 50 updates / frame, with a frame rate of 20 /s that was 50 * 50 * 20 = 50.000 messages in a second!!! that was making the server collapse, so I had an idea, why not nest packets? when a "/upda" is generated I made the server to don't send that message instantly , instead of that it "packed"  into a bigger message, when more "/upda" messages arraived they were packed up to 10 updates in a bigger one, so in the same "crowded"  are messages sent will be reduced to 5.000, that was much more reasonable.

 With that pair of improvements server was able to handle 1000 clients at the same time with almost no delays , traffic had spikes of 18.000 incoming messages and 200.000 messages sent in a second, but then I reached blocking socket's limits...


<Blocking versus non blocking sockets in the next post>