I'm alive!!

I could tell you I had a very bad car accident, maybe that I got into the drugs , got kidnapped or abducted but I really can't say other thing than the truth:

I have not made anything in the project for about two months :(

Got quite frustrated trying to make the project to work the non-blocking way, and in fact I made it to "work" quite easy, I was very happy because I thought I could make the server work faster and use less resources than I was using...

WRONG!, server worked fine with a low load (say less than 50 users), but when I started to add more users it just collapsed with nasty segmentation fault errors, sometimes when a client disconnected it just blocked the server, I didn't had all those problems with the "old" server and I although I tried to debug the code, errors never went away..

But that's not the worst thing, worst think is performance was poor.. I never felt the non-blocking approach was faster, basically because my blocking approach used many, many threads, a good approach if you use a multi-core processor, but the non blocking used only 3-4 threads for everything, with separate functions like accepting new clients, updating player states or sending data, so some threads have little load while other threads couldn't handle the load at all.

What happens when you have problems? usually it starts to appear more like mushrooms (Damn Murphy's law ), and I was just trying to solve other problems that were not related to the blog...
   
And then I let the blog die slowly, but even without posting more blogs many people entered and read it, and today I arrived to 3k visits and I knew I had to do something, like trying to reanimate the zombie-blog.

Then I remembered a comment I got from wise Amit Patel : How many users do I plan to have? maybe the problem is not being able to handle 10k users·

Now I can say I was distracting myself from my biggest objective, actually making the game!!! so I put my hands on the blocking sockets code again and re-started again, it has been hard, but in this week I expect to launch two updates for the game that will make it a little more user friendly and usable..

Thanks for reading and see you soon (promise)

Blocking-sockets Windows binaries released!

        I have compiled the projects under MS Windows, surprisingly almost everything worked as expected and the binaries are available in  Downloads(-RPGServer / Client / Bot Tutorial ->Windows Binaries), feel free to download and test it.

      There were only two problems:

1. Server include paths were wrong, I had <SDL/SDL.h> so I replaced them with <SDL.h> and compiled fine
2. The bot was giving me errors in the line srand(time(NULL)), solved with a #include <time.h>, curious it worked fine under linux.

When running there is only an issue: no cout output under windows so server starts, and actually works! but there is no output at all, anyway it's ok to test.

        The server has certain limits like not being able to accept more than 1018 clients and to use too much resources, but if you plan to work with a limited amount of users (200-300) it will make the job.

        I would be glad to hear (constructive) comments about the code, the bugs you may found and well, basically your opinion.

Thanks to everybody!

Blocking-Sockets server code released

Hello again!

         After I decided to stop the development for a while I tought it would be nice to share the code with everybody interested, so after a huge cleanup (probably not enought) I have uploaded the code for the server, the client and a silly bot I did to test multiple client connections, they are now in Downloads (-RPGServer / Client / Bot Tutorial -> Source code), feel free to download and test it.

        The server has certain limits like not being able to accept more than 1018 clients and to use too much resources, but if you plan to work with a limited amount of users (200-300) it will make the job.

     Server is untested under windows (it works!), client and bot wont care too much for the OS (will work in linux / windows), I don't have a machine with Mac OS X to test so I couldn't test it (any volunteer to try with an apple computer?)

        I would be glad to hear (constructive) comments about the code, the bugs you may found and well, basically your opinion.

Thanks to everybody!

STOP & GO

Blocking socket limits

After a long time trying to fix errors / improve performance I have arrived to a sad point: My server with blocking sockets wont support more than 1.018 users, I arrived that limit making an stress test trying to arrive to the 2.000 users barrier, no more clients were accepted after client 1.018.

The problem comes from the design I choose to make the server, I was using blocking sockets, that means for every client we need a thread to wait for data, 1.018 users = 1.018 threads + threads used by the server reached the default limit in Linux (1.024), I have been reading about that , and I know that limit can be modified but it's not a real solution for the problem because my test computer used 90% processor time to handle "only" 1.000 users, so I have to switch to another technology..

That technology is non-blocking sockets, because basically with a single thread you could theoretically handle 1.000 maybe 2.000 connected users, I have been speaking with people that has worked in the game industry and for linux servers the best / fastest way is to use epoll  , epoll is basically used to handle I/O from many file descriptors (connections) in real time, but using epoll to me would have meant throwing away all what I have been working on in the last 3 months.... (Thanks Neils for showing me the right way!)

Finally I decided to use socket sets  , that is the SDL_net non-blocking approach  for that problem , so basically I have to re-write all the networking code for my server...

I have decided to take a rest in the development and  I have started a collaboration with a friend to make a multiplayer platformer, it is very good because I have time to think in other things and learn, I'm just finding that I was making many errors and many things could be improved.

See you soon

So I want make an MMORPG, Where do I start from? PART 4

Messaging and optimizations:

<Messaging:>

Networking is the core of an mmorpg, and just in the top of the network we have the messaging system. What we need is to create a way to communicate between server and clients.

Clients will send commands to the server for example: move forward, turn right, atack!... In the server we could have comands to send the map, the clients data like player / npc position,etc...

Most of the people (should) use serialization to send data, for example an item could be packed in an object / serialize it and send over the net, in the destination it would be deserialized and here we have our object/item again, but I have used another way to transfer messages that is hard to code but easier to understand: Plain-text messages.

Probably now you are wondering about how does it works, suppose we want our client to send to the server we are moving forward, then we will just send a command telling the server we have pressed "forward", suppose we decide the text command is going to be "/acce", when the server receives a string "/acce" from our client it will understand the client is moving forward so it will move the client forward in the map and return the new updated position to the client; The server will send "/ping" to the clients every x seconds ,when the clients receive "/ping" it will return "/pong" to measure delay in the communication between server/clients.

What are the advantages when using this method? basically it simplifies debugging , it's easier to understand an incoming packet when you can actually see the command and not a binary-format hex value, basically it is good for didactic purposes.

The disadvantages? performance and scalability , "/acce" uses 5 bytes to send "forward", too much for a simple commands in the other side for every new command you have to modify both client and server, that can be overwhelming!

I resume here all the commands I used for the demo:

Client side:

• /nick > Set players nickname
• /imag > Set players image
• /acce  > Move forward
• /dece > Move backward
• /righ > Turn right
• /left > Turn left
• /idle > Stay idle
• /pong > Answer a /ping request
• /quit > Exit server

Server side:

• /SUID > Send Unique identifier to the player
• /upda > Send non-player updates (X,Y,Nicknames...)
• /updb > Send player updates (X,Y...)
• /ping > Send a /ping request to calculate client latency
• /kill > Sent when a player logoffs

So we have our server running, in our client we create a player and then connect to the server, then the client will wait until our server sends "/SUID" with the UID for our client, the UID wont change until we logoff, after receiving the UID our client will send our nick and image, that is how our "enemies" will see us, then we are ready to explore.

We will start moving around, when we move we will be sending "/acce", "/dece", "/righ" and "/left" to the server that will update our position in the map, if we don't move a "/idle" message will be sent, when the server updates our position it will mark if the client has changed state so it will send the command "/upda" with our updated x and y to the clients in a range (500px), so server wont send updates to clients that can't see our moves, that will limit too network traffic.

When a clients exits the game it sends a "/quit" to the server, server will clean the players data and then it will send a "/kill" command to the clients so they will clean the exiting client...

<9 messages for the client + 5 messages for the server = total 14 diferent messages! , Probably the simplest messaging system you will have ever seen>


<Optimizations:>

After I made the server I saw two simple ways to improve performance: first one in the client second one in the server:

1. In the client I was sending 25 messages/ second to the server: I was sending data no mather if it was moving or if it was idle, but I thought that instead sending messages all the time I could only send data when a change was detected, for example: the client is idle, so the first time, it will send "/idle", but if it keeps standing still it wont keep sending "/idle", server will save the command until a new one arrives, so after a while I start to move forward, I send a "/acce" to the server first time, but again if I keep on moving forward it wont send again "/acce".. that divided outgoing messages from 25 to 2 or 3 messages when moving and 0! standing still.
2. In the server I checked the messages and found that "/upda" >> (client updates) was about 99% of the outgoing traffic, that generated a huge problem when a client was in a crowded area, suppose there are 50 clients near, all them are moving so every client is generating 50 updates / frame, with a frame rate of 20 /s that was 50 * 50 * 20 = 50.000 messages in a second!!! that was making the server collapse, so I had an idea, why not nest packets? when a "/upda" is generated I made the server to don't send that message instantly , instead of that it "packed"  into a bigger message, when more "/upda" messages arraived they were packed up to 10 updates in a bigger one, so in the same "crowded"  are messages sent will be reduced to 5.000, that was much more reasonable.

 With that pair of improvements server was able to handle 1000 clients at the same time with almost no delays , traffic had spikes of 18.000 incoming messages and 200.000 messages sent in a second, but then I reached blocking socket's limits...


<Blocking versus non blocking sockets in the next post>

So I want make an MMORPG, Where do I start from? PART 3

<It had been a long time since I made my last post , part of the problem was because I've been on a trip to Paris (A city everybody should visit at least once in their life), another part of the problem is my fight with non - blocking sockets, but now i'm back and I feel refreshed , with energy to continue with the project.>

Today I'm going to speak about a few things: the server structure ,about messaging  and finally about blocking sockets vs non blocking sockets:

Server Structure:

I plan to make my server architecture based in something like that:

Clients (a.k.a. players) will connect to a login/ authentication server , when validated they will connect to the proxy server, proxy will redirect players to the less loaded server, (balancing server load), server instances will be deployed across the servers on demand ( if a server is more powerful it could handle more instances), all server instances + login server + proxy srv will access the SQL server to write /read all persistent data to be stored, such as:

• Usernames / passwords for the login server
• Maps , players and objects for the server instances
• Number of instances, server instance sockets and health status of the instances (load and failures) for the proxy server.

That is a draft of the design I plan to do, at the moment I'm just working to make a server instance + sql server working, the remaining features will be done later (LOGIN / PROXY).

<Messaging in the next post>

So I want make an MMORPG, Where do I start from? PART 2

Welcome back  to the MMORPG series, part 2!

<Incoming connections....>

After expending one year to learn c++ and SDL I felt ready for the next step: Networking.

I did a chat server just to learn how SDL_net works, It is very interesting and I recommend to try to code one to everybody who dares to create an RPG/MMORPG server because it is basically the same with lower processing needs..

In a chat server (basically ) what happens is:

1. Clients connect to the server
2. They Login with user name / password
3. Every time they type something and press <ENTER> message is sent to all the other connected clients.

(I'm not including here channels and other things... it is just for demonstration purposes only)

If you want you can Download the sources for the server and client and test it, (Unfortunately it only works in Linux because if you run it under windows, it will redirect all the output to a file, so basically is useless..).

But for the MMORPG server I had to analyze it before starting....

What happens (basically )in a MMORPG server is:

(Client side:)

1. Clients connect to the server
2. They Login with user name / password (Or they can create a new user...)
3. Wait for the server to send UID (See below), and world data
4. Transmit actions to the server (update position, use skills, pic objects, attack, etc)
5. Update world state from data received from the server.
6. Goto point 4...

 (Server side:)

1. Allocate resources to accept and process data from incoming clients  (That includes queues and threads to process data)
2. For every client that logs into the server, assign an Unique identifier that is not going to change until client disconnects, the UID is going to be the identifier for all the process in the server (and the client too), the client will receive too the basic world data to start playing (like map data, players and enemies...)
3. Server processes incoming data from connected clients (that includes moving around / and all the skills the player have), for example a player moves forward, server receives a message from client(x) that wants to move forward, so it's going to increase client(x)->Speed
4. Update world state , here it comes were timing is important, the amount of times a world update is applied will change the speed of the game for ALL PLAYERS, for example, players wont send position changes to the server, but  move forward, backward, turn right/left commands, what happens when a world update is executed is that clients will move according to their speed and direction and update X/Y position, so no matter if a client tries to hack the client to run at 100 FPS and not the 25FPS set as default, it wont run faster...
5. Make a list of the players that have changed state (position, skills used)..
6. Send updated data from the clients with changed state to all the clients connected and in a X range...
7. Go to point 3....

I know it is a very raw way to describe what the server does, and you may be missing many features such as maps, skills, enemies / NPCS, AI, etc... but my aim is to make it a simple yet stable network server, then I will add the persistence to do world with  a database (Mysql).

To describe how my server works I'm going to explain how it evolved so it's easier to understand why I changed things...

So let's start to speak with the 3 versions / milestones my server has reached:

<Version Number 0.1:>

Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 1xProcessor thread:It processes the data into the input queues and updates wold data
• 1xUpdater thread: Insert updated data to the output queues
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.
• 1xSender thread/Client:Sends data from the output queue to the client.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
• Input queue:Every client has one, used to store incoming data
• Output Queue:Every client has one, used to store outgoing data

Good things of the design:

• Very simple.
• Functional.

Bad Things of the design:

• Very bad performance, when there was more than 3 clients connected you started to feel it lagging even working with the loop-back interface...
• There is no time control, so the server will have to be confident in the clients <Horrid mistake>
• Wasted data: Two threads per client, two queues per client.. too much

What I learned from this version:
Threads are a must when speaking of a socket server, so in this very first version what I saw is I needed to use them in a more efficient way, I was using many threads to just send and receive data to the client but only three to process data.. other big mistake was to use so many queues in the clients, it was a pain in the a** to check them and probably not efficient at all.., with all this things in my head I decided to make an updated version:


<Version Number 0.1.7:>


Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
  
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 2xProcessor thread:It processes the data into the input queue and updates wold data
• 1xUpdater Thread, inserts into output queue the data to send
• XxSender threads: Send data from the output queue
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
  
• Input queue:One in the server, used to store incoming data
• Output Queue:One in the server, used to store outgoing data

Good things of the design:

• Centralizing queues make it to waste less data.
• Having more threads to process data makes it to be able to handle more clients (watch the screen-shot in the top of the post), 101 clients connected...
• Timing centralized in the server, more secure now
  

Bad Things of the design:

• Still bad performance,after adding 20 clients it started to degrade performance no matter how many threads you added to the server

What I learned from this version:
Threads collide, if you use semaphores / mutexes you will prevent it from happening but, after adding 5 threads to process data from a queue, there is no performance improvement because threads have to wait each-other. It was a design fault, to continue increasing performance I had to add more queues with different semaphores so the threads wont collide so much, so that brings me to the last version:



<Version Number 0.1.12:>

<867 Clients connected at the same time, everyone constantly moving sending data to the server, 25 incoming packets /s per client, total, +2.000 incoming messages processed/s,  peak:18.000 messages SENT in a second, that is like 18 packets /millisecond, latency/lag for the clients from 49ms to 113ms average 70-80ms>



Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
  
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 2xProcessor threads:It processes the data into the input queue and updates wold data
• 1xUpdater Thread: inserts clients with updated data to a queue to update
• 4xUpdaterQueues:it gets the client UID from the UpdaterQueue and check which clients are near and need to get an update state, so it inserts the outgoing data to the Pool of Output queues
• Pool of Sender threads:The thread gets data from the Pool of Queues and Send data to the clients
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
  
• Input queue:One in the server, used to store incoming data
• <vector>Pool Output Queue:It can handle many queues so if the client number rise, you can add more queues to prevent thread collisions.
• <vector>Pool Sender thread Queue: Used to store / add Sender threads dynamically.

Good things of the design:

• Finally threads start to unleash their power, as many threads / output queues you add, as many users can handle, 867 clients (BOTs) are starting to be something good for a server....

Bad Things of the design:

• No optimized at all, there are many parts in the server that could change to make it work smoother, but for didactic purposes, I wont change them at the moment.

What I learned from this version:
I have learned to be very careful with the threads, many threads can access to a variable and read from it and processing power will go up, but when writing data you are forced to use semaphores, so speed stops increasing after a certain number of threads.

I added a command for the server that was "AddPower" it adds one output queue and 5 sender threads, so you can adjust performance, but speaking with a friend he told me that I could automate it so automatically it added more queues /threads, so I did! checking if the queues were filled and needed to be processed faster was easy, and the result is the server you have in the picture absorving 867 clients data....

Thanks Jhonny D!

See you soon in the MMORPG series....