So I want make an MMORPG, Where do I start from? PART 3

<It had been a long time since I made my last post , part of the problem was because I've been on a trip to Paris (A city everybody should visit at least once in their life), another part of the problem is my fight with non - blocking sockets, but now i'm back and I feel refreshed , with energy to continue with the project.>

Today I'm going to speak about a few things: the server structure ,about messaging  and finally about blocking sockets vs non blocking sockets:

Server Structure:

I plan to make my server architecture based in something like that:

Clients (a.k.a. players) will connect to a login/ authentication server , when validated they will connect to the proxy server, proxy will redirect players to the less loaded server, (balancing server load), server instances will be deployed across the servers on demand ( if a server is more powerful it could handle more instances), all server instances + login server + proxy srv will access the SQL server to write /read all persistent data to be stored, such as:

• Usernames / passwords for the login server
• Maps , players and objects for the server instances
• Number of instances, server instance sockets and health status of the instances (load and failures) for the proxy server.

That is a draft of the design I plan to do, at the moment I'm just working to make a server instance + sql server working, the remaining features will be done later (LOGIN / PROXY).

<Messaging in the next post>

So I want make an MMORPG, Where do I start from? PART 2

Welcome back  to the MMORPG series, part 2!

<Incoming connections....>

After expending one year to learn c++ and SDL I felt ready for the next step: Networking.

I did a chat server just to learn how SDL_net works, It is very interesting and I recommend to try to code one to everybody who dares to create an RPG/MMORPG server because it is basically the same with lower processing needs..

In a chat server (basically ) what happens is:

1. Clients connect to the server
2. They Login with user name / password
3. Every time they type something and press <ENTER> message is sent to all the other connected clients.

(I'm not including here channels and other things... it is just for demonstration purposes only)

If you want you can Download the sources for the server and client and test it, (Unfortunately it only works in Linux because if you run it under windows, it will redirect all the output to a file, so basically is useless..).

But for the MMORPG server I had to analyze it before starting....

What happens (basically )in a MMORPG server is:

(Client side:)

1. Clients connect to the server
2. They Login with user name / password (Or they can create a new user...)
3. Wait for the server to send UID (See below), and world data
4. Transmit actions to the server (update position, use skills, pic objects, attack, etc)
5. Update world state from data received from the server.
6. Goto point 4...

 (Server side:)

1. Allocate resources to accept and process data from incoming clients  (That includes queues and threads to process data)
2. For every client that logs into the server, assign an Unique identifier that is not going to change until client disconnects, the UID is going to be the identifier for all the process in the server (and the client too), the client will receive too the basic world data to start playing (like map data, players and enemies...)
3. Server processes incoming data from connected clients (that includes moving around / and all the skills the player have), for example a player moves forward, server receives a message from client(x) that wants to move forward, so it's going to increase client(x)->Speed
4. Update world state , here it comes were timing is important, the amount of times a world update is applied will change the speed of the game for ALL PLAYERS, for example, players wont send position changes to the server, but  move forward, backward, turn right/left commands, what happens when a world update is executed is that clients will move according to their speed and direction and update X/Y position, so no matter if a client tries to hack the client to run at 100 FPS and not the 25FPS set as default, it wont run faster...
5. Make a list of the players that have changed state (position, skills used)..
6. Send updated data from the clients with changed state to all the clients connected and in a X range...
7. Go to point 3....

I know it is a very raw way to describe what the server does, and you may be missing many features such as maps, skills, enemies / NPCS, AI, etc... but my aim is to make it a simple yet stable network server, then I will add the persistence to do world with  a database (Mysql).

To describe how my server works I'm going to explain how it evolved so it's easier to understand why I changed things...

So let's start to speak with the 3 versions / milestones my server has reached:

<Version Number 0.1:>

Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 1xProcessor thread:It processes the data into the input queues and updates wold data
• 1xUpdater thread: Insert updated data to the output queues
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.
• 1xSender thread/Client:Sends data from the output queue to the client.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
• Input queue:Every client has one, used to store incoming data
• Output Queue:Every client has one, used to store outgoing data

Good things of the design:

• Very simple.
• Functional.

Bad Things of the design:

• Very bad performance, when there was more than 3 clients connected you started to feel it lagging even working with the loop-back interface...
• There is no time control, so the server will have to be confident in the clients <Horrid mistake>
• Wasted data: Two threads per client, two queues per client.. too much

What I learned from this version:
Threads are a must when speaking of a socket server, so in this very first version what I saw is I needed to use them in a more efficient way, I was using many threads to just send and receive data to the client but only three to process data.. other big mistake was to use so many queues in the clients, it was a pain in the a** to check them and probably not efficient at all.., with all this things in my head I decided to make an updated version:


<Version Number 0.1.7:>


Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
  
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 2xProcessor thread:It processes the data into the input queue and updates wold data
• 1xUpdater Thread, inserts into output queue the data to send
• XxSender threads: Send data from the output queue
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
  
• Input queue:One in the server, used to store incoming data
• Output Queue:One in the server, used to store outgoing data

Good things of the design:

• Centralizing queues make it to waste less data.
• Having more threads to process data makes it to be able to handle more clients (watch the screen-shot in the top of the post), 101 clients connected...
• Timing centralized in the server, more secure now
  

Bad Things of the design:

• Still bad performance,after adding 20 clients it started to degrade performance no matter how many threads you added to the server

What I learned from this version:
Threads collide, if you use semaphores / mutexes you will prevent it from happening but, after adding 5 threads to process data from a queue, there is no performance improvement because threads have to wait each-other. It was a design fault, to continue increasing performance I had to add more queues with different semaphores so the threads wont collide so much, so that brings me to the last version:



<Version Number 0.1.12:>

<867 Clients connected at the same time, everyone constantly moving sending data to the server, 25 incoming packets /s per client, total, +2.000 incoming messages processed/s,  peak:18.000 messages SENT in a second, that is like 18 packets /millisecond, latency/lag for the clients from 49ms to 113ms average 70-80ms>



Classes:

• cSockServer: Stores global data , a vector to store all the clients connected, server socket, etc...
  
• cSockClient:Stores data for every client such as socket to communicate

Threads:

• 1x Master thread: Master thread listen for the clients and assigns a new socket to everyone.
• 2xProcessor threads:It processes the data into the input queue and updates wold data
• 1xUpdater Thread: inserts clients with updated data to a queue to update
• 4xUpdaterQueues:it gets the client UID from the UpdaterQueue and check which clients are near and need to get an update state, so it inserts the outgoing data to the Pool of Output queues
• Pool of Sender threads:The thread gets data from the Pool of Queues and Send data to the clients
• 1xListener Thread/Client:listen for incoming data and inserts it into the Input queue.

Queues / vectors:

• Clients vector:Here I store clients data (sockets, UIDs, etc)
  
• Input queue:One in the server, used to store incoming data
• <vector>Pool Output Queue:It can handle many queues so if the client number rise, you can add more queues to prevent thread collisions.
• <vector>Pool Sender thread Queue: Used to store / add Sender threads dynamically.

Good things of the design:

• Finally threads start to unleash their power, as many threads / output queues you add, as many users can handle, 867 clients (BOTs) are starting to be something good for a server....

Bad Things of the design:

• No optimized at all, there are many parts in the server that could change to make it work smoother, but for didactic purposes, I wont change them at the moment.

What I learned from this version:
I have learned to be very careful with the threads, many threads can access to a variable and read from it and processing power will go up, but when writing data you are forced to use semaphores, so speed stops increasing after a certain number of threads.

I added a command for the server that was "AddPower" it adds one output queue and 5 sender threads, so you can adjust performance, but speaking with a friend he told me that I could automate it so automatically it added more queues /threads, so I did! checking if the queues were filled and needed to be processed faster was easy, and the result is the server you have in the picture absorving 867 clients data....

Thanks Jhonny D!

See you soon in the MMORPG series....

Debian Versus Kubuntu Versus Windows 8

Debian 6.03

Kubuntu 11.10

Windows 8 beta developer preview

In the last weeks I have tried many "new" OSes, the first one was Kubuntu 11.10:

After the decision from the Ubuntu team to use Unity in the standard distro 11.04 I have been waiting to see if in the 11.10 version all the mistakes have been removed, unfortunately I had exactly the same feeling that with the previous version...

One of the things I like from Linux is the freedom: freedom to choose what do you want to install and how to configure it to adapt to your personal tastes and needs

With the infamous Unity I feel like if I was a toy inside a Box: I can see beautiful glitter-filled walls, but I can't see anything else!, I thing that when in your process table Xorg is taking MORE CPU than your mmorpg server, something is completely wrong..

I downloaded and installed Kubuntu 11.10, hell I don't like windows-style from KDE but I was so disgusted with regular "Unitybuntu 11.10" that I decided to try it... now I regret I didn't went for Debian before..

Debian is the mother of Ubuntu, and many other Linux flavors, but I have been always afraid to download the tons of gigabytes the full distro uses, but when I entered www.debian.org last day I realized the network installer only occupied like 400 MB, so I downloaded , burned and installed it..

When I did I felt like my first times with Linux (with Slackware), everything was not made "for dummies", you had to set up many things, like repositories to be able to install apps, but well after goggling a bit I was able to install all the tools I use day by day....

It is not as beautiful as Ubuntu or Kubuntu but now my CPU cycles are used for my apps, not for openGL....


I downloaded today Windows 8 beta developer preview and installed it, after all the things I have heard from it I was almost sure it's gonna be very different from Windows 7, but  all the people that has been MS OSes since a few years ago will realize it shares many things from previous versions, the feeling I got from it is that it was completely designed not for pcs, but for Tablets... yes, forget about pcs, the interface is made so you can use it with a touch screen...

Another thing is forget to use it with any computer with less than a 2 dual core fast (2Ghz+ processor) with 2 GB RAM, I installed it in a virtual computer and it took 5 minutes to boot using one core from my I5 core processor (3.2Ghz), assigning the 4 cores to Windows made it boot in "only" 2 minutes... OMG! I hope MS optimize it.. or get ready for a "Vista 8"....

Well I have to test it more but the first feeling is Microsoft wants to change mind of the users.. again, time will say if they are right...


Well , enough OS review for today..

See you!

So I want make an MMORPG, Where do I start from? PART 1

That is indeed a good question, asked so many times and answered many times too, in many sites the answer is like:

<If you have a team>

1. Get a team: coders + artists + world designers +well like 10 persons.
2. Get ready to expend 2 years (at least) on it
3. Pay big amount of money to the ISP
4. Have a design document

<Lone developer>

1. Forget about it :(

 What can I add to that? I basically decided to made it no matter how hard it was; I'm not somebody that has never programmed and think that their "RPG maker experience is enough", I'm not asking how to make it because If you are asking that, you probably are so far from the objective you desire that you wont get it never.

I tried to divide the BIG puzzle into small pieces, c++, SDL, SDL_image, SDL_ttf, SDL_NET, SDL_thread, Gimp, MYSQL....

With those pieces I made some bigger ones: A tileset, a simple game engine, game states, player creation process, a simple chat server....

And now it comes the good thing, I got a Simple (RPG DEMO) server with a client, I didn't made it on time (1 year was my objective), but now , after millions of doubts I know I will finish the game...

My objectives now are:

1. Start a series of posts explaining how I did it.
2. Clean Server / client code and upload them (Yes, I'm going to release it soon)
3. Set up MYSQL server

So first thing first:

Q:What do I need to learn to send / receive date across the network?
A:Network sockets

Q: How does network sockets work?
A: Network sockets handles data send/received to an IP network, they are both used by TCP (Connection oriented ) and UDP (Connectionless oriented), in the server you "open" a socket to listen for incoming data, open a socket means bind the program to a particular IP port of the computer, so we could open a socket at port 80 to create a web server, for example, when we create a server socket it waits until a Client machine (program) connects to it, then we create a client sockets that is going to actually communicate with the client. The clients just open a client socket and establish a connection with the server socket.

Q:Why do I need threads?
A: Threads are used to do many things at the same time, for example, a Client for the MMORPG, could have a thread to draw the images on the screen while another one sends/receives network data , the server could have one thread (maybe more?) to handle network data, another one for the AI, another one for Environment events... In the modern multi-core computers, it's a way to use 100% of the processor power

Q:Why are you using SDL_NET? isn't it an ancient library?
A:It is old, but network sockets are older and are still used today (and probably are going to be used for many, many years), just thing the new IP protocol (IPv6) uses them too... SDL_NET has other advantages too, it can be used in a multi-platform environment (I use it in my client and I can compile it under windows / Linux without touching a single line from the code to make it work :) ), finally it is a low-level library so you feel you have the control over what you are working, there are libraries made upon SDL_NET that add things (NET2 uses multi-threading for example), but you have less control about what is happening, another problem is find documentation , SDL_NET is so small that when you read the manual , you realize how easy is to use it.

<More Q/A soon...>

In the next post: Server code and explanations..

JB

RPG Client Alpha!!!

<Client V0.11>

<Client V0.1>

<Doesn't it looks nice??>


So after two weeks fighting against the elements I have been able to connect my Rpg Demo with the chat server (and basically modifying all the code), but now you can move around with your own player...


You can't do much things, in fact you can just MOVE, but that's exactly the base where I wanted to start from.


I have uploaded the Windows version binaries so basically everybody (windows /linux + wine) can test it, just go to the downloads page and get it!

For alpha testers: To test it, you are going to need an IP number, PM please so I will tell you...


See you

A Simple chat server, code uploaded...

<A chat server / client>


I've just uploaded the full code for the simple chat server and client projects, it is a very simple project, but shows the core needed to create a server including multi-threading and connections /disconnections from clients.

At the moment the clients can do:

• Send messages to other clients.
• Send private messages to another client (/priv nick message)
• Ask for the list of connected users (/List)
• Change nick (/nick newnick)
• And of course, exit the server (/quit)

I don't have plans to upgrade the chat server, instead I'm going to adapt it so many players can connect with a client (rpg demo 4) to the server and anybody can walk around and chat.... That is my next milestone, when I achieve it I'll enable the fights to spice it a little ;)

JB