Troubleshooting guide
In this guide we provide a few tips and tricks to overcome typical issues that may arise while setting up or running the Server.
Unable to reach the server
1) Blocked ports
Once your SmartFoxServer is running you should make sure that no firewall (software or hardware) is blocking the TCP ports in use. Specifically you should open the TCP ports 9977 and 8088 to the "outside world". If this is something you can't do we recommend asking your system administrator.
We have a detailed breakdown of the ports used in our installation guide that can be used as reference.
2) Port forwarding / NAT
If your server is not connected directly to the internet you should make sure that the port forwarding (a.k.a. NAT) is configured correctly. The NAT allows to expose services running in your LAN to the internet.
More details are provided at the end of our installation guide
3) Telnet test
In order to determine if something is blocking your server you can try telnet-ing its public address from outside. Telnet is a popular command-line utility available on most operating systems that allows to establish a TCP connection to a remote host.
Launch a terminal window and execute the following:
If telnet is not available on your system, you can alternatively use netcat (nc), another popular utility to test connections.
Typically telnet should return something like this, which indicates success:
Otherwise you'll get an error:
Server Startup problems
If the server doesn't complete the startup phase we recommend to check the log files which are found under server/logs/. Log files are rotated on a daily basis and easy to identify by the date appended to the filename.
To check the latest log messages open smartfox.log
Dropped packets
Sometimes you might notice a high value in the number of dropped packets in the the AdminTool's Dashboard.
Incoming dropped packets are simple to explain: SmartFoxServer drops any request that doesn't conform to its protocol and discards packets whose size is exceeds the value set in the server/config/core.xml file for the maxIncomingRequestSize parameter.
Outgoing dropped packets are due to clients lagging behind and not being able to keep up with the rate of messages sent by the server. While it's not uncommon to see a 1-5% outgoing dropped packet rate, any value higher than that is indicative of a larger issue. Typically this problem arises from custom server side code sending too many updates too frequently, filling each User's queue too quickly while the client struggles keeping up with the pace.
The following diagram illustrates how packet queues work:
SmartFoxServer associates an outgoing message queue to each connected client in order to store data that cannot be sent immediately. The server attempts to deliver each packet as quick as possible but there are times where the User connection is either too slow or congested and only a part of the data can be sent.
In these situations SmartFoxServer is forced to store the remaining bits of data in the queue and wait for the network pipe to become available again for a new transmission.
When a client queue reaches its capacity packets must be dropped to preserve the integrity of the server. This mechanism allows SmartFoxServer to protect itself from unlimited memory allocation which would eventually slow everything down until it would crash.
Causes of outgoing dropped packets
-
Bad or slow client connection: the user has a slow response time and not enough bandwidth to keep up with the server responses. This causes the socket to become busy very soon and forces the server to keep the data in the queue.
-
Too much data being sent too frequently: this is a slight variation on the previous theme. Only this time we can not necessarily blame the client connection but it is probably depending on the server logic sending too large data or too frequent updates.
-
Lack of bandwidth on the server side: as you reach the capacity of your available bandwidth you will experience a general slow down of the network performance. Players will likely start to complain about the game getting slower and you should see a rising number of dropped packets.
Addressing these problems depend on multiple factors: what your game/app does, the inner workings of your server side code, the kind of devices that your game targets and more. Typically they can be solved by fine tuning the packet rate and reducing the amount of data sent as appropriate.
Ghost Users
Ghost Users can be generated when an incomplete TCP disconnection occurs. The TCP transport protocol uses a 4-way exchange between client and server to shut down a previously opened connection. If during this exchange the communication is interrupted the disconnection will not complete, leaving either side of the communication in a state of partial disconnection.
The low level TCP stack uses several flags to maintain the status of each connection which can be inspected using the netstat utility from the commandline. It's not the scope of this section to go into the technical details of the TCP inner workings, but if you want to learn more you can check the following resources:
Common causes of Ghost Users
Ghost Users are exceedingly rare when working in a LAN/WAN environment because the connections are direct, and there's unlikely slow-down or congestion. On the other hand over the internet the situation can be different because as client goes through a number of network hops, which represent routers/firewalls/gateways that the packets go through from one end of the connection (e.g. the client) to the other (e.g. the server).
In this scenario there is a significant number of factors that can lead to a loss of connection, including network congestion, timeouts, resets, unrecoverable packet loss etc... When this happens the user socket can remain in a "lingering state" for a long time, as each of the two parties wait for the final bits of the communication.
Until this process is completed the underlying OS can't notify a disconnection event and therefore SmartFoxServer remains unaware of what is happening. Each OS (and its TCP implementation) has slightly different rules and settings to deal with lingering connections but, eventually, all these sockets will be forcefully closed, triggering the disconnection event up to SmartFoxServer and its Extensions.
How to deal with Ghosts
SmartFoxServer attempts to reduce the impact of pending connections by running a background task (called Ghost Hunter) that checks the integrity of each connection and removing those that are deemed stale.
The server's configuration is also important to reduce this problem, in particular there are two settings that can be fine tuned via the AdminTool:
-
ServerConfigurator > Session Max Idle Time: sets the maximum idle time for a single session. A session is only used by clients to login in the Server. Once they are logged in the client is turned into a User. For this reason it is recommended to keep this value in a range of 10-40 seconds. Any other client that might open a connection to the Server without logging in will be removed quickly.
-
ZoneConfigurator > User Max Idle Time: sets the maximum idle time for a User. This value can change a lot depending on how the application works and how long a User is allowed to remain idle without being disconnected. In general we suggest to avoid values greater than 30 minutes.