HRC: the reconnection system

HRC (Highly Resilient Connection) is a reconnection system built in SmartFoxServer that enables players to retry multiple connections (if they get disconnected) without losing their current game state.

In this article we're going to discuss the ways in which HRC works, how it is done technically, and learn how to configure and test it.

Why was I disconnected?

The main reason why a connection is dropped without notice is traffic congestion. The player connection usually travels through multiple network nodes to reach its destination (i.e. the game server), and each of these links can be a potential point of failure. In fact, if any of these routers get overwhelmed with traffic they can either respond too slowly or be forced to drop packets, thus causing a stall and ultimately a disconnection.

If we add to the mix WiFi and mobile connections the scenario becomes even more delicate, as a drop in the strength of the signal can also cause an abrupt disconnect from the game server.

All these situations are handled primarily at the low level, in the operating system's TCP stack which in turn dispatches events at the application level, where other programs can react. SmartFoxServer is no exception in these regards, as it depends on those low level notifications as all.

Later in this article we will see what might happen if some of those events are not fired in a timely fashion.

HRC to the rescue

The diagram below illustrates what happens when a player is suddenly disconnected from the server:

Starting on the left side we have three users connected to SmartFoxServer, all playing together as Player A, B, C in the same Zone. Suddenly User B looses its connection, the Player object on the server is frozen and messages that should be received are queued.

While client B API attempt to re-establish a connection to the server, the other players can continue with their game (where the game logic allows it) and can be notified that Player B is trying to re-enter the game.

On the right side of the diagram Player B is finally reconnected. The new session is linked with the old Player object, the client is updated with all the messages that were cached, and other players can be notified that B is back.

What might go wrong

Not all disconnections are made equal. There is small percentage of cases in which a disconnection can remain in an incomplete state for some time, thus not triggering the reconnection system.

We have dedicated an entire section of our documentation to the problem of ghost connections, also known as half-closed connections. Without going into too much detail, the TCP protocol employs several message exchanges in order to disconnect the client and server. In case the network becomes suddenly unavailable to one side, this exchange cannot be completed and the TCP connection can remain in a "waiting" state for a certain amount of time.

Issues like this can be easily demonstrated by pulling the network cable from the computer or shutting down the WIFI interface (or mobile connection). In all these cases it is possible to end up with a ghost connection, where the TCP socket on one side is waiting for the close operation to be completed by the other side.

Why is this so relevant for the Reconnection system? It is critical because a TCP socket in this state won't notify any disconnection event at the application level, and therefore SmartFoxServer remains unaware of this new state. Eventually the TCP's own timeout, or the SmartFox idle socket timeout, will trigger (depending on which fires earlier) and the disconnection event will be dispatched.

Proper ways of testing disconnections

Counterintuitively, unplugging the network cable or shutting down the WiFi connection will not generate a sudden disconnection that can be used for testing.

In fact you can monitor your computer's connection using the netstat utility (available on Windows, Mac and Linux) before and after unplugging the cable and you will be surprised to see that nothing changes!

In the following section we discuss how to properly configure and test the reconnection system.

Configuring a Zone to support HRC

HRC can be activated on a per-Zone basis, meaning that any of your application can choose to use or not use the feature. In order to activate HRC login your AdminTool, select the Zone Configurator module, choose or create a Zone and under General set the value for the User Reconnection Timeframe:

The value represents the number of seconds that the client will be allowed to reconnect. If no connection is established during that period an SFSEvent.CONNECTION_LOST will be triggered.

Testing the reconnection

In order to test HRC you can simulate a sudden disconnection by invoking the client side API method killConnection() exposed by the main SmartFox class. We recommend doing so at any time in your client application once you've connected and logged into a Zone. If you get disconnected before joining a Zone the reconnection system will not work.

Example:

C#

var sfs = new SmartFox();
sfs.AddEventListener(SFSEvent.CONNECTION, OnConnection);
sfs.AddEventListener(SFSEvent.CONNECTION_LOST, OnConnectionLost);
sfs.AddEventListener(SFSEvent.CONNECTION_RETRY, OnConnectionRetry);
sfs.AddEventListener(SFSEvent.CONNECTION_RESUME, OnConnectionResume);
sfs.AddEventListener(SFSEvent.LOGIN, OnLogin);
sfs.AddEventListener(SFSEvent.LOGIN_ERROR, OnLoginError);

var cfgData = new ConfigData();
cfgData.Zone = "BasicExamples";

sfs.Connect(cfgData);

private void OnConnection(ApiEvent evt) {
    var success = (Boolean) evt.Param[EventParam.Success]!;

    if (success)
    {
        Log("Connection success");
        sfs.Send(new LoginRequest("Kermit the Frog"));
    } else {
        Log("Connection Failed: " + evt.Params[EventParam.ErrorMessage]);
    }
}

private void OnConnectionLost(ApiEvent evt)
{
    string? reason = (string?) evt.GetParam(EventParam.DisconnectionReason);

    Log("Connection lost, reason: " + reason);
}

private void OnConnectionRetry(ApiEvent evt)
{
    Log("Connection lost. Trying to reconnect... ");
}

private void OnConnectionResume(ApiEvent evt)
{
    Log("Connection Resumed!");
}

private void OnLogin(ApiEvent evt) {
    Log("Logged in successfully as: " + sfs.MySelf.Name);

    // Wait for 2 seconds then kill the current connection
    Thread.sleep(2000);
    sfs.KillConnection();
}

private void onLoginError(ApiEvent evt) {
    log.info("Login error: " + evt.Params[EventParam.ErrorMessage]);
}

Here we've extended the connection + login example with a few extra event handlers:

• onConnectionRetry: triggered by SFSEvent.CONNECTION_RETRY, signals the disconnection and that the API has started the reconnection loop. Here the game developer can notify the player about what's happening and block the interaction with UI, if necessary, until the connection is resumed

• onConnectionResume: triggered by SFSEvent.CONNECTION_RESUME, signals the successful reconnection to the server. When resumed the server sends all the updates that were queued during the connection blackout to keep the client up to date.

• onConnectionLost: triggered by SFSEvent.CONNECTION_LOST when the connection is lost or after all reconnection attempts have failed.

Please Note:

Do not attempt to test the reconnection system by pulling the ethernet cable or shutting down the WIFI connection. Both operations will not terminate your current socket connections and therefore the disconnection event may take a long time before it's recognized by the system.