Problem with High Latency Links

In my test environment I am running Lightstreamer on a server with a Packet Storm box between it and a client. The link is 256K and the latency is 1000 ms. It is taking a really long time to get data over that link. The problem doesn't seem to be with the amount of data but with the latency itself. When asking for a subscription, it take ~2200 ms for each subscription and then the data still seems to be slow getting there. If I run the client with the same bandwidth but 500 ms latency, it takes half the time to get the data. What settings should I be using (lightstreamer_conf.xml, client configuration, tcp, etc.) to increase my speed? I am running Windows server 2008 as my lightstreamer server and Windows 7 with the .Net Client on the client machine.

Thanks,
Denison

Denison, please help us to clarify the scenario.
Did you force a latency of 1000 ms only on the packets flowing from the Server to the Client or in both ways?
Is the bandwidth limitation 256Kbytes or 256Kbits per second?
When you test the subscriptions, are you actually starting the client, so that a session has to be established as well, or is the subscription request performed on an established an idle session?
What delay do you notice on the arrival of the updates? Is it increasing in time?
Have you set up a test Data Adapter with a specific frequency for the generated updates, or are you using one of our demo Adapters?

The latency is 500 ms each was for a RTT of 1000 ms.

The link is 256kbits per second.

The data adapter(s) we are using are .Net data adapters created by us.

For this test, the client is connecting to Lightstreamer and after a connection is established, then asking for 50+ subscriptions, waiting for a response between each subscription. Lightstreamer is sending a snapshot for each subscription. It doesn't seem to matter if Lightstreamer already has the subscription or it has to go to the Data Adapters.

Unfortunetly, I cannot tell from using fiddler if the snapshot arrivals increase in time or not. Just seems steadily slow.

If you measure 2 seconds from the start of the page to the arrival of the first update, this is to be expected, as two roundtrips in sequence are needed: one for session establishment and one for the subscription.
After that, if you keep the previous session open and only issue a subscription request, you should expect a 1 second delay.
The above is from a logical point of view; it may not be the same from the TCP/IP point of view.
So, if you notice anything different, we should analyze a network capture. To start with, you could also show us the corresponding Server log, after setting the LightstreamerLogger.connections logger at INFO level and the LightstreamerLogger.pump logger at DEBUG level.

Please expand on your observation of data arrival being slow.
In these test conditions, we expect any real-time update issued by the Data Adapter to reach the client with a 500 ms delay.
However, depending on the update size, the bandwidth limitation of 256Kbit/sec can also play a role.
Do you notice a higher delay or an update flow with a bit rate lower than expected?

By the way, if you use fiddler, ensure that it has been configured to support streaming connections.

First of all, we are not using web pages, but a .Net thick client. First we establish the connection and then after we get the OK, we loop through and wait for each of the 55 subscriptions to return "OK" before proceeding to the next subscription. We are still seeing 2 second delays for each subscription, which you can see in the log.

On the plus side, I changed the "sendbuf" setting in lightstreamer_conf.xml file to a larger than default value and that had dramatically increased the throughput of the pipe. However, I still need to figure out the reason for the latency on creating a subscription. The log file is attached.

[attachment: OK, received by Lightstreamer support, then removed]

Ok, now I see that the subscription requests are issued in https.
This explains the 2 seconds delay; in fact, the SSL handshake accounts for the extra roundtrip.
However, this only holds for the first subscription, upon which the control connection is established;
for the other subscriptions, the existing connection is reused and only one roundtrip should be expected.

Unfortunately, I see that only in few cases is the snapshot sent immediately;
I found one at 18:38:12, for instance.
It seems that some delay is really in the data arrival from the Data Adapter; may you please confirm that?
By the way, you said that you noticed extra delays even on subscriptions of items already subscribed to, but I can't find in the log any such case suitable for a check: at a first sight, all the reported subscriptions seem to belong to different items.

We could investigate the case further, if you sent us a log also including duplicate subscriptions and with also the LightstreamerLogger.subscriptions logger at DEBUG level.

About the send buffer setting: we keep it small in order to cope exactly with these network limitations scenarios.
However, the best setting depends on the kind of data to be pushed.
A small buffer is suitable for subscriptions in MERGE mode: it avoids that too many subsequent updates are queued in the TCP buffer, waiting for the network to deliver them, hence accumulating delay.
In fact, rather than being stuck in the TCP buffer, these updates can be merged.
On the other hand, in your case, most of the data is made of COMMAND mode snapshots.
This data cannot be merged and must all reach the client; so a larger TCP buffer allows for a higher throughput, as the TCP can send more snapshot lines while waiting for the (delayed) ACK packets.
Unfortunately, the send buffer setting is global and constant, so you have to find the best trade-off.