Accumulated Delay

[ManKeer]

Hi,

My customers start complaining that there's an accumulated delay in receiving data, the client can be connected to different domains.

I have the logs for both the server and the client, How can I know from the logs THAT THERE'S A DELAY IN THE STREAMING PROCESS,

I can upload the logs.

Regards

[giuseppe.corti]

Hi ManKeer,

If the delays involve only a subset of the clients and then we can exclude a general problem affecting the whole server you should look for the "NIO write queue" and "NIO write queue wait" metrics in the MonitorText logs.
When these metrics show high values, and are constantly growing, it means that a part of the client sessions can not cope with the frequency of incoming real-time messages may be due to poor connectivity conditions or very slow devices, and the server accumulates delays in sending messages to them.

Below an example of MonitorText log line with the mentioned metrics in evidence:

"MonitorText|Timer-0 |Total threads = 460, Total heap = 231735296 (free = 16710304), Sessions = 3148 (max = 3175), New sessions = [+16, -12], Connections = 3473 (max = 3575), New connections = [+52, -37], In-pool threads = 128, Active threads = 4, Available threads = 124, Queued tasks = 0, Pool queue wait = 0, NIO write queue = 64, NIO write queue wait = 2457239, NIO write selectors = 32, NIO total selectors = 128, Subscribed items = 47 (for 25363 subscriptions), Inbound throughput = 116.3 updates/s (pre-filtered = 116.3), Outbound throughput = 13418.99 updates/s (6259.6 kbit/s, max = 11693.98), Lost updates = 0 (total = 0), Total bytes sent = 9265303321, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0 "

In any case if the size of the logs is not very big you can send them to support@lightstreamer.com

Regards,
Giuseppe

[ManKeer]

Thank you very much @giuseppe.corti, I have sent the logs from my personal email "mankeer@gmail.com", I have attached the logs for both LS server and client,

I need to know if this is a general problem [for all clients, I have about 7 to 10 clients keep claiming of this] or it's a specific problem for those clients only.

I appreciate your help in this problem, I will be so thankful if you advice how can I distinguish this problem and how it can be solved.

Regards

[DarioCrivelli]

Hi ManKeer,

We have looked at the logs, but we too cannot find evidence of increasing delays, nor do we notice health problems on LS Server.
Do you confirm that the client log you have sent us comes from one of your users who observed delays?
The updates logged by the client library contain a timestamp coming from the Data Adapter and the difference between the client log timestamp and the data timestamp is stable at less than one second (apart from a timezone difference).
As far as we can see, if a delay was in place, it must have occurred either in the Data Adapter, when acquiring the data, or in the client page, when visualizing the data.
In fact, the update flow that reaches the client is significant, of various dozens of updates per second; hence slowness in processing, in some client scenarios, is a possibility.
But, since the logs are huge, please ask your user to provide the exact time of a moment in which they see a delay and, if possible, a quantification of the delay; so we can focus the inspection.

BTW, are you aware of the 61750 refusals of create_session requests by the Metadata Adapter occurring in a few hours, according to LS Server log?

[ManKeer]

Dear Dario,

Thank you for your response, I have attached the logs of the data adapter in my emails.

Unfortunately, I couldn't find any evidence for a delay from the data adapter side.

Regarding the 61750 refusals, I have seen all of these but do these refusals can affect the LS performance?

Regards

[DarioCrivelli]

The Adapter-side logs regard custom code and we aren't able to find clues about the update flow.
Until we have a clear identification of the problem as the user perceives it, I don't think that we can proceed further.
If you confirm that the issue affects only a few clients and doesn't affect the other ones, it is more likely that the problem is related with the client application's ability to keep the pace of the incoming updates while trying to process them.
Please clarify if the client application's job is to visualize the data or to elaborate it internally.

BTW, the client log regards February 27, that is, the day before the Server and Adapter logs.
Could it be a mistake and perhaps the issue was observed on February 28 and never on February 27 ?

The refusals don't seem to affect the performance. Only in case a client application gets several refusals before succeeding, this may be seen by the user as a delay. Until we have a clear identification of the problem as the user perceives it, this is also a possibility.
Or perhaps this is just the evidence of another issue to be investigated.