Sync errors in server log

[krishnamani]

Hi DarioCrivelli,

I have created a page with lsengine and 2 non-visual push tables.

When user login successfully, about 900KB of data pushed to first push table in single subscription. Then 10 to 120 of market watch details of 10 KB each are subscribed through second non-visual table. Market watch details are pushed one by one each of 10 KB when received by the server (Not altogether).

Streaming mode works fine when only 10 market watch details are subscribed. For more than that, I was getting Sync error as mentioned in above posts.

Polling mode works when it subscribes more, but still, it also doesn't work for 120 market watch details subscription.

Hope it ends up with bottleneck at client. If this can be handled by changing content-length? Is there any other way to handle this case in such way to remove the bottleneck at client?

[DarioCrivelli]

The content-length setting should play no role in your streaming issue.

In polling, the content-length may limit the size of the response to a poll. This doesn't happen in a normal case, but if you handle such big data, this limitation may come into play.
This may cause the Server to need more or less polls to dequeue the snapshots.
Yours is an uncommon case and, honestly, I can't guess whether limiting the size of a polling response could be beneficial.
Anyway, if a single event were responsible for the 900KB of snapshot data caused by your first subscription, they would be sent in a single poll, regardless of the content-length configuration.

What does it happen with 120 market watch detail subscriptions in long polling?
Do you still notice "sync error" messages? Or is it just too slow to be acceptable?

With 120 market watch detail subscriptions, the processing of the updates in your custom code may be heavy; I suppose that the client CPU usage is at 100%.
However, Lightstreamer library itself may need a lot of processing time. Please try removing all your callbacks from the NonVisualTables and see if the client CPU usage is still at 100%.

[krishnamani]

What does it happen with 120 market watch detail subscriptions in long polling?
Do you still notice "sync error" messages? Or is it just too slow to be acceptable?

By Long Polling do you mean increased Polling interval (max_polling_millis)?
If yes, I have tried increasing max_polling_millis, even with 60 seconds, I am getting same Sync Error when market watch details are pushed.

With 120 market watch detail subscriptions, the processing of the updates in your custom code may be heavy; I suppose that the client CPU usage is at 100%.
However, Lightstreamer library itself may need a lot of processing time. Please try removing all your callbacks from the NonVisualTables and see if the client CPU usage is still at 100%.

Even with 120 market watch details, it takes maximum of 50% of CPU. It takes 0 to 2% normally, and 50% while processing and showing it.

[DarioCrivelli]

I wrote "Long Polling" as another name for the "Asynchronous Polling", which, in an Ajax context, can be named as just "Polling".
So, I just referred to your previous post, in which you had reported the behavior of the polling case.
In polling mode, we expect that the browser, if overwhelmed by the processing of the outcome of a poll request, delays the next poll request (but we have no control on what the browser actually chooses to delay).
This would avoid that the updates queue up on the client: they would try to queue up on the Server, which, however, can conflate them.

If the next request is delayed too long, the Server will have discarded the Session and will get a "sync error".
The client tries heuristically to measure its own processing delay and may request the Server to delay the Session discarding. Your setting of <max_polling_millis> is correct, because it avoids that the Server refuses to obey a client delay request which is too long.
However, the client heuristic may be underestimating the delay.

May we check your Server log taken during a polling run with 120 market watch details?
You should set the priority of the LightstreamerLogger.requests.polling logger to INFO.

If you see the CPU usage steadily at 50%, it may be that only half of the available processors can be used by the browser; hence, this would still represent a bottleneck.
You could still try removing all your callbacks from the NonVisualTables and see if the client CPU usage is still at 50%.

[krishnamani]

I wrote "Long Polling" as another name for the "Asynchronous Polling", which, in an Ajax context, can be named as just "Polling".
So, I just referred to your previous post, in which you had reported the behavior of the polling case.
In polling mode, we expect that the browser, if overwhelmed by the processing of the outcome of a poll request, delays the next poll request (but we have no control on what the browser actually chooses to delay).
This would avoid that the updates queue up on the client: they would try to queue up on the Server, which, however, can conflate them.

We are ok even if the updates are queued up in client and process it in sequence, but we need to handle the browser close event.

How is to be coded at both client and LS side to handle the delay at client side?

If the next request is delayed too long, the Server will have discarded the Session and will get a "sync error".
The client tries heuristically to measure its own processing delay and may request the Server to delay the Session discarding. Your setting of <max_polling_millis> is correct, because it avoids that the Server refuses to obey a client delay request which is too long.
However, the client heuristic may be underestimating the delay.

May we check your Server log taken during a polling run with 120 market watch details?
You should set the priority of the LightstreamerLogger.requests.polling logger to INFO.

I will be sending you the Server log files as early as possible.

If you see the CPU usage steadily at 50%, it may be that only half of the available processors can be used by the browser; hence, this would still represent a bottleneck.
You could still try removing all your callbacks from the NonVisualTables and see if the client CPU usage is still at 50%.

We are using Firefox for testing purpose. But it allows more than 50% for Firefox and it goes 52% or 56% for this case but in an average of 50%.

Can you please let me know what you mean by callbacks and how to clear it at client side?

[DarioCrivelli]

I mean that you should try detaching (i.e. not definig at all) the onItemUpdate callback of your NonVisualTable,
so that all streaming data are read by LS library but not processed at all;
then see if the CPU usage decreases
and, in that case, the Sync Error also disappear.
This test may not be feasible, as the overall page behavior may be altered; it depends on how your page actually works.

[krishnamani]

I mean that you should try detaching (i.e. not definig at all) the onItemUpdate callback of your NonVisualTable,
so that all streaming data are read by LS library but not processed at all;
then see if the CPU usage decreases
and, in that case, the Sync Error also disappear.
This test may not be feasible, as the overall page behavior may be altered; it depends on how your page actually works.

Hi Dario Crivelli,

If we remove the callbacks of market watch NonVisualTable, then it takes 0 to 2% of CPU and Sync Error disappears, but this will totally turns off the actual requirement.

If you can tell us that what is the maximum time to be taken to process each market watch at client side, we will try to optimize and confirm if we can do it in required time and still receive such Sync Errors.

Regards,
Krishnamani

[krishnamani]

I will be sending you the Server log files as early as possible.

Please find the attached server log of this case.