client refresh data content incomplete

[paulwiersema]

Giuseppe,

Thanks for the heads-up. Unfortunately, even after the recent changes we are still seeing missing rows after a client browser refresh.

In our use case, the clients need all updates, so they subscribe to a generic item called "FLIGHTS", so there is only 1 subscription per client. The only time there will be no subscriptions to "FLIGHTS" is if all clients disconnect, which would be fine. The initial load URL returns a list of flight objects, with a unique key and snapshot=true on each flight object. After browser refresh, we never see any of these flights missing.

There are instances where a new flight needs to be pushed to the clients. The system that pushes this data does not have insight into whether or not the flight was already pushed or not, so it sets the snapshot=true to err on the side of caution (even though there may be some data missing from that row that needs to be filled in later). These "new" flights are the ones that are sometimes missing after a browser refresh.

There are other systems that send other partial updates, but these systems "know" that and so the snapshot is not set in those updates.

The question that sticks out in my mind is "Does the server see the SNAPSHOT as related to the subscribed item ("FLIGHTS") or the individual flight row (unique by key)? If it is by subscribed item, how do we ensure that any new flights are added to the "group"? if it is by key, do we need to ensure that only the first data push is marked as a snapshot? That would require a bit of re-architecture, but it could be done, if needed.

Thanks again for your wisdom and direction,
Paul

[DarioCrivelli]

Hello Paul,

The Data Adapter is not supposed to set the "isSnapshot" flag to notify that an update introduces a new key in a COMMAND mode item, which should appear in the snapshot of future subscriptions.
The meaning of the "isSnapshot" flag is that the data being passed is not a real-time update, but it occurred earlier, hence it is passed only as part of the current item state (i.e. the item snapshot).
As a consequence, the Data Adapter should set the "isSnapshot" flag only at the beginning, just after receiving the "subscribe" call, to send the accumulated data. After sending them, it should declare the end-of-snapshot and only send real-time updates, without the flag.
The same information will be forwarded by Lightstreamer to the client.

Then, if it happens that

1. a real-time update introduces a new key in a COMMAND mode item;
2. the item remains subscribed to the Data Adapter;
3. a new client subsequently subscribes to the item;

Lightstreamer will take care of ensuring that the new client will receive the key as part of the snapshot (provided that it requests the snapshot).
So, if you are in this case and don't see the key, this is unexpected; but only the detailed log, as requested by Giuseppe, can shed more light.

[paulwiersema]

Dario,

Thanks for the insight. I will revert that snapshot code for new flights and increase logging to see if we can figure out what's happening. Are there any requirements for setting the keys? For example, do the keys need to include the item name (FLIGHTS.key1) ? Just wanting to make sure we are doing what's expected.

Thanks,
Paul

[DarioCrivelli]

No, the key names are free; they are String objects and their equality is String equality. Only null keys or wrong commands are refused.
You should obey the constraint about using the ADD command the first time you use a key (or after a DELETE) and UPDATE subsequently, but this is not mandatory and the Server just warns if it finds that your command doesn't match its observations, but doesn't discard any event because of this.

[paulwiersema]

Guys,

I just realized that I am not able to reproduce this issue locally using version 7.0.1, but our enterprise is still on version 6.1. From the release notes, this may have been fixed in 7.0:

"Fixed a bug affecting update dispatching, which could have been triggered by the combination of a very fast unsubscription-resubscription sequence and significant delays in internal update processing, possibly due to high load. The bug could have caused one or more updates for the previous subscription to be used in place of an equal number of updates for the new subscription."

Can someone please confirm?

Thanks,
Paul

[DarioCrivelli]

Hi Paul,
That case happened very rarely, but it is possible.
In that case, a warning message saying "Unexpected duplicate event" was also observed.
Can you find it in the logs?