Handling client update when snapshot not present

Hi there,


I am building a trading platform and doing something very similar to your recent MarketDepth demo. I am however using the node js adapters instead of the Java adapter that you have used as the backbone of that demo.

I have read through the docs quite thoroughly and still have the following question:

I want to show the user his orders. For this the client subscribes to userID_orders (he is only allowed to see his orders and also needs to be authenticated to do so via the lightstreamer data adapter). The first time that he subscribes, the node js data adapter looks in the database for all of this current orders and then publishes them with isSnapshot = true and finally calls endOfSnapshot. This results in the client receiving multiple item updates where the isSnapshot parameter is set to false, as it is the very first subscription. In the client the endOfSnapshot is also not called, again I understand that this is because it is the very first subscription and the snapshot has thus not been created in the Lightstreamer server yet. This causes an issue in the client because for the first subscription the client has no way to tell when the itemUpdates will stop so that it can repaint the user interface. It is not efficient to rebuild the user interface on every item update when downloading the initial population of orders, there may be many orders and we certainly do not want to keep redrawing the grids in the user interface. If the client then refreshes his browser, because he is already subscribed to userID_orders he gets all the updates and an endOfShapshot callback so that he knows when to rebuild the grid in the GUI efficiently. What is your recommended approach on how to efficiently handle the initial population of the user’s orders when he subscribes for the first time and receives very many item updates?

The market depth demo loads the data into the grid very quickly but I am guessing that is because I am not the first client to subscribe to it and the snapshot is already available, meaning that the GUI can behave efficiently in terms of rebuilding its grids.

Thanks in advance.

Doraintech

Hi Doraintech,

I think that the problem is due to the isSnapshotAvailable function implementation in your DataProvider node class.
Please note that the default value for is function(itemName) {return false;} so the Lightstreamer server assumes that no snapshot is available for any Item, and also if you sent updates with the isSnapshot flag = true, these will be turned into regular updates (by the way you should see warning messages in the server log).

You should change the DataProvider initialization in your node adapter with something like this:
<br /> var dataProvider = new DataProvider(reqRespStream, notifyStream, function(itemName) {return true;});

HTH,
Giuseppe

Hi Giuseppe,

Thank you for your reply.

I have actually already included the function

var isSnapshotAvailable =function( itemName ){ return true };

in the data provider constructor. I am not sure I understand why if you send isSnapshot flat = true they will be turned into regular updates? I thought that they the whole idea is that you needs to set this flag to true create the initial snapshot in the Lightstreamer server?

Just to make it clear, the snapshot is working fine for the 2nd and 3rd clients that subscribe, I only have the problem mentioned above when the first client subscribes. I believe that this is a problem due to the limitation that a snapshot is only available in the LS server if at least one client is subscribed.

Look forward to hearing from you and thanks in advance.

Regards,

Doraintech

Hi Doraintech,

Sorry for the misunderstanding, updates received from the Adapter with isSnapshot flag set to true are turned in regular update only if the isSnapshotAvailable return false (or the server has already determined the EndOfSnapshot).
To recap, when the Item is subscribed for the first time the Lightstreamer server calls the Adapter for isSnapshotAvailable; if the reply is false:
- the server assumes empty snapshot, and EndOfSnapshot is immediately determined;
- all subsequent updates, regardless of the value of the isSnapshot flag true or false, are considered as regular.
If the reply is true:
- the server expects one or more updates with isSnapshot flag set to true, that make up the initial image of the Item;
- the Snapshot completion can be inferred by the endOfSnapshot call or from the first update with isSnapshot flag set to false.
- from this moment all the updates are considered regular.

For all subsequent subscriptions the server does not ask anything about the snapshot to the Adapter, but relies on its in-memory image.

Now in your case, it is very strange that if your isSnapshotAvailable implementation always return true, you do not get the expected behavior.
There is probably something wrong with the code, please could you confirm that there is in your code something like this;

var isSnapshotAvailable = function( itemName ){ return true };<br /> var dataProvider = new DataProvider(reqRespStream, notifyStream, isSnapshotAvailable);

Alternatively you can set this log level, to check the interations between the server and the Adpaters:

Code:

---

<logger name="LightstreamerProxyAdapters" level="DEBUG">

---

Regards,
Giuseppe

Hi Giuseppe,

Yes I can confirm that i have the code you mentioned. I am definitely passing the isSnapshotAvailable function to the data providers when they are constructed.

I have turned on that logging and now i see something interesting:

20-May-15 12:32:47,906|WARN |LightstreamerLogger.preprocessor |#9 Notify Receiver |Unexpected snapshot event for COMMAND item user1_fills
20-May-15 12:32:47,979|WARN |LightstreamerLogger.preprocessor |#9 Notify Receiver |Unexpected snapshot event for COMMAND item user1_fills
20-May-15 12:32:47,983|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 7219c140-f983-11e4-b1e1-f3f54ebbc21a. Event propagated as an ADD command.
20-May-15 12:32:47,992|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key c1332fa0-fa37-11e4-8606-5330024fde49. Event propagated as an ADD command.



20-May-15 12:32:48,039|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 723b2bf0-f983-11e4-b1e1-f3f54ebbc21a. Event propagated as an ADD command.
20-May-15 12:32:48,041|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 77387c20-f983-11e4-b1e1-f3f54ebbc21a. Event propagated as an ADD command.
20-May-15 12:32:48,050|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 953136e0-f983-11e4-b1e1-f3f54ebbc21a. Event propagated as an ADD command.
20-May-15 12:32:48,053|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key da85f510-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,057|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key df027c80-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,064|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key e4ff3b50-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,065|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key e67dc500-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,070|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key edf738c0-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,085|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key f86eaa40-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,090|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key fe6b4200-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,091|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key ffe97d90-fa4f-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,094|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 00b028f0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,117|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 0d5ca2e0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,118|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 0edd0150-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,119|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 105a7990-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,122|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 14d5c880-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,123|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 16612370-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,123|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 17d302f0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,126|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 1bd37c90-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,128|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 1dcd77d0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,134|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 26c7bf30-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,135|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 28470c30-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,135|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 29c37300-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,136|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 2b570b50-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,137|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 2cc05f50-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,139|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 2fbdc0d0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,140|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 313d0dd0-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,141|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 32bd1e20-fa50-11e4-8606-5330024fde49. Event propagated as an ADD command.
20-May-15 12:32:48,142|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key e74ad060-fafc-11e4-85e1-6f07505b6809. Event propagated as an ADD command.
20-May-15 12:32:48,143|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 4afd9480-fafd-11e4-85e1-6f07505b6809. Event propagated as an ADD command.
20-May-15 12:32:48,144|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 89c517f0-fafe-11e4-85e1-6f07505b6809. Event propagated as an ADD command.
20-May-15 12:32:48,145|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 0b7f78d0-fd4d-11e4-8fb8-c702e2d8d5df. Event propagated as an ADD command.
20-May-15 12:32:48,146|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 18a78b60-fd4d-11e4-8fb8-c702e2d8d5df. Event propagated as an ADD command.
20-May-15 12:32:48,150|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 8cefd890-fd4f-11e4-8fb8-c702e2d8d5df. Event propagated as an ADD command.
20-May-15 12:32:48,152|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 45f7bad0-fd58-11e4-8139-e5eb28eb14ff. Event propagated as an ADD command.
20-May-15 12:32:48,153|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 533e6700-fd5a-11e4-915c-53e64e042fe5. Event propagated as an ADD command.
20-May-15 12:32:48,155|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key fe414860-fd5b-11e4-915c-53e64e042fe5. Event propagated as an ADD command.
20-May-15 12:32:48,155|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 051a8480-fd5c-11e4-915c-53e64e042fe5. Event propagated as an ADD command.
20-May-15 12:32:48,156|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key adc86ed0-fd5c-11e4-915c-53e64e042fe5. Event propagated as an ADD command.
20-May-15 12:32:48,157|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key d0fb8d50-fd5d-11e4-91c4-e3f7805c1089. Event propagated as an ADD command.
20-May-15 12:32:48,158|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected UPDATE event for key 9c0e6620-fd5e-11e4-be81-3b57ddf5910b. Event propagated as an ADD command.
20-May-15 12:32:48,158|WARN |LightstreamerLogger.preprocessor |#12 Notify Receiver |Unexpected snapshot event for COMMAND item user1_orders
(END)


Do you know why the logs are showing this Unexpected snapshot event? I am sending my snapshot events with command = ADD. Is that correct?

Thanks if advance,

Doraintech

Hi Doraintech,

The messages of the first type, "Unexpected snapshot event for COMMAND item user1_fills", are due to updates with the isSnaspshot flag set to true when the server has already determined the 'End Of Snapshot'. The update is still propagated to the client but with isSnapshot set to false.

The messages of the second type, "Unexpected UPDATE event for key ... . Event propagated as an ADD command.", are due to updates with the field command set to UPDATE for keys that actually have not been added to the Item with an event type ADD. The server in this case accepts the update changing the field command from UPDATE to ADD.

It is absolutely correct that all snapshot events for a command Item should be ADD.

I'm quite puzzled about the behavior that you are experiencing. In order to check the dynamics underway in your case, please could you arrange a test like this:

1. Shut down the Lightstreamer server and your Adapters;
2. Set these levels in the log configuration file (lightstreamer_log_conf.xml):
   
   • <logger name="LightstreamerProxyAdapters" level="DEBUG"> (for Lightstreamer 6)
   • <logger name="com.lightstreamer.adapters.remote" level="DEBUG"> (for Lightstreamer 5)
   • <logger name="LightstreamerLogger.subscriptions" level="DEBUG"/>
   • <logger name="LightstreamerLogger.pump" level="DEBUG"/>
3. Restart the Lightstreamer server and your Adapters;
4. Open only one client (and if it is possible perform only a single subscription for the Item involved);

From the log we should understand exactly what happens to your updates.

Regards,
Giuseppe

Hi Giuseppe,

Please find the logs attached.

I have done a find and replace to remove some sensitive data but that should not make any difference to the part you are interested in.

Thanks for your help.

Doraintech

Hi Giuseppe,

I think I have found the problem. Adding the loggers you suggested really helped to narrow it down.

Thank you very much for your help.

Hi Doraintech,

Thank you for the feedback.