Difference between two adapters

[ManKeer]

I have to java adapters running on the same Lightstreamer server, the first one working with "MERGE" mode, while the other working with "COMMAND" mode, the first one pushing data directly, while the other converts the data to JSON then published to the subscribers.

The first one has two main items for subscriptions, each one has it's own DB thread and listener, the other has 7 subscriptions each one has it's own listener.

now both of them reading from the same DB tables, but unfortunately, the first one is faster than the second, client for the first one receives about 6 updates in 2-3 seconds, while the client of the second receives about 2 updates in 2-3 seconds.

when I check the logs, the data retrieved successfully without difference, the adapter logs show that the data is pushed....


Why do we have such a behavior?????

[DarioCrivelli]

The only scenario that I can recall is if the json data in the COMMAND item is much heavier than the field-based data in the MERGE item and the client is very slow at handling the updates.

Apart from that, we should follow the updates trip to see what happens.
We can start from the Server side with the following settings in lightstreamer_log_conf.xml
- LightstreamerLogger.subscriptions at DEBUG
- LightstreamerLogger.subscriptions.upd at DEBUG
- LightstreamerLogger.pump at DEBUG

Note that the log is verbose, but we assume that you can reproduce the issue with only the two mentioned clients connected.
On the other hand, if you can replicate the issue only in a production scenario with many clients connected, please clarify that first.

[ManKeer]

From client-side Is there any configurations that limit the received number of updates?

I will re-configure the logging based on what you have mentioned, and I will show the logs.

Thanks

[DarioCrivelli]

Yes, the client application can set a maximum frequency on the updates received on an item basis.
Also the Metadata Adapter can set such a maximum frequency on an item basis, which may further restrict the limit.
However, none of these limits is set by default.

[ManKeer]

Based on what I have seen in the logs, L.subscriptions.upd refers to an update from the adapter sent to LS engine, while L.pump refers to the pumped message by LS engine to the clients.

if the above is true then, I have an issue in the following log records:


22-Aug-23 13:19:19,137|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"11\":\ "3.978\",\"12\":\"5817\",\"13\":\"1014770\",\"5\": \"2023-08-22 13:19:18.000\",\"6\":\"2023-08-22 13:19:19.086\",\"7\":\"94.1\",\"9\":\"440\",\"10\" :\"3.6\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,148|DEBUG|L.subscriptions.upd|LMARKETS |INCOMING DATA for LMARKETS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"TBNI\":{\"11\":\ "9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7 \":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\ "}}","removed":null}, key=LMARKETS, command=UPDATE}
22-Aug-23 13:19:19,148|DEBUG|L.pump |PUMP POOLED THREAD 8 |Pumping event in session S8c12f8acdc0d0c2eMfccT1556739: U,42,1,||{"added":null,"changed":"{\"TBNI\":{\"11\ ":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7 \":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\ "}}","removed":null}

22-Aug-23 13:19:19,152|DEBUG|L.pump |PUMP POOLED THREAD 5 |Pumping event in session Se80d7908765fb794MfccT1557247: U,13,1,||{"added":null,"changed":"{\"TBNI\":{\"11\ ":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7 \":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\ "}}","removed":null}
U,12,4,||{"added":null,"changed":"{\"2010\":{\"6\" :\"2023-08-22 13:19:18.951\"}}","removed":null}

22-Aug-23 13:19:19,294|DEBUG|L.subscriptions.upd|LDEPTH |INCOMING DATA for LDEPTH.1120,MBO on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"1120,MBO\":{\"0\ ":\"1120\",\"1\":\"MBO\",\"2\":\"TASI\",\"3\":\"20 23-08-22 13:19:19.226\",\"4\":\"1100\\u003d20,1101\\u003d24 27,1102\\u003d72.7,1103\\u003d140,1104\\u003d72.7, 1105\\u003d45,1106\\u003d72.7,1107\\u003d3800,1108 \\u003d72.8,1109\\u003d345,1110\\u003d72.8,1111\\u 003d900,1112\\u003d72.8,1113\\u003d10,1114\\u003d7 2.8,1115\\u003d4000,1116\\u003d72.8,1117\\u003d700 1,1118\\u003d72.8,1119\\u003d800,1120\\u003d72.8,1 121\\u003d280,1122\\u003d72.8,1123\\u003d700,1124\ \u003d72.8,1125\\u003d92550,1126\\u003d72.8,1127\\ u003d17,1128\\u003d72.8,1129\\u003d1,1130\\u003d72 .8,1131\\u003d197,1132\\u003d72.8,1133\\u003d376,1 134\\u003d72.8,1135\\u003d300,1136\\u003d72.8,1137 \\u003d227,1138\\u003d72.8,1139\\u003d1,1140\\u003 d72.8,1141\\u003d20,1142\\u003d34,1143\\u003d72.6, 1144\\u003d2000,1145\\u003d72.6,1146\\u003d1,1147\ \u003d72.6,1148\\u003d2,1149\\u003d72.6,1150\\u003 d495,1151\\u003d72.6,1152\\u003d1377,1153\\u003d72 .6,1154\\u003d100,1155\\u003d72.6,1156\\u003d100,1 157\\u003d72.6,1158\\u003d500,1159\\u003d72.6,1160 \\u003d25,1161\\u003d72.6,1162\\u003d45,1163\\u003 d72.6,1164\\u003d4,1165\\u003d72.6,1166\\u003d50,1 167\\u003d72.6,1168\\u003d23,1169\\u003d72.6,1170\ \u003d50,1171\\u003d72.6,1172\\u003d10,1173\\u003d 72.6,1174\\u003d100,1175\\u003d72.6,1176\\u003d66, 1177\\u003d72.6,1178\\u003d10,1179\\u003d72.6,1180 \\u003d7001,1181\\u003d72.6\"}}","removed":null}, key=LDEPTH, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"4142\":{\"16\":\ "724\",\"6\":\"2023-08-22 13:19:19.220\"},\"1120\":{\"16\":\"2612\",\"6\":\" 2023-08-22 13:19:19.222\"},\"1010\":{\"16\":\"7278\",\"6\":\" 2023-08-22 13:19:19.226\"},\"2010\":{\"14\":\"4940\",\"6\":\" 2023-08-22 13:19:19.214\"},\"7040\":{\"16\":\"2497\",\"17\":\ "94.2\",\"6\":\"2023-08-22 13:19:19.223\"},\"4291\":{\"16\":\"37\",\"6\":\"20 23-08-22 13:19:19.229\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.1120 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"1120\":{\"16\":\ "2612\",\"6\":\"2023-08-22 13:19:19.222\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.2010 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"2010\":{\"14\":\ "4940\",\"6\":\"2023-08-22 13:19:19.214\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,305|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"16\":\ "2497\",\"17\":\"94.2\",\"6\":\"2023-08-22 13:19:19.223\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,389|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"12\":\ "5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,389|DEBUG|L.subscriptions.upd|LSTOCKS |INCOMING DATA for LSTOCKS.7040 on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"7040\":{\"12\":\ "5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}, key=LSTOCKS, command=UPDATE}
22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 8 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,3,1,||{"added":null,"changed":"{\"TBNI\":{\"11\" :\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7 \":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\ "}}","removed":null}

22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S6f06ad1777945360MfccT1526937: U,25,1,||{"added":null,"changed":"{\"TBNI\":{\"11\ ":\"9962679\",\"3\":\"2023-08-22 13:19:19.080\",\"5\":\"11059.5\",\"6\":\".54\",\"7 \":\".005\",\"8\":\"16671\",\"10\":\"387636431.36\ "}}","removed":null}

22-Aug-23 13:19:19,397|DEBUG|L.pump |PUMP POOLED THREAD 5 |Pumping event in session Se80d7908765fb794MfccT1557247: U,12,2,||{"added":null,"changed":"{\"1120\":{\"16\ ":\"2612\",\"6\":\"2023-08-22 13:19:19.222\"}}","removed":null}

22-Aug-23 13:19:19,400|DEBUG|L.subscriptions.upd|LMARKETS |INCOMING DATA for LMARKETS on FEEDERSERVER.FEEDER_ADAPTER --> {Data={"added":null,"changed":"{\"TASI\":{\"11\":\ "116942017\",\"3\":\"2023-08-22 13:19:19.321\",\"5\":\"11403.05\",\"6\":\"-5.4\",\"7\":\"-.047\",\"8\":\"217779\",\"10\":\"2864023605.05\"}} ","removed":null}, key=LMARKETS, command=UPDATE}
22-Aug-23 13:19:19,428|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,1,1,||{"added":null,"changed":"{\"7040\":{\"12\" :\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}

22-Aug-23 13:19:19,428|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session S59600dae1b9ab544MfccT1906549: U,4,1,||{"added":null,"changed":"{\"1120,MBO\":{\" 0\":\"1120\",\"1\":\"MBO\",\"2\":\"TASI\",\"3\":\" 2023-08-22 13:19:19.226\",\"4\":\"1100\\u003d20,1101\\u003d24 27,1102\\u003d72.7,1103\\u003d140,1104\\u003d72.7, 1105\\u003d45,1106\\u003d72.7,1107\\u003d3800,1108 \\u003d72.8,1109\\u003d345,1110\\u003d72.8,1111\\u 003d900,1112\\u003d72.8,1113\\u003d10,1114\\u003d7 2.8,1115\\u003d4000,1116\\u003d72.8,1117\\u003d700 1,1118\\u003d72.8,1119\\u003d800,1120\\u003d72.8,1 121\\u003d280,1122\\u003d72.8,1123\\u003d700,1124\ \u003d72.8,1125\\u003d92550,1126\\u003d72.8,1127\\ u003d17,1128\\u003d72.8,1129\\u003d1,1130\\u003d72 .8,1131\\u003d197,1132\\u003d72.8,1133\\u003d376,1 134\\u003d72.8,1135\\u003d300,1136\\u003d72.8,1137 \\u003d227,1138\\u003d72.8,1139\\u003d1,1140\\u003 d72.8,1141\\u003d20,1142\\u003d34,1143\\u003d72.6, 1144\\u003d2000,1145\\u003d72.6,1146\\u003d1,1147\ \u003d72.6,1148\\u003d2,1149\\u003d72.6,1150\\u003 d495,1151\\u003d72.6,1152\\u003d1377,1153\\u003d72 .6,1154\\u003d100,1155\\u003d72.6,1156\\u003d100,1 157\\u003d72.6,1158\\u003d500,1159\\u003d72.6,1160 \\u003d25,1161\\u003d72.6,1162\\u003d45,1163\\u003 d72.6,1164\\u003d4,1165\\u003d72.6,1166\\u003d50,1 167\\u003d72.6,1168\\u003d23,1169\\u003d72.6,1170\ \u003d50,1171\\u003d72.6,1172\\u003d10,1173\\u003d 72.6,1174\\u003d100,1175\\u003d72.6,1176\\u003d66, 1177\\u003d72.6,1178\\u003d10,1179\\u003d72.6,1180 \\u003d7001,1181\\u003d72.6\"}}","removed":null}

22-Aug-23 13:19:19,450|DEBUG|L.pump |PUMP POOLED THREAD 7 |Pumping event in session Sc4736f28d89e4ee3MfccT1547256: U,30,1,||{"added":null,"changed":"{\"TASI\":{\"11\ ":\"116942017\",\"3\":\"2023-08-22 13:19:19.321\",\"5\":\"11403.05\",\"6\":\"-5.4\",\"7\":\"-.047\",\"8\":\"217779\",\"10\":\"2864023605.05\"}} ","removed":null}
U,29,1,||{"added":null,"changed":"{\"7040\":{\"12\ ":\"5821\",\"13\":\"1015118\",\"5\":\"2023-08-22 13:19:19.000\",\"6\":\"2023-08-22 13:19:19.326\",\"9\":\"150\"}}","removed":null}


The highlighted green message has been lost, and my java client didn't receive it at all, from the logs there's no pump to this message.

is that true????

[ManKeer]

In addition to the above, I can confirm that the adapter's code is running perfectly, and the data retrieved without any problems, however the clients for the same adapter behaves differently, java script client not receiving most of the messages, while java client is receiving around 95% of the messages.

in my last reply, I have mentioned that one message has not sent from the LS engine side [this explains why java client not receiving the messages], but why the java script client not receiving messages which are received by java client.

both clients have been configured to receive MAXFREQUENCYUPDATE = UNLIMITED.

[ManKeer]

How can I attach the lightstreamer.log file so you can check whatever you need.

[DarioCrivelli]

The log snippet includes 5 sessions, whereas I understood that two sessions were enough to exploit the issue.
This complicates the analysis. Can you please produce a log with only two sessions?
Then, assuming that it will be short, you can send it to support@lightstreamer.com
Please ensure that the log includes the Server startup and, in particular, the subscription requests by the two clients.

Note that, unless the client explicitly requests all updates, updates not sent are not lost, but just filtered out in order to always send the latest state.
This is possible also when specifying "unlimited" frequency. Only the "unfiltered" setting forces all updates.
Obviously, such filtering is expected only when there is some bottleneck or some race condition.
With these conditions, it is possible that two different clients receive a different number of updates.
Apparently, these conditions don't apply to your case; but perhaps the full log can help to clarify.

[ManKeer]

Hi Dario,

By two sessions, you mean that only two subscriptions [not two clients] are connected to the LS server, Am I right? please confirm...

if the above is correct, I will reproduce the logs again.

BTW, what will happen if I configuered both clients for "unfiltered" subscriptions, how this will affect the performance?

[DarioCrivelli]

Sorry; if my reference to "two sessions" raises doubts, then please disregard it and send what you have available.
What's important is that the subscriptions are included in the log, so that we can better compare subscriptions and updates received.
On the other question I will answer tomorrow.