server bottleneck

[PaulW]

Hi,

I am using Lightstreamer Moderato in Command Mode as a server for a search engine that typically returns thousands of records per search. To get the information to the client's browser, we are sending thousands of ADD commands in each client session. The adapter generates the commands very quickly, but the transfer takes several minutes with an outbound throughput of only ~50 kilobytes per second and 100% CPU usage on a single core.

I tried a number of parameter mods to improve performance, such as increasing the max_buffer_size and sendbuf parameters (which were also causing problems) but the throughput is still only 50 KB/s. I debugged the server with VisualVM and found that the Pump Pool threads seem to be the source of the bottleneck. The server is creating a Pump Pool thread for each CPU core, but only one thread is active at a time and that thread uses 100% CPU. The obfuscated names of the methods that are consuming the CPU are com.lightstreamer.g.b.b.a and com.lightstreamer.o.b.g.j We are using Lightstreamer 6 but the same problem occurs with previous versions.

Can you suggest a fix for this?

Thanks very much,
Paul

[DarioCrivelli]

If you mean that a huge list is to be sent to a single client, I confirm that the delivery operation is sequential and involves a single thread of the pump pool.
This is not the typical usage and, for instance, I confirm that you have to enlarge <max_buffer_size> and <sendbuf>, whose defaults are for an intermittent update flow for many clients.
Are the ADD events supplied by the Data Adapter marked as part of the snapshot or as real-time updates?

Anyway, a CPU bottleneck was not expected.
Please specify the build of the Server related with the reported class names. We cannot find them in the latest 6.0 beta.
Could you provide more information? for instance, a screenshot from the Visual VM ?

[PaulW]

Thank you for your reply. I am attaching a screenshot of the VisualVM output showing high CPU usage by the PUMP POOLED threads and o.b.e.d and o.b.g.j methods. The method g.b.b.a also has a very high "Self Time" but does not appear to be the CPU bottleneck. I am using the latest version of Lightstreamer Moderato (Lightstreamer 6.0 b1 build 20140801). If you open the lightstreamer.jar file as a zip archive, the file /com/lightstreamer/o/b/e.class contains the method "d" and /com/lightstreamer/o/b/g.class contains the method "j". I am sending the ADD commands as real-time updates.

[DarioCrivelli]

The results from the VisualVM are quite puzzling, because they don't mention other methods invoked beside the two marked as hot spots;
and also because the first one is a two-line-long final method that I expected to be expanded inline by the JVM.
But, indeed, they spot an operation that is likely to be involved in this CPU problem; and it is not easy to fix it.
Actually, your usage of the COMMAND mode, by sending thousands of real-time ADD events in a loop, is a use case that was not expected and was not optimized properly.

You could resort to the snapshot case and send the events that make up a search result as part of a snapshot.
This means that you cannot have a persistent subscription on which to send the search results, but that you have to perform a one-shot subscription for each search, so that you can leverage the snapshot for the answer.
The snapshot in COMMAND mode is handled in a different way that does not involve the reported bottleneck.