Stream data in bulk to client using LightStreamer (.NET)

Dear all,

How would you use the Lightstreamer technology to stream some bulk-data to a web-client?

Here is an explanation of the scenario I am trying to accomplish:

I have a large dataset sitting somewhere server-side - e.g. 500.000 records - that I want to deliver to a web-client (JavaScript, Flex, etc...). This is too large a file to simply download with a vanilla-AJAX call. So I want to stream this file in small little chunks (say, in bits of json or xml).

As chunks of that data come in, the client is doing stuff with it: Drawing a graph, calculating some aggregates, keeping relevant bits&pieces in memory - depending on the app logic.

Once all 500.000 records have arrived, I want for that particular session to close the connection to the server.

So in a sense, I am not really interested in subscribing to some sort of "real-time pulse" of data (like the Stock example - or most other Lightstreamer demos). I just want to take advantage of streaming technology in Lightstreamer to deliver a large dataset in small pieces.

Think of it as watching little Youtube videos online - each user gets the same video streamed to him start to finish. This as opposed to watching a live video-stream. I want to emulate the former as opposed to the latter for json/xml/x-amf data.

I need some help. Most of the samples on LightStreamer I can find are along the lines of displaying real-time data - sometimes filtered on user-preferences - but still not exactly what I am looking for.

Many many thanks to anybody who can put me on my way - to sample code (.NET preferably), other threads, ... whatever you think might help.

thank you,

Thomas

You can certainly use Lightstreamer to send a sequence of pieces which make up a full content, encoded as a sequence of updates for a field of an item,
provided that your code takes care of forming the pieces from the original content on the adapter side and of rebuilding the original content on the client side.
After receiving the last piece, from your client side callback you can unsubscribe from the item or even close the connection to the Server.

What's important is that you turn off any conflation and request your item either in RAW mode or in DISTINCT mode with the "unfiltered" flag set (the latter is done in different ways, depending on the client API).
In both cases, the Data Adapter, upon subscription, just has to send all the (pseudo)updates. The Server ensures that all updates will reach the client, provided that only one client is currently subscribed to the item. If more than one client need the same data, you have to use different items each time (actually, different clients can ask for the same item name and the "getItems" method of the Metadata Adapter can be leveraged to force the differentiation internally).

By setting frequency constraints on the item, you can ensure that the pieces reach the client gradually, with no need to supply the updates gradually from the Data Adapter.
Note that editions other than Vivace already impose a maximum frequency constraint.
Hence, you should ensure that the internal buffer size limit (<max_buffer_size> in the Server configuration file) is set large enough.

Actually, by using DISTINCT mode, you could have Lightstreamer Server keep the (pseudo)updates in the item snapshot (which must be configured large enough).
In this way, you could even use the same item to serve requests from multiple clients: clients subscribing to the item while others are already receiving it will get all the pieces already produced through the item snapshot; just note that custom frequency limits do not apply to the snapshot.