Page 1 of 2 12 LastLast
Results 1 to 10 of 11
  1. #1
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17

    Bandwidth and event pushing

    Hi all,

    I have implemented my own adapter and my own client using the lighstreamer server and client APIs. So far, so good, everything works.

    In the client API I used the proxy approach to see the bandwidth usage. I've seen that the average usage is 1.5Kbit/sec. Currently I'm pushing Java objects from the server (ok, the attributes, I build the objects afterwards) and currently I'm getting an average of 3 objects/second. That seems quite slow, so probably I'm quite sure that I'm doing something wrong or that there is a way to tune this.

    I tried to change the max_bandwidth, max_frequency and buffer_size parameters in the adapters.xml file but with no apparent effect.

    So the question is, is there a way to improve the bandwidth/throughput? would it be a solution to implement more items and instead of doing a single subscription do many subscriptions?

    Thanks,
    Martin

  2. #2
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17
    I forgot to say that I'm running the Vivace edition, quite important due to bandwidth restrictions I guess.

  3. #3
    Power Member
    Join Date
    Jul 2006
    Location
    Cesano Maderno, Italy
    Posts
    784
    Hi,

    please, could you modify your lightstreamer_log_conf.xml file to set the "subscriptions" and "pump" categories to DEBUG level and then attach a server log showing the issue?

  4. #4
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17

    Logs attached

    Sure.

    I'm currently trying to simulate a trading environment. I've deployed an adapter that contains an embedded rate generator. This generator creates rates at variable speeds, i.e. Some rates are generated more frequently than others. My Lighstreamer adaptor listens to that generator and each time it receives a rate pushes it to the client side.

    I have been doing some tests and also looking at the log files and now the problem seems clear to me (of course, probably it's not a problem). Basically I'm pushing all the rates objects through the same channel ("rates"). So, in the log file you can see how some currency pair rates are being discarded. See attached file "first-try.txt". In the client, I only receive the data that is pumped in the event.

    So I guess I should create a channel per currency pair. Does this have any consequence in the communication? (i.e. does this mean more sockets open? does this mean more bandwidth usage? etc.)

    Thanks for the useful help and regards,
    Martin

    PS. Note that I have stripped down the log for being able to upload it.
    Attached Files Attached Files

  5. #5
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17

    Puzzled about second test

    Hi again.

    I have upgraded my test to create six different subscription channels one for each currency pair I have.

    However, I'm now puzzled because the adapter seems to successfully send all the rates in the currency pairs some times, but there are other times that it performs throttling and only sends some of the rates. In the attached file "second-try.txt" see how the first time only one rate is sent, but in the second time the four rates are sent, the next time only 2 of five rates are sent, the next time 4/7, etc.

    I'm sure there is some logic behind of this behaviour. Any tips?

    Thanks,
    Martin
    Attached Files Attached Files

  6. #6
    Power Member
    Join Date
    Jul 2006
    Location
    Cesano Maderno, Italy
    Posts
    784

    max_delay_millis

    Hello,

    Are you using the MERGE subscription mode right?

    The problem is that since the updates are all published on the same item, Lightstreamer merges the updates if needed:

    Code:
    08.May.07 09:02:05,934 <TRACE> INCOMING DATA for rates --> {bidValue=0.07943651574153077, sellCCY=USD, bidCCY=EUR, sellValue=0.19836981819406774}
    08.May.07 09:02:05,935 <TRACE> INCOMING DATA for rates --> {bidValue=0.04185744998798879, sellCCY=JPY, bidCCY=EUR, sellValue=0.002155323147410959}
    08.May.07 09:02:05,936 <TRACE> INCOMING DATA for rates --> {bidValue=0.9033005599346077, sellCCY=USD, bidCCY=JPY, sellValue=0.3281485803285823}
    08.May.07 09:02:05,937 <TRACE> INCOMING DATA for rates --> {bidValue=0.947976333651295, sellCCY=GBP, bidCCY=EUR, sellValue=0.38439026792729647}
    08.May.07 09:02:06,044 <TRACE> INCOMING DATA for rates --> {bidValue=0.9040453255654483, sellCCY=USD, bidCCY=EUR, sellValue=0.9458336664422791}
    08.May.07 09:02:06,045 <TRACE> INCOMING DATA for rates --> {bidValue=0.9708380297991672, sellCCY=GBP, bidCCY=EUR, sellValue=0.215475738803115}
    08.May.07 09:02:06,134 <TRACE> Pumping event in session S1321751661N1: 1,1,1||0.9708380297991672||0.215475738803115
    In the example 6 updates arrives to the server but only one is forwarded to the clients.

    If yoo take a look at the timings of the "Pumping in Session" messages you will see a 200ms pattern:

    09:02:05,537
    09:02:05,734
    09:02:05,934
    09:02:06,134
    and so on...

    Those 200ms are set in the Lightstreamer_conf.xml file:
    Code xml:
    1.  <max_delay_millis>200</max_delay_millis>

    What appens is that Lightstreamer is enabled to send just one packet each 200ms (this is a session related delay) unless there is too much data to be sent in one single packet (due to frequency limits bandwidth limits etc this is just a simple explanation of the delivering algorithm). Btw since you publish every update to the same item, after 200ms from the last "Pumping event in session" - from Lightstreamer point of view - there is only one relevant update that must be sent (the latest).


    Quote Originally Posted by churrusco
    So I guess I should create a channel per currency pair.
    This is indeed a good idea.

    Quote Originally Posted by churrusco
    does this mean more sockets open?
    The socket is always the same as Lightstreamer protocol take care of handling more items on the same data flow

    Quote Originally Posted by churrusco
    does this mean more bandwidth usage?
    Obviously more bandwidth will be used if more updates reach the client, btw you can limit the bandwidth both client and server side.

  7. #7
    Power Member
    Join Date
    Jul 2006
    Location
    Cesano Maderno, Italy
    Posts
    784
    Hi,

    You wrote another post while I was writing the response for the previous one

    In the new log you can see how the merge algorithm works when different items are used:
    Code:
    08.May.07 09:56:57,239 <TRACE> INCOMING DATA for EUR/GBP --> {bidValue=0.3952132682553846, sellCCY=GBP, bidCCY=EUR, sellValue=0.1582634326389245}
    08.May.07 09:56:57,338 <TRACE> INCOMING DATA for EUR/USD --> {bidValue=0.9052316638725739, sellCCY=USD, bidCCY=EUR, sellValue=0.47263148734258653}
    08.May.07 09:56:57,339 <TRACE> INCOMING DATA for EUR/GBP --> {bidValue=0.7117531450899011, sellCCY=GBP, bidCCY=EUR, sellValue=0.5475364773823609}
    08.May.07 09:56:57,340 <TRACE> INCOMING DATA for USD/NZD --> {bidValue=0.3461941736979528, sellCCY=NZD, bidCCY=USD, sellValue=0.19362361881184487}
    08.May.07 09:56:57,439 <TRACE> INCOMING DATA for EUR/USD --> {bidValue=0.7626206088735283, sellCCY=USD, bidCCY=EUR, sellValue=0.5145519477153373}
    08.May.07 09:56:57,440 <TRACE> INCOMING DATA for JPY/USD --> {bidValue=0.204416519895124, sellCCY=USD, bidCCY=JPY, sellValue=0.2764089416286716}
    08.May.07 09:56:57,440 <TRACE> INCOMING DATA for EUR/GBP --> {bidValue=0.5949231423651182, sellCCY=GBP, bidCCY=EUR, sellValue=0.5972830556785825}
    08.May.07 09:56:57,450 <TRACE> Pumping event in session S-725498679N1: 4,1,1||0.7626206088735283||0.5145519477153373
    3,1,1||0.5949231423651182||0.5972830556785825
    6,1,1|USD|0.3461941736979528|NZD|0.19362361881184487
    2,1,1||0.204416519895124||0.2764089416286716
    Inside this 200ms you can see that the server accumulated 7 events but 4 are sent to the client. If you look further you can see that those 7 updates pertains to only 4 items because there are 3 EUR/GBP and 2 EUR/USD events so that the server to send the latest update per each item have to send only 4 updates.

    Hope that helps.

  8. #8
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17
    Absolutely it helps, and it makes quite a lot of sense indeed. Great feature and quite easy to configure. Actually it was my fault haven't re-read the documentation. I read it one or two months ago and almost forgot everything about these modes. I believe it has to do with the age

    Happily enough the simulation is now working fine so I can start stressing it Also I would like to comment some minor things:

    - I found that with the RAW mode the server log messages are weird as it seems as if Lighstreamer was throttling messages. But the reality is that all of them arrive at the client even when in the logs it seems that only one is pushed. See attached file.

    - I also see that there is a timeout and the session is maintained idle for 30 seconds. I guess this feature is to optimize AJAX communication, isn't it? (I say it because we are targeting an Applet environment so our ideal is just a "0" timeout with no back buttons and this browser stuff.)

    Cheers,
    Martin
    Attached Files Attached Files

  9. #9
    Member
    Join Date
    Apr 2007
    Location
    Kerry
    Posts
    17
    Hi Mone,

    Just forget the first minor issue. I just have realized the obvious, that there are different pump threads and this is not a sequential process

    Cheers,
    Martin

  10. #10
    Power Member
    Join Date
    Jul 2006
    Location
    Cesano Maderno, Italy
    Posts
    784
    The session is kept open for some time.
    There is a set of session configurations that you can tune to let Lightstreamer behave as you desire.

    If you have the original (i.e. the one in the distributed zip) Lightstreamer_conf.xml file you will see an entire section called "SESSION CONFIGURATION".
    Read the comments inside it to have an overview of the session handling.

    Feel free to ask if something is unclear.

 

 

Similar Threads

  1. Bandwidth in iOS
    By bor306 in forum Client SDKs
    Replies: 7
    Last Post: July 11th, 2012, 03:28 PM
  2. AMF and Consuming Network Bandwidth?
    By ernivan in forum Client SDKs
    Replies: 1
    Last Post: March 11th, 2011, 10:35 AM
  3. Replies: 1
    Last Post: March 11th, 2010, 04:53 PM
  4. Replies: 4
    Last Post: August 14th, 2009, 04:28 AM
  5. Pushing complex data structure
    By shreyaspurohit in forum Adapter SDKs
    Replies: 4
    Last Post: March 30th, 2009, 12:06 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT +1. The time now is 04:46 PM.