Page 1 of 2 12 LastLast
Results 1 to 10 of 18
  1. #1

    R Client for lightstreamer

    Hi. I have setup an R client for lightstreamer towards IG,
    to subscribe on 1 minute OHLCV (Open, High, Close, Volume) data.

    Background to move into lightstreamer is that IG:s REST API
    sends to many NULL in their content, resulting in that my system removes 30% OHLC lines.

    I have a client lightstreaming working that looks as below.
    Since this is my first time to setup a client following the TLCP,
    i am not familiar with how the stream exactly should look like.
    I thought that if I subscribe on 1 minutes data I would get PROBE-lines and the OHLCV data,
    but I get also all the data changes in between.

    My parameter settings:

    LS_session=Sa9447dbaae6d3456T2643148
    LS_op=add
    LS_id=CHART:IX.D.OMX.IFM.IP:1MINUTE
    LS_schema=UTM + BID_OPEN + BID_HIGH + BID_LOW + BID_CLOSE + CONS_END
    LS_mode=MERGE
    LS_table=1
    LS_reqId=1
    LS_subId=1
    LS_group=item1
    LS_data_adapterQUOTE ADAPTER

    Question1) Could you please confirm if a subscription of 1 minutes data should have data in between the 1 minutes screenshots?

    Question2) Above each trading-data line there is a character/number combination.
    I do not find in the documentation what they mean. Could you please clarify or refer where
    to find explanation.

    Question3) Is it correct to assume that an empty filed between two "pipe" characters should be
    interpreted as that value is unchanged thus if having a dataframe one should populate the unchanged value with previous changed value.

    #### Extraction from the data stream

    << C
    << 1,1||||||1
    <<

    << 7
    << PROBE
    <<

    << 1D
    << 1,1|1497276480000|$|$|$|$|0
    <<

    << 27
    << 1,1||1639.75|1639.75|1639.63|1639.63|
    <<

    << 19
    << 1,1|||1639.88||1639.88|
    <<

    << 19
    << 1,1|||1640.13||1640.13|
    <<

    << 19
    << 1,1|||1640.38||1640.25|
    <<

    << 12
    << 1,1|||||1640.38|
    <<

    << 12
    << 1,1|||||1640.13|
    <<

    ###############################

    My final goal is to have the data as below, in a time-series format. (called xts in R).
    I am aware that I might be need to decode the pipe-separated list-of-values.

    [open], [high], [low], [close], [volume]
    line1-1min resolution - [streamvalue - open], [streamvalue - high], [streamvalue - low], [streamvalue - close], [streamvalue - volume]
    line2-1min resolution - [streamvalue - open], [streamvalue - high], [streamvalue - low], [streamvalue - close], [streamvalue - volume],
    etc...etc....

    / Regards, Alonso
    Last edited by toolbox; June 12th, 2017 at 04:49 PM.

  2. #2
    Hi Alonso,

    1) The update maximum frequency is set by the server according to the data feed frequency. So if you notice a difference from the expected frequency, you should contact the server maintainers.
    Alternately you could explicitly ask the server for a lower frequency adding the parameter LS_requested_max_frequency to your subscription requests (see the section 4.3.1 Subscription Control Connections of the document linked below).

    2) Observing the excerpted log, I warn you that
    a) IG server is not using the TLCP protocol but the old text protocol. The document explaining the old protocol is not listed in the Docs page anymore,but you could retrieve a copy looking inside the distributions of the servers below version 6.1 (for example here).
    The complete syntax is explained in the document Network Protocol Tutorial under DOCS-SDKs/sdk_client_generic/doc.

    b) the hex numbers as C, 7,1D, 27 above the protocol messages are the sizes of the HTTP 1.1 chunks and they are not part of the text protocol.

    3) The “pipe format” is explained in the section 4.5.1 Update messages of the above document.


    Regards
    Alessandro Carioni
    Last edited by Alessandro Carioni; June 13th, 2017 at 10:14 AM.

  3. #3
    Administrator
    Join Date
    Jul 2006
    Location
    Milan, Italy
    Posts
    515
    The documentation of the right version of the network protocol used by IG is linked on IG's site: https://labs.ig.com/lightstreamer-downloads

    The link sends here.

  4. #4
    Thanks Alessandro . It looks like I will be able to finalize the R client for lightstreamer with
    some more hard work. I still need to transform the stream into a OHLCV data.frame for my
    automated trading system to use that as a base for analysis.

    4) Regarding avoiding tick-data and force client to show 1MINUTE or 5MINUTE
    I have now tested with IG:s Streaming Companion and despite if putting the {scale} to either [1MINUTE, 5MINUTE or HOUR], I always see the tick changes. I did change the setting, in my client, of [LS_requested_max_frequency] to both 60 seconds and changed it later for 300 seconds. There is not really any change in regards to visible tick-data in the stream. Adjusting the above parameter does not remove the stream of tick-data. The parameter settings seem otherwise fine since it looks like [BID_OPEN, BID_HIGH, etc] is allocated to candlestick data, having the {scale} after the {epic}. I assume I need to contact IG for clarifications of the [LS_requested_max_frequency].

    5) Regarding using the old text protocol instead of TLCP
    Since I come from using IG API REST and loosing up to 30% OHLC lines, it is still a great improvement for my client solution using lightstreamer. Are there significant disadvantages to use the old protocol? Do you recommend that I request IG to upgrade to TLCP?

    6) Regarding setting up a lightstreamer to use the test scenarios.
    Would you recommend for me to setup the lightstreamer server to understand
    more the traffic flow? Since I am using HTTPS i cannot always sniff the TCP/HTTPS stream with an dedicated tools such as wireshark.

    / Regards, Alonso
    Last edited by toolbox; June 14th, 2017 at 11:30 AM.

  5. #5
    Hi Alonso,

    4) It is not clear to me what is the exact meaning of the scale granularity (1MINUTE, 5MINUTE, …) you mention. I interpret is as the number of updates per unit of time, but maybe you should consult IG for the right meaning. If we stick with the above interpretation, then to limit frequency of updates you can leverage LS_requested_max_frequency. But keep in mind that LS_requested_max_frequency specifies the number of updates per second. So if you want just one update per minute, you must set the parameter to 0.017 (i.e. 1/60).

    5) TLCP protocol has a more uniform and compact syntax, but it is only supported by server version 6.1 or above. So IG should update the server in order that the clients may use TLCP.

    6) A local server is surely useful while you are learning the technology. You can install the Stock-List Demo, which simulates stock quotes, on your local server, and then connect with your client.
    The server 6.1 can talk TLCP and the old protocol, so you can experiment with both of them.

    Regards
    Alessandro Carioni
    Last edited by Alessandro Carioni; June 15th, 2017 at 10:27 AM.

  6. #6
    Administrator
    Join Date
    Jul 2006
    Location
    Milan, Italy
    Posts
    515
    5) To directly answer your question, there are no significant disadvantages to use the old protocol.

  7. #7
    Thanks Alessandro.

    4) The scale granularity or resolution is by IG defined to as an item, with following syntax: CHART:{epic}:{scale}, whereof {scale} is the item. See IG streaming API reference: https://labs.ig.com/streaming-api-reference

    It is basically the time interval between 2 candlesticks. From a timestamp perspective it is the timewise startpoint of a candlestick, but from a value perspective it is the LastTradePrice meaning the OHLC values as a screenshot exactly before next (new timestamp arrives). This means that taken account the updates in Lightstreamer one would need to take the timestamp + all updates until next timestamp arrives. Unless IGs candlestick solution would work because then they should send only the LastTradePrice values, which currently does not work for me.

    I have tested setting the [LS_requested_max_frequency] to 0.017, for the stream to go down to 1 minute updates, but without success. I thought of change the [LS_mode] and try Command, Distinct or RAW, but seems that the IG [Chart subscriptions - Consolidated (candle) data] is supposed to be set with [LS_mode] Merge.
    Last edited by toolbox; June 19th, 2017 at 09:47 AM.

  8. #8
    4) Here is an example of scale with 5 minutes resolution, see bold text. To simplify reading I have transformed UTM format to POSIXct,and left the stock values same for all the 3 lines.

    1,1|2017-06-19 09:40:00 CEST|1652.05|1652.18|1649.93|1650.18|0
    1,1|2017-06-19 09:45:00 CEST|1652.05|1652.18|1649.93|1650.18|0
    1,1|2017-06-19 09:50:00 CEST|1652.05|1652.18|1649.93|1650.18|0

  9. #9
    Administrator
    Join Date
    Jul 2006
    Location
    Milan
    Posts
    958
    Hi, If I understand correctly, the Consolidated candle item provides the state of the current candle.
    So, upon each new contract that causes the current candle to change, the whole candle coordinates are sent (as I infer from the traffic shown on the first post).
    Then, when a candle is completed (according to the time-slot implied in the item name), the updates begin to refer to the next candle, as identified by the increased timestamp field.

    Do you agree on this interpretation?
    If so, please clarify what is the alternative behavior that you would like to achieve.
    Perhaps you would like to get candle updates only for consolidated candles, while you are not interested in temporary candle states?

  10. #10
    Clarifications of goals:
    8) Move from REST API to Streaming API due to REST API loosing 30% datalines (removing lines with NA).
    9) Being in Streaming API secure candlestick OHLC data. Basically data when each candlestick is closed and do not update more.
    10) In streaming API secure stability if data with 1 minute OHLC data.
    11) If not getting all candlestick lines in streaming API, use the stream to construct the candlestick.

    To summarize: I am using the lightstreamer technique to significantly improve the data quality and stability. However, my tool summarize all the lightstream updates within a candlestick resolution, eg. 5 minutes and stores the closure status of the candlestick.
    Last edited by toolbox; June 21st, 2017 at 12:35 AM.

 

 

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT +1. The time now is 08:00 PM.