R Client for lightstreamer

[DarioCrivelli]

Ok, so point 9) seems to match my assumption in the previous post.
But operating on the client side to reduce the updates of candlestick data to only the ones related to candlestick close seems impossible to me. For sure, you cannot achieve that by leveraging LS_requested_max_frequency.
I think that your only possibility is to:

• subscribe normally and read all updates,
• keep the state of the current candlestick by integrating the updates,
• however, whenever an update carries a new timestamp value, save the previous state as a full candlestick and clear the state to prepare for the next candlestick (but the incoming update should itself take care of the clearing).

I cannot understand point 10), but it seems just a reinforcement of point 9).

I also can't understand point 11). So far, the problem seems that, in streaming API you get more lines than needed, not fewer.
Perhaps you mean that if you cannot satisfy point 9) you have to give up subscribing to candlesticks and subscribe to ticks only and use the ticks to build the candlesticks yourself on the client side. Guidance on this is obviously out of our scope as Lightstreamer support.

[toolbox]

Yes, I understand that the guidance is limited, but I think it might be valid to keep the thread open so I can potentially update it. I assume that it is of benefit for Lightstreamer as a product that the users creates clients that are not officially released nor supported to empower and grow the usage of Lightstreamer.

[Grine1973]

Thats right , there are no significant disadvantages to use the old protocol.

[Alessandro]

@toolbox: Yes, we agree that it's good to keep this thread open, as this discussion is quite interesting to several Lightstreamer users.

Thanks,
Alessandro

[toolbox]

With some backwards-engineering I have now a working solution. The IG Streaming API shows now exactly the same 5 minutes OHLC(+Volume) data as the IG REST API. It will probably need major tuning to be optimized. I will summarize my findings and solution and post it here. Probably in generic terms to avoid confusion.

[Alessandro]

That's great news, thanks!

[toolbox]

Below summary is only valid based on following environment:
Server:
- Lightstreamer server positioned at IG premises.
- Server version less then 6.1
Client:
- old text protocol (not TLCP)
- OS - Linux
- Programming environment - "R".

################################################
# Summary
################################################
I am working on setting up an "automated trading system" completely
in the programming environment "R". As first step I created a download
environment based on REST API. During the test phase it was obviously that
the robustness was not acceptable, missing 5%-30% downloaded OHLC lines.
The resolution was set to 5 minutes candlesticks. The 5 min resolution means that the REST API
sends out "historical data" that is not meant change dynamically.

The reason for the missing lines was that IG sends out values "NULL" in their datalines without
any explanation of how to handle those "NULL" values. Due to that my client is
programmed to delete any line that contains minimum one "NULL".

To continue with my project I initiated the task to replace REST API with Streaming API.
(same bank though). To benchmark increase in quality of downloaded lines, the aim was
to set streaming API to 5 minutes candlesticks. You should be able to subscribe on "5 min candlestick" but I encountered 2 problems:

1) The streaming API from IG sends out all ticks, despite if you subscribe on "5 min candlesticks",
you do get the timestamps per each 5 minutes though.

2) Since you need to work with the stream as source, you need to let each line,
accumulate the changes until you "freeze" the line to be historical and will never change.
This is of course not valid for the last timestamp (since it will keep changing).

...further on tried to set the clients "LS_requested_max_frequency" in order to reduce the
stream intensity but the change did not have any impact on the stream at all.

################################################## ####################
# The basics of building the client in "R", without technical details.
################################################## ####################
1) Open up a stream connection using create_session.txt, that continuously saves to file (e.g. stream1.txt)
2) Request for a subscription using control.txt
3) Poll data from stream connection using bind_session.txt
4) Script to transform the stream1.txt to 5 minutes OHLC data.
5) Compare result between Streaming API <-> REST API.
6) Develop function to steer the stream (start, stop, etc).

################################################## ####################
# Result and needed improvements
################################################## ####################
I have managed to reduce missing OHLC data lines from REST API (5% - 30%) to Streaming API (roughly <5%). These <5% includes potential bottlenecks, such as programming language, OS, network, scheduler, modeling, architecture, etc. Benchmarking between REST API and Streaming API is currently not possible since Streaming API is "self-healing". If a line is missing, next production of line will include all historical lines, resulting in total success of 100%. From a "automated trading system" -persective, if the line is not there at a specific time, the loss for that line is 100%, and next decision is to be taken next successful line. To stresstest the system i now run 1 minute resolution.

The biggest back-draws right now is:
1) If the stream stops (hangs or breaks), the system stops to have updated download data.
2) Currently the transformation function uses a screen-shot of complete accumulated data,
at 5 p.m. it screen-shots the whole day.

[Alessandro]

Great, thanks for the update!