Python: lightstream data and have another process access last 15minutes histo

Hello,
I followed your github example for Python and it works great with gainCapital (thank you).
That said, I am trying to keep the last 15 minutes of historics (or other timeperiod), in a shared memory class, which i would be able to access from another processor to compute statistics. Ideally I would use a third processor to manage orders sent/book positions.
So, I would ideally use mulitprocessing, where 1 processor would be used to handle the lightstreamer and fill histo of 15 minutes period, another processor to check values and calculated statistics and a 3rd one to handle the orders sent, and check positions, as well as P&L.
Any help would be greatly appreciated.

So far i modified the "_on_item_update" class method to add any new data to a database but I realised that calls to database takes too much time, hence looking at other alternative.
Modified code (using pymysql):

def _on_item_update(self, item_update):

self.insert_to_mdb_with_pymysql(
(
item_update["values"]["MarketId"],
_Datetime.utcfromtimestamp(
int(
str(
str(item_update["values"]["TickDate"]).split(sep="(")[1]
).split(")")[0]
)
/ 1000
),
item_update["values"]["Bid"],
item_update["values"]["Price"],
item_update["values"]["Offer"],
)
)

That bit of code works but it won't work for what i am trying to achieve.

I would greatly appreciate help regarding, program structure, libraries, and if possible an example.

Thank you for you help.
NB: I was thinking to keep live in memory the last 15 minutes of data from my subscribed stocks but it might be better to save data to harddrive in csv or other best practise and make a call for the desired time line: I am exploring here, so would apreciate the help.

Best
Nono

Hi Nono,

unfortunately, I'm not an expert in Python development, but it looks like a multithreading approach is the best way to achieve your targets.
If inserting data into the database takes too much time, then probably you should switch to a sort of asynchronous task that will be in charge of only operating on the DB. Otherwise, you risk blocking too often the thread calling the _on_item_update method.
For example, you could enqueue items onto a buffer which will be processed by another thread, whose task is to update the DB.

Best,
Gianluca