Hi, I'm new to algo trading and getting started with different systems. So far the plan is to get real-time data for multiple symbols using Fyers V3 API and send it Kafka from where I can consume and run my logic on different nodes. As part of this, I want to subscribe and unsubscribe to some of the symbols in real-time based on my strategies.
But the problem I'm facing is, if I run the websocket in Foreground mode I'm not able to run the socket's subscribe and unsubscribe commands. If I run the websocket in Background mode, the data is getting written to a file (which I'm not able to get it working yet), I'm unable to use the on_message callback function to write the symbol details to Kafka brokers for further processing.
Can you please help with what is the best way to subscribe and unsubscribe from the symbols while still being able to use something like on_message callbacks so that I can write data to Kafka for further processing?
Programming Language: Python