Qlib-Server
: Quant Library Data Server¶
Introduction¶
Qlib-Server
is the assorted server system for Qlib
, which utilizes Qlib
for basic calculations and provides extensive server system and cache mechanism. With Qlib-Server
, the data provided for Qlib
can be managed in a centralized manner.
Framework¶
The Client/Server
framework of Qlib
is based on WebSocket
considering its capability of bidirectional communication between client and server in async mode.
Qlib-Server
is based on Flask, which is a micro-framework for Python and here Flask-SocketIO is used for websocket connection.
Qlib-Server
provides the following procedures:
Listening to incoming request from client¶
The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. Qlib
use RabbitMQ as this pipe. The tasks will be published to a channel task_queue.
RequestListener is used for this function:
After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use Redis to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, Redis_Lock is imported to make sure no tasks in redis will be read and written at the same time.
Responding clients with data¶
The server consumes the result from message_queue and get the session-ids of the clients requring this result. Then it responds to these clients with results.
RequestResponder is used for this method.
The two class above is combined as RequestHandler, which is responsible for communicating with clients.
Accepting tasks from RabbitMQ and processing data¶
The server will automatically collect tasks from RabbitMQ and process the relevant data. RabbitMQ provides a mechanism that when the server consumes a task, a callback function is triggered. The data processing procedure is implemented within these callbacks and it currently supports the three types of tasks corresponding to above:
- Calendar
- Instruments
- Features
DataProcessor is used for this function.
The server will use qlib.data.Provider to process the data. RabbitMQ also provides a mechanism that can make sure all tasks is succesfully consumed and completed by consumers. This requires the consumer call ch.basic_ack(delivery_tag=method.delivery_tag) after succesfully processing the data. If the task is not acked, it will return to the pipe and wait for another consuming.
Once the task is finished, a result (could be data or uri) will be published to another channel message_queue.