Qlib-Server: Quant Library Data Server

Introduction

Qlib-Server is the assorted server system for Qlib, which utilizes Qlib for basic calculations and provides extensive server system and cache mechanism. With Qlib-Server, the data provided for Qlib can be managed in a centralized manner.

Framework

_images/framework.png

The Client/Server framework of Qlib is based on WebSocket considering its capability of bidirectional communication between client and server in async mode.

Qlib-Server is based on Flask, which is a micro-framework for Python and here Flask-SocketIO is used for websocket connection.

Qlib-Server provides the following procedures:

Listening to incoming request from client

The clients will propose several types of requests to server. The server will parse the requests, collect the identical requests from different clients, record their session-ids, and submit these parsed tasks to a pipe. Qlib use RabbitMQ as this pipe. The tasks will be published to a channel task_queue.

RequestListener is used for this function:

After receiving these requests, the server will check whether different clients are asking for the same data. If so, to prevent repeated generation of data or repeated generation of cache files, the server will use Redis to maintain the session-ids of those clients. These session-ids will be deleted once this task is finished. To avoid IO conflicts, Redis_Lock is imported to make sure no tasks in redis will be read and written at the same time.

Responding clients with data

The server consumes the result from message_queue and get the session-ids of the clients requring this result. Then it responds to these clients with results.

RequestResponder is used for this method.

The two class above is combined as RequestHandler, which is responsible for communicating with clients.

Accepting tasks from RabbitMQ and processing data

The server will automatically collect tasks from RabbitMQ and process the relevant data. RabbitMQ provides a mechanism that when the server consumes a task, a callback function is triggered. The data processing procedure is implemented within these callbacks and it currently supports the three types of tasks corresponding to above:

  • Calendar
  • Instruments
  • Features

DataProcessor is used for this function.

The server will use qlib.data.Provider to process the data. RabbitMQ also provides a mechanism that can make sure all tasks is succesfully consumed and completed by consumers. This requires the consumer call ch.basic_ack(delivery_tag=method.delivery_tag) after succesfully processing the data. If the task is not acked, it will return to the pipe and wait for another consuming.

Once the task is finished, a result (could be data or uri) will be published to another channel message_queue.