Qlib Client-Server Framework

Introduction

Client-Server is designed to solve following problems

  • Manage the data in a centralized way. Users don’t have to manage data of different versions.
  • Reduce the amount of cache to be generated.
  • Make the data can be accessed in a remote way.

Therefore, we designed the client-server framework to solve these problems. We will maintain a server and provide the data.

You have to initialize you qlib with specific config for using the client-server framework. Here is a typical initialization process.

qlib init commonly used parameters; nfs-common must be installed on the server where the client is located, execute: sudo apt install nfs-common:
  • provider_uri: nfs-server path; the format is host: data_dir, for example: 172.23.233.89:/data2/gaochao/sync_qlib/qlib. If using offline, it can be a local data directory
  • mount_path: local data directory, provider_uri will be mounted to this directory
  • auto_mount: whether to automatically mount provider_uri to mount_path during qlib init; You can also mount it manually: sudo mount.nfs provider_uri mount_path. If on PAI, it is recommended to set auto_mount=True
  • flask_server: data service host; if you are on the intranet, you can use the default host: 172.23.233.89
  • flask_port: data service port

If running on 10.150.144.153 or 10.150.144.154 server, it’s recommended to use the following code to init qlib:

>>> import qlib
>>> qlib.init(auto_mount=False, mount_path='/data/csdesign/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
 [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
 [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
 Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
 Out[5]:
                            $close
 instrument datetime
 SH600000   2008-01-02  119.079704
            2008-01-03  113.120125
            2008-01-04  117.878860
            2008-01-07  124.505539
            2008-01-08  125.395004

If running on PAI, it’s recommended to use the following code to init qlib:

>>> import qlib
>>> qlib.init(auto_mount=True, mount_path='/data/csdesign/qlib', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
 [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
 [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
 Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
 Out[5]:
                            $close
 instrument datetime
 SH600000   2008-01-02  119.079704
            2008-01-03  113.120125
            2008-01-04  117.878860
            2008-01-07  124.505539
            2008-01-08  125.395004

If running on Windows, open NFS features and write correct mount_path, it’s recommended to use the following code to init qlib:

1.windows System open NFS Features
  • Open Programs and Features.
  • Click Turn Windows features on or off.
  • Scroll down and check the option Services for NFS, then click OK

Reference address: https://graspingtech.com/mount-nfs-share-windows-10/

2.config correct mount_path
  • In windows, mount path must be not exist path and root path,
    • correct format path eg: H, i…
    • error format path eg: C, C:/user/name, qlib_data…
>>> import qlib
>>> qlib.init(auto_mount=True, mount_path='H', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
 [39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
 [39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
 [39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
 [39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
 Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
 Out[5]:
                            $close
 instrument datetime
 SH600000   2008-01-02  119.079704
            2008-01-03  113.120125
            2008-01-04  117.878860
            2008-01-07  124.505539
            2008-01-08  125.395004

The client will mount the data in provider_uri on mount_path. Then the server and client will communicate with flask and transporting data with this NFS.

If you have a local qlib data files and want to use the qlib data offline instead of online with client server framework. It is also possible with specific config. You can created such a config. client_config_local.yml

provider_uri: /data/csdesign/qlib
calendar_provider: 'LocalCalendarProvider'
instrument_provider: 'LocalInstrumentProvider'
feature_provider: 'LocalFeatureProvider'
expression_provider: 'LocalExpressionProvider'
dataset_provider: 'LocalDatasetProvider'
provider: 'LocalProvider'
dataset_cache: 'SimpleDatasetCache'
local_cache_path: '~/.cache/qlib/'

provider_uri is the directory of your local data.

>>> import qlib
>>> qlib.init_from_yaml_conf('client_config_local.yml')
>>> from qlib.data import D
>>> D.features(['SH600001'], ['$close'], start_time='20180101', end_time='20190101').head()
 21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [__init__.py:16] - default_conf: client.
 [21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
 [21232:MainThread](2019-05-29 10:16:05,067) INFO - Initialization - [__init__.py:56] - provider_uri=/data/csdesign/qlib
 Out[9]:
                           $close
 instrument datetime
 SH600001   2008-01-02  21.082111
            2008-01-03  23.195362
            2008-01-04  23.874615
            2008-01-07  24.880930
            2008-01-08  24.277143

Limitations

  1. The following API under the client-server module may not be as fast as the older off-line API.
    • Cal.calendar
    • Inst.list_instruments
  2. The rolling operation expression with parameter 0 can not be updated rightly under mechanism of the client-server framework.

API

The client is based on `python-socketio<https://python-socketio.readthedocs.io>`_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure.

Class

class qlib.data.client.Client(host, port)

A client class

Provide the connection tool functions for ClientProvider.

__init__(host, port)

Initialize self. See help(type(self)) for accurate signature.

connect_server()

Connect to server.

disconnect()

Disconnect from server.

send_request(request_type, request_content, msg_queue, msg_proc_func=None)

Send a certain request to server.

Parameters:
  • request_type (str) – type of proposed request, ‘calendar’/’instrument’/’feature’.
  • request_content (dict) – records the information of the request.
  • msg_proc_func (func) – the function to process the message when receiving response, should have arg *args.
  • msg_queue (Queue) – The queue to pass the message after callback.