If you need to use the latest data of yahoo finance, you can download and dump:
download data: https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo
dump data: https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-format
When downloading data and dump data, the target directory should not overwrite the "origin data" directory, it will not affect the "origin data"
You can use them for your real time data.
https://github.com/microsoft/qlib/blob/090b68e44ea081c6802e91daa2821ba480490e6a/scripts/dump_bin.py#L225
In terms of install from source, could you provide the version of cython and gcc on your machine? Thanks 😊
Ref($close, -2)/Ref($close, -1)-1 (also used in Alpha158 and Alpha360), which is inconsistent with "the label is formulated as Ref($close, -1)/$close - 1" in the note. Thanks.
Ref($close, -1)/$close - 1 was used a lot in https://qlib.readthedocs.io/en/latest/reference/api.html . May I know the purpose of the difference?
Ref($close, -1)/$close - 1 means the change from T to T+1, which can be used in backtest.In model training, we use the label
Ref($close, -2)/Ref($close, -1)-1that means the change from T+1 to T+2 rather than Ref($close, -1)/$close - 1, of which the reason is that when you get T day close price of a china stock, you can buy it in T+1 day and sell it in T+2 day.
For example, you can change ge label in the config.
https://github.com/microsoft/qlib/blob/main/examples/benchmarks/LSTM/workflow_config_lstm_Alpha158.yaml#L32
@BigW
The expression engine of Qlib provides similar problem like https://twopirllc.github.io/pandas-ta/#indicators-by-category
Users are encouraged to implement technical indicators with it.
But Qlib doesn't force users to use it. If you prefer pandas-ta, you can implement your technical indicates in the datahandler after retrieving the raw data.
code sample below Running the model: LightGBM for iteration 1...
40048:MainThread INFO - qlib.Initialization - [config.py:276] - default_conf: client.
40048:MainThread WARNING - qlib.Initialization - [config.py:292] - redis connection failed(host=127.0.0.1 port=6379), cache will not be used!
40048:MainThread INFO - qlib.Initialization - [init.py:46] - qlib successfully initialized based on client settings.
40048:MainThread INFO - qlib.Initialization - [init.py:47] - data_path=/home/warren/.qlib/qlib_data/cn_data
Killed
Deleting the environment: /tmp/tmp01wsrv_h...
Retrieving results...
39831:MainThread ERROR - qlib.workflow - [utils.py:35] - An exception has been raised[ValueError: No valid experiment has been found, please make sure the input experiment name is correct.].
File "run_all_model.py", line 305, in <module>
fire.Fire(run) # run all the model
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(varargs, **kwargs)
File "run_all_model.py", line 62, in _return_wrapped
return function_to_decorate(args, **kwargs)
File "run_all_model.py", line 287, in run
results = get_all_results(folders)
File "run_all_model.py", line 148, in get_all_results
exp = R.get_exp(experiment_name=fn, create=False)
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/workflow/init.py", line 239, in get_exp
return self.exp_manager.get_exp(experiment_id, experiment_name, create)
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/workflow/expm.py", line 149, in get_exp
exp, is_new = self._get_exp(experiment_id=experiment_id, experiment_name=experiment_name), False
File "/home/warren/anaconda3/envs/QLIB3TEST/lib/python3.8/site-packages/qlib/workflow/expm.py", line 298, in _get_exp
raise ValueError(
ValueError: No valid experiment has been found, please make sure the input experiment name is correct.
@you-n-g 以下是代码部分：import qlib
import pandas as pd
from qlib.config import REG_CN
from qlib.workflow import R
from qlib.contrib.report import analysis_model
provider_uri = "~/.qlib/qlib_data/cn_data" # target_dir
qlib.init(provider_uri=provider_uri, region=REG_CN)
my_rid = "3f999767c71b499dbe0dc3c3480067d3"
recorder = R.get_recorder(my_rid, experiment_name="backtest_analysis")
pred_df = recorder.load_object("pred.pkl")
label_df = recorder.load_object("label.pkl")
label_df.columns = ['label']
pred_label=pd.concat([label_df,pred_df],axis=1,sort=True).reindex(label_df.index)
graph_names: list = ["group_return", "pred_ic", "pred_autocorr","pred_turnover"],
Error : 'XAxis' object has no attribute '_gridOnMajor'
f1 = analysis_model.model_performance_graph(pred_label,show_notebook=False)
graph_names=["group_return"],OK !
f2 = analysis_model.model_performance_graph(pred_label,graph_names=["group_return"],\
show_notebook=False)
graph_names=["pred_ic"],Error : 'XAxis' object has no attribute '_gridOnMajor'
f3 = analysis_model.model_performance_graph(pred_label,graph_names=["pred_ic"],\
show_notebook=False)
谢谢了!
@you-n-g df = D.features(D.instruments(market="all"), ["$DWJZ", "$LJJZ"],start_time='2018-01-01', end_time='2021-03-22', freq="day")
market = "all"
benchmark = "000001"
data_handler_config = {
"start_time": "2018-01-01",
"end_time": "2020-08-01",
"fit_start_time": "2018-01-01",
"fit_end_time": "2018-12-31",
"instruments": market,
}
task = {
"model": {
"class": "LGBModel",
"module_path": "qlib.contrib.model.gbdt",
"kwargs": {
"loss": "mse",
"colsample_bytree": 0.8879,
"learning_rate": 0.0421,
"subsample": 0.8789,
"lambda_l1": 205.6999,
"lambda_l2": 580.9768,
"max_depth": 8,
"num_leaves": 210,
"num_threads": 20,
},
},
"dataset": {
"class": "DatasetH",
"module_path": "qlib.data.dataset",
"kwargs": {
"handler": {
"class": "Alpha158",
"module_path": "qlib.contrib.data.handler",
"kwargs": data_handler_config,
},
"segments": {
"train": ("2018-01-01", "2018-12-31"),
"valid": ("2019-01-01", "2019-12-31"),
"test": ("2020-01-01", "2020-08-01"),
},
},
},
}
model initiaiton
model = init_instance_by_config(task["model"])
dataset = init_instance_by_config(task["dataset"])
@gxxuej Hi,
I can’t reproduce this bug in the same environment.
You can use the latest qlib(https://github.com/microsoft/qlib) for more information: cd scripts && python collect_info.py all
@mozi7 Can you give us more details about your error?
Thanks
We have include pip install test in our CI
https://github.com/microsoft/qlib/blob/70c84cbc77e52bea67e4528d6f79e2e462a4ffa6/.github/workflows/test.yml#L39
It suceeded in all kinds of platforms. You can check the commands it uses.