These are chat archives for thunder-project/thunder

31st
Mar 2016
Bradley Zhou
@bradleyzhou
Mar 31 2016 05:18
Help on loading HDFS files:
The problem : When I try to load images or series by tsc.loadSeries("hdfs://node01:8020/path/to/images") I get NotImplementedError
Enviroment: We have a local cluster, running Hadoop 2.6.3 and Spark 1.4.1 using YARN. Thunder version is 0.6.0
Notes: Loading from NFS mounted drives is fine.
Bradley Zhou
@bradleyzhou
Mar 31 2016 10:10

Hmm ... I found this in thunder/rdds/fileio/readers.py

"""Classes that abstract reading from various types of filesystems.
Currently two types of 'filesystem' are supported:
* the local file system, via python's native file() objects
* Amazon's S3 or Google Storage, using the boto library (only if boto is installed; boto is not a requirement)"""

Seems that reading files from HDFS is not supported.