These are chat archives for thunder-project/thunder

25th
Sep 2015
Gilles Vanwalleghem
@Yassum
Sep 25 2015 02:57

Hi there, I don't know if it is related to the upgrade to 5.0.1, but now I get a NameError: name 'tsc' is not defined, if I manually start a SparkContext, I get another error
sc = SparkContext tsc=thunder.ThunderContext(sc)
I get

/home/ubuntu/anaconda/lib/python2.7/site-packages/thunder/utils/context.pyc in loadImages(self, dataPath, dims, dtype, inputFormat, ext, startIdx, stopIdx, recursive, nplanes, npartitions, renumber, confFilename)
    182         from thunder.rdds.fileio.imagesloader import ImagesLoader
--> 183         loader = ImagesLoader(self._sc)
 # Checking StartIdx is smaller or equal to StopIdx

/home/ubuntu/anaconda/lib/python2.7/site-packages/thunder/rdds/fileio/imagesloader.pyc in __init__(self, sparkContext)
     23         from thunder.utils.common import AWSCredentials
     24         self.sc = sparkContext
---> 25         self.awsCredentialsOverride = AWSCredentials.fromContext(sparkContext)
     26 
     27     def fromArrays(self, arrays, npartitions=None):

/home/ubuntu/anaconda/lib/python2.7/site-packages/thunder/utils/common.pyc in fromContext(cls, sparkContext)
    244     def fromContext(cls, sparkContext):
    245         if sparkContext:
--> 246             awsAccessKeyId = sparkContext._jsc.hadoopConfiguration().get("fs.s3n.awsAccessKeyId", "")
    247             awsSecretAccessKey = sparkContext._jsc.hadoopConfiguration().get("fs.s3n.awsSecretAccessKey", "")
    248             return AWSCredentials(awsAccessKeyId, awsSecretAccessKey)
AttributeError: type object 'SparkContext' has no attribute '_jsc'

From my understanding it is attempting to connect to an AWS server even though I am running thunder locally ? (I did not change any of the spark conf files though). Any idea on what is happening ?