data-8/datascience

A Python library for introductory data science

Chris Holdgraf

@choldgraf

and then for encoding there may also be this: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.str.encode.html

e.g.: df['Kingdom'].str.encode('utf-8')

so you could loop through your columns, and if it's a column of strings change the encoding to utf-8

then write back to disk

Carl Boettiger

@cboettig

Cool. Though for the students I think it's best if I can give them utf-8 encoded data whenever possible; like you say, it's kind of the dark underbelly, I'm sure no one actually enjoys battling string encodings...

Chris Holdgraf

@choldgraf

oh definitely

I meant just for you/me

Carl Boettiger

@cboettig

right

Chris Holdgraf

@choldgraf

e.g. that seems to work properly

and if I write that dataframe to file, I can now read it back in like before

but w/o the extra 'encoding' flag

df = extinct.to_df()
for col, vals in df.iteritems():
    try:
        df.loc[:, col] = vals.str.encode('utf-8')      
    except:
        print(col)
df.to_csv('./test.csv')
ds.Table.read_table('./test.csv')

there's the code to do it

Carl Boettiger

@cboettig

@choldgraf @SamLau95 Hmm, somehow I can no longer do an apply with a function that takes multiple arguments:

letter = ['a', 'b', 'c', 'z']
count = [9, 3, 3, 1]
points = [1, 2, 2, 10]

t = Table([letter, count, points], ['letter', 'count', 'points'])

# Works
t.apply(lambda x: x * x, 'count')

## throws error
t.apply(lambda x, y: x * y, ['count', 'points'])

Error is:


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-8fe518b79c55> in <module>()
      7 t.apply(lambda x: x * x, 'count')
      8 
----> 9 t.apply(lambda x, y: x * y, ['count', 'points'])
     10 
     11 #array([ 9,  6,  6, 10])

/usr/local/lib/python3.4/dist-packages/datascience/tables.py in apply(self, fn, column_label)
    447         """Returns an array where fn is applied to each element
    448         of a specified column."""
--> 449         return np.array([fn(v) for v in self[column_label]])
    450 
    451     ############

/usr/local/lib/python3.4/dist-packages/datascience/tables.py in __getitem__(self, label)
    352 
    353     def __getitem__(self, label):
--> 354         return self.values(label)
    355 
    356     def __setitem__(self, label, values):

/usr/local/lib/python3.4/dist-packages/datascience/tables.py in values(self, label)
    438             An instance of ``numpy.array``.
    439         """
--> 440         return self._columns[label]
    441 
    442     def column_index(self, column_label):

TypeError: unhashable type: 'list'

Sam Lau

@SamLau95

looks like you might have an old version of the package @cboettig

do you have 0.3.dev21?

Chris Holdgraf

@choldgraf

I can confirm that @cboettig's code works on the latest version for me

Chris Holdgraf

@choldgraf

Also - quick quibble, but I often forget to open this package with python 3 sourced, and the error message you get for doing so is a bit cryptic (it just throws an import error)

think we could just put:

import sys
if sys.version_info < (3, 0):
    raise ValueError('This package requires python >= 3.0')

at the root __init__.py?

Sam Lau

@SamLau95

ah, that’s a good idea

Carl Boettiger

@cboettig

@SamLau95 yup, running on ds8.berkeley.edu and it says '0.3.dev21'

Chris Holdgraf

@choldgraf

@SamLau95 want me to make a PR?

Carl Boettiger

@cboettig

It's very strange. I have in my git history successful runs of the notebook with the very same code that works with multiple columns, so I'm not sure why it is failing for me now.

Chris Holdgraf

@choldgraf

hmm, that is weird - and it works for me

problem is I can't really debug it w/o the code breaking for me

maybe just to be sure you can pull the latest changes from the git repo

Sam Lau

@SamLau95

@choldgraf that’d be great

Carl Boettiger

@cboettig

You can see the error here in context, if that helps: https://github.com/dsten/ecology-connector/blob/0947c5464e76998d135babdb92c5321e8ac9cc76/climate/climate.ipynb

Chris Holdgraf

@choldgraf

@SamLau95 hrmmm, actually it might be more complicated

because it looks like some of the changes actually result in a syntaxerror in 2.7

so it won't get to the point that it's running any code

perhaps a better option would be to prevent it from installing on any versions of python < 3.0?

Sam Lau

@SamLau95

ah i see

would you like to look into that?

if not, feel free to make an issue

Chris Holdgraf

@choldgraf

@cboettig so it looks like the problem is that the function is passing a list of column names to the table. E.g., this returns the same error you get:

t[['letter', 'count', 'points']]

and it looks like that function has indeed been updated. Here's the new code for it: np.array([fn(*[self.take(i)[col][0] for col in column_label]) for i in range(self.num_rows)])

note that there is another for loop: for col in column_label so now it's passing each column that you give individually, that way it never tries to pass a list

so try pulling and it should work

Carl Boettiger

@cboettig

interesting... so somehow I am running an old version of datascience even though it says 0.3.dev21 ?

I'm running this on ds8.berkeley.edu, so I have no ability to install there, but let me try locally. What version are you on?

Chris Holdgraf

@choldgraf

ah interesting

same version

but w/o knowing much about how the repo is organized, that's probably just the current dev branch

and presumably the version of that code being used in ds8 is a few commits behind the branch on github

Carl Boettiger

@cboettig

maybe version number didn't get bumped;

right.

Where communities thrive

People

Repo info

Activity