@WacoHolve_gitlab thanks for reporting this. I've posted it as an issue on the Featuretools repo #254
@WacoHolve_gitlab We're leaning towards option 1 or 2 listed there. Do you have a strong preference for one or the other?
Just discovered the feautretools package. Looks very interesting! I have been doing a similar, but a bit less automated approach when creating features. One slight difference, that I so far haven't found in featuretools, is that I tend to aggregate the child data in time periods, e.g. last 30 days before the event, as well as the 30-90 days previous, depending on the outcome, and creating variables for lag of value, difference between lag value and most recent value, percentage difference, etc. This will e.g. give me a variable for percentage change in minimum balance between the last two or more time periods before the event. As far as I can tell, featuretools will aggregate "all" the child elements in the dataset, without regard to splitting the data in time buckets, thus not be able to create features that will pick up on say a temporal change in the minimun balcance of a custumer. Have I overlooked something? I can see how I could force something like this on the featuretools framework by feeding it several datasets from different time periods, and then create variables for the difference between the time periods, but is there some option to specify time periods, or event better, let the algoritm find not just the most relevant aggregations (mean, max, etc), but also the optimal length of temporal aggregations (last 30 days? last 60? change between last 3 weeks and the previous 6?)
@Tullsokk to get aggregate over time different time periods, you can pick a
cutoff_time you'd like to create features at and a training_window which specifies how much historical data to use. So, you can create the different time period features you want by make multiple calls to ft.calculate_feature_matrix for each window. you can read more about handling time here: https://docs.featuretools.com/automated_feature_engineering/handling_time.html
to create the lag features you want, you'd need to create a custom transform primitive. there is info on doing that here: https://docs.featuretools.com/automated_feature_engineering/primitives.html#defining-custom-primitives
if you're interested in trying to make custom primitives, i'd love to find a time to talk!
@kmax12 sure. not at all https://stackoverflow.com/questions/52463225/featuretools-handling-of-multiple-join-keys here is it on SF
@kmax12 thanks for the clarification. One more question is open https://stackoverflow.com/questions/52463356/featuretools-categorical-handling as it is unclear to me which type of categorical is better supported by featuretools
@kmax12 one more question https://stackoverflow.com/questions/52472930/featuretools-multiple-cutoff-times-aggregation
actually, I believe i understand the error. can you put on stack overflow and I'll answer there
the error essentially comes down to the categories being different between the categorical variables you are trying to relate. See this code example
import pandas as pd
from pandas.api.types import is_dtype_equal
s = pd.Series(["a","b","a"], dtype="category")
s2 = pd.Series(["b","b","a"], dtype="category")
s3 = pd.Series(["a","b","c"], dtype="category")
is_dtype_equal(s.dtype, s2.dtype) # this is True
is_dtype_equal(s.dtype, s3.dtype) # this is FalseYou need update your dataframe before loading it into Featuretool to make sure the Pandas Categoricals have the same values category values. Here's how you do that
# if s is missing categories from s3
new_s = s.astype(s3.dtype)
is_dtype_equal(new_s.dtype, s3.dtype) # this is True
# if both are missing categories from each other
import pandas.api.types as pdtypes
s4 = pd.Series(["b","c"], dtype="category")
categories = set(s.dtype.categories + s4.dtype.categories)
new_s = s.astype("category", categories=categories)
new_s4 = s4.astype("category", categories=categories)
is_dtype_equal(new_s.dtype, new_s4.dtype) # this is Trueplease also post on SO where I can give a more detailed answer for everyone else
@kmax12 no it is not really that tough. It means that if both types are of pandas.Category type it cant ensure that a join works i.e. if it joins via category codes it would fail to join as they are not necessarily the same in both datasets (as pandas infers the codes in order of reading the file)
@kmax12 as you whished for an SF posting: https://stackoverflow.com/questions/52505484/pandas-join-on-category-columns-however-different-cat-codes-in-featuretools
might not matter that much
@Tullsokk @geoHeil I just answered this question about using multiple training windows on Stack Overflow. Hopefully, it's helpful for you two: https://stackoverflow.com/questions/52472930/featuretools-multiple-cutoff-times-aggregation
you may need to increase max depth
Hi everyone! I love featuretools and the idea to automically engeineer features. Unfortunately I can't seem to add interesting variables and I would be happy if someone could help out :)
I suspect that it has something to do with my data because I can reproduce the example in the docs just fine..
Maybe this is an easy question for Pythonistas.. I am an ardent R user so maybe there is something I am just not seeing.
It's not high priority though, just something that I couldn't figure out. Enjoy your sunday everyone :)