Folks, I just spent a couple of hours uploading 43 datasets. It was a very frustrating to find that only 3 of those datasets made it to the datahub website, even though the data utility uploaded everything without an issue. Here are the results:
@MAliNaqvi Hi Ali! As I can see all datasets was uploaded successfully, however, most of them have validation/processing issues. You need to be logged in to see those errors. I know that you’re using an org account so the best way to check would be to pass your JWT within query params, e.g., try this https://datahub.io/JohnSnowLabs/chicago-traffic-tracker/v/2?jwt=<your-jwt> so that you are able to see FAILED dataset page.
I'm thinking of a data set for each disease, his different levels and his symptoms, in order to design a tool for medical diagnostic.
medical diagnostic. or for only one disease like heart and like that ?
@Malikah95 Hi! Could you please send a request using our service here - https://datahub.io/requests ?
You could also take a look at existing datasets, e.g., some can be found here - https://datahub.io/machine-learning
You could also take a look at existing datasets, e.g., some can be found here - https://datahub.io/machine-learning
@akariv dataflows' sort_by processor does not seem to be working as expected, any ideas?
from dataflows import Flow, printer, sort_rows
data = [
{'data': 'B'},
{'data': 'E'},
{'data': 'C'},
{'data': 'D'},
{'data': 'A'},
]
f = Flow(
data,
sort_rows('data'),
printer()
)
f.process()results with
res_1:
# data
(string)
--- ----------
1 B
2 E
3 C
4 D
5 A
hm... OK, so looking at the docs here seems like I need
{} around data https://github.com/frictionlessdata/datapackage-pipelines#sortfrom dataflows import Flow, printer, sort_rows
data = [
{'data': 'B'},
{'data': 'E'},
{'data': 'C'},
{'data': 'D'},
{'data': 'A'},
]
f = Flow(
data,
sort_rows('{data}'),
printer()
)
f.process()
and the name of the dataset is 2016 Survey of Consumer Finances (Summary Extract) and the link is
Usually usernames on DataHub and GitHub match as people register with their GitHub accounts
Hi @KonradHoeffner could you please fill in this form - https://datahub.io/docs/features/teams-and-permissions
S&P 500 Companies with Financial Information
@MDeck06_twitter Thanks for writing to us. Our data engineer @Branko-Dj has done some work in this area and will get back to you tomorrow. We can respond on this thread or if you fill out a data request we can get back to you via email https://datahub.io/requests
@Senousy_gitlab Thanks for reaching out. On this one the best option is to send us a https://datahub.io/requests and one of our data engineers will get back to you
Hey there. I changed which email address is listed as the primary address on my GitHub account, which is linked to my DataHub account, and now when I log in to DataHub w/ GitHub... I am a different person with a different username, and all of my uploaded datasets have disappeared. This seems like it might not be the desired behavior.
Hi gitters! I was trying to play around with GMSL dataset:
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
@zaneselvans thanks for letting us know! We’ll need to discuss it with the team. Would you mind to open an issue here https://github.com/datahq/datahub-qa/issues ?
@sebaswm Hey! Not sure if such APIs are available at all. Try to look through existing questions here may be it’ll help - https://opendata.stackexchange.com/search?q=food
@gaurav-gireesh Hi there! Good to know you’re playing around with it :smile: :+1: Unfortunately, I’m not an expert on it so cannot help but I hope somebody in the community will :pray:
@Branko-Dj any ideas?
@Branko-Dj any ideas?
@sebaswm Partial answer to your question (better than nothing I guess): https://openfoodfacts.org
(it's a French project but increasingly popular, I can introduce to the team if needed)
(it's a French project but increasingly popular, I can introduce to the team if needed)
Hi! Appreciated if anyone can shine some more light on Datapackages and DBMS.
We have 100s of datapackages with different schemas and want to make them accessible behind one API with at least basic filtering across available dimensions (for production web application use). Are there some readymade solutions for this? (Does for example https://datahub.io/data-factory help with parts of this or are there other solutions to automatically ingest into Postgres, MongoDB etc?)
We have 100s of datapackages with different schemas and want to make them accessible behind one API with at least basic filtering across available dimensions (for production web application use). Are there some readymade solutions for this? (Does for example https://datahub.io/data-factory help with parts of this or are there other solutions to automatically ingest into Postgres, MongoDB etc?)
Thank you @anuveyatsu! @akariv: Any recommendations very appreciated here or on DM.
For a good api, I recommend using
babbage, which provides facts and aggregation endpoints over such DBs.
@akariv So is the idea with
dump_to_sql that you can automatically reconstitute a relational database with all of the same constraints and types and inter-table relationships that are specified within a data package? Is there any mechanism for specifying relationships between tables that are originating from resources in different data packages?
@zaneselvans data packages have the concept of foreign keys, which allows defining these sort of inter resource relationships. dump_to_sql will load all data tables to the DB for you, but you might need to add some of these constraints yourself (I'll check the code once I'm in front of my PC)
