Hey there. I changed which email address is listed as the primary address on my GitHub account, which is linked to my DataHub account, and now when I log in to DataHub w/ GitHub... I am a different person with a different username, and all of my uploaded datasets have disappeared. This seems like it might not be the desired behavior.
Hi gitters! I was trying to play around with GMSL dataset:
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
@zaneselvans thanks for letting us know! We’ll need to discuss it with the team. Would you mind to open an issue here https://github.com/datahq/datahub-qa/issues ?
@sebaswm Hey! Not sure if such APIs are available at all. Try to look through existing questions here may be it’ll help - https://opendata.stackexchange.com/search?q=food
@gaurav-gireesh Hi there! Good to know you’re playing around with it :smile: :+1: Unfortunately, I’m not an expert on it so cannot help but I hope somebody in the community will :pray:
@Branko-Dj any ideas?
@Branko-Dj any ideas?
@sebaswm Partial answer to your question (better than nothing I guess): https://openfoodfacts.org
(it's a French project but increasingly popular, I can introduce to the team if needed)
(it's a French project but increasingly popular, I can introduce to the team if needed)
Hi! Appreciated if anyone can shine some more light on Datapackages and DBMS.
We have 100s of datapackages with different schemas and want to make them accessible behind one API with at least basic filtering across available dimensions (for production web application use). Are there some readymade solutions for this? (Does for example https://datahub.io/data-factory help with parts of this or are there other solutions to automatically ingest into Postgres, MongoDB etc?)
We have 100s of datapackages with different schemas and want to make them accessible behind one API with at least basic filtering across available dimensions (for production web application use). Are there some readymade solutions for this? (Does for example https://datahub.io/data-factory help with parts of this or are there other solutions to automatically ingest into Postgres, MongoDB etc?)
Thank you @anuveyatsu! @akariv: Any recommendations very appreciated here or on DM.
For a good api, I recommend using
babbage, which provides facts and aggregation endpoints over such DBs.
@akariv So is the idea with
dump_to_sql that you can automatically reconstitute a relational database with all of the same constraints and types and inter-table relationships that are specified within a data package? Is there any mechanism for specifying relationships between tables that are originating from resources in different data packages?
@zaneselvans data packages have the concept of foreign keys, which allows defining these sort of inter resource relationships. dump_to_sql will load all data tables to the DB for you, but you might need to add some of these constraints yourself (I'll check the code once I'm in front of my PC)
@jhofra11_twitter the best way of overviewing datasets from a publisher is to go to a publisher page. Use your username on datahub: https://datahub.io/username. On that page you can see all published datasets (and only publisher himself/herself can see unlisted/private datasets).
Number of datasets you can publish is unlimited but total size of your datasets is limited (eg, 5GB for basic plan).
@jhofra11_twitter Hi, I can confirm that you definately have more than 2 datasets but for some reason your publisher page isn’t showing up all of them. We are having a look at our metastore service to identify the problem. Thank you for reporting this and for your patience :+1: I’ll get back to you once it’s resolved
@jhofra11_twitter Hi, it seems that metastore service was down for a while for some yet unknown reasons. We are trying to investigate the root cause of it, but right now it seems to be resolved. Could you please try re-pushing them - they should appear in search once re-pushed
Hi, I have 2 questions. First of all, it's possible search in datahub, like in the old version, specifying parameter as "organization", "tags", "formats", etc. etc.? The second question is if there is a sparql endpoint (like "http://semantic.ckan.net/sparql") to send queries. Thank you all
@clamar14 and there is no sparql endpoint at present. If you or someone else were interested in creating that let us know :smile:
@clamar14 and let us know the contexst of these questions - i.e. what you are trying to do :smile:
I am working in a open and linked data project, and I have a question: How I can implement linked data in this new version of Datahub? How I can publish the url? How I have to upload the RDF files? In one last post, I read that in this version of datahub, we can not build endpoints.
I'm having issue trouble not inferring types on load in dataflows when I am loading the file directly without a datapackage. Can someone help me out?
From the documentation, validate option should control that but it isn't working unless I am misunderstanding.
- tried validate=True and False
- also tried force_strings from tabulator since the doc also mentions tabulator options
Using dataflows Version: 0.0.32
data.csv
cruise,station,date,time,lat,lon,cast,pump_serial_num
FK160115,4,1/19/2016,20:00,10,204,MP01,12665-01
FK160115,4,1/19/2016,20:00,10,204,MP01,12665-02
FK160115,4,1/19/2016,20:00,10,204,MP01,ML12371-01
FK160115,4,1/19/2016,20:00,10,204,MP01,ML 10820-02
FK160115,4,1/19/2016,20:00,10,204,MP01,ML 11000-01
FK160115,4,1/19/2016,20:00,10,204,MP01,ML 11515-02
FK160115,4,1/19/2016,20:00,10,204,MP01,ML 11934-02
FK160115,5,1/20/2016,18:00,8,156,MP02,12665-01
FK160115,5,1/20/2016,18:00,8,156,MP02,12665-02
FK160115,5,1/20/2016,18:00,8,156,MP02,ML 10820-02
FK160115,5,1/20/2016,18:00,8,156,MP02,ML 11515-02
FK160115,5,1/20/2016,18:00,8,156,MP02,ML 11000-01
FK160115,5,1/20/2016,18:00,8,156,MP02,ML 11491-02from dataflows import Flow, load, printer
def flow():
flow = Flow(
load('data.csv', format='csv',validate=False,force_strings=True),
printer(num_rows=1),
)
flow.process()
if __name__ == '__main__':
flow()getting:
data:
# cruise station date time lat lon cast pump_serial_num
(string) (integer) (string) (string) (number) (number) (string) (string)
--- ---------- ----------- ---------- ---------- ---------- ---------- ---------- -----------------
1 FK160115 4 1/19/2016 20:00 10 204 MP01 12665-01
2 FK160115 4 1/19/2016 20:00 10 204 MP01 12665-02
...
66 FK160115 14 2/4/2016 13:00 -4.23 142.23 MP14 ML12371-01