Is there an api or a function in data utility?
At the moment I am scraping the list of pages, but it would be great to somehow get an exhaustive list of what has been uploaded. My use case is that we have a list of 217 datasets that we want uploaded. However, only 197 were uploaded. How do we identify the ones there weren't processed?
I went through the data utility logs which seemed to have uploaded everything.
@MAliNaqvi you can use our API to list the datasets here's an example https://api.datahub.io/metastore/search?datahub.owner="core"&size=100&from=0
Where:
- datahub.owner is owner_id, not username
- Size is max items per page (Max is 100)
- From is the page number starting from 0
You can also send the JWT token in headers
Auth-Token=your_jwt or as query parameter &jwt=your_jwt to list the unlisted/private datasets as well
Folks, I just spent a couple of hours uploading 43 datasets. It was a very frustrating to find that only 3 of those datasets made it to the datahub website, even though the data utility uploaded everything without an issue. Here are the results:
| dataset | url | AVAILABLE |
|--------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+-----------|
| nasa-temperature-anomalies-by-latitude-bands-time-series-1880-2017 | https://datahub.io/JohnSnowLabs/nasa-temperature-anomalies-by-latitude-bands-time-series-1880-2017/v/2 | No |
| chicago-annual-taxpayer-location-list | https://datahub.io/JohnSnowLabs/chicago-annual-taxpayer-location-list/v/2 | No |
| nasa-global-temperature-anomalies-time-series-1880-2018 | https://datahub.io/JohnSnowLabs/nasa-global-temperature-anomalies-time-series-1880-2018/v/2 | No |
| nj-residents-leading-causes-of-death | https://datahub.io/JohnSnowLabs/nj-residents-leading-causes-of-death/v/2 | No |
| uk-properties-for-sale-by-ministry-of-defense | https://datahub.io/JohnSnowLabs/uk-properties-for-sale-by-ministry-of-defense/v/2 | No |
| tree-debris-requested-by-311-service | https://datahub.io/JohnSnowLabs/tree-debris-requested-by-311-service/v/2 | No |
| tree-trims-requested-by-311-service | https://datahub.io/JohnSnowLabs/tree-trims-requested-by-311-service/v/2 | No |
| garbage-carts-requested-by-311-service | https://datahub.io/JohnSnowLabs/garbage-carts-requested-by-311-service/v/2 | No |
| pot-holes-reported-by-311-service | https://datahub.io/JohnSnowLabs/pot-holes-reported-by-311-service/v/2 | No |
| eicu-collaborative-research-admissions-summary-statistics | https://datahub.io/JohnSnowLabs/eicu-collaborative-research-admissions-summary-statistics/v/1 | Yes |
| chicago-taxi-trips | https://datahub.io/JohnSnowLabs/chicago-taxi-trips/v/2 | No |
| chicago-beach-weather-stations-automated-sensors | https://datahub.io/JohnSnowLabs/chicago-beach-weather-stations-automated-sensors/v/2 | No |
| chicago-beach-water-quality-automated-sensors-report | https://datahub.io/JohnSnowLabs/chicago-beach-water-quality-automated-sensors-report/v/2 | No |
| all-countries-latitude-longitude | https://datahub.io/JohnSnowLabs/all-countries-latitude-longitude/v/4 | No |
| estimates-emissions-of-co2-at-country-and-global-level | https://datahub.io/JohnSnowLabs/estimates-emissions-of-co2-at-country-and-global-level/v/2 | No |
| energy-consumption-by-mode-of-transportation-and-type-of-energy | https://datahub.io/JohnSnowLabs/energy-consumption-by-mode-of-transportation-and-type-of-energy/v/2 | No |
| relocated-vehicles-in-chicago-last-90-days | https://datahub.io/JohnSnowLabs/relocated-vehicles-in-chicago-last-90-days/v/1 | No |
| nys-english-and-mathematics-exam | https://datahub.io/JohnSnowLabs/nys-english-and-mathematics-exam/v/2 | No |
| schools-for-life-safety-evaluations | https://datahub.io/JohnSnowLabs/schools-for-life-safety-evaluations/v/2 | No || food-affordability-for-households-led-by-females | https://datahub.io/JohnSnowLabs/food-affordability-for-households-led-by-females/v/2 | No |
| chicago-business-licenses | https://datahub.io/JohnSnowLabs/chicago-business-licenses/v/1 | No |
| city-population-annual-time-series | https://datahub.io/JohnSnowLabs/city-population-annual-time-series/v/3 | No |
| bloomington-animal-care-and-control-adopted-animals | https://datahub.io/JohnSnowLabs/bloomington-animal-care-and-control-adopted-animals/v/2 | No |
| legally-operating-businesses | https://datahub.io/JohnSnowLabs/legally-operating-businesses/v/2 | No |
| cta-ridership-bus-routes | https://datahub.io/JohnSnowLabs/cta-ridership-bus-routes/v/1 | Yes |
| most-popular-baby-names-by-gender-and-mother-ethnic-group | https://datahub.io/JohnSnowLabs/most-popular-baby-names-by-gender-and-mother-ethnic-group/v/2 | No |
| eicu-collaborative-research-available-tables-and-data | https://datahub.io/JohnSnowLabs/eicu-collaborative-research-available-tables-and-data/v/1 | Yes |
| nj-traffic-counts-data | https://datahub.io/JohnSnowLabs/nj-traffic-counts-data/v/2 | No |
| austin-adult-and-children-vaccinations | https://datahub.io/JohnSnowLabs/austin-adult-and-children-vaccinations/v/2 | No |
| euro-4-cars-emissions-traded-on-uk-market-2000-2012 | https://datahub.io/JohnSnowLabs/euro-4-cars-emissions-traded-on-uk-market-2000-2012/v/2 | No |
| lobbyist-agency-report | https://datahub.io/JohnSnowLabs/lobbyist-agency-report/v/2 | No |
| windsor-transit-bus-stops | https://datahub.io/JohnSnowLabs/windsor-transit-bus-stops/v/2 | No |
| omha-receipts-for-fiscal-year-2011-2013 | https://datahub.io/JohnSnowLabs/omha-receipts-for-fiscal-year-2011-2013/v/2 | No |
| impaired-driving-death-rate-by-age-and-race | https://datahub.io/JohnSnowLabs/impaired-driving-death-rate-by-age-and-race/v/2 | No |
| chicago-red-light-and-speed-camera-violations | https://datahub.io/JohnSnowLabs/chicago-red-light-and-speed-camera-violations/v/2 | No |
| us-employment-and-unemployment-rates | https://datahub.io/JohnSnowLabs/us-employment-and-unemployment-rates/v/2 | No |
| chicago-affordable-rental-housing-developments | https://datahub.io/JohnSnowLabs/chicago-affordable-rental-housing-developments/v/2 | No |
| vehicle-occupant-safety-data | https://datahub.io/JohnSnowLabs/vehicle-occupant-safety-data/v/2 | No |
| chicago-traffic-tracker | https://datahub.io/JohnSnowLabs/chicago-traffic-tracker/v/2 | No |
| imf-world-economic-outlook-database | https://datahub.io/JohnSnowLabs/imf-world-economic-outlook-database/v/2 | No || chicago-bike-racks-map | https://datahub.io/JohnSnowLabs/chicago-bike-racks-map/v/2 | No |
| us-states-and-territories | https://datahub.io/JohnSnowLabs/us-states-and-territories/v/2 | No |
| chicago-alternative-fuel-locations | https://datahub.io/JohnSnowLabs/chicago-alternative-fuel-locations/v/2 | No |
Folks, I just spent a couple of hours uploading 43 datasets. It was a very frustrating to find that only 3 of those datasets made it to the datahub website, even though the data utility uploaded everything without an issue. Here are the results:
@MAliNaqvi Hi Ali! As I can see all datasets was uploaded successfully, however, most of them have validation/processing issues. You need to be logged in to see those errors. I know that you’re using an org account so the best way to check would be to pass your JWT within query params, e.g., try this https://datahub.io/JohnSnowLabs/chicago-traffic-tracker/v/2?jwt=<your-jwt> so that you are able to see FAILED dataset page.
I'm thinking of a data set for each disease, his different levels and his symptoms, in order to design a tool for medical diagnostic.
medical diagnostic. or for only one disease like heart and like that ?
@Malikah95 Hi! Could you please send a request using our service here - https://datahub.io/requests ?
You could also take a look at existing datasets, e.g., some can be found here - https://datahub.io/machine-learning
You could also take a look at existing datasets, e.g., some can be found here - https://datahub.io/machine-learning
@akariv dataflows' sort_by processor does not seem to be working as expected, any ideas?
from dataflows import Flow, printer, sort_rows
data = [
{'data': 'B'},
{'data': 'E'},
{'data': 'C'},
{'data': 'D'},
{'data': 'A'},
]
f = Flow(
data,
sort_rows('data'),
printer()
)
f.process()results with
res_1:
# data
(string)
--- ----------
1 B
2 E
3 C
4 D
5 A
hm... OK, so looking at the docs here seems like I need
{} around data https://github.com/frictionlessdata/datapackage-pipelines#sortfrom dataflows import Flow, printer, sort_rows
data = [
{'data': 'B'},
{'data': 'E'},
{'data': 'C'},
{'data': 'D'},
{'data': 'A'},
]
f = Flow(
data,
sort_rows('{data}'),
printer()
)
f.process()
and the name of the dataset is 2016 Survey of Consumer Finances (Summary Extract) and the link is
Usually usernames on DataHub and GitHub match as people register with their GitHub accounts
Hi @KonradHoeffner could you please fill in this form - https://datahub.io/docs/features/teams-and-permissions
S&P 500 Companies with Financial Information
@MDeck06_twitter Thanks for writing to us. Our data engineer @Branko-Dj has done some work in this area and will get back to you tomorrow. We can respond on this thread or if you fill out a data request we can get back to you via email https://datahub.io/requests
@Senousy_gitlab Thanks for reaching out. On this one the best option is to send us a https://datahub.io/requests and one of our data engineers will get back to you
Hey there. I changed which email address is listed as the primary address on my GitHub account, which is linked to my DataHub account, and now when I log in to DataHub w/ GitHub... I am a different person with a different username, and all of my uploaded datasets have disappeared. This seems like it might not be the desired behavior.
Hi gitters! I was trying to play around with GMSL dataset:
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
https://datahub.io/core/sea-level-rise/r/csiro_alt_gmsl_mo_2015.csv
I was trying to train a simple neural network to carry out poisson regression on GMSL data. As a loss function I am using PoissonNLLLoss function. While training I can see the loss decreasing with time, which is as expected. However, I am not able to understand, what I am trying to predict here and how do I test accuracy of this model?
Any ideas are welcome!
Thanks,
Gaurav
@zaneselvans thanks for letting us know! We’ll need to discuss it with the team. Would you mind to open an issue here https://github.com/datahq/datahub-qa/issues ?
@sebaswm Hey! Not sure if such APIs are available at all. Try to look through existing questions here may be it’ll help - https://opendata.stackexchange.com/search?q=food
@gaurav-gireesh Hi there! Good to know you’re playing around with it :smile: :+1: Unfortunately, I’m not an expert on it so cannot help but I hope somebody in the community will :pray:
@Branko-Dj any ideas?
@Branko-Dj any ideas?