Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Mark Silverberg
@marks
just making sure :)
Tom Schenk Jr
@tomschenkjr
Also, FYI, write.socrata is failing the unit tests, but only because Travis-CI and Appveyor are simutaneously running these tests, which Socrata obviously isn’t going to like. NBD, just have to re-run the test that failed to make sure that was the only problem
Mark Silverberg
@marks
ah yeah. Trying to think if there is a better way to do it.
We do queue updates but I can see why it would mess up tests. Hmmm
Tom Schenk Jr
@tomschenkjr
Are you queuing updates when it’s happing at the exact same time?
Could understand if you’re processing an update when another one comes by, but seems like the auth and upload process is happening too close together. Not a big burden on my side, just have to click an icon
Mark Silverberg
@marks
We queue the updates server side.. but it’s going to mess up tests no matter what if one runner has written an update and the other reads for the first time and gets something unexpected
What I’m trying to say, is that the system is safe in normal use cases but the tests I wrote arent smart enough to wait for an all clear to run themselves
Tom Schenk Jr
@tomschenkjr
ah, that’s right. but you’re immediately re-downloading the data, so will fail the expect_equal
Mark Silverberg
@marks
right
Tom Schenk Jr
@tomschenkjr
that’s alright. Because if you wait until it’s over, that might lead to longer build/test times, which causes CRAN to get grumpy (they get grumpy over a lot of things)
Mark Silverberg
@marks
sure sounds like it
Tom Schenk Jr
@tomschenkjr
they’re a bit of that old-school open source. perpetually angry
Mark Silverberg
@marks
@tomschenkjr any word on when master will be in CRAN?
Tom Schenk Jr
@tomschenkjr
Oddly, no
Tom Schenk Jr
@tomschenkjr
Will follow-up to see if there is an issue.
Mark Silverberg
@marks
had a few folks ask. take your time of course!
Tom Schenk Jr
@tomschenkjr
Did hear back last night. They need a quick modification to Suggest: readr (this didn't pop-up on any of our checks for some reason). Resubmitting today
Mark Silverberg
@marks
:bow:
Tom Schenk Jr
@tomschenkjr
Hrm, I made the quick adjustment, but am getting failures on Travis and AppVeyor that I cannot replicate on my machine https://travis-ci.org/Chicago/RSocrata/builds/105699970 & https://ci.appveyor.com/project/tomschenkjr/rsocrata/build/1.0.141
@marks - can you see if you can replicate these errors (scroll to bottom) in the unit testing? I'm flummoxed.
Tom Schenk Jr
@tomschenkjr
Ok, I found the issue. Hadley updated httr, which introduced readr, which broke the package.
Mark Silverberg
@marks
cool. so you dont need me to test anymore?
Tom Schenk Jr
@tomschenkjr
nah, I’m certain of the issue. Problem is, it’s an issue with httr, so I’ll either need to write a wierd work-around or wait for Hadley to fix it
opened an issue on httr repo: hadley/httr#329
Tom Schenk Jr
@tomschenkjr
Deployed a work-around for #75. Will submit it Sunday afternoon for CRAN consideration
Mark Silverberg
@marks
awesome
Tom Schenk Jr
@tomschenkjr
I submitted 1.7.0 (build 10) to CRAN the other day again with a workaround for the bug on hadley/httr#329
Tom Schenk Jr
@tomschenkjr
There appears to be a fix in a pull request hadley/httr#331 that I verified will work and lets me undo my "fix". In anticipation of that pull request, I've deployed 1.7.1 to iss75 branch. Will merge into dev once httr fix is formally deployed to CRAN.
Mark Silverberg
@marks
thanks man
Tom Schenk Jr
@tomschenkjr
FYI- Appveyor is down with some issues, so the builds will appear as failing.
Tom Schenk Jr
@tomschenkjr
RSocrata has been updated and now on CRAN.
Tom Schenk Jr
@tomschenkjr
@geneorama - the current master and dev branch reads as 1.7.2. I missed this, but we should keep it at 1.7.1 (the current CRAN version is 1.7.0). I'm going to adjust the version number in dev and master.
@geneorama - likewise, it looks like I missed updating NEWS.md as well. I'm going to note some of the work I did. I'll let you know when I finish the clean-up
Tom Schenk Jr
@tomschenkjr
We're now enforcing that RSocrata shouldn't store any data as factors, that anything string-like should be handled as character vectors. Where does everyone think this is a good place to document this decision (besides here)?
Tom Schenk Jr
@tomschenkjr
Nevermind, I think that's now implicit in the new stringAsFactors option
Tom Schenk Jr
@tomschenkjr
@geneorama - I made the documentation fixes and opened pull request #88. Take a look and accept if it's alright.
Tom Schenk Jr
@tomschenkjr
FYI: I've deleted branches issue77, iss75, iss81 and iss62 as clean-up
Mark Silverberg
@marks
Hey @tomschenkjr - looking in the code for read.socrata and validateUrl… if passed a human-readable URL, does RSocrata download 1k records at a time?
Tom Schenk Jr
@tomschenkjr
I certianly hope not
Mark Silverberg
@marks
Hmmm. How does it decide on paging?
I don't see any reference to $limit
Mark Silverberg
@marks
Anyone seen duplicate rows show in RSocrata results?
Tom Schenk Jr
@tomschenkjr
haven't seen that. though, possible because of #15, but would need the data to be updated during the download, so under specific circumstances
Mark Silverberg
@marks
i dont think thats it. someone is mentioning it happens on a private dataset and I’m asking them to replicate on a public one so we can submit a ticket
thanks for the response. its very very odd.
Tom Schenk Jr
@tomschenkjr
@marks - some digging. For your comment on 2016-01-24T13:41, it's not because the queuing isn't working. Instead, what is happening is that the unit tests generate random numbers, upload those numbers to the portal with write.socrata, then downloads the data set with read.socrata and compares it to the numbers it just uploaded. However, a conflict arises because Travis CI tests generates numbers and uploads it. In the meantime, AppVeyor also generates numbers and uploads it. Then Travis CI downloads the numbers, but is actually grabbing the AppVeyor-generated numbers, so the comparison fails.
Tom Schenk Jr
@tomschenkjr
@/all - The team is working on our next sprint to be finished by the end of October. It will be a bug-fix release, so the version number is targeted as 1.7.1. See the scope of this sprint at https://github.com/Chicago/RSocrata/milestone/2. The next version--planned as v1.8.0--will begin to incorporate the most recent Socrata API functionality.
Tom Schenk Jr
@tomschenkjr
Version 1.7.1 is now on CRAN. Thanks to those who reported bugs. https://github.com/Chicago/RSocrata/releases/tag/v1.7.1
Tom Schenk Jr
@tomschenkjr
Version 1.7.2 is now on CRAN. Thanks to those who reported bugs. http://dev.cityofchicago.org/rsocrata/data%20portal/2017/04/20/rsocrata-172-now-available.html