Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Mar 16 2017 10:22
  • Mar 16 2017 10:22
    AlexxNica starred s3ththompson/berliner
  • Feb 07 2017 17:49
    s3ththompson commented #91
  • Oct 22 2016 21:53
    jondashkyle starred s3ththompson/berliner
  • Jun 21 2016 21:27
    s3ththompson opened #91
  • Jun 21 2016 14:23

    s3ththompson on master

    reproducible builds and updated… (compare)

  • Jun 21 2016 13:55

    s3ththompson on master

    repro 2 (compare)

  • Jun 21 2016 13:51

    s3ththompson on master

    test reproducible build (compare)

  • Jun 21 2016 13:29

    s3ththompson on master

    update dependencies and fix .tr… (compare)

  • Jun 21 2016 11:00

    s3ththompson on master

    go fmt (compare)

  • Jun 14 2016 08:59
    s3ththompson opened #90
  • Jun 14 2016 08:58
    s3ththompson opened #89
  • Jun 14 2016 08:58
    s3ththompson opened #88
  • Jun 14 2016 07:54
    s3ththompson opened #87
  • Jun 14 2016 07:51
    s3ththompson edited #83
  • Jun 14 2016 07:50
    s3ththompson opened #86
  • Jun 14 2016 07:50
    s3ththompson opened #85
  • Jun 14 2016 07:49
    s3ththompson opened #84
  • Jun 14 2016 07:48
    s3ththompson opened #83
Geoffrey Litt
@geoffreylitt
yeah, sounds reasonable. although probalby only necessary once it starts to become a problem
@s3ththompson have you done anything with table of contents?
Geoffrey Litt
@geoffreylitt
i've started dabbling with table of contents, it's on the simple-theme branch
Geoffrey Litt
@geoffreylitt
hey @s3ththompson quick update: pushed some improvements to my simple-theme branch.. still a long way to go until i'm really happy with how it looks, but the basics are there, enough to dogfood... trying to decide when it should merge in
my thought is to improve the table of contents a bit and then just merge it in for now
then i'm gonna build some sort of topic/keyword filter or something
one thought i had would be that it would be AWESOME to have different "sections" in your berliner.. the weird thing right now is that so many different types of articles are mixed together. I love the idea of a "sports section", but pulled from all sorts of different news sources, and deduplicated as much as possible...
lmk if you have any thoughts on that
i'm also seriously thinking about a tinder filter or renderer. shows you headlines/summaries, you quickly decide to read or skip, and then once you're done tindering you just read the ones that you chose
(maybe there's three options: excited about this, i'll read it, and skip)
anyway, obviously a lot of possible ideas, and the first step is to dogfood and solidify the framework
Seth Thompson
@s3ththompson
Feel free to merge theme in whebever
I think we'll want to really polish our "default" images by the time we launch, but we should go nuts for now and experiment
Sections make sense. I think we should have a tags array in article object
And a smart default which does some lightweight t
Tfdiff
Unclear to me rn how tinder should work
Seth Thompson
@s3ththompson
I don't think we should set the precedent (yet) for interactive html output (if that's what you were suggesting)
And Im not certain there is
scratch that last line
What are the other hing
Ack gitter on mobile is tough
Seth Thompson
@s3ththompson
What else might require interactivity via the CLI during the berliner creation process? I'm not sure we want to do something special just for the tinder-style news picking alone
Seth Thompson
@s3ththompson
minor milestone change:
I'd like to lock down filter format relatively soon, and add support for different types of articles: "articles", "vidoes", "gallery", "code", etc.
so I'm going to move horizontally scaling our sources and filters to the v0.1.0 milestone
and have 0.0.5 focus on locking down how sources and filters are written
also, as an aside: thinking about post types raises the recurring question of how tightly coupled this stuff is: should every source be required to differentiate between post types? what if someone wants to add a post type themselves? do they have to rewrite every source to recognize it?
how do that add support for that new post type to their favorite renderer?
Seth Thompson
@s3ththompson
This message was deleted
package main

import (
    "fmt"
    "github.com/headzoo/surf"
    "os"
)

func main() {
    // Start by creating a new bow.
    bow := surf.NewBrowser()

    // Add additional request headers.
    bow.AddRequestHeader("Accept", "text/html")
    bow.AddRequestHeader("Accept-Charset", "utf8")

    // Requesting a page.
    err := bow.Open("https://myaccount.nytimes.com/auth/login")
    if err != nil { panic(err) }
    fmt.Println(bow.Title())
    // Outputs: "reddit: the front page of the internet"

    // Login to the site via their login form. Again, we're using the goquery selector
    // syntax.
    fm, err := bow.Form("form.loginForm")
    if err != nil { panic(err) }
    fm.Input("userid", "s3th.thompson@gmail.com")
    fm.Input("password", "redacted")
    err = fm.Submit()
    if err != nil { panic(err) }

    bow.Open("http://www.nytimes.com/2015/01/29/sports/football/nfl-tries-to-reassure-mothers-as-polls-and-studies-rattle-them.html")

    fmt.Println(bow.Title())
    file, err := os.Create("nyt.html")
    if err != nil { panic(err) }
    defer file.Close()
    bow.Download(file)
}
^ this is actually working for me... I'm kinda thinking this could work!
Seth Thompson
@s3ththompson
This message was deleted
This message was deleted
example new-yorker.go:
Seth Thompson
@s3ththompson
package berliner

type NewYorker struct {
    Source
}

func (ny *NewYorker) Parse(e Entry) Article {
    doc, _ := goquery.NewDocument(e.url)
    title := doc.Find("#title")
    author := doc.Find("#author")
    return Article{
        Title: title,
        Author: author
    }
}

func (ny *NewYorker) Init() {
    ny.Feed = "http://www.newyorker.com/feed/news"
    ny.Title = "The New Yorker"
    ny.Homepage = "http://www.newyorker.com/"
}
just a thought. I'm 50% bought into the idea...
Seth Thompson
@s3ththompson
(also just an example of how good the stdlib is... even if we don't use headzoo/surf we could roll our own mechanize from scratch: http://golang.org/pkg/net/http/cookiejar/)
Seth Thompson
@s3ththompson
also, while we're on the topic of other languages, this is an awesome lib: http://newspaper.readthedocs.org/en/latest/
Seth Thompson
@s3ththompson
to summarize offline decision: we're sticking with Ruby
Geoffrey Litt
@geoffreylitt
Nice hack sesh tonight!
basic tinder filter is done, on tinder-filter branch for now (going to improve it a bit before merging)
Seth Thompson
@s3ththompson
much cleaner abstraction around executing multiple requests (as opposed to mapping arbitrary work across n threads)
plus it has built-in caching!!!!
and stubbing out remote server responses... beautifu
this is going to make testing so much better
Seth Thompson
@s3ththompson
it's been a while since we've posted in the gitter, but for posterity...
Screen Shot 2015-09-28 at 2.46.31 AM.png
this is the first time that the go branch has surpassed the ruby branch in sophistication of renderer output