Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Kyle P. Johnson
    @kylepjohnson
    Yes, I know this isn't perfect ... but I don't know what would be better. There's something else, I'm sure.
    The central problem is: How can we
    ... How can we convey
    (Sorry, iPhone app keeps sending my message too soon!)
    How can we mark the beginning of a new speaker? In my NLP work I could care less who the speaker is, however for reading this is obviously necessary!
    Luke, when you're ready to work on this, I expect you'll have better ideas
    Luke Hollis
    @lukehollis
    Oh man, that's a good question! Hrm... well, it seems like if it works for now, I'm all for it.
    We can work for a more complex solution after we get to a simple one, I think
    One reflection with this also: we're currently sorting on all the keys for book-chapter, chapter-section, book-line, etc.
    So when we have something like
    {
       'text' : {
          '1' : {
             '1' : "This is the first line",
             '2' : "This is the second line",
             ... 
          }
       }
    }
    Our current queries look like this:
    Luke Hollis
    @lukehollis
    Texts.find({work:"the_work"}, {sort:{ n_1 : 1, n_2 : 1}})
    Which means generally sort by the first nested keys and then sort by the second nested keys
    The simplest possible solution (maybe not optimal?) that I could imagine if we wanted to keep sorting on the keys would just be to do something like this:
    Luke Hollis
    @lukehollis
    {  
          "1":{
                        "speaker" : "Actor 1",
                         "text" : "This is the first line"
           },
          "2":{
                        "speaker" : "Actor 2", 
                         "text" : "This is the second line"
          },
          "3":{
                        "speaker" : "Actor 2", 
                         "text" : "This is the third line"
          },
    }
    But, we can ingest data from either data format--whatever makes the most sense for the document conversion.
    When we move things to Text objects in the database, I think we'll just have to add a speaker property that we can use across genres when appropriate
    I was thinking about this back in the day with the speakers in the Eclogues
    Sameer Chaudhari
    @sameeriitkgp
    I agree with Luke.
    Sameer Chaudhari
    @sameeriitkgp
    @kylepjohnson -- If you agree, we can create an issue for this actor-line
    I think I can take care of the document conversion and the database part.
    What do you think @lukehollis ?
    Kyle P. Johnson
    @kylepjohnson
    @SameerIITKGP and @lukehollis I agree, too. I like you solution better much than mine
    @SameerIITKGP I think you can go for it. I thank you both for being so helpful. I knew this would be a problem and had avoided it
    Luke Hollis
    @lukehollis
    sounds good! I think we should just add the speaker parameter to the Text objects collection schema and add the check for ingesting "actor-line" documents in the text sync
    Kyle P. Johnson
    @kylepjohnson
    Wonderful! Seeing drama in the frontend will be powerful
    Luke Hollis
    @lukehollis
    totally! that'll be good
    Was wondering about involving the cltk core more in the API
    So when you GET api.cltk.org:5000/core/stemmer?input="quid faciat laetas segetes quo sidere...etc."
    it will run something like this:
    def stem_input(input):
       s = Stemmer()
       return s.stem(input)
    and you will get back "Quid fac laet seget quo sid ... etc."
    Seems like pretty low-hanging fruit if we could keep it simple.
    Idk about the path /core/stemmer... maybe we could just make that match the directory structure of the cltk core package
    Luke Hollis
    @lukehollis
    This is in my head as I look more at getting scansion and text reuse on the frontend
    Kyle P. Johnson
    @kylepjohnson
    @lukehollis In and of itself, this isn't wrong.
    If you are OK testing out some ideas now in the API and don't mind if it gets redone, definitely go for implementing the routing you're thinking of.
    I think I'll need to sit down with pen and paper to come to a good API proposal that will be flexible enough for all processing and all texts
    Comment/criticism about your URI in particular: I think we'll be happier posting text as data objects, then calling the processing type in URI
    Kyle P. Johnson
    @kylepjohnson
    Because there will come a time when we could be posting very large data packets to the API
    Luke Hollis
    @lukehollis
    Ah sure, that sounds great! well, I'll create some issues for this
    makes sense with the long uris sometimes breaking things in the future
    Luke Hollis
    @lukehollis
    Okay, created an issue to explore the stemmer and cltk api so that we can start moving forward there: cltk/cltk_api#20
    Luke Hollis
    @lukehollis
    There have been a few requests for more comprehensive web application architecture diagramming, so here's what I propose:
    I'm not married to any of this, but I think that this solves the problem of leveraging the CLTK advanced NLP functionality to render data to the frontend interface.
    Kyle P. Johnson
    @kylepjohnson
    This looks very good
    I'll point people here too if they need
    Manvendra Singh
    @manu-chroma
    @lukehollis what is the use of mongoDB database here ?
    can you explain it's function briefly
    can't the meteor application query api in realtime and serve whatever is required rather than getting the whole set of data and storing it the DB
    Kyle P. Johnson
    @kylepjohnson
    @manu-chroma querying in real time might be too slow, so Luke is planning on pre-processing some text