Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Luke Hollis
    @lukehollis
    totally! that'll be good
    Was wondering about involving the cltk core more in the API
    So when you GET api.cltk.org:5000/core/stemmer?input="quid faciat laetas segetes quo sidere...etc."
    it will run something like this:
    def stem_input(input):
       s = Stemmer()
       return s.stem(input)
    and you will get back "Quid fac laet seget quo sid ... etc."
    Seems like pretty low-hanging fruit if we could keep it simple.
    Idk about the path /core/stemmer... maybe we could just make that match the directory structure of the cltk core package
    Luke Hollis
    @lukehollis
    This is in my head as I look more at getting scansion and text reuse on the frontend
    Kyle P. Johnson
    @kylepjohnson
    @lukehollis In and of itself, this isn't wrong.
    If you are OK testing out some ideas now in the API and don't mind if it gets redone, definitely go for implementing the routing you're thinking of.
    I think I'll need to sit down with pen and paper to come to a good API proposal that will be flexible enough for all processing and all texts
    Comment/criticism about your URI in particular: I think we'll be happier posting text as data objects, then calling the processing type in URI
    Kyle P. Johnson
    @kylepjohnson
    Because there will come a time when we could be posting very large data packets to the API
    Luke Hollis
    @lukehollis
    Ah sure, that sounds great! well, I'll create some issues for this
    makes sense with the long uris sometimes breaking things in the future
    Luke Hollis
    @lukehollis
    Okay, created an issue to explore the stemmer and cltk api so that we can start moving forward there: cltk/cltk_api#20
    Luke Hollis
    @lukehollis
    There have been a few requests for more comprehensive web application architecture diagramming, so here's what I propose:
    I'm not married to any of this, but I think that this solves the problem of leveraging the CLTK advanced NLP functionality to render data to the frontend interface.
    Kyle P. Johnson
    @kylepjohnson
    This looks very good
    I'll point people here too if they need
    Manvendra Singh
    @manu-chroma
    @lukehollis what is the use of mongoDB database here ?
    can you explain it's function briefly
    can't the meteor application query api in realtime and serve whatever is required rather than getting the whole set of data and storing it the DB
    Kyle P. Johnson
    @kylepjohnson
    @manu-chroma querying in real time might be too slow, so Luke is planning on pre-processing some text
    We may move away from this, but it's the best way to get started for us
    Luke Hollis
    @lukehollis
    Yes, that's all
    We have to keep the text somewhere so for each Text object created we can request definitions, commentary, translations, etc. from the API.
    It'd be awesome if we could just get it all at once!
    And in the future, we may be able to, but that seems like it would require the Flask API application also having a database
    Also, Meteor needs a Mongo database for User account, user notes/comments, and other things that are only on the frontend
    Manvendra Singh
    @manu-chroma
    And in the future, we may be able to, but that seems like it would require the Flask API application also having a database @lukehollis I had this doubt in my mind. Thanks for your response!
    Luke Hollis
    @lukehollis
    Ah, sure--well, I think there's a lot of ways that we can do it..
    Manvendra Singh
    @manu-chroma
    To retrieve data faster, we can use a very simple hack of querying various data from API async fashion. I don't know if that's implemented already in the frontend.
    Luke Hollis
    @lukehollis
    The nice part about only rendering from MongoDb is we can leverage the Meteor websocket DDP connection
    and then put our queries in our react .jsx templates
    Manvendra Singh
    @manu-chroma
    That sure will make site more responsive while the data is being fetched in real-time
    Luke Hollis
    @lukehollis
    yeah! and should be easy to program
    Manvendra Singh
    @manu-chroma
    Caching can also be implemented in front end with aid of the mongoDB
    If I'm not wrong, it will help in reducing the load on API by some amount
    Luke Hollis
    @lukehollis
    Yes, I think you're correct
    Actually, I imagine that we'll only have to update content rarely.. and won't really have to do the interval sync I proposed
    We'll more have to load all content from the API once
    Manvendra Singh
    @manu-chroma
    We'll more have to load all content from the API once
    @lukehollis What do you mean by this ?
    Luke Hollis
    @lukehollis
    and then just put a webhook on the CLTK git repos that tells our API to update the corpus with git and then the frontend application to invalidate it's cache of resources for that document and reingest them
    we can make it as granular as we need to
    Manvendra Singh
    @manu-chroma
    @lukehollis Webhook idea is a great one.
    Luke Hollis
    @lukehollis
    The text server sync gets everything from the API and stores it in the Mongodb database: https://github.com/cltk/cltk_frontend/blob/master/server/text-server-sync.js