Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Alok Kumar
    @iFlameing
    Hi, everyone
    Alok Kumar
    @iFlameing
    @datakurre I am going through the development readme.md and found that we need to update it. Because when we run the make build command we get bunch of error and we don't provide any solution for the problem. It is very hard for the newcomer to guess what causes the build error and how to solve it. There is also some null case arises because local Plone Site environment don't provide any document data on "/index/" path and breadcrumbs item on "/front-page/" path.
    Asko Soukka
    @datakurre
    Did you run make init-backend
    I should also run, whether it still works OOTB... (it should)
    But sure, README enhancements are welcome.
    make init-backend should only be run once (or after backend needs reset) so including it into make build would have made it more complex than necessary
    Asko Soukka
    @datakurre
    @iFlameing make purge init-backend build :-)
    or watch instead of build
    And later make build or make watch should be enough
    Alok Kumar
    @iFlameing
    @iFlameing make purge init-backend build :-)
    this is the right sequence of command to build the project
    I spent too much time to manually feed the data into my plone environment and debug the error. thank you!
    Asko Soukka
    @datakurre
    Great to hear that you got it running.
    I'm sorry for the complexity of that build, but it was necessary to have it automatically acceptance tested against real Plone.
    Now you are probably able to see that this https://collective.github.io/gatsby-source-plone/ is build from that repository.
    Alok Kumar
    @iFlameing
    @datakurre Yes! I am able to see the same site in my local environment :) I read the code in utils.js folder and gatsby-node.js and understand most of it. But some keywords like batching expanded didn't understand. we are creating one node at a time using es7 generator function but I didn't get it where we are calling .next property to create makeBreadcrumbsNode and makeNavigationNode. After reading the code of both files, I found currently when we are rebuilding the site we are fetching the whole content and checking with the cache value if the item is already present or not with the cache value. If any node is different from previous either we add the new node, update the node with the updated value or delete the node which is not necessary. we also have to implement the same functionality as I guess when we get trigger by webhook. what you say? please! suggest me if I am missing something or leading towards wrong path.
    Asko Soukka
    @datakurre
    @iFlameing Yes, how I recall it currently works:
    • on the first run it uses Plone REST API search endpoint to go through every item (sorted in path order to optimize navigation updates later)
    • GatsbyJS support caching, so the plugin store the timestamp of the first run and URL of every found page
    • on the subsequent runs it still needs to use Plone REST API search endpoint to get all existing paths to be able to: 1) remove pages gone missing, 2) update page with modification date after the last fetch timestamp 3) when folder has been updated, fetch also navigation and breadcrumbs of every children of that folder 4) update all collections, because it is impossible guess if they have been changed or not
    @iFlameing What changes with websocket events is that Plone can tell when page has been removed (not it is only possible to deduct that from URLs gone missing from the search endpoint results).
    Asko Soukka
    @datakurre

    @iFlameing About navigation and breadcrumbs. In Plone REST API they are called "expansions". They can be called separately or they can be fetched with the main page results. So, they exists as "expansions" to allow optimization of queries.

    When ever a page contents are fetched, I recall, the plugin fetches also navigation and breadcrumb extensions (to allow extra requests). But on subsequent runs, when a folder has been modified, we also need to update breadcrumbs and navigation of its children (because the title of the page may have been changed and might affect those). In this case, we only call navigation and breadcrumbs endpoints directly. (Calling those two separately should be faster than calling the main page with the expansions. That would be only one request, but it could be slower than those two smaller ones combined.)

    @iFlameing About caching in Gatsby develop. If you run gatsby develop (or make develop) or gatsby build (or make build) multiple times, you should see the cache in use: instead of seeing log messages about fetching the nodes, you should see only messages about "touching" the nodes (if cached node is not touched, GatsbyJS would remove it on subsequen run).
    @iFlameing And back to your original question: yes, we need to implement adding, updating and removing pages (and updating navigation and breadcrumbs of affected pages).
    Asko Soukka
    @datakurre
    @iFlameing I believe that you should look for a solution where you need to touch the existing code as little as possible and instead write new functions to respond to those websocket events. That could lead to code duplicating code, but that is not issue in the beginning. Once the new code works, it would be time to check if it makes sense to merge the old and new code with some shared functions.
    @iFlameing That said, obviously you will also need to make changes at least to the main functions called by GatsbyJS (at least sourceNodes) to make them support similar lazy updates from websocket as gatsby-source-filesystem supports updates from filesystem events.
    I hope that us choosing the websocket approach makes this easy enough for this GSOC, because subscribing to websocket events should make this use case pretty similar to gatsby-source-filesystem subscribing file system events.
    Alok Kumar
    @iFlameing
    ok thanks for explaining all these things :)
    Alok Kumar
    @iFlameing
    @datakurre I read the documentation of Gatsby related to creating source plugin, gatsby node api, various action provided by Gatsby. I also read the plone Rest Api for better understanding how things are going on when we fetch the data from Plone CMS and converting them into Gatsby-Node. Just One problem I read the documentation of webhook but I don't understand it . What we get when webhook trigger? we will get only one message that something is changed or we get a specific url that is changed?. Can you explain the webhook addon. please!
    Asko Soukka
    @datakurre
    @iFlameing We can forget the webhook plugin. I should be able to add a Plone with websocket support into the gatsby-source-development environment, if not already next week, then after the first week of June.

    @iFlameing And for the websocket, we can define the protocol we need.

    Maybe a websocket message could be JSON encoded payload with something like {"event": "created", "id": "http://url/of/the/page", "uuid": "uuid of the page"}

    @iFlameing For development, it might still be good for you to develop trivial websocket server so that you can manually send websocket messages the plugin.
    Asko Soukka
    @datakurre

    @iFlameing For a development websocket server a minimal chat server could work https://socket.io/get-started/chat/

    Just create that chat example separately from gatsby-source-plone

    Next make gatsby-source-plone connect that server with websocket (believing that it is Plone)

    Finally use the chat client example to publish messages from browser to gatsby-source-plone through that websocket server.

    @iFlameing So even before I can provide a websocket supporting Plone server for you, you can simulate it with minimal nodejs websocket example.
    Asko Soukka
    @datakurre

    @iFlameing Something else you probably need to learn for this is using node debugger https://nodejs.org/en/docs/guides/debugging-getting-started/

    For example, you should be able to run gatsby develop with debugger with node --inspect node_modules/.bin/gatsby develop. With debugging mode you should be able to add "debugger" break points into the code and e.g. have interactive repl to figure things out when websocket connection gets a message.

    Asko Soukka
    @datakurre

    @iFlameing Please, try to transform the main tasks from your GSOC project plain into tasks or real issues into https://github.com/collective/gatsby-source-plone/projects/2

    I'm working on Docker image with Plone and websockets on next Tuesday, but it might still take another week for me to complete it. So you should probably start as you planned.

    Let's try to find a time when I would be available here every week so that we could prevent blocking issues. I am on UTC+3 so I assume that we could have that time around you Monday noon or aftertnoon. Something between 6am UTC and 8am UTC should fit for me.

    Yet, the best way to ask questions is either using GitHub issues or here. Remember to tag me so that I get notifications.

    Asko Soukka
    @datakurre

    @iFlameing Today is the start day of the coding phase.

    You should start by re-reading the rules on how to publish the results at the end of the GSOC:

    • Make your work visible by using GitHub project board tasks, GitHub issues and commit all your work into feature branches so that they could be discussed and merged with pull requests.

    • During the last years it has been ok to make the final report as GitHub Gist referring and explaining all the pull request done for the code.

    I'm sorry that the "community bonding period" was a bit tame, but luckily you already learned practices for both Plone and GatsbyJS communities before and during the application period.
    Asko Soukka
    @datakurre
    @iFlameing Please, try to also plan your summer by transforming your project proposal into issues (or just tasks).
    And if you manage to write test websocket server before I have a websocket Plone, that can also be included in gatsby-source-plone (in some custom folder) to make also that work visible.
    Alok Kumar
    @iFlameing
    @datakurre thank you very much for all these suggestion. I am thinking that I should start with the chat app and first publish message from browser to gatsby-source-plone throught websocket server. And also update the readme.md section and writing test for the added fetching private image part which you mentioned in the pull request.
    Asko Soukka
    @datakurre
    @iFlameing Looking good! Making a push into some branch at the end of the day would be a good practice. Even if the code is broken. Branches and be deleted / merged / rebased etc. and cost nothing. And from GSOC perspective that would make your work visible and easy to evaluate.
    Asko Soukka
    @datakurre

    @iFlameing I got an MVP for websocket notifications like defined in https://github.com/collective/gatsby-source-plone/pull/176#issuecomment-496275066 implemented, I did not get it dockerized yet. But most probably you will get a Docker image with these notifications in 1-2 weeks.

    The first version requires no interaction nor authentication with the server.

    Once I have automated Docker image building for that I'll add authentication (Authorization-token during websocket connect) and require some subscription call before notifications are being sent.

    Alok Kumar
    @iFlameing
    @datakurre that's good. Just unclear about where should I add the boolean websocketUpdates that we mention in the pull request collective/gatsby-source-plone#176. Currently I am working on sending json data from mock server to gatsby-plugin using setTimeout and setInterval, I will make a commit once I tested successfully.
    Alok Kumar
    @iFlameing
    @datakurre Currently I am using make watch for testing with websocket. When I ran the command node --inspect node_modules/.bin/gatsby develop I am getting error module gatsby not found. when I downloaded the gatsby once again and I ran the command once again it shows me module react not found and other dependencies on subsequent run. What should I do now?
    Asko Soukka
    @datakurre
    @iFlameing "websocketUpdates" would go into gatsby-config.js and could then be included in gatsby-node.js next to the other options read from gatsby-config.js
    Asko Soukka
    @datakurre

    @iFlameing I'm sorry for not trying that --inspect beforehand.

    It seems that must be run in ./tests/gatsby-starter-default to work.

    So, you need to remember to:

    cd tests/gatsby-starter-plone
    node --inspect node_modules/.bin/gatsby develop

    I should add a make target for that on some day...

    I'm only familiar with the previous version of that debugger , so I'm not sure how that works, but I assume that the documentation explains.
    Asko Soukka
    @datakurre
    @iFlameing Now, I actually tried it:
    1. add "debugger" keyword into your code as breakpoints
    2. cd tests/gatsby-starter-plone
      node inspect node_modules/.bin/gatsby develop
    3. enter one time "c" to allow it to start
    4. you should see the execution being stopped at your "debugger" breakpoint
    5. enter "repl" to enter into interactive repl at the line of your "debugger" keyword
    I believe some IDEs support more interactive debugging (like adding breakpoints directly from the editor in-fly), but I don't have experience of those yet.