@datakurre I am going through the development readme.md and found that we need to update it. Because when we run the make build command we get bunch of error and we don't provide any solution for the problem. It is very hard for the newcomer to guess what causes the build error and how to solve it. There is also some null case arises because local Plone Site environment don't provide any document data on "/index/" path and breadcrumbs item on "/front-page/" path.
I should also run, whether it still works OOTB... (it should)
But sure, README enhancements are welcome.
make init-backend should only be run once (or after backend needs reset) so including it into make build would have made it more complex than necessary
or
watch instead of build
And later
make build or make watch should be enough
this is the right sequence of command to build the project
I spent too much time to manually feed the data into my plone environment and debug the error. thank you!
I'm sorry for the complexity of that build, but it was necessary to have it automatically acceptance tested against real Plone.
Now you are probably able to see that this https://collective.github.io/gatsby-source-plone/ is build from that repository.
@datakurre Yes! I am able to see the same site in my local environment :) I read the code in utils.js folder and gatsby-node.js and understand most of it. But some keywords like batching expanded didn't understand. we are creating one node at a time using es7 generator function but I didn't get it where we are calling .next property to create makeBreadcrumbsNode and makeNavigationNode. After reading the code of both files, I found currently when we are rebuilding the site we are fetching the whole content and checking with the cache value if the item is already present or not with the cache value. If any node is different from previous either we add the new node, update the node with the updated value or delete the node which is not necessary. we also have to implement the same functionality as I guess when we get trigger by webhook. what you say? please! suggest me if I am missing something or leading towards wrong path.
@iFlameing Yes, how I recall it currently works:
- on the first run it uses Plone REST API search endpoint to go through every item (sorted in path order to optimize navigation updates later)
- GatsbyJS support caching, so the plugin store the timestamp of the first run and URL of every found page
- on the subsequent runs it still needs to use Plone REST API search endpoint to get all existing paths to be able to: 1) remove pages gone missing, 2) update page with modification date after the last fetch timestamp 3) when folder has been updated, fetch also navigation and breadcrumbs of every children of that folder 4) update all collections, because it is impossible guess if they have been changed or not
@iFlameing What changes with websocket events is that Plone can tell when page has been removed (not it is only possible to deduct that from URLs gone missing from the search endpoint results).
@iFlameing About navigation and breadcrumbs. In Plone REST API they are called "expansions". They can be called separately or they can be fetched with the main page results. So, they exists as "expansions" to allow optimization of queries.
When ever a page contents are fetched, I recall, the plugin fetches also navigation and breadcrumb extensions (to allow extra requests). But on subsequent runs, when a folder has been modified, we also need to update breadcrumbs and navigation of its children (because the title of the page may have been changed and might affect those). In this case, we only call navigation and breadcrumbs endpoints directly. (Calling those two separately should be faster than calling the main page with the expansions. That would be only one request, but it could be slower than those two smaller ones combined.)
@iFlameing About caching in Gatsby develop. If you run gatsby develop (or make develop) or gatsby build (or make build) multiple times, you should see the cache in use: instead of seeing log messages about fetching the nodes, you should see only messages about "touching" the nodes (if cached node is not touched, GatsbyJS would remove it on subsequen run).
@iFlameing And back to your original question: yes, we need to implement adding, updating and removing pages (and updating navigation and breadcrumbs of affected pages).
@iFlameing I believe that you should look for a solution where you need to touch the existing code as little as possible and instead write new functions to respond to those websocket events. That could lead to code duplicating code, but that is not issue in the beginning. Once the new code works, it would be time to check if it makes sense to merge the old and new code with some shared functions.
@iFlameing That said, obviously you will also need to make changes at least to the main functions called by GatsbyJS (at least
sourceNodes) to make them support similar lazy updates from websocket as gatsby-source-filesystem supports updates from filesystem events.
I hope that us choosing the websocket approach makes this easy enough for this GSOC, because subscribing to websocket events should make this use case pretty similar to gatsby-source-filesystem subscribing file system events.
@datakurre I read the documentation of Gatsby related to creating source plugin, gatsby node api, various action provided by Gatsby. I also read the plone Rest Api for better understanding how things are going on when we fetch the data from Plone CMS and converting them into Gatsby-Node. Just One problem I read the documentation of webhook but I don't understand it . What we get when webhook trigger? we will get only one message that something is changed or we get a specific url that is changed?. Can you explain the webhook addon. please!