Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 03:21
    davidlatwe synchronize #528
  • 03:16
    davidlatwe commented #528
  • Feb 20 17:52
    tokejepsen commented #527
  • Feb 20 15:36
    cameronpd commented #503
  • Feb 20 14:25
    tokejepsen commented #527
  • Feb 20 11:43
    tokejepsen commented #503
  • Feb 20 11:41
    tokejepsen commented #503
  • Feb 20 09:52
    cameronpd commented #503
  • Feb 19 14:02
    jasperges commented #466
  • Feb 18 10:23
    mkolar commented #528
  • Feb 18 04:01
    davidlatwe commented #528
  • Feb 18 03:51
    davidlatwe commented #528
  • Feb 17 19:44
    BigRoy commented #528
  • Feb 17 17:58
    mottosso commented #528
  • Feb 17 17:40
    davidlatwe review_requested #528
  • Feb 17 17:40
    davidlatwe review_requested #528
  • Feb 17 17:40
    davidlatwe review_requested #528
  • Feb 17 17:40
    davidlatwe review_requested #528
  • Feb 17 17:40
    davidlatwe opened #528
  • Feb 17 12:50
    tokejepsen synchronize #524
Aleks
@ddesmond
lets dive more. Streams ,or before ,oplog, is to get changes in mongo only, basicly youre subscribing(like in redis) to changes in mongo. what hapens to postgre changes? This is is syncing only changes from mongo layer of avalon to Cgwire. You dont have streams on kitsu server? also what hapends with kitsu setup? I have it on my server, but can a mortal with hosted kitsu make changes like so?
Aleks
@ddesmond
lets look this from avalon setup scenario, i have 10 pcs runing avalon, will they all query mongo each minute to get "None" back? Better solution is to ask for changes when needed and push data on changes back to kitsu when job is done. (runing avalon 8hrs will query kitsu 60 times per hour, for 8 hours, and user will maybe publish once or twice? even 10 publishes to kitsu is less than 8x60)
Aleks
@ddesmond
proper solution would be to have both mongo and kitsu read and write from same mongo :)
Roy Nieterau
@BigRoy
I'd imagine a Stream in mongo is built in such a way that it listens to an answer or callback from MongoDB as opposed to doing a query every second. So like, tell the mongodb server that whenever a specific update/insert matches my needs please notify me. And that a Stream basically means that your machine is listening to when you get a ping/answer for that from the MongoDB regarding that.
So the server just knows if this happens then send to client at IP X as opposed to the client continuously asking the server, and? And? And? Did it happen?
Toke Jepsen
@tokejepsen

This sounds like the behaviour you need? https://www.mongodb.com/blog/post/an-introduction-to-change-streams

I did try the change streams, but its only for replica sets :( Else it would have been perfect. Actually considered what the implications would be of having a replica set just for this reason, but might be overkill.

Toke Jepsen
@tokejepsen

@ddesmond The reasons why I wanna get away from using actions in Avalon is; clarity and future proofing.

Clarity
If we can map directly from MongoDB to another database like CGWire, it'll be very clear what needs to be implemented for other databases.

Future Proofing
Using actions in Avalon like publishing etc to sync, means we need to either emit a signal or add a "syncing" call when we change something in the database. Granted this could happen at the io.py stage to minimize calls.
But what happens if we built something different in the future that makes changes in the database but does not sync accidentally?
Hooking directly into the changes into the database would allow us to having a syncing mechanism that lasts.

Although if we cant listen to changes in MongoDB, then the io.py method might be better than the brute force method. Sorry, just realising some things while writing :)

For the CGWire side we can listen to events; https://gazu.cg-wire.com/events.html

Marcus Ottosson
@mottosso
(This topic would make a good forum thread)
Aleks
@ddesmond
good youve found about replicas, totaly forgot to mention that... @tokejepsen , this is a great discusion about a not so easy topic, its a two way street.
Jakub Trllo
@iLLiCiTiT

Although if we cant listen to changes in MongoDB.

That would require to implement session based database connection.

Toke Jepsen
@tokejepsen

That would require to implement session based database connection.

Could you elaborate on this?

Jakub Trllo
@iLLiCiTiT
Well session base connection for most of databases is what you described (kind of). When change happens in database, all sessions connected to db get event with this change, plus in sql based databases: changes you made during your session are not written to database until you commit them.
Toke Jepsen
@tokejepsen
So would it not work writing a syncing mechanism into io.py ?
Jakub Trllo
@iLLiCiTiT
Mongo is NoSQL database, session is possible but hard to keep consistent since you can totally change object type or remove key in one action, which needs to be handled, you can do it but it's not easy task. I think it would be possible with access that @BigRoy posted https://emptysqua.re/blog/driver-features-for-mongodb-3-6/. When that will work, you can do it in io (who called it io and named its current context as Session btw?) but that would require thread because watch method is actually while loop that waits for messages from mongo server.
Toke Jepsen
@tokejepsen

I was thinking that we could discard the whole listening directly to MongoDB ( cause it tricky and a faff), and just have a syncing call before/after writing to the database.

Something like (https://github.com/getavalon/core/blob/master/avalon/io.py#L335):

@auto_reconnect
def insert_one(item):
    assert isinstance(item, dict), "item must be of type <dict>"
    schema.validate(item)
    result = self._database[Session["AVALON_PROJECT"]].insert_one(item)
    # Syncing to available registered sync modules here.
    return result
Jakub Trllo
@iLLiCiTiT
how that would help you with the problem?
Toke Jepsen
@tokejepsen
It would sync to CGWire here. Or any sync modules you register.
The problem is syncing to CGWire in a non-brute force method.
Jakub Trllo
@iLLiCiTiT
if you want to know only about insertions and deletions then it's probably ok but updates are tricky
Toke Jepsen
@tokejepsen
Sounds like you have been down this route?
Jakub Trllo
@iLLiCiTiT
I've tried and realized how much work must be done to make it at least work in the most basic way... not mentioning about changes that may happen during publishing. It's similar task like to be able use avalon with SQL database. Possible, but not easy at all...
Toke Jepsen
@tokejepsen

not mentioning about changes that may happen during publishing

Would the changes during publishing not go through io.py?

Would imagine that we just forward the event to the syncing module. Its the syncing module's responsibility to handle writing and updating. For a "simple" database like CGWire, that could be just to create items and register the MongoDB id.
Jakub Trllo
@iLLiCiTiT
I have a tendency to make tasks more complicated then they really are, hopefully you'll find a way :) ...
Toke Jepsen
@tokejepsen
Haha, fair enough. I tend to oversimplify.
Will explore this io.py route a bit.
Toke Jepsen
@tokejepsen
Update on the Photoshop stdin/stdout. I could not get anything to work, to abandoning this experiment.
What I might take from this though, is to have the Python process still be the parent so its easier hook up the server on Photoshop launch.
Toke Jepsen
@tokejepsen
@BigRoy how did the event go last night?
Did you manage to record anything?
Roy Nieterau
@BigRoy

Yes, I did. I will be looking into getting this prepped for online as soon as possible, and get things online. Only the Autodesk Arnold render talk had some confidential bits which I unfortunately was not allowed to record. But I think I got the crucial bits of it at least since those were allowed.

There were some very impressive talks, so I'm looking forward to sharing it.

Toke Jepsen
@tokejepsen
Amazing! Looking forward to it
Kaspar
@kguyaux
image.png
Hello everyone, I want to publish an imagesequence but the pyblish-tool seems to have a bug: It does not show the error because the text-box is too small:
(And I really would like to see what I can do to fix the error :-) )
Toke Jepsen
@tokejepsen
That does look buggy. Can you click the arrows?
Marcus Ottosson
@mottosso
Which distro and desktop manager is that?
QML runs on the GPU, so it could be driver related
Kaspar
@kguyaux
I am using fedora30, with Cinnamon as desktop-manager
I'll check with another graphical (login)options
O, correction: Cinnamon is the desktop-environment, which is built on gnome3
Kaspar
@kguyaux
..And as you can see, the other text-box('Path') expands correctly when I click the arrow.
Marcus Ottosson
@mottosso
Try upgrading/downgrading PyQt, not sure what that could be
Toke Jepsen
@tokejepsen
Yeah, which version of PyQt are you on?
Salvador
@salvaom
I had this same issue and I think upgrading fixed it. I'm using 5.9 and I don't remember experiencing it anymore
Kaspar
@kguyaux
Yes! Thanks all, downgrading from 5.12 to 5.9 did the trick.
Marcus Ottosson
@mottosso
If you're up for it, you would be most welcome to have a look at why it didn't work in a more recent version
And if you can, submit a pull-request to fix it
Kaspar
@kguyaux
I'll see if I can have a look :-)
David Lai
@davidlatwe
Does anyone using 3DsMax ?
Marcus Ottosson
@mottosso
What is that?
Is it a Maya plug-in?
David Lai
@davidlatwe
:laughing: