Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
icyvapor
@icyvapor
Hey Marcus, thanks for getting back. I think I was lost in a code path I shouldn't be in: The cause of my problem was loader initialization couldn't get a filepath, all three internal functions returning None: https://github.com/getavalon/core/blob/91f5ae87120f61d8beb57af099d321e28178399f/avalon/pipeline.py#L1348 Turned out AVALON_HIERARCHY was used in my template but didn't get passed into context, thus the second call didn't work. By the way, i'm a MongoDB novice, the official mongodb compass worked fine for my simple data viewing: https://www.mongodb.com/products/compass
Marcus Ottosson
@mottosso
@icyvapor Great! That sounds like something that should throw a more descriptive error dialog, at least showing the path it's trying to open. Or at the very least check that template arguments actually have values. Would you be interested in patching this?
icyvapor
@icyvapor
Marcus Ottosson
@mottosso
Brilliant! I've made some comments
kstrandli
@kstrandli

Hi gang! So, I've spent the last evenings reading the code and API and documentation of various pipelines, gathering inspiration for rebuilding my own pipeline from scratch.
Half because I have some very specific ideas of what I want my pipeline to do, but also as a learning experience, to graduate past my hard-coded pipelines and code.
Avalon is pushing a lot of the right buttons for me, and by far the most enjoyable to study. The structure makes sense, and is very similar to what I hope my intuition would eventually lead me to with a few more years experience :p
And the documentation is also just fantastic.

So I'm considering to drop my ego and build upon Avalon instead, which makes the most sense in terms of quality, support, time and money.
But I'm wondering, how doable would it be to make it work without MongoDB?
I have a very small team, often consisting mostly of myself, or up to 15 other people. Some remote.
So I had imagined the pipeline being portable and require minimum infrastructure to assist whatever artists I'm working with.
Does Avalon make sense to me? Or do I need to bite the bullet and setup a server to run mongo 24/7?

Marcus Ottosson
@mottosso

Hello @kstrandli, good to hear you're enjoying it!

But I'm wondering, how doable would it be to make it work without MongoDB?

Possibly, there's nothing inherently dependent on Mongo itself, only that the data being passed around are dictionaries which Mongo does a good job hosting in a way it can be queried with reasonable performance.

As for setting up a server 24/7, it could just be your own machine, with a port open for others to access it. It could be more complicated, you could host it on AWS or rent a physical server off of Hetzner. And long-term you probably would. But getting started needn't be more complicated than that.

@davidlatwe is working on a file-based alternative to the standard Mongo, with the same API. We had a look at building our demos on that a while back, but it wasn't quite ready yet. It's possible, maybe even likely, that it's ready now (ping David!). If so, you could avoid the server, and host the data in a file alongside your assets. You'd still want access from other machines to it though, so it deosn't necessarily change the need for infrastructure. But a file might be easier to grok and maintain?

The main reason you want a centralised server though, and why you probably want to bite that bullet, is because the data is centralised. You want to publish somewhere central, you want to load assets somewhere central. The only reasonable and common alternative that I know of is to do what game studios do and use Perforce. But it's got its own problems, with locking and merging of assets.

One cost-effective option of a server, now that I think of it, would be to just expose a Raspberry Pi to the internet, it can run Mongo just fine and the performance wouldn't be an issue for 15 people (or even 100 people). The bandwidth is small too, in the kb range so even with a metered internet connection you would be fine.
kstrandli
@kstrandli
Ooh, that sounds interesting! Yeh I want centralized project data, but so far we've been able to keep the project on Dropbox, or sync the project root with Resilio. Then the pipeline works relative to where each user has their copy of the pipeline and project saved.
Marcus Ottosson
@mottosso
Personally I would recommend biting the mongo bullet, I don't think you'll regret it. It'll open up new doors to pipeline you thought were exclusive to big shops. Like making interesting queries on the data, "I want all assets that are green, are lowpoly models, has been used on at least 2 shows and have exactly 4 wheels"
kstrandli
@kstrandli
Having my own pipeline server is super tempting, but so far mostly for tinkering, so Im still thinking "what could go wrong?". Thus I've been leaning towards file-based pipelines so far. But I gotta grow up some time.
Marcus Ottosson
@mottosso
I did that too, I was adamant on keeping things file-based. Really didn't like anything with "database" in it.
But now I wish I had bitten the bullet sooner. Spent a lot of time forcing the filesystem to do things it does poorly.
xD
kstrandli
@kstrandli
There's a tickling sensation in the back of my brain that knows you are right, but it is resistant to change. I also adopted Git painfully late because I had imagined as a bigger learning curve than it was.
Marcus Ottosson
@mottosso
Nowadays, I'm adamant about avoiding anything with "SQL" in it. And I don't expect to bite any of those bullets anytime soon.
Mm, the thing I like about Mongo is that it's basically just a big dumb Python dictionary.
kstrandli
@kstrandli
Haha, yeh I looked into Sqlite for its assumed portability, but I would rather stick to dictionaries and Mongo
yeh exactly
Marcus Ottosson
@mottosso
There might be best-of-both, once David sees this he can tell you about https://github.com/davidlatwe/montydb
kstrandli
@kstrandli
That said, I am also working a lot with students at the university I'm teaching at, and being the only person there that have an unnatural excitement over pipeline, I try to stick with solutions I could easily pass onto students.
Marcus Ottosson
@mottosso
It might not be the right fit for you though, because if you store it on Dropbox there can (and very likely will) still be conflicts, where two users updates the file at the same time. Dropbox won't know how to merge those conflicts, and now you have to sift through an opaque database block to figure out what the differences are.
kstrandli
@kstrandli
Currently the school has begun to push Prism onto students, which is filebased, but students find it causes more trouble than its worth
Marcus Ottosson
@mottosso
Mm, I would argue that for anything "pipeline" you really can't avoid a centralised database. It's the bread and butter of any production, especially past 3-5 people.
kstrandli
@kstrandli
I think many "pipelines" are glorified file browsers, which doesn't really provide much value past a prettier interface and reduced clicks
My rookie pipelines have been mostly been about keeping as much data about shots/assets available outside scene files in dicts, so I can automate everything I wish I had a junior to do for me.
So working with a dict based database fits that mentality really well
kstrandli
@kstrandli
Speaking of Automation, could Pyblish be used for stuff not related to publishing? I really like the interface and structure of registering steps. Would be cool to use that for a scene setup tool.
The way I understood pyblish is that you register actions that are associated with certain family types. So if you are in a modelling family, it runs one collection of commands. Could it perform a different set of operations associated with a "setup" phase of a modelling family?
Marcus Ottosson
@mottosso
You probably could, but it's "data-driven" and needs some data to know what to do. E.g. if there's a model in your scene, it would call on modeling-related commands to publish it.
That's right, you register commands to a given "family", such as a model or a rig, and then those are called whenever the artists tries to publish a model or a rig.
You could use it to build scenes, it just might require some creativity. For example, you could use the stage ahead of publishing typically called "collection" to build a scene, as they would run regardless of what data there is.
kstrandli
@kstrandli
Hmm, think I just have to have a play with it!
It is a super nice tool either way!
Thanks a lot for the answers! I have a lot more questions, but I think most of them will be answered by just throwing myself into it. :)
Marcus Ottosson
@mottosso
Enjoy :)
David Lai
@davidlatwe
Haha, I only used montydb once with Avalon for enabling lighting artists doing Avalon + Deadline submission in a remote render farm.
MontyDB could not work with current Avalon Loader, since it requires MongoDB's aggregation, and MontyDB haven't implement it yet. :D
David Lai
@davidlatwe
https://github.com/abstractfactory/openmetadata could be an alternative for not using centralized database I think.
As long as you can be sure that no one allowed to modify those published files in Dropbox, and the path of the metadata/publishes are in same pattern which is formatted from name of project/asset/subset/version .ect.

As long as you can be sure that no one allowed to modify those published files in Dropbox

If you could somehow publish file to the dropbox sync folder, and change the file sharing rules ?

David Lai
@davidlatwe
And this might be another good tool to help achieve that
https://github.com/BigRoy/pather
kstrandli
@kstrandli
Thanks @davidlatwe ! :D I will investigate. Did download montydb, and it is super neat, may use it for some other small scale projects :)
Having some issues with Avalon though, but its probably a user error on my end.
"Runtime Error!
Program: C:\Python37\python.exe
R6034
An application has made an attempt to load the C runtime library incorrectly.
Please contact the application's support team for more information."
I get this error whenever trying to do anything with Avalon that runs a gui.
When I open the Launcher from command line
When opening the Asset Loader from the Launcher (Batman example project)
And when trying to open Maya through the Launcher
kstrandli
@kstrandli
Otherwise I've gotten the interfaces to open, and imported Batman to MongoDB
I can try to run it on a virtual machine or a different system and create a ticket on the github if the problem persists
kstrandli
@kstrandli
@davidlatwe Pather looks absolutely brilliant!
Marcus Ottosson
@mottosso
Hm, haven't seen that error before. It looks like an issue with Python itself. Are you able to launch that python.exe just standalone?
kstrandli
@kstrandli
No errors opening the python.exe
icyvapor
@icyvapor
@kstrandli a quick search on the specific error code turned up many posts about this problem, here's a detailed one: https://support.esri.com/en/technical-article/000013127
kstrandli
@kstrandli
Cool, thanks @icyvapor ! :) I just sadly don't know what to do with that information, as I think the solution is a bit outside my field. Since the warning told me to contact the software developer, I figured I'd be a good sheep and do so.
Was as I suspected after all though, that there's something funky with my paths and system, which is in due need of a clean up either way.
icyvapor
@icyvapor
Yea I'd start with cleaning up the installed Microsoft Visual C++ Runtime Libraries.. those can get out of control quickly.