another article on marquez: (good read)
https://www.dataplatformschool.com/blog/w0y8g0-the-data-governance-zoo
https://www.dataplatformschool.com/blog/w0y8g0-the-data-governance-zoo
Hi @wslulciuc. I'm checking to see if you need help on MarquezProject/marquez-airflow#49. I'd like to add some extractors to that library but that PR changes a lot of the underlying code that an extractor would use. If you need someone to fix that PR up and get it out, I can work on that.
2 replies
@kartheesh27_twitter I was able to get it running on EKS using this helm chart: https://github.com/MarquezProject/marquez-chart/tree/feature/chart/templates
I set up an RDS Postgres database to serve as the data storage system. That resulted in two changes to the helm chart:
- In values.yaml, add the
portandadminPortvalues and change the database connection values to match your RDS instance - I switched from a
ClusterIPset up toLoadBalancerso I could access the API from outside of the VPC
3 replies
Also splitting it will make it easier to collaborate
@rbriski I closed MarquezProject/marquez-airflow#49. As @julienledem mentioned, we'll be taking a more incremental approach to updating the extractor interface to ensure others can provide feedback and help evolve the library!
Hi guys, I've got a question - could anyone give more examples of differences between Source Types and Dataset Types? We are using both MySQL tables,CSV + Apache Parquet files on AWS S3, CSV + Apache Parquet files on MapR-FS (lets assume Hadoop FS) and MapR-DB (lets assume Hbase). Could you help me with listing the correct dataset types and source types? Any help would be really appreciated. Example dataset A: a parquet folder on Hadoop; Example dataset B: a table on Hbase. What would be their source and dataset types? (I can create new ones if needed)
SourceType is changed from an Enum (restricted) to a String (free form).
you are most welcome to create new types that fits your use case.
We have a Meetup group, too. Please fee free to join, https://www.meetup.com/marquez-hq/events/273937252/
Hi there, happy to join the Marquez community and congratulation for this awesome work! Really useful. Nonetheless I'm a bit disappointed by the marquez-ui, and more specifically by the fact the the UI presented in the meetup https://www.youtube.com/watch?v=BIVUXruv5io&t=2207s&ab_channel=PreziConferenceTeam is absolutely not the same. My question is, why WeWork didn't share the UI as well? Wouldn't be much easier for the community? Instead of building a UI from the ground up? Just curious to have your thoughts
Welcome, @marclamberti (really appreciate your honest feedback). Yeah, you're absolutely right that the current UI needs work. A concern that was brought up / discussed in our community meeting last week. For context, the Marquez UI went through various development phases internally at WeWork. What I presented at the Prezi conference last year was considered more of a "prototype" (more to get early feedback about the Marquez API from end users). We worked with an internal design team to reevaluate the UI flow and usability, with the new UI eventually deployed and used, then open sourced. We are now taking a far more incremental approach to greatly improving the usability of the current UI (@phixMe is leading this effort)
Our next community meeting will be Thursday, October 29th from 10:00am to 11:00am that will cover more of the roadmap for the UI and ways for others to get involved: (https://lists.lfai.foundation/g/marquez-technical-discuss/viewevent?repeatid=32038&eventid=929509&calstart=2020-10-29)
We wrote our own marquez clients. We like to request to reserve
marquez-python-client and marquez-java-client as project names to align with our internal names; for when the time comes to open source them. Thank you.
Hello Team,
How do I add a review label to a Marquez PR MarquezProject/marquez-python#123 ? I'm trying to follow the steps listed here - Add Label, but unable to add it.
How do I add a review label to a Marquez PR MarquezProject/marquez-python#123 ? I'm trying to follow the steps listed here - Add Label, but unable to add it.
I have to install Marquez and then install Marquez-web. Then in which in Marquez POSTGRES_PORT is 5432 and in Marquez-web POSTGRES_PORT is 5432 .so there is conflict and shows error message port is not available when we install Marquez-web.Kindly help.
9 replies
i want use Marquez for job dependencies visualisation. I downloaded marquez-0.11.3 but all the dependency java jars are missing. Can i have reference for those jars? Thanks
33 replies
@/all We've made the decision to move to slack! This will ensure we promptly respond to your questions and feedback on Marquez (slack has also become the standard for online communities). We will make sure all questions in this channel are answered before fully transitioning over. Below you will find the link to join the Marquez slack channel. Looking forward to continuing our discussion on metadata there!
Slack Invite: https://join.slack.com/t/marquezproject/shared_invite/zt-izlcs3ar-V_mHjWkAUBBGEHuezX~ZgQ
3 replies
Also, don't forget to join our LF AI mailing lists :)
Mail Lists
Hi, I'm a data engineer at edX and looking at Marquez, and one of the features we're looking at is the ability to link job runs to specific code commits. Alternatively, if we could link a job run to a historical version of a job that would suffice because jobs have a "location" attribute which can provide the code commit or URI of the job invocation.
2 replies
Hello, we are looking at marquez as a data catalog & lineage solution. Our databases are all hosted on AWS hosted platform. We have APIs to connect to the database/tables using an API key. From what I understand, Marquez doesn't really need to access the db/tables. Would that mean we need to create a separate python application where we fetch all the db/tables information & then use marquez APIs to populate the needed metadata?
Also, we don't use Airflow but use a similar workflow orchestrator tool called digdag. Since there is no marquez digdag plugin, would we need to follow a similar approach where we fetch all job/task information via APIs & push those to marquez? Is it possible to re-use the marquez-airflow plugin for digdag?
Also, we don't use Airflow but use a similar workflow orchestrator tool called digdag. Since there is no marquez digdag plugin, would we need to follow a similar approach where we fetch all job/task information via APIs & push those to marquez? Is it possible to re-use the marquez-airflow plugin for digdag?