Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

Marquez/community

People

Repo info

Activity

Sree Vaddi

@sreev

Proposal: Log Events described at, MarquezProject/marquez-python#88

Sree Vaddi

@sreev

MarquezProject/marquez-python 's default branch is main now.

MANDATORY STEPS:
Please update your local settings accordingly.

daviddhc20120601

@daviddhc20120601

hi , the galaxy demo, is there a dataset available?

https://github.com/MarquezProject/marquez-web

https://github.com/MarquezProject/marquez-web/blob/master/docs/images/preview.png

daviddhc20120601

@daviddhc20120601

how to create a continuous job like the preview shown above???

2 replies

Sree Vaddi

@sreev

recent talk on marquez:

https://www.youtube.com/watch?v=VI5pVT-dB-k

Sree Vaddi

@sreev

another article on marquez: (good read)
https://www.dataplatformschool.com/blog/w0y8g0-the-data-governance-zoo

kartheeswaran

@kartheesh27_twitter

can you please tell , how i can install /deploy Marquez in AWS cloud , any input is appreciated

rbriski

@rbriski

Hi @wslulciuc. I'm checking to see if you need help on MarquezProject/marquez-airflow#49. I'd like to add some extractors to that library but that PR changes a lot of the underlying code that an extractor would use. If you need someone to fix that PR up and get it out, I can work on that.

2 replies

rbriski

@rbriski

@kartheesh27_twitter I was able to get it running on EKS using this helm chart: https://github.com/MarquezProject/marquez-chart/tree/feature/chart/templates

I set up an RDS Postgres database to serve as the data storage system. That resulted in two changes to the helm chart:

In values.yaml, add the port and adminPort values and change the database connection values to match your RDS instance
I switched from a ClusterIP set up to LoadBalancer so I could access the API from outside of the VPC

3 replies

kartheeswaran

@kartheesh27_twitter

@rbriski thanks appreciate it :)

Sree Vaddi

@sreev

@rbriski , can you think of add extractors outside that PR ?

rbriski

@rbriski

I can, but the code will all change after @wslulciuc PR goes in. I'd rather not refactor something a week after I write it.

Julien Le Dem

@julienledem

@rbriski after some discussion with Willy, we're going to take this more incremental. There are several things going on in that PR that can be implemented independently. There should be more discussions on an issue on github before changing the Extractor contract.

Also splitting it will make it easier to collaborate

rbriski

@rbriski

:+1:

Willy Lulciuc

@wslulciuc

@rbriski I closed MarquezProject/marquez-airflow#49. As @julienledem mentioned, we'll be taking a more incremental approach to updating the extractor interface to ensure others can provide feedback and help evolve the library!

rbriski

@rbriski

OK, I saw that @sreev opened another PR. Should I use that as a template for adding extractors? Or would you like me to take on some of the pieces that were part of PR 49

6 replies

Sree Vaddi

@sreev

MarquezProject/marquez#814

Sree Vaddi

@sreev

simple issue, for any beginners takers:
MarquezProject/marquez-python#107

Michał Januszewski

@Meehau

Hi guys, I've got a question - could anyone give more examples of differences between Source Types and Dataset Types? We are using both MySQL tables,CSV + Apache Parquet files on AWS S3, CSV + Apache Parquet files on MapR-FS (lets assume Hadoop FS) and MapR-DB (lets assume Hbase). Could you help me with listing the correct dataset types and source types? Any help would be really appreciated. Example dataset A: a parquet folder on Hadoop; Example dataset B: a table on Hbase. What would be their source and dataset types? (I can create new ones if needed)

Sree Vaddi

@sreev

eg. of SouceType is MYSQL, BIGQUERY, S3, ....; eg. of Dataset Type is DB_TABLE, STREAM, HTTP_ENDPOINT .... and cannot exists without a Source.

SourceType is changed from an Enum (restricted) to a String (free form).

you are most welcome to create new types that fits your use case.

Sree Vaddi

@sreev

Marquez Community Meeting

When:
Thursday, 15 October 2020
10:00am to 11:00am
(UTC-07:00) America/Los Angeles
Repeats: Every 2 weeks on Thursday

Where:
Zoom

Organizer: Marquez TSC

Sree Vaddi

@sreev

We have a LinkedIn group, now. Please feel free to join. https://www.linkedin.com/groups/13913506/

We have a Meetup group, too. Please fee free to join, https://www.meetup.com/marquez-hq/events/273937252/

Sree Vaddi

@sreev

We have successfully launched our first Community Meeting with curious crowd. Thank you for your participation and all your great questions. :pray:

Marc Lamberti

@marclamberti

Hi there, happy to join the Marquez community and congratulation for this awesome work! Really useful. Nonetheless I'm a bit disappointed by the marquez-ui, and more specifically by the fact the the UI presented in the meetup https://www.youtube.com/watch?v=BIVUXruv5io&t=2207s&ab_channel=PreziConferenceTeam is absolutely not the same. My question is, why WeWork didn't share the UI as well? Wouldn't be much easier for the community? Instead of building a UI from the ground up? Just curious to have your thoughts

Willy Lulciuc

@wslulciuc

Welcome, @marclamberti (really appreciate your honest feedback). Yeah, you're absolutely right that the current UI needs work. A concern that was brought up / discussed in our community meeting last week. For context, the Marquez UI went through various development phases internally at WeWork. What I presented at the Prezi conference last year was considered more of a "prototype" (more to get early feedback about the Marquez API from end users). We worked with an internal design team to reevaluate the UI flow and usability, with the new UI eventually deployed and used, then open sourced. We are now taking a far more incremental approach to greatly improving the usability of the current UI (@phixMe is leading this effort)

Our next community meeting will be Thursday, October 29th from 10:00am to 11:00am that will cover more of the roadmap for the UI and ways for others to get involved: (https://lists.lfai.foundation/g/marquez-technical-discuss/viewevent?repeatid=32038&eventid=929509&calstart=2020-10-29)

Sree Vaddi

@sreev

Hi,
My name is Sree Vaddi. We are a startup based in San Francisco Bay Area and in stealth. I am one of the founder. We are in Data Lineage field.

We wrote our own marquez clients. We like to request to reserve marquez-python-client and marquez-java-client as project names to align with our internal names; for when the time comes to open source them. Thank you.

Sree Vaddi

@sreev

You are welcome to join:
https://www.linkedin.com/groups/13917603/

Sree Vaddi

@sreev

Successfully, we had another Community Meeting with curious crowd. Thank you for your participation and all your great questions.

ajithpunnakula

@ajithpunnakula

Hello Team,
How do I add a review label to a Marquez PR MarquezProject/marquez-python#123 ? I'm trying to follow the steps listed here - Add Label, but unable to add it.

Willy Lulciuc

@wslulciuc

Welcome, @ajithpunnakula! Looks like the contributing guide for the python client is a bit out of date (i.e. you no longer have to add the review label). Thanks for pointing this out and also for the PRs :) (which I'll review this week)

2 replies

AvianshKumar

@AvianshKumar

Hi,

I have to install Marquez and then install Marquez-web. Then in which in Marquez POSTGRES_PORT is 5432 and in Marquez-web POSTGRES_PORT is 5432 .so there is conflict and shows error message port is not available when we install Marquez-web.Kindly help.

9 replies

Sree Vaddi

@sreev

https://www.meetup.com/marquez-hq/events/lrntzrybcpbqb/

sharad-malviya

@sharad-malviya

Hi,

i want use Marquez for job dependencies visualisation. I downloaded marquez-0.11.3 but all the dependency java jars are missing. Can i have reference for those jars? Thanks

33 replies

Willy Lulciuc

@wslulciuc

@/all We've made the decision to move to slack! This will ensure we promptly respond to your questions and feedback on Marquez (slack has also become the standard for online communities). We will make sure all questions in this channel are answered before fully transitioning over. Below you will find the link to join the Marquez slack channel. Looking forward to continuing our discussion on metadata there!

Slack Invite: https://join.slack.com/t/marquezproject/shared_invite/zt-izlcs3ar-V_mHjWkAUBBGEHuezX~ZgQ

3 replies

Also, don't forget to join our LF AI mailing lists :)

Mail Lists