Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Sree Vaddi
    @sreev
    Proposal: Log Events described at, MarquezProject/marquez-python#88
    Sree Vaddi
    @sreev

    MarquezProject/marquez-python 's default branch is main now.

    MANDATORY STEPS:
    Please update your local settings accordingly.

    daviddhc20120601
    @daviddhc20120601
    hi , the galaxy demo, is there a dataset available?
    daviddhc20120601
    @daviddhc20120601
    how to create a continuous job like the preview shown above???
    2 replies
    Sree Vaddi
    @sreev
    Sree Vaddi
    @sreev
    kartheeswaran
    @kartheesh27_twitter
    can you please tell , how i can install /deploy Marquez in AWS cloud , any input is appreciated
    rbriski
    @rbriski
    Hi @wslulciuc. I'm checking to see if you need help on MarquezProject/marquez-airflow#49. I'd like to add some extractors to that library but that PR changes a lot of the underlying code that an extractor would use. If you need someone to fix that PR up and get it out, I can work on that.
    2 replies
    rbriski
    @rbriski
    @kartheesh27_twitter I was able to get it running on EKS using this helm chart: https://github.com/MarquezProject/marquez-chart/tree/feature/chart/templates
    I set up an RDS Postgres database to serve as the data storage system. That resulted in two changes to the helm chart:
    1. In values.yaml, add the port and adminPort values and change the database connection values to match your RDS instance
    2. I switched from a ClusterIP set up to LoadBalancer so I could access the API from outside of the VPC
    3 replies
    kartheeswaran
    @kartheesh27_twitter
    @rbriski thanks appreciate it :)
    Sree Vaddi
    @sreev
    @rbriski , can you think of add extractors outside that PR ?
    rbriski
    @rbriski
    I can, but the code will all change after @wslulciuc PR goes in. I'd rather not refactor something a week after I write it.
    Julien Le Dem
    @julienledem
    @rbriski after some discussion with Willy, we're going to take this more incremental. There are several things going on in that PR that can be implemented independently. There should be more discussions on an issue on github before changing the Extractor contract.
    Also splitting it will make it easier to collaborate
    rbriski
    @rbriski
    :+1:
    Willy Lulciuc
    @wslulciuc
    @rbriski I closed MarquezProject/marquez-airflow#49. As @julienledem mentioned, we'll be taking a more incremental approach to updating the extractor interface to ensure others can provide feedback and help evolve the library!
    rbriski
    @rbriski
    OK, I saw that @sreev opened another PR. Should I use that as a template for adding extractors? Or would you like me to take on some of the pieces that were part of PR 49
    6 replies
    Sree Vaddi
    @sreev
    Sree Vaddi
    @sreev
    simple issue, for any beginners takers:
    MarquezProject/marquez-python#107
    Michał Januszewski
    @Meehau
    Hi guys, I've got a question - could anyone give more examples of differences between Source Types and Dataset Types? We are using both MySQL tables,CSV + Apache Parquet files on AWS S3, CSV + Apache Parquet files on MapR-FS (lets assume Hadoop FS) and MapR-DB (lets assume Hbase). Could you help me with listing the correct dataset types and source types? Any help would be really appreciated. Example dataset A: a parquet folder on Hadoop; Example dataset B: a table on Hbase. What would be their source and dataset types? (I can create new ones if needed)
    Sree Vaddi
    @sreev
    eg. of SouceType is MYSQL, BIGQUERY, S3, ....; eg. of Dataset Type is DB_TABLE, STREAM, HTTP_ENDPOINT .... and cannot exists without a Source.
    SourceType is changed from an Enum (restricted) to a String (free form).
    you are most welcome to create new types that fits your use case.
    Sree Vaddi
    @sreev

    Marquez Community Meeting

    When:
    Thursday, 15 October 2020
    10:00am to 11:00am
    (UTC-07:00) America/Los Angeles
    Repeats: Every 2 weeks on Thursday

    Where:
    Zoom

    Organizer: Marquez TSC

    Sree Vaddi
    @sreev
    We have a LinkedIn group, now. Please feel free to join. https://www.linkedin.com/groups/13913506/
    We have a Meetup group, too. Please fee free to join, https://www.meetup.com/marquez-hq/events/273937252/
    Sree Vaddi
    @sreev
    We have successfully launched our first Community Meeting with curious crowd. Thank you for your participation and all your great questions. :pray:
    Marc Lamberti
    @marclamberti
    Hi there, happy to join the Marquez community and congratulation for this awesome work! Really useful. Nonetheless I'm a bit disappointed by the marquez-ui, and more specifically by the fact the the UI presented in the meetup https://www.youtube.com/watch?v=BIVUXruv5io&t=2207s&ab_channel=PreziConferenceTeam is absolutely not the same. My question is, why WeWork didn't share the UI as well? Wouldn't be much easier for the community? Instead of building a UI from the ground up? Just curious to have your thoughts
    Willy Lulciuc
    @wslulciuc
    Welcome, @marclamberti (really appreciate your honest feedback). Yeah, you're absolutely right that the current UI needs work. A concern that was brought up / discussed in our community meeting last week. For context, the Marquez UI went through various development phases internally at WeWork. What I presented at the Prezi conference last year was considered more of a "prototype" (more to get early feedback about the Marquez API from end users). We worked with an internal design team to reevaluate the UI flow and usability, with the new UI eventually deployed and used, then open sourced. We are now taking a far more incremental approach to greatly improving the usability of the current UI (@phixMe is leading this effort)
    Our next community meeting will be Thursday, October 29th from 10:00am to 11:00am that will cover more of the roadmap for the UI and ways for others to get involved: (https://lists.lfai.foundation/g/marquez-technical-discuss/viewevent?repeatid=32038&eventid=929509&calstart=2020-10-29)
    Sree Vaddi
    @sreev
    Hi,
    My name is Sree Vaddi. We are a startup based in San Francisco Bay Area and in stealth. I am one of the founder. We are in Data Lineage field.
    We wrote our own marquez clients. We like to request to reserve marquez-python-client and marquez-java-client as project names to align with our internal names; for when the time comes to open source them. Thank you.
    Sree Vaddi
    @sreev
    Sree Vaddi
    @sreev
    Successfully, we had another Community Meeting with curious crowd. Thank you for your participation and all your great questions.
    ajithpunnakula
    @ajithpunnakula
    Hello Team,
    How do I add a review label to a Marquez PR MarquezProject/marquez-python#123 ? I'm trying to follow the steps listed here - Add Label, but unable to add it.
    Willy Lulciuc
    @wslulciuc
    Welcome, @ajithpunnakula! Looks like the contributing guide for the python client is a bit out of date (i.e. you no longer have to add the review label). Thanks for pointing this out and also for the PRs :) (which I'll review this week)
    2 replies
    AvianshKumar
    @AvianshKumar
    Hi,
    I have to install Marquez and then install Marquez-web. Then in which in Marquez POSTGRES_PORT is 5432 and in Marquez-web POSTGRES_PORT is 5432 .so there is conflict and shows error message port is not available when we install Marquez-web.Kindly help.
    9 replies
    sharad-malviya
    @sharad-malviya
    Hi,
    i want use Marquez for job dependencies visualisation. I downloaded marquez-0.11.3 but all the dependency java jars are missing. Can i have reference for those jars? Thanks
    33 replies
    Willy Lulciuc
    @wslulciuc

    @/all We've made the decision to move to slack! This will ensure we promptly respond to your questions and feedback on Marquez (slack has also become the standard for online communities). We will make sure all questions in this channel are answered before fully transitioning over. Below you will find the link to join the Marquez slack channel. Looking forward to continuing our discussion on metadata there!

    Slack Invite: https://join.slack.com/t/marquezproject/shared_invite/zt-izlcs3ar-V_mHjWkAUBBGEHuezX~ZgQ

    3 replies
    prajval kulkarni
    @prajvalkulkarni_twitter
    Hi team, I'm new to Marquez.. I would understand more on the product and start using it. What is the minimum system requirements for using this?
    Evanto
    @Evanto
    Hi guys, a quick question: do you have a project development roadmap somewhere? I was looking for it and couldn't find it.
    walter-hernandez
    @walter-hernandez
    @wslulciuc could you share the slack invite again please? It looks like the one shared in your message has expired
    1 reply
    Ruben Laguna
    @ecerulm
    Is there any screenshot of the Marquez UI somewhere?
    1 reply