Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 19:29
    coveralls commented #2780
  • 19:27
    jakhani commented #2780
  • 19:26
    codecov[bot] commented #2780
  • 19:18
    codecov[bot] commented #2780
  • 19:18
    jakhani synchronize #2780
  • 09:31
    coveralls commented #2781
  • 09:27
    codecov[bot] commented #2781
  • 09:19
    sshardool opened #2781
  • 00:46
    coveralls commented #2780
  • 00:43
    codecov[bot] commented #2780
  • 00:35
    jakhani opened #2780
  • Feb 26 20:36
    coveralls commented #2779
  • Feb 26 20:35
    codecov[bot] commented #2779
  • Feb 26 20:27
    aditya1105 opened #2779
  • Feb 26 20:02

    aditya1105 on master

    Changed start status to READY i… (compare)

  • Feb 26 20:02
    aditya1105 closed #2778
  • Feb 26 19:58
    codecov[bot] commented #2778
  • Feb 26 19:52

    aditya1105 on master

    Add probes from the podTemplate… (compare)

  • Feb 26 19:52
    aditya1105 closed #2777
  • Feb 26 19:50
    aditya1105 opened #2778
Ayoub
@I_am_ayoub_twitter
Hi, I have small issue after upgrading from 3.43 to 3.71.1. Scheduling a flow via the API/UI doesn't work anymore and I get the following error message "Error looking for flow trigger of flow: MY_PROJECT
There is no mention about flow trigger in the doc and how they relate to normal flows.
Any idea on how to fix the issue ?
Cheng Ren
@burgerkingeater
do you have server error log? @I_am_ayoub_twitter
Toby Allsopp
@toby-allsopp
Is there a way to override properties that are set in azkaban.properties that isn't overwritten when upgrading? It looks like azkaban.private.properties can't be used to change things that are set in the default azkaban.properties - is there something else or am I missing something?
Toby Allsopp
@toby-allsopp
I created a PR (azkaban/azkaban#2196) to add another properties file that allows one to avoid editing azkaban.properties. Not sure this is the right direction or not though.
xieqifeng
@xieqifeng
hi
Toby Allsopp
@toby-allsopp
:wave:
Hai-long.Guo
@ghl0451
Dear Azkabaners,
Is there a way assign job between different executors?
I find AZ just assign a flow to an exec, and all jobs will be execute on assigned exec
Juho Autio
@juhoautio
@abti Or maybe you could help – could someone have a look at this, please? azkaban/azkaban#2243
Abhishek Tiwari
@abti
@juhoautio folks are out due to our annual shutdown, will request someone after we are back in office
Poorva Lunawat
@PoorvaLunawat

Hi All,

I am using the azkaban-solo-server and want to implement the custom-user-manager. As per the document, we need to implement the UserManager interface, create a jar and add it to extlib folder.
This folder did not exist and hence I created one in azkaban-solo-server->src->main and tried to follow the steps, but for some reason, it did not reflect the changes.

Can anyone guide me on this? If I am missing on some step or did anything wrongly?

Poorva Lunawat
@PoorvaLunawat

Tried using a CustomUserManager with multiple-executor mode as well. No luck on that. Is there any document or any sample code that I can refer too?
Currently getting an error - Error: Could not find or load main class azkaban.execapp.AzkabanExecutorServer. But when I check the JAR file that is generated, it does contain this file and at the same location the script is looking for.

Is there something that I'm missing on.

Really appreciate any help or guidance on this. Stuck on this for a long time now.

Juho Autio
@juhoautio
@PoorvaLunawat https://azkaban.readthedocs.io/en/latest/userManager.html#custom-user-manager
– did you follow this guide? Creating extlib under azkaban-solo-server/src/main seems unexpected to me. Usually the jar would be placed at extlib/ which is under the azkaban installation folder (after extracting the package). You can create the folder if it doesn't exist.
Lzm1996
@Lzm1996

Hi All,

I am studying the schedule algorithm and want to apply to the AZ, Have the class ExecutorManager and ExecutorSelector been deprecated completely? And the new dispatch strategy is poll model, I want to know if the PSO algorithm can be applied to it? Thank you very much!

Jitendra Sarswat
@sarsjits
Hello everyone, can someone help me with a query? I am not able to find a property or a way in which I can enable graceful shutdown of an executor.
Sundeep Kumar Paulraj
@sundeepk1

hi Community members,

Greetings for the day,

I am new to azkaban, in our organization, we are using it to execute batch jobs and I am being tasked with maintainence of the azkaban platform .. I am trying to setup a local vagrant environment for testing and learning purpose, however facing issue with login part

the webserver and executor are running and I get to the UI ... the page. seems distorted, after providing the credentials, its not allowing me to login ... can someone pls guide me with ref to this
Deepak Jaiswal
@djaiswal83
@sundeepk1 can you please share screenshot? Did you check if your javascript environment is set correctly in azkaban.properties in conf?
Sundeep Kumar Paulraj
@sundeepk1
hi @djaiswal83 , will share in few mins
I am running AZ 3.85.1 ... in vagrant machines - centos/7 ... mysql installed on local machine ... ( no errors with reference to the db connectivity)
Sundeep Kumar Paulraj
@sundeepk1
image.png
image.png
shishir goyal
@shishir-22
Hi all, Is there any way to create tmp files like jobname_props_3833914518406956051_tmp, jobname_output_3833914518406956051_tmp files in other directory other than working directory. I looked into it and find out JOB_OUTPUT_PROP_FILE and JOB_PROP_FILE environment variables stores the location of these files which is by default same as working directory. So is there any way to set values of these environment variables other than working directory location.
thiru
@thirumalai_10_twitter
Hi all,
Is there any way to pass system/executor's env variables to jobType command ?
Juho Autio
@juhoautio
@djaiswal83 do you know how to develop the UI side "live"? I've set velocity.dev.mode=true and it's used in here: https://github.com/azkaban/azkaban/blob/master/azkaban-web-server/src/main/java/azkaban/webapp/AzkabanWebServerModule.java#L58
But it doesn't seem to work? Also, it's not only .vm that I'd like to edit on the fly; also .js is loaded from azkaban-web-server/build/install/azkaban-web-server/web/ so if I edit the .js source it is not reflected on the fly on the running app.
This is with AzkabanSingleServer run in IntelliJ IDEA btw
Juho Autio
@juhoautio
Hmm.. azkaban.css is also compiled from .less files
Deepak Jaiswal
@djaiswal83
Not an expert of UI development, let me find out and get back.
Arvind Pruthi
@arvindpruthi
/all Hello Azkaban users! I wanted to inform everyone that we at LinkedIn are working on "Containerization" project. This project entails adding: "Executions on Kubernetes" apart from bare-metal servers. Bare metal servers have entailed a lot of tech debt and the containerization project will address a lot of limitations from an architecture based on bare-metal servers. Will be sharing a design doc soon. Stay tuned!
Arvind Pruthi
@arvindpruthi
@/all ^^
hopechols
@hopechols
Where can I download the distributed tar file for Azkaban 3.9.0, due to the network issue, I'm not able to build it from source
appreciate anyone who can send me the distribution file
arvindpruthi @arvindpruthi Soliciting speakers for big data meetup on Feb 23 (6 - 7 PM PST). Looking for someone to share their experience with Azkaban in the upcoming big data meetup sponsored by Linkedin. This will be a 5 min slot and the topic is around: "Scaling pains in growing data"
Mukund Thakur
@mukund-thakur
Hi @arvindpruthi I can talk about Executor grouping feature in Azkaban. Full details here azkaban/azkaban#670
Arvind Pruthi
@arvindpruthi
@mukund-thakur awesome! Thanks for volunteering. Will get in touch with you offline. My email is: apruthi@linkedin.com
@mukund-thakur can you pls share your email id that I can use for offline communication?
Mukund Thakur
@mukund-thakur
arvindpruthi @arvindpruthi We have been making progress on Containerization Project at Linkedin. This is about enabling Kubernetes instead of bare-metal machines to orchestrate flows. The design documentation has been updated and can be found at: https://github.com/azkaban/azkaban/blob/master/docs/containerization-design.rst
arvindpruthi @arvindpruthi Also pls note that we are unlocking a very important capability as part of containerization. With this release, image owners (Azkaban core, jobtype owners, platform image owners, etc.) will have the capability to do a fine-tuned (0-100%) ramp-up of their binaries. The documentation for the image-management API can be found at: https://github.com/azkaban/azkaban/blob/master/docs/imageMgmtAPI.rst
Arvind Pruthi
@arvindpruthi
We plan to talk about this in a big-data meetup on Feb 23
whoisjohngalt-dev
@whoisjohngalt-dev
Hi all.
When "join" is written down into my hive script, the azkaban workflow reports:
--java.io.IOException: Cannot run program "/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/bin/hadoop" (in directory "/data/software/azkaban-exec-server-3.8.0/ak/execution/52"): error=13, not enough permissions
nevertheless ,the other sql like union,etc are normal all the time.
Arvind Pruthi
@arvindpruthi
@/all Hello everyone. On behalf of Azkaban team at Linkedin, I am happy to announce that the "Containerization project" is fully merged into master now. This implies you can use Kubernetes back-end instead of bare-metal executors for orchestration in Azkaban. The design doc and image management API are published on readthedocs. Checkout: https://azkaban.readthedocs.io/en/latest/containerization-design.html and https://azkaban.readthedocs.io/en/latest/imageMgmtAPI.html
We hope others in Azkaban community can take advantage of this key innovation
@/all Another key upcoming innovation that we will be merging into the master soon are: "Partial Execution REST API". These are especially formulated to facilitate machine learning workflows (Although it can benefit everyone). With these APIs you can do the following:
  • Define Experiments (Ex: Training new Machine Learning Model).
  • Define flows with data dependencies (Input/Outputs) for each job and spec. how input of a job depends on the output of another job
Arvind Pruthi
@arvindpruthi
  • Request a partial execution (Only execute a portion of the flow. Also requesting the execution to pick up inputs from a previous execution of the flow)
  • Track evolution of models/flows by tracking multiple versions of the same flow.