Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 16:03

    mperham on master

    Initial support for building an… (compare)

  • 16:02

    mperham on master

    Fix build (compare)

  • 15:58

    mperham on master

    Remove Faktory Pro (compare)

  • 15:26
    mperham commented #366
  • 14:42
    clarkdave commented #366
  • Jun 18 20:47
    mperham commented #367
  • Jun 18 20:44
    rakeshkarnati1 opened #367
  • Jun 18 20:34
    rakeshkarnati1 closed #354
  • Jun 18 19:50
    mperham commented #366
  • Jun 18 16:30
    clarkdave edited #366
  • Jun 18 16:29
    clarkdave edited #366
  • Jun 18 16:29
    clarkdave opened #366
  • Jun 16 16:10
    mperham commented #365
  • Jun 16 16:10

    mperham on master

    Expose queue information on cli… Check if hash is in expected st… Return integers instead of floa… and 2 more (compare)

  • Jun 16 16:10
    mperham closed #365
  • Jun 16 06:40
    robinvdvleuten synchronize #365
  • Jun 15 20:20
    robinvdvleuten synchronize #365
  • Jun 15 20:18
    robinvdvleuten synchronize #365
  • Jun 15 19:24
    robinvdvleuten commented #365
  • Jun 15 19:22
    robinvdvleuten synchronize #365
Mike Perham
@mperham
1.5.1 is out with some minor fixes.
Mike Perham
@mperham
The Elixir client just picked up Mutate API support. I love to see it!
Mike Perham
@mperham
NEWS: I'm going to sunset Faktory Pro. Most customers are opting for Enterprise so there's little value in keeping a second commercial tier.
  1. Existing Pro customers will have their subscriptions upgraded to Enterprise for the life of their current subscription. Expect an email in the next few weeks.
  2. Existing Pro builds and binaries will stay but will get no further releases.
Cary Landholt
@CaryLandholt
👍
Mike Perham
@mperham
Faktory Pro has been removed. Faktory Enterprise has a new lower price. Existing customers will need to cancel their old subscription and purchase a new subscription to get the lower price. Unfortunately Stripe does not have any way to migrate pricing plans AFAIK.
3 replies
Neth Six
@nethsix
Would any kind soul be able to help me understand the difference between faktory_options reserve_for (https://github.com/contribsys/faktory/blob/master/docs/protocol-specification.md#optional-fields) and throttling timeout (https://github.com/contribsys/faktory/wiki/Ent-Throttling#throttles). Is the latter a global setting for all jobs enqueued in the queue, while the former is just a way to override the latter on a per job type (assuming jobs includes Faktory::Job)
Mike Perham
@mperham
reserve_for is how long Faktory will let the worker process the job before assuming it has died. The worker must ACK/FAIL the job before this time runs out. So that individual job is reserved for N seconds.
The throttle limits concurrency for the entire queue. All workers on that queue are blocked by the throttle lock so holding that lock is a much higher cost. That's why the default throttle timeout is 60 seconds. The default reserve_for is 30 minutes.
So yes, your last statement is absolutely correct.
Krishna Parajuli
@shaardul
Hi Everyone, in faktory golang worker, can we use same manager to manage multiple processes? or do I have to use seperate binaries for each processes? Thanks in advance.
processes = faktory processes
Krishna Parajuli
@shaardul
@mperham
any help from the community will be appreciated. :)
Brian Ploetz
@bploetz
@shaardul yes, a single manager instance can manage multiple queue processor functions. This is how we run it in production.
1 reply
Mike Perham
@mperham
@shaardul You will have one Faktory process. You can have many golang worker processes, all talking to that one Faktory.
1 reply
coleman
@coleman:de.limited
[m]
@shaardul:
create a single manager
register each job you want that process to handle
control the concurrency to determine how many background threads your golang process will use
Also, regarding how you tell them apart:

I think you should stay focused on the job id for each job that is spawned. Try to keep your worker processes themselves stateless if at all possible.

If that is not possible, then launch multiple processes and pass in an identifier on startup

perhaps something like:
./my-worker -name worker-01
./my-worker -name worker-02
You have a lot of flexibility here. Faktory's api is not very big.
Also, I am not an expert. But this is just my experience with light usage.
Krishna Parajuli
@shaardul
@coleman:de.limited that is about multiple queue. I am asking about multiple faktory processes so I can use multiple cpu cores. I am not sure if golang worker processes under single faktory process use multi core.
Krishna Parajuli
@shaardul
We have 20 threads running in a default queue of a process. Generally these jobs finish within a second. but there could be some jobs that would take 20-30 minutes, based on the volume of data the customer has. These long running jobs could hog the queue for general users who would otherwise finish within seconds. So I want to separate them, I could use 2 queues, but if there is resource usage advantage on running 2 processes I want to do that instead.
End goal is to run long running jobs separately and not disturb general flow.
Mike Perham
@mperham
The golang worker will use multiple cores. Python and Ruby are the main languages that are locked to one core.
Stefan Aurori
@saurori
:wave: Any interest in upgrading the Faktory UI from Bootstrap 3 --> 5? I have some spare time to give it a try. I did a 3->5 and 4->5 upgrade recently. @mperham
Mike Perham
@mperham
Absolutely, very interested but I do have a request or two to minimize maintenance. Please open an issue and I will explain more.
Stefan Aurori
@saurori
@mperham sounds good, here's the issue: contribsys/faktory#359
sloan-dog
@sloan-dog
Looking for some basics, running faktory server in docker. How to create a queue? If there's docs send them my way, didn't see it in node or go worker documentation
Mike Perham
@mperham
A queue is automatically created when you push a job to it.
sloan-dog
@sloan-dog
Thx Mike, faktory is so simple coming from a kafka/pulsar background i need to think less complicated thoughts
Mike Perham
@mperham
@sloan-dog Ha! I'm a fan of reliability through simpler infrastructure. Complexity breeds downtime.
sloan-dog
@sloan-dog

We're about to start using faktory enterprise, are there examples of a k8s based enterprise setup? We just bought our license.

If not, I think I'm going to begin with the config here https://github.com/contribsys/faktory/wiki/Kubernetes-Deployment-Example, and modify to support remote redis, etc. Will probably use the existing config map for redis, and then mount to deployment via ENV variable, create Secret and mount that to ENV with enterprise license key

Mike Perham
@mperham
That's the place to start.
Ben Chiciudean
@benydc
is it possible to remove queued jobs with expire?
Mike Perham
@mperham
I'm not sure what you mean but expiring jobs are documented right here: https://github.com/contribsys/faktory/wiki/Ent-Expiring-Jobs
Ara Hacopian
@ahacop
Can you use an ActiveSupport::Duration with unique_for in the ruby client?
Mike Perham
@mperham
Only if it serializes to a float in JSON
Mike Perham
@mperham
It looks like it will not work. custom: { unique_for: 10.minutes } serializes to unique_for: "600" and Faktory does not recognize a String as a valid amount of time. Needs to be an int or float of seconds. I'd suggest you use 10.minutes.to_i as a workaround.
I updated the Unique Jobs wiki page to include this gotcha.
Ara Hacopian
@ahacop
Thank you!
sloan-dog
@sloan-dog
Do you see any need for faktory to run as a quorum? We run faktory with a remote redis instance, so if faktory dies, the data lives, but nothing gets processed in the mean time. I'm curious of the pros and cons for maybe running faktory instances as a part of a raft, backed by the same redis instance.
Mike Perham
@mperham
I've had that idea once before. We couldn't do it before when Faktory owned the Redis process but with a remote Redis, it's possible. Open an issue if you'd like.
coleman
@coleman:de.limited
[m]
Can I use a closure as a worker.Perform function? The data I intend to close over will be immutable, so I believe that qualifies as thread safe. But are there any gotchas?
I'd like to share some config data, specifically the filepath to a sqlite database
Mike Perham
@mperham
I don't see why not, as long as it's goroutine safe.