mperham/sidekiq

Unofficial support only, please open an issue or send an email!

Mike Perham

@mperham

https://rubydocs.org/d/rails-6-0-0-rc1/classes/ActiveSupport/ExecutionWrapper.html

that class wraps both web request processing (in puma) and job execution (in sidekiq).

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/rails.rb#L3

Garrett T

@film42

Awesome! Thank you. I didn't realize there was a new pattern.

We had started to move clear_active_connections to an async timer task allowing the request path to run without connection mgmt overhead. I noticed that wasn't in place for sidekiq, but I guess at this point, it doesn't even matter :)

Mark Cheshier

@mark.cheshier_gitlab

Anyone here using k8s+istio w/sidekiq? Having all sorts of trouble getting the sidekiq UI to display properly from a Sinatra app. The resources are there but the UI keeps sending everything to HOST/sidekiq, even if I try and explicitly rewrite from either the virtual service or in the Rack command.

Works great in ECS, just hoping someone has already gone through the pain of migrating to k8s and has some pointers :)

Mark Cheshier

@mark.cheshier_gitlab

In case anyone runs into this, the solution ended up being kind of hacky: make a dummy route in the virtual service and tell Rack to URLMap sidekiq to that spot. The key is to put your dummy route first as istio matches in order.

Cadu Ribeiro

@duduribeiro

Hey folks 👋

I'm using sidekiq (4.2.10) and I have this configuration


Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add MyModule::SidekiqLoggingMiddleware
  end
end

to add this SidekiqLoggingMiddleware as a middleware. this middleware relies on another class (FilterArguments)

module MyModule
  class SidekiqLoggingMiddleware
    def call(worker_class, msg, *)
      yield
    ensure
      filtered_args = FilterArguments.new(msg['args']).filter
      Rails.logger.info "Enqueued #{worker_class}##{msg['jid']} with args: #{filtered_args}"
    end
  end
end

this works fine..

but in development, when I change some code, (probably the hot reload) crashes something, and I get this error

ArgumentError in Api::ApplicationPlansController#masterize A copy of MyModule::SidekiqLoggingMiddleware has been removed from the module tree but is still active!

and I need to restart my server

do you folks have any idea why?

Cadu Ribeiro

@duduribeiro

I think that I fixed explicitly adding the module for the class that the middleware relies:

 filtered_args = MyModule::FilterArguments.new(msg['args']).filter

instead of :

filtered_args = FilterArguments.new(msg['args']).filter

Mike Perham

@mperham

Please upgrade to Sidekiq 5.

dev mode code reloading works much better in sidekiq 5.

William Flanagan

@wflanagan

hey all.. i've got a question.. I've got logs that look like this...

https://gist.github.com/wflanagan/98f85b1cf86d65a4a6bc6701c80bd563

William Flanagan

@wflanagan

basically, my JSON is intermixed with non JSON messages that have "start" and "done XXX.xx" in them. I'm not sure how to get the start and done to stop.. without cutting off all INFO messages all together.. all other cases for the most part are handled and return JSON-formatted log entries..

Adil Ali

@adilw3nomad

This message was deleted

Adil Ali

@adilw3nomad

Hello, I've been encountering an unusual problem. Scheduled jobs are being remove from the scheduled set, but never get pushed to their respective queue. Here's an example from my redis monitor logs

$ cat redis.log.16 | grep -a -i dd9bb66082f6a4a34a03e9e1
1576007694.107665 [0 127.0.0.1:49956] "sadd" "b-IMhn1NRyOCqBYQ-jids" "dd9bb66082f6a4a34a03e9e1"
1576007694.107711 [0 127.0.0.1:49956] "zadd" "schedule" "1576007994.0583317" "{\"class\":\"WorkerClass\",\"args\":[72654,4611361],\"retry\":true,\"queue\":\"first_queue\",\"backtrace\":true
,\"jid\":\"dd9bb66082f6a4a34a03e9e1\",\"created_at\":1576007694.058427,\"bid\":\"IMhn1NRyOCqBYQ\"}"
1576007996.291126 [0 127.0.0.1:49926] "zrem" "schedule" "{\"class\":\"WorkerClass\",\"args\":[72654,4611361],\"retry\":true,\"queue\":\"first_queue\",\"backtrace\":true,\"jid\":\"dd9bb66082
f6a4a34a03e9e1\",\"created_at\":1576007694.058427,\"bid\":\"IMhn1NRyOCqBYQ\"}

I've ruled out latency issues, since there's no connection errors in the sidekiq logs or my rails log. Is this a bug in sidekiq? I noticed that the code for queuing scheduled jobs has this comment

# We need to go through the list one at a time to reduce the risk of something
# going wrong between the time jobs are popped from the scheduled queue and when
# they are pushed onto a work queue and losing the jobs.

Is this just something "going wrong"? I'm running 10 sidekiq processes, but this bug only seems to appear with scheduled jobs. Would appreciate any tips.

Sidekiq version 6.0.3, pro 5.0.1, redis 5.0.7.

Adil Ali

@adilw3nomad

Also ruby version 2.6.5

suryateja01

@suryateja01

encountered above error while working with sidekiq in crystal project with lucky framework. do you folks have any idea why?

This is my ./web.cr file

require "sidekiq/web"

# Basic authentication:
#
# require "kemal-basic-auth"
# basic_auth "username", "password"

Kemal.config do |config|
  # To enable SSL termination:
  # ./web --ssl --ssl-key-file your_key_file --ssl-cert-file your_cert_file
  #
  # For more options, including changing the listening port:
  # ./web --help
end
Kemal::Session.config.secret = "my_super_secret"
# Exact same configuration for the Client API as above
Sidekiq::Client.default_context = Sidekiq::Client::Context.new

Kemal.run

Dominik Goltermann

@Goltergaul

Hello folks, we were wondering if we could just enable Sidekiq unique via "Sidekiq::Enterprise.unique!" for all our sidekiq enterprise setups even if some of them would not use the unique feature. Would turning this on have any downsides if no worker would actually use the uniqueness? Like for example performance implications? Maybe someone here can help me out, i could not find information about this on the wiki :)

Mike Perham

@mperham

Yes, it's safe to enable. If the job has no uniqueness it's close to zero overhead.

Dominik Goltermann

@Goltergaul

Awesome, thank you :)

Abraham Milano

@amilano

Hi everyone, I hope everyone is ok... Is there a way to programmatically set a custom value for the retry attribute to a worker when executing perform_async?

Mike Perham

@mperham

SomeWorker.set(retry: 3).perform_async(...)

Exactly like ActiveJob...

Abraham Milano

@amilano

awesome!!! thanks a lot @mperham

Thomas Chrétien

@tchret

Hi there, I'm dealing with Frozen jobs as stated here https://www.mikeperham.com/2013/05/03/dealing-with-stuck-workers/

The jobs I'm running work fine locally, but they just get "stuck" in production.

Adil Ali

@adilw3nomad

how are you running it in production? can you view the logs from the process?

have they never worked in production, or does the error occur after some time of working normally?

What have you tried to fix it yourself? fyi when you ask for help online, you should give as much info as possible. people will be more likely to help if they have more information, especially on what you've tried to do that hasnt worked

Lior Rabin

@LiorRabin

Hello, is there a way to make sidekiq restart on code changes? It's running in a docker container if it makes any difference

Ben Chiciudean

@benydc

I'm getting WRONGPASS invalid username-password pair when connect to redis kubernetes, anyone seen this issue beforee?

my url is in this format: redis://:some-password@url:6379/0

acejam

@acejam_twitter

@benydc Did you ever figure this out?

Rachel Lanman

@RSR312-206

In Sidekiq Pro - I'm a little confused about the role of the workflow class which kicks off the first step of the workflow. Documentation here: https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches

Why is this class needed?
As written, it calls batch.jobs, but why does it need a batch block when it's already inside the overall batch block?

Mike Perham

@mperham

Please open an issue.

Yusuf Celik

@YusufCelik

So, I have been requested to maintain a project. The project in question has Sidekiq configured. However, while I am able to see tasks being enqueued, these tasks never get to be performed. 5 threads, 5 busy, 0 processed and 0 failed.

Now, I have obviously used google, and there are some suggestions to set a parameter related to testing. However, this parameter was not used in the project.

(Unless that is some default state that things are faked and in some kind of test mode when a specific environment variable is not set)

Yusuf Celik

@YusufCelik

yeah, it was a matter of rails_env production versus development...

Jakub Collier

@jakubcollier_gitlab

Is sidekiq a job scheduler?

Mike Perham

@mperham

https://github.com/mperham/sidekiq/wiki

"Sidekiq is a full-featured background job framework for Ruby. It aims to be simple to integrate with any modern Rails application and much higher performance than other existing solutions."

Sinjae LEE

@Jay9393

Hi there! Our company is using sidekiq very usefully.

By the way, I wonder how sidekiq handle pick the job data in redis.
In my opinion, it seems that sidekiq pushes job data to redis, using redis methods like sadd, lpush. and then Several methods are subsequently executed and the poller is started.
after poller started, is the poller getting the job information through the BRPOP method while polling the jobs accumulated in redis?

I'm confused that how sidekiq gets the job information exactly from redis. 🥺

Mike Perham 🇺🇦

@getajobmike_twitter

See lib/sidekiq/fetch.rb

Where communities thrive

People

Repo info

Activity