Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 25 16:31

    mperham on master

    Improve dark Web UI (#4674) * … (compare)

  • Aug 25 16:31
    mperham closed #4674
  • Aug 25 16:26

    mperham on master

    Allow middleware to yield argum… (compare)

  • Aug 25 16:26
    mperham closed #4673
  • Aug 25 16:26
    mperham commented #4673
  • Aug 25 16:11
    mperham synchronize #4674
  • Aug 25 16:11

    mperham on darkui-contrast

    More constrast fixes (compare)

  • Aug 25 14:49
    mperham commented #4675
  • Aug 25 14:48
    mperham closed #4675
  • Aug 25 14:48
    mperham commented #4675
  • Aug 25 14:25

    mperham on master

    document update using YARD anno… (compare)

  • Aug 25 14:25
    mperham closed #4676
  • Aug 25 12:41
    akanshak984 reopened #4676
  • Aug 25 12:38
    akanshak984 closed #4676
  • Aug 25 12:27
    akanshak984 edited #4676
  • Aug 25 12:27
    akanshak984 opened #4676
  • Aug 25 05:41
    mperham commented #4675
  • Aug 25 05:08
    adampal commented #4675
  • Aug 25 00:49
    adampal opened #4675
  • Aug 24 23:38
    eugeneius commented #4673
Mike Perham
@mperham
more polling traffic to/from redis
Andrey Kryachkov
@kryachkov
No performance or reliability drawbacks?
Mike Perham
@mperham
more traffic is a perf drawback, especially if your Redis is very busy. :-)
Andrey Kryachkov
@kryachkov
Thanks Mike!
Mikael Henriksson
@mhenrixon
Regarding my question over a month ago the error I posted then was just a fallout of the following: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.. Out of Sync in other words.
Mike Perham
@mperham
@mhenrixon Thanks for following up, TIL!
Bulat-Gumerov
@Bulat-Gumerov
Hey @mperham what limits best for sidekiq pods cpu/ram usage in k8s?
TMG
@themainguru
@Bulat-Gumerov Depends on what you are trying to achive. We had huge background jobs + Sidekiq pods assigned for them. In events of high ram usage(crossing 4gb or 8gb), the pods restarted and jobs were lost.On the basis of trial and error we were able to set a limit.
Also refactored the code :)
Ilya Vassilevsky
@vassilevsky
Hello :) We need an in-memory cache inside the Sidekiq process. We plan to use some thread-safe map implementation. Just put it in a singleton and use it from all jobs. Is this a viable plan? Any pitfalls to avoid? Thank you!
Mike Perham
@mperham
@vassilevsky If you only have one Sidekiq process, sure. You can use Redis or memcached if you need something that is accessible from all processes.
Ismael Correa Castro
@Iarfen
hi! how I can open sidekiq? I've already runned bundle exec sidekiq but it doesn't open the browser tab
The Web UI runs in your app server so you need to start puma and go to /sidekiq.
Mikael Henriksson
@mhenrixon
@mperham any gotchas with on hooking into Sidekiq.on(:heartbeat)? Anything I should have in mind if I do?
Garrett T
@film42
I noticed v6 is planning on removing the server/active_record middleware. Is there documentation somewhere showing what has changed in rails?
Mike Perham
@mperham
the middleware has not been necessary since rails 5.
Garrett T
@film42
:+1: do you know if there's an issue that references the behavior changes in rails? I'm currently tracking down some optimizations around the clear_active_connections! path for rails 4
Mike Perham
@mperham
connection cleanup is handled by ActiveSupport::Executor now.
there's been a dozen issues around Executor over the last 2 years.
that class wraps both web request processing (in puma) and job execution (in sidekiq).
Garrett T
@film42
Awesome! Thank you. I didn't realize there was a new pattern.
We had started to move clear_active_connections to an async timer task allowing the request path to run without connection mgmt overhead. I noticed that wasn't in place for sidekiq, but I guess at this point, it doesn't even matter :)
Mark Cheshier
@mark.cheshier_gitlab
Anyone here using k8s+istio w/sidekiq? Having all sorts of trouble getting the sidekiq UI to display properly from a Sinatra app. The resources are there but the UI keeps sending everything to HOST/sidekiq, even if I try and explicitly rewrite from either the virtual service or in the Rack command.
Works great in ECS, just hoping someone has already gone through the pain of migrating to k8s and has some pointers :)
Mark Cheshier
@mark.cheshier_gitlab
In case anyone runs into this, the solution ended up being kind of hacky: make a dummy route in the virtual service and tell Rack to URLMap sidekiq to that spot. The key is to put your dummy route first as istio matches in order.
Cadu Ribeiro
@duduribeiro

Hey folks 👋

I'm using sidekiq (4.2.10) and I have this configuration


Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add MyModule::SidekiqLoggingMiddleware
  end
end

to add this SidekiqLoggingMiddleware as a middleware. this middleware relies on another class (FilterArguments)

module MyModule
  class SidekiqLoggingMiddleware
    def call(worker_class, msg, *)
      yield
    ensure
      filtered_args = FilterArguments.new(msg['args']).filter
      Rails.logger.info "Enqueued #{worker_class}##{msg['jid']} with args: #{filtered_args}"
    end
  end
end

this works fine..

but in development, when I change some code, (probably the hot reload) crashes something, and I get this error

ArgumentError in Api::ApplicationPlansController#masterize A copy of MyModule::SidekiqLoggingMiddleware has been removed from the module tree but is still active!

and I need to restart my server

do you folks have any idea why?
Cadu Ribeiro
@duduribeiro

I think that I fixed explicitly adding the module for the class that the middleware relies:

 filtered_args = MyModule::FilterArguments.new(msg['args']).filter

instead of :

filtered_args = FilterArguments.new(msg['args']).filter
Mike Perham
@mperham
Please upgrade to Sidekiq 5.
dev mode code reloading works much better in sidekiq 5.
William Flanagan
@wflanagan
hey all.. i've got a question.. I've got logs that look like this...
William Flanagan
@wflanagan
basically, my JSON is intermixed with non JSON messages that have "start" and "done XXX.xx" in them. I'm not sure how to get the start and done to stop.. without cutting off all INFO messages all together.. all other cases for the most part are handled and return JSON-formatted log entries..
Adil Ali
@adilw3nomad
This message was deleted
Adil Ali
@adilw3nomad

Hello, I've been encountering an unusual problem. Scheduled jobs are being remove from the scheduled set, but never get pushed to their respective queue. Here's an example from my redis monitor logs

$ cat redis.log.16 | grep -a -i dd9bb66082f6a4a34a03e9e1
1576007694.107665 [0 127.0.0.1:49956] "sadd" "b-IMhn1NRyOCqBYQ-jids" "dd9bb66082f6a4a34a03e9e1"
1576007694.107711 [0 127.0.0.1:49956] "zadd" "schedule" "1576007994.0583317" "{\"class\":\"WorkerClass\",\"args\":[72654,4611361],\"retry\":true,\"queue\":\"first_queue\",\"backtrace\":true
,\"jid\":\"dd9bb66082f6a4a34a03e9e1\",\"created_at\":1576007694.058427,\"bid\":\"IMhn1NRyOCqBYQ\"}"
1576007996.291126 [0 127.0.0.1:49926] "zrem" "schedule" "{\"class\":\"WorkerClass\",\"args\":[72654,4611361],\"retry\":true,\"queue\":\"first_queue\",\"backtrace\":true,\"jid\":\"dd9bb66082
f6a4a34a03e9e1\",\"created_at\":1576007694.058427,\"bid\":\"IMhn1NRyOCqBYQ\"}

I've ruled out latency issues, since there's no connection errors in the sidekiq logs or my rails log. Is this a bug in sidekiq? I noticed that the code for queuing scheduled jobs has this comment

# We need to go through the list one at a time to reduce the risk of something
# going wrong between the time jobs are popped from the scheduled queue and when
# they are pushed onto a work queue and losing the jobs.

Is this just something "going wrong"? I'm running 10 sidekiq processes, but this bug only seems to appear with scheduled jobs. Would appreciate any tips.

Sidekiq version 6.0.3, pro 5.0.1, redis 5.0.7.

Adil Ali
@adilw3nomad
Also ruby version 2.6.5
suryateja01
@suryateja01
image.png
encountered above error while working with sidekiq in crystal project with lucky framework. do you folks have any idea why?
This is my ./web.cr file
require "sidekiq/web"

# Basic authentication:
#
# require "kemal-basic-auth"
# basic_auth "username", "password"

Kemal.config do |config|
  # To enable SSL termination:
  # ./web --ssl --ssl-key-file your_key_file --ssl-cert-file your_cert_file
  #
  # For more options, including changing the listening port:
  # ./web --help
end
Kemal::Session.config.secret = "my_super_secret"
# Exact same configuration for the Client API as above
Sidekiq::Client.default_context = Sidekiq::Client::Context.new

Kemal.run
Dominik Goltermann
@Goltergaul
Hello folks, we were wondering if we could just enable Sidekiq unique via "Sidekiq::Enterprise.unique!" for all our sidekiq enterprise setups even if some of them would not use the unique feature. Would turning this on have any downsides if no worker would actually use the uniqueness? Like for example performance implications? Maybe someone here can help me out, i could not find information about this on the wiki :)
Mike Perham
@mperham
Yes, it's safe to enable. If the job has no uniqueness it's close to zero overhead.
Dominik Goltermann
@Goltergaul
Awesome, thank you :)
Abraham Milano
@amilano
Hi everyone, I hope everyone is ok... Is there a way to programmatically set a custom value for the retry attribute to a worker when executing perform_async?
Mike Perham
@mperham
SomeWorker.set(retry: 3).perform_async(...)
Exactly like ActiveJob...