Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Ken Hibino
    @hibiken
    Unfortunately, the current implementation doesn't build an index with this task ID; so it would be very inefficient to support that operation. Currently, ID is merely an identifier for each task to uniquely identify a task from others.
    I'll take this feedback and consider supporting this query via API.
    In the meantime, a workaround would be to use Inspector to list tasks from the queue using ListActiveTasks, ListPendingTasks, etc
    Apan Cyborg
    @apancyborg
    Many thanks for taking the time to respond and your very clear explanation. Thank you for all the hard and wonderful work you are doing on asynq.
    Ken Hibino
    @hibiken
    No problem! Thank you for a valuable feedback!!
    Patrick Barnum
    @pbarnum
    hey @hibiken , im running into a retry issue. essentially i have an integration test that will return an error in the ProcessTask function, and then will return a nil the second time it is processed. my expectation is that it will retry the task automatically within the same ProcessTask function, but the task never leaves the retry state. can you provide any insight?
    Patrick Barnum
    @pbarnum
    just found out that the retry delay logic was longer than expected. no issue!
    Ken Hibino
    @hibiken
    Hi @pbarnum! Glad you found out the reason :) If you want to customize retry delay duration, you can specifiy in the Config when you create your server. https://pkg.go.dev/github.com/hibiken/asynq#Config.
    srv := asynq.NewServer(asynq.Config{
        RetryDelayFunc: func(n int, err error, t *Task) time.Duration {
             return 1 * time.Second // retry after 1 second
        },
    })
    newlife
    @newlife
    hi @hibiken I`m writing a client library with python https://github.com/newlife/asynq-py
    however ,I meet a error like this
    asynq: pid=66175 2021/08/24 09:39:10.711036 ERROR: Dequeue error: UNKNOWN: redis eval error: ERR Error running script (call to f_8bb6156bb828c6c6bbf533f870a078d044cd390e): @user_script:13: user_script:13: attempt to perform arithmetic on local 'timeout' (a nil value)
    actually I have set the timeout in task
    newlife
    @newlife
    could you give me some advice or any infomation?
    newlife
    @newlife
    ok ,fix stupid mistake, I give the wrong name of TaskKey
    Mark Raymund N. Tejero
    @bayugyug:matrix.org
    [m]

    hi go-async community,

    does anybody experience same as me when installing the asynq CLI ?

    go get -u github.com/hibiken/asynq/tools/asynq

    currently, i can't install it successfully..

    _```
    go: github.com/spf13/viper upgrade => v1.9.0

    github.com/hibiken/asynq/tools/asynq/cmd

    ../../../../../../pkg/mod/github.com/hibiken/asynq/tools@v0.0.0-20210902125602-b3ef9e91a9ce/asynq/cmd/migrate.go:91:42: cannot use context.Background() (type context.Context) as type string in argument to r.Client().Exists
    ../../../../../../pkg/mod/github.com/hibiken/asynq/tools@v0.0.0-20210902125602-b3ef9e91a9ce/asynq/cmd/migrate.go:95:27: too many arguments in call to r.Client().Rename
    have (context.Context, string, string)
    want (string, string)
    ```_

    did i miss something here?

    bayugyug
    @bayugyug_twitter

    hi @hibiken

    hope you can guide me on how to successfully install the asynq CLI ?

    always get this error below

    type string in argument to r.Client().Exists
    ../../../../../../pkg/mod/github.com/hibiken/asynq/tools@v0.0.0-20210902125602-b3ef9e91a9ce/asynq/cmd/migrate.go:95:27: too many arguments in call to r.Client().Rename
    have (context.Context, string, string)
    want (string, string)

    thanks

    Ken Hibino
    @hibiken
    Sorry for the delayed response. I just released a new version of asynq and that should be able to resolve this CLI download issue. Please let me know if you continue to see the issue :)
    bayugyug
    @bayugyug_twitter
    thank you very very mucho @hibiken
    its working now the CLI :-)
    zeroallox
    @zeroallox
    Love this library so much <3
    1 reply
    Shandon Anderson
    @CheeseL0ver
    Has anyone here had success passing pointers in a payload? For example my application has a ton of db queries so I would want my tasks to be able to reference my db connection. Now on first thought I figured I could add a db pointer to the payload I pass to my task, but because pointer aren't safe to use between go routines is this really a good method? Any suggestions/thoughts?
    zeroallox
    @zeroallox
    You can pass pointers between goroutines all day long. What you have to keep in mind when you do this is who is accessing the pointer and where. Why would you need to pass a pointer in a payload though? As far as i know the payloads are meant to be sent over the network as json to another process.
    If you are looking to do in-process task scheduling (say you have one app, one binary that does XYZ at certain times) i dont think that’s what asynq was designed for. Something like gocron might be better.
    zeroallox
    @zeroallox

    For my use case, I have a single "controller" app that generates tasks and submits them to Async. Worker processes living on other physical machines / containers pick them up over the network and execute. Each worker will create its own connection to the DB if it needs to and is entirely self contained.

    The controller app uses gocron internally to create the tasks. So like "every 15 mins, inspect the DB. If some condition was met, create and submit an Aasynq task that -some-worker-somewhere-else-not-me- will consume”.

    Thats where this lib absolutely nails it.

    Ken Hibino
    @hibiken
    Thanks for jumping in to answer the question @zeroallox ! Yes, Asynq tasks are sent over the network to be handled by another process (most likely running on a separate node), so if you encode pointer into payload, it doesn't mean anything on the other side (points to some random memory location on a different machine!). If you are okay creating a goroutine within the same process to handle task asynchronously, then you don't need this library to handle that :) Let me know if you have more questions @CheeseL0ver !
    Shandon Anderson
    @CheeseL0ver
    Okay, thanks for the responses. I think what I will do is have a "shared" db pointer that all of the handlers for my tasks can use. It makes completely since to think of the processes being run on different nodes, thanks again @zeroallox and @hibiken
    zeroallox
    @zeroallox

    @hibiken Hey Hey! Is there a way to view which tasks have failed or need to be retry-d?

    http://share.dkai.dk/Screen-Shot-2021-12-29-06-24-39.10.png

    Sometimes i can find them by looking through the pending queue, others they’re just “gone”.

    Currently in this queue i can see a few tasks that failed and can find them with a retry count set to 1. Yet the Retry tab shows 0.

    Ken Hibino
    @hibiken
    it's most likely because the failed tasks are being retried within a few seconds and web UI polls redis for every x seconds (by default 8s I think)
    You can do a few things to make sure that you can manually inspect failed tasks
    Option 1) Set Config.RetryDealyFunc so that failed task are delayed for longer period
    Option 2) In the web UI, go to the settings page, and set the polling interval to lower number (e.g. fetch data every 2s)
    Option 3) You can set the MaxRetry(0) option (i.e. no retry) so that failed tasks go directly to archive
    2 replies
    Let me know if this doesn't help :)
    Mahdi Dibaiee
    @mdibaiee

    Hey @hibiken ! :wave:
    First off, thanks for asynq! And thank you for the quick review on my pull-request.

    I'm interested to contribute further... are there any specific issues that you think have a higher priority?

    Ken Hibino
    @hibiken

    Hi @mdibaiee, thank you for contributing to the project! I'm planning on working on a larger feature this year, namely,
    1) Support workflow (issue #244)
    2) Support groups (issue #339)

    But before we work on these features, I want to do some work to make sure that the project is in a maintainable state, so I'll be doing some work to clean up tech debts and adding more docs to get more contributions.
    But there's one low hanging fruit (#359) that you can work on, let me know if you are interested in tackling this!

    Mahdi Dibaiee
    @mdibaiee
    @hibiken yep that sounds good! I'll hopefully have a PR by tomorrow
    zeroallox
    @zeroallox

    Hihi.
    I’m trying to access asyncmon over a wireguard VPN however the page doesnt load. It’s just blank.

    When i check the logs of the container i get:

    2022/04/08 00:54:46 http: superfluous response.WriteHeader call from github.com/hibiken/asynqmon.(*uiAssetsHandler).ServeHTTP (static.go:46)
    ayncmon works PERFECTLY when accessed locally within the network it lives. It’s just over the VPN. I am a zillion% confident it’s nothing to do with the VPN itself, the connection, or anything like that.
    zeroallox
    @zeroallox

    Created a network traffic capture through chrome:

    10.99.66.12.har.zip

    Ken Hibino
    @hibiken
    Hi @zeroallox. thanks for reporting! Would you mind creating a Github issue so we can move the discussion there?
    zeroallox
    @zeroallox
    Yes NP will do today / tomorrowish
    Herb Stahl
    @ghstahl
    I have a 2 step pipeline. Kafka=>redis, where I wrote the kafka handlers with backoff. I was going to do the same for consuming out of redis, until I found asynq. Is there anything like asynq for kafka.
    Ken Hibino
    @hibiken
    @ghstahl thanks for asking! It's in the long-term project roadmap to explore support for other message brokers (Kafka, RabbitMQ, etc) but currently it only supports Redis (See hibiken/asynq#173 for similar feature request)
    Herb Stahl
    @ghstahl
    What are folks doing when it comes to unittesting the engine during a build. I have used miniredis (https://github.com/alicebob/miniredis) in the past where I swapped out the redis-go client for the miniredis one during a github actions build.
    3 replies
    Herb Stahl
    @ghstahl
    This is one of the best OSS I have seen in a very long time, and it was a dream to integrate and get up and working. You really ought to submit it to https://gohack.devpost.com/?utm_source=devpost&utm_medium=twitter&utm_campaign=gohack
    zeroallox
    @zeroallox
    @ghstahl Agreed on all points. Not only does Async “just work”, the documentation does a clear job explaining what Async is and how to use it. It’s just not fancy web3.0cloudnativeelonmuskmonetizeyourNFTassets. So many githib pages now. dont actually tell you what a piece of kit does. Its just words.
    Herb Stahl
    @ghstahl
    Is there a setting to auto purge messages that find themselves in the archive queue? I.e. instead of going to the archive they get thrown out. One of my options is to return a success in the handler even though I failed to process it.
    Ken Hibino
    @hibiken
    Thanks @ghstahl and @zeroallox for the feedback!!
    @ghstahl to answer your question: Currently this is not configurable by the user and hard coded to check for archive size (# of tasks in archive state in a given queue) and age of messages (# of days the task is sitting in the archive): Code is here: https://github.com/hibiken/asynq/blob/5c723f597e01e028f84744bff625e1be803e4ad5/internal/rdb/rdb.go#L819-L822
    We can make this configurable through Config. If you are interested, please file a feature request!
    If you are interested in querying all the archived tasks, and delete them programmatically, you can use Inspector.ListArchivedTasks and Inspector.DeleteTask :)
    Herb Stahl
    @ghstahl
    The power of asynq. So asynq backed by a live redis cluster is cool and all, but I also started using asynq as an in-memory task runner backed by miniredis. You lose asynqmon for observability unless someone can suggest a technique to expose miniredis so it looks like a real redis server to the outside world.
    ChnMig
    @ChnMig
    Hello, developer, I have enabled several different work on the same machine, and I get an error not found task on the task, how should I adjust it?
    ChnMig
    @ChnMig
    handler not found for task, It looks like it was accepted by another mux, but the other mux is not set up to receive this pattern