Limiting the worker jobs per time #720

Activity

1. evantahler commented on Dec 12, 2021

   

   evantahler

   on Dec 12, 2021

   Member

   More actions

   Willkommen @kkretsch!

   Out of the box, Node Resque doesn't support worker rate-limiting. Since this is a queue system, the workers, just try to work one job at a time until they are done.

   Here are some ideas:

   • Some of the plugins included here support uniqueness in either enqueuing or working based on the arguments of the job, but that doesn't quite seem like what you are looking for.
   • If there was a very special kind of job that really could only be run one-at-a-time (like interacting with something on the filesystem), you could use a unique queue name for just this job, and only have one worker work this queue (eg: don't work "*" for this one worker). But this limits you to 1 worker per server in your cluster
   • Otherwise, if you have something more complex like a distributed system in which only 10 workers max can work a certain type of job at once (eg: you are hitting an external API's rate limit), then I agree that custom plugin is a great idea.
     • You can use a redis key which is incremented and decremneted at the beforePerform and afterPerform steps
     • The redis key's name could be based on the job name
     • Be sure to but a TTL on this key as well so if the workers hang up or crash, it will be cleared eventually

   Let me know if you want to talk strategy more!

   If you do end up writing a rate-limiting plugin, it would be great to get that checked back into node-resque so others can use it!
2. evantahler

   added

   question

   on Dec 12, 2021
3. kkretsch commented on Dec 13, 2021

   

   kkretsch

   on Dec 13, 2021

   Author

   More actions

   Thanks for the ideas.

   My case is the mentioned API rate limit I would catch in some cases. Currently I just summarize the events per job worker run when seeing I have more in the queue.
   I have several event subscribers, one is a mqtt client which still sends everything to some other server. The other one is a mastodon bot agent which just posts textual articles per each new entry. As those entries would get too much per time and even in total I think summarizing it is a good intermediate solution. (seen in action from now on as @bintra@m.kretschmann.social )

   Next I have to try the real limiting stuff like you suggested.
4. evantahler commented on Dec 14, 2021

   

   evantahler

   on Dec 14, 2021

   Member

   More actions

   Good luck!

   I'm going to move this to a discussion rather than an issue. Keep us posted!
5. actionhero

   locked and limited conversation to collaborators on Dec 14, 2021
6. evantahler

   converted this issue into a discussion #728 on Dec 14, 2021