-
Notifications
You must be signed in to change notification settings - Fork 69
ThreadsOnFork worker
nesquena edited this page Nov 13, 2012
·
12 revisions
Backburner now supports a ThreadsOnFork worker thanks to ShadowBelmolve. This worker will create a fork for each tube being watched and then spawn up a configurable number of threads to process jobs for each tube. After a configurable number of processed jobs, a new fork will be started to keep the environment clean and protect against memory bloat.
You can select ThreadsOnFork
this as the default worker for processing with:
Backburner.configure do |config|
config.default_worker = Backburner::Workers::ThreadsOnFork
end
or determine the worker on the fly when invoking work:
Backburner.work('newsletter_sender', :worker => Backburner::Workers::ThreadsOnFork)
or through the rake tasks with:
$ QUEUES=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
You can also configure more advanced options to take control of the threads and garbage for each individual tube in the format of QUEUE=TUBE_NAME:THREADS_LIMIT:GARBAGE_LIMIT:RETRIES_LIMIT
. For example:
QUEUE=twitter:10:50:5,parse_page:20:10,send_mail:5,verify_bithday::10 THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
With this:
- twitter tube will have 10 threads, garbage after 50 executions and retry jobs 5 times.
- parse_page will have 20 threads, garbage after 10 executions and retry as the default.
- send_mail will have 5 threads, garbage after 1k and retry as the default.
- verify_birthday will have 2 threads, garbage after 1k and retry 10 times.