-
Notifications
You must be signed in to change notification settings - Fork 577
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manager can send SIGQUIT to worker while exiting, causing coredump #2046
Comments
I was able to prevent this behavior with the following patch --- Server/Prefork.pm.bak 2023-03-21 16:51:31.027344078 +0000
+++ Server/Prefork.pm 2023-03-21 16:54:24.053345048 +0000
@@ -194,6 +194,7 @@
next unless my $w = $self->{pool}{$1};
@$w{qw(healthy time)} = (1, $time) and $self->emit(heartbeat => $1);
$w->{graceful} ||= $time if $2;
+ $w->{quit}++ if $2;
}
} As I understand it, the control flow here is:
This patch "combines" steps 7 and 8, so that
Assumption (2) appears to be fine, but I'm not sure about (1), would appreciate some input from you folks on that point This corresponds to (1) in the Possible Suggestions from the original comment |
This does indeed seem like the correct solution for the problem from #1883. Please make this a PR. |
Steps to reproduce the behavior
SIGQUIT
to a worker process to end it gracefullySIGQUIT
SIG_DFL
before catching the secondSIGQUIT
, causing it to dump its coreIn more detail:
Unpack the attached tarfile which has my app and update
mojo_server_wrapper.pl
to have the correct$ENV{REPRO_PATH}
(you could most likely use any Hypnotoad/Prefork app, but I figured a working example would be best)hypno.tar.gz
Start the app from terminal (A)
strace
one of the workers from another terminal (B)Issue a
kill -QUIT
from a third terminal (C)Observe that the kill signal is received in the strace output (note that two SIGQUIT are received, one from terminal (C) and one from the manager)
The manager will print status information about the killed worker
Expected behavior
The app should not generate a coredump when it is sent a graceful shutdown signal
Actual behavior
The second SIGQUIT from the manager process is being caught by
SIG_DFL
and creating a lot of cores, filling disk space on my serverOpen comments
Issue first seen on Supervillain in our product
Reproduced on Ubuntu in a fresh install
I suspect that this is due to Perl deregistering all signal handlers when
exit
is called at the end ofMojo::Server::Prefork::_spawn
, so the secondSIGQUIT
is being caught by the default handler (core dump) instead of the Perl handler (graceful stop). This is a little racy, since we can sometimes get theSIGQUIT
from the manager before uninstalling the signal handler, so you might need to try the repro a couple times.Possible suggestions:
SIGQUIT
(not sure why this is triggered)SIGTERM
or something that does not create coredumps when the signal handlers are deregistered at exit_spawn
so we catch any late signals before uninstalling the signal handlers duringexit
The text was updated successfully, but these errors were encountered: