Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add possibility to automatically stop processing if catched up #22

Open
breml opened this issue Apr 24, 2019 · 0 comments
Open

Add possibility to automatically stop processing if catched up #22

breml opened this issue Apr 24, 2019 · 0 comments

Comments

@breml
Copy link

breml commented Apr 24, 2019

When the dead_letter_input plugin is used to investigate failing log events in the dead letter queue, I normally use the commit_offsets option set to false and I start an additional instance of Logstash with a minimal config to make the content of the DLQ visible as suggested in https://www.elastic.co/guide/en/logstash/7.0/dead-letter-queues.html#processing-dlq-events.

The problem I have is, that with this config Logstash will never complete the task because the default behaviour of the dead_letter_input plugin is to poll the DLQ infinitly for new arriving events. Due to this I have to kill Logstash with Ctrl+c after some (hopefully long enough) period of time.
During the investigation phase it would be nice to have an option, which instructs the dead_letter_input plugin to stop processing when it has catched up to the end of the DLQ in a similar way as the stdin input plugin does it, when stdin is closed.
A possible implementation of this feature could be based on a duration, during which no new events are seen in the DLQ. This also would make it possible to post-process the output of Logstash with other tools (in our case with a pipe to jq ).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant