Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpu overhead increase 10X when log increase outburst #960

Open
smileusd opened this issue Sep 29, 2024 · 0 comments
Open

cpu overhead increase 10X when log increase outburst #960

smileusd opened this issue Sep 29, 2024 · 0 comments

Comments

@smileusd
Copy link
Contributor

smileusd commented Sep 29, 2024

We find the regexp tool take lots cpu if log flush too much. We make a test for 6400 logs in 1 second to test the performance for simulating the case of process issue to report too much logs.

root@hostname:~# tail  -n 1000000 /var/log/messages  |egrep "Sep 29 01:02:40" |grep "audit"| wc -l 
6400

It take 1.95 cpu and 88.8 from regexp execution:
Screenshot 2024-09-29 at 17 07 30
Screenshot 2024-09-29 at 16 41 31

Then I try to filter the audit log by strings.containe funcition to skip the regexp, it is reduced to 0.15 cpu which reduce exceed 10X:

		line = buffer.String()
		buffer.Reset()
		if strings.Contains(line, " audit:") || strings.Contains(line, " audit[") {
			continue
		}
		log, err := s.translator.translate(strings.TrimSuffix(line, "\n"))
		if log == nil {
			if err != nil {
				glog.Warningf("Unable to parse line: %q, %v", line, err)
			}
			continue
		}

Screenshot 2024-09-29 at 15 58 56

So I think we need add a black list to simple fliter the log to skip regexp to protect the npd perforcemance. I will make a pr soon.

@smileusd smileusd changed the title cpu overhead increase 10X when log file increase cpu overhead increase 10X when log increase fast Sep 29, 2024
@smileusd smileusd changed the title cpu overhead increase 10X when log increase fast cpu overhead increase 10X when log increase outburst Sep 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant