Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compatibility issues with fluentd plugins #4645

Open
PranavBhatSF opened this issue Sep 25, 2024 · 2 comments
Open

Compatibility issues with fluentd plugins #4645

PranavBhatSF opened this issue Sep 25, 2024 · 2 comments
Assignees
@daipom
Labels
plugin issue work-in-progress

Comments

@PranavBhatSF
Copy link

PranavBhatSF commented Sep 25, 2024 •
edited
Loading

Usecase

We need to push kubernetes pod logs to Azure, S3 or Splunk based on the configuration.

What we have done

Built a custom image with Fluentd image as base and included the required plugins.
Dockerfile

FROM ghcr.io/kube-logging/fluentd:v1.16-4.9-full as base-fluentd

# add azure storage append blob plugin
RUN gem install fluent-plugin-azure-storage-append-blob -v 0.2.1
# add aws s3 plugin
RUN gem install fluent-plugin-s3 -v 1.7.2
# add splunk plugin
RUN gem install fluent-plugin-splunk-hec -v 1.3.2

Tried with https://github.com/fluent/fluent-plugin-splunk as well. But hitting the same issue.

Issue

Fluentd is pushing logs to S3 and Azure. But when we configure splunk as destination, fluentd-configcheck pod fails with the below error

....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)

To Reproduce

  • Use a fluentd image with Azure, S3 and Splunk plugins installed
  • Configure Splunk as destination by creating ClusterOutput and ClusterFlow objects. Configurations are added below.

Expected behavior

Fluentd should send logs to Splunk server

Your Environment

- Fluentd version: 1.16.3

This is a kubernetes 1.30.1 environment. But this is reproducible in older kubernetes versions as well.
We are using kube-logging stack. kube-logging version is 4.9.1

Your Configuration

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: splunk-output
spec:
  splunkHec:
    buffer:
      tags: '[]'
      timekey: <splunk_hec_timekey>
      timekey_use_utc: true
      timekey_wait: 10s
      type: file
    hec_host: <splunk_hec_host>
    hec_port: <splunk_hec_port>
    hec_token: <splunk_hec_token>
    index: <splunk_hec_index>
    insecure_ssl: true
    protocol: <splunk_hec_protocol>
    source: <splunk_hec_source>
    sourcetype: <splunk_hec_source_type>
EOF

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: splunk-flow
  namespace: logging
spec:
  filters:
  - tag_normaliser:
      format: ${namespace_name}/${pod_name}.${container_name}
  globalOutputRefs:
  - splunk-output
  match:
  - select:
      container_names:
      - istio-proxy
      namespaces:
      - istio-system
  - exclude:
      container_names:
      - istio-proxy
      - istio-init
      - aicenter-hit-count-update
      - istio-configure-executor
      - on-prem-tenant-license-update
      - curl
      - recovery
      - aicenter-oob-scheduler
      - cert-trustor
  - exclude:
      namespaces:
      - default
  - exclude:
      labels:
        app: csi-snapshotter
  - exclude:
      labels:
        app: csi-resizer
  - select: {}
EOF

Your Error Log

➜  ~ kubectl logs logging-fluentd-configcheck-691ab8a2  -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
2024-09-25 07:20:53 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-25 07:20:53 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gelf-best' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.9.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.6.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.2'
2024-09-25 07:20:53 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.2.4"
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:210:in `rescue in try_activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:203:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:204:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
@daipom
Copy link
Contributor

daipom commented Sep 26, 2024

Thanks for your report!
It appears that some dependencies conflict with each other.
We need to check it.

@daipom daipom self-assigned this Sep 26, 2024
@daipom
Copy link
Contributor

daipom commented Sep 26, 2024

At least, fluent-plugin-azure-storage-append-blob requires faraday v1.
On the other hand, swd v2 requires faraday v2.
Using swd v1 may solve the error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daipom daipom

Labels
plugin issue work-in-progress
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@daipom @PranavBhatSF