Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nacos Client Causes Database Connections to Be Consumed and Not Released During Configuration Refresh #12674

Open
radish172521 opened this issue Sep 20, 2024 · 1 comment

Comments

@radish172521
Copy link

Describe the bug
I am experiencing an issue where every time the Nacos Client refreshes the configuration, a database connection is consumed but not released. This problem leads to the active connection count increasing in the database pool, causing eventual connection exhaustion. The connections remain in the active state even after the refresh completes, and when the connection pool is exhausted, new requests cannot acquire a connection.

This issue occurs when using the Nacos Client along with the Druid DataSource in a Spring Boot application. Even after disabling other factors (such as Jasypt), the issue persists whenever the configuration is refreshed.

Expected behavior
The database connection should be released after the configuration refresh completes, and no connection leaks should occur in the connection pool.

Actually behavior
After each configuration refresh, a new database connection is consumed and remains active, without being released. Over time, this causes the connection pool to run out of available connections, preventing new database requests from acquiring a connection.

How to Reproduce
Steps to reproduce the behavior:
1.Use the Nacos Client(2.3.x or 2.4.x) in a Spring Boot application with a Druid DataSource.
2.Configure Nacos to refresh configurations dynamically
3.Enable database connection monitoring using Druid’s StatViewServlet.
4.Observe that every time the configuration is refreshed, a database connection is consumed but not released.

Desktop (please complete the following information):

  • OS: [e.g. Centos]
  • Version [e.g. nacos-server 2.4, nacos-client 2.4]
  • Module [e.g. naming/config]
  • SDK [e.g. original, spring-cloud-alibaba-nacos, dubbo]

Additional context
Debugging Information:

I enabled Druid’s connection pool monitoring using the StatViewServlet and observed that the active connections increase after each configuration refresh.
I also enabled Druid's connection leak detection (remove-abandoned=true, log-abandoned=true) and saw errors indicating that connections were not being released correctly.
The issue appears to occur when NacosContextRefresher triggers a configuration refresh, and Spring’s @ConfigurationProperties beans, which depend on database connections, are re-initialized.
Example stack trace:
com.alibaba.druid.pool.DruidDataSource - abandon connection, owner thread: nacos.client.config.listener.task-0, connected at : 1726819679731, open stackTrace at java.base/java.lang.Thread.getStackTrace(Thread.java:1621) at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1565) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5074) at com.alibaba.druid.filter.logging.LogFilter.dataSource_getConnection(LogFilter.java:915) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5070) at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2759) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5070) at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:724) at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5070) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1477) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1463) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.boot.context.properties.bind.JavaBeanBinder$BeanProperty.lambda$getValue$0(JavaBeanBinder.java:383) at org.springframework.boot.context.properties.bind.DefaultBindConstructorProvider.getBindConstructor(DefaultBindConstructorProvider.java:46) at org.springframework.boot.context.properties.bind.ValueObjectBinder$ValueObject.get(ValueObjectBinder.java:208) at org.springframework.boot.context.properties.bind.ValueObjectBinder.bind(ValueObjectBinder.java:75) at org.springframework.boot.context.properties.bind.Binder.lambda$bindDataObject$6(Binder.java:480) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.AbstractList$RandomAccessSpliterator.tryAdvance(AbstractList.java:706) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647) at org.springframework.boot.context.properties.bind.Binder.fromDataObjectBinders(Binder.java:488) at org.springframework.boot.context.properties.bind.Binder.lambda$bindDataObject$7(Binder.java:479) at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:597) at org.springframework.boot.context.properties.bind.Binder$Context.withDataObject(Binder.java:583) at org.springframework.boot.context.properties.bind.Binder.bindDataObject(Binder.java:479) at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:418) at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:350) at org.springframework.boot.context.properties.bind.Binder.lambda$bindDataObject$5(Binder.java:477) at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:99) at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:87) at org.springframework.boot.context.properties.bind.JavaBeanBinder.bind(JavaBeanBinder.java:63) at org.springframework.boot.context.properties.bind.Binder.lambda$bindDataObject$6(Binder.java:480) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.AbstractList$RandomAccessSpliterator.tryAdvance(AbstractList.java:706) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647) at org.springframework.boot.context.properties.bind.Binder.fromDataObjectBinders(Binder.java:488) at org.springframework.boot.context.properties.bind.Binder.lambda$bindDataObject$7(Binder.java:479) at org.springframework.boot.context.properties.bind.Binder$Context.withIncreasedDepth(Binder.java:597) at org.springframework.boot.context.properties.bind.Binder$Context.withDataObject(Binder.java:583) at org.springframework.boot.context.properties.bind.Binder.bindDataObject(Binder.java:479) at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:418) at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:350) at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:339) at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:269) at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:256) at org.springframework.boot.context.properties.ConfigurationPropertiesBinder.bind(ConfigurationPropertiesBinder.java:94) at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.bind(ConfigurationPropertiesBindingPostProcessor.java:96) at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.postProcessBeforeInitialization(ConfigurationPropertiesBindingPostProcessor.java:79) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1780) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:412) at org.springframework.cloud.context.properties.ConfigurationPropertiesRebinder.rebind(ConfigurationPropertiesRebinder.java:138) at org.springframework.cloud.context.properties.ConfigurationPropertiesRebinder.rebind(ConfigurationPropertiesRebinder.java:97) at org.springframework.cloud.context.properties.ConfigurationPropertiesRebinder.rebind(ConfigurationPropertiesRebinder.java:85) at org.springframework.cloud.context.properties.ConfigurationPropertiesRebinder.onApplicationEvent(ConfigurationPropertiesRebinder.java:170) at org.springframework.cloud.context.properties.ConfigurationPropertiesRebinder.onApplicationEvent(ConfigurationPropertiesRebinder.java:53) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) at org.springframework.cloud.context.refresh.ContextRefresher.refreshEnvironment(ContextRefresher.java:103) at org.springframework.cloud.context.refresh.ContextRefresher.refresh(ContextRefresher.java:94) at org.springframework.cloud.endpoint.event.RefreshEventListener.handle(RefreshEventListener.java:72) at org.springframework.cloud.endpoint.event.RefreshEventListener.onApplicationEvent(RefreshEventListener.java:61) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) at com.alibaba.cloud.nacos.refresh.NacosContextRefresher$1.innerReceive(NacosContextRefresher.java:126) at com.alibaba.nacos.api.config.listener.AbstractSharedListener.receiveConfigInfo(AbstractSharedListener.java:40) at com.alibaba.nacos.client.config.impl.CacheData$1.run(CacheData.java:440) at com.alibaba.nacos.client.config.impl.CacheData.safeNotifyListener(CacheData.java:483) at com.alibaba.nacos.client.config.impl.CacheData.checkListenerMd5(CacheData.java:324) at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.refreshContentAndCheck(ClientWorker.java:941) at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.refreshContentAndCheck(ClientWorker.java:922) at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.lambda$checkListenCache$7(ClientWorker.java:1031) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:842) ownerThread current state is WAITING, current stackTrace at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341) at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506) at [email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465) at [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436) at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623) at [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435) at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062) at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122) at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at [email protected]/java.lang.Thread.run(Thread.java:842)

@KomachiSion
Copy link
Collaborator

It's not nacos' problem, nacos only push the new configuration to your app. How your app to use the new configuration is not nacos' workflow.

This problem from your stack is your application get the new config and want to rebuild your conneciton , but your DruidDataSource not allow to rebuild because you use different thread.

I think you should recive the push config and notify the owner thread to do this rebuild operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants