FileExistsMode.REPLACE not working

[haidangbk]

In version 6.0.3

I'm new to spring integration

FileExistsMode.REPLACE throw exception instead it changes the original file.
I don't know if this is a bug.

[artembilan]

Can we see the whole stack trace, please?

[haidangbk]

Can we see the whole stack trace, please?

This is full stack trace.

org.apache.sshd.sftp.common.SftpException: Failure
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:217) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:212) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:186) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkCommandStatus(AbstractSftpClient.java:164) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.rename(AbstractSftpClient.java:647) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.apache.sshd.sftp.client.SftpClient.rename(SftpClient.java:617) ~[sshd-sftp-2.9.2.jar:2.9.2]
	at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:162) ~[spring-integration-sftp-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:252) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:606) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.sendFileToRemoteDirectory(RemoteFileTemplate.java:570) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:353) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.lambda$send$0(RemoteFileTemplate.java:314) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:452) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:314) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:302) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.RemoteFileTemplate.send(RemoteFileTemplate.java:294) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:207) ~[spring-integration-file-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.AbstractMessageHandler.doHandleMessage(AbstractMessageHandler.java:105) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:59) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:136) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.AbstractMessageHandler.doHandleMessage(AbstractMessageHandler.java:105) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.channel.AbstractMessageChannel.sendInternal(AbstractMessageChannel.java:373) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:327) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:297) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:151) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:143) ~[spring-messaging-6.0.5.jar:6.0.5]
	at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:467) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.gateway.GatewayProxyFactoryBean.sendOrSendAndReceive(GatewayProxyFactoryBean.java:669) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:584) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:550) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:540) ~[spring-integration-core-6.0.3.jar:6.0.3]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.5.jar:6.0.5]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:218) ~[spring-aop-6.0.5.jar:6.0.5]
	at jdk.proxy2/jdk.proxy2.$Proxy66.sendToSftp(Unknown Source) ~[na:na]
	at com.example.springintegration.controller.UploadFileController.upload(UploadFileController.java:30) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207) ~[spring-web-6.0.5.jar:6.0.5]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152) ~[spring-web-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:705) ~[tomcat-embed-core-10.1.5.jar:6.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.0.5.jar:6.0.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:814) ~[tomcat-embed-core-10.1.5.jar:6.0]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:223) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.0.5.jar:6.0.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.5.jar:6.0.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.0.5.jar:6.0.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.5.jar:6.0.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.0.5.jar:6.0.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.5.jar:6.0.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:119) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:400) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:859) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1734) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-10.1.5.jar:10.1.5]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

[artembilan]

So, it fails on at org.apache.sshd.sftp.client.SftpClient.rename(SftpClient.java:617) ~[sshd-sftp-2.9.2.jar:2.9.2].
Would be great to know what is going on the server side: perhaps your user doesn't have permissions for renaming...

[haidangbk]

@artembilan
I have set chmod read, write, execute for user in sftp directory.

[artembilan]

OK. Then this one is out of this project scope.
In fact the FileExistsMode.REPLACE is a default one and if it didn't work, everyone would complain.
We have many tests in the project against Apache MINA SFTP server and nothing fails.
I only can suggest to consult with your SFTP server to determine why rename command is rejected.

[der-markus]

Just came along this scenario in my project .

We recently switched to use IntegrationFlow with FileExistsMode.FAIL in the first step.
After a couple of days business came along and requested to set FileExistsMode.REPLACE instead.

As of now we are facing exactly the same issue.

We do have enabled useTemporaryFilename.

• The SFTP Integration is writing the filename.ext.writing file to SFTP server. ✅
• After writing is finished the file is rename to filename.ext ✅
• We need to write a file with the same filename again.
• The SFTP Integration is writing the filename.ext.writing file to SFTP server. ✅
• After writing is finished the file is rename to filename.ext 🛑

So in the end the old filename.ext named file is still on SFTP.
Next to it there is the filename.ext.writing file with the new content.

And the log says:

Caused by: org.springframework.messaging.MessagingException: Failed to write to '/develop/export/XXXXXXX.json.writing' while uploading the file
at org.springframework.integration.file.remote.RemoteFileTemplate.sendFileToRemoteDirectory(RemoteFileTemplate.java:573)
at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:353)
... 74 common frames omitted
Caused by: org.apache.sshd.sftp.common.SftpException: Failure
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:217)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:212)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:186)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkCommandStatus(AbstractSftpClient.java:164)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.rename(AbstractSftpClient.java:647)
at org.apache.sshd.sftp.client.SftpClient.rename(SftpClient.java:617)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:165)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:252)
at org.springframework.integration.file.remote.RemoteFileTemplate.doSend(RemoteFileTemplate.java:606)
at org.springframework.integration.file.remote.RemoteFileTemplate.sendFileToRemoteDirectory(RemoteFileTemplate.java:570)
... 75 common frames omitted

We do have all the rights we need on SFTP Server to perform

Are we missing any configuration? SFTP Documentation tells me to remove the target file before renaming. I thought SFTP Integration would handle this for me?

[artembilan]

at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:165)

The logic there is like this:

	@Override
	public void rename(String pathFrom, String pathTo) throws IOException {
		if (this.sftpClient.getVersion() >= SftpConstants.SFTP_V5) {
			this.sftpClient.rename(pathFrom, pathTo, SftpClient.CopyMode.Overwrite);
		}
		else {
			try {
				this.sftpClient.rename(pathFrom, pathTo);
			}
			catch (SftpException sftpex) {
				if (SftpConstants.SSH_FX_FILE_ALREADY_EXISTS == sftpex.getStatus()) {
					remove(pathTo);
					// attempt to rename again
					this.sftpClient.rename(pathFrom, pathTo);
				}
				else {
					throw sftpex;
				}
			}
		}
	}

You fail exactly on that this.sftpClient.rename(pathFrom, pathTo);, but looks like the status of that failure is not a SftpConstants.SSH_FX_FILE_ALREADY_EXISTS, but just general SSH_FX_FAILURE.
In the old Spring Integration version 5.5.x we really just treat all the exceptions as failure and call remove before trying to rename again.

However I don't think it is always legit to remove old file for any exceptions thrown from the server.
We always can use a retry advice on our channel adapter to attempt sending the file again.

Would be great if you can consult with your SFTP server to see why its rename command does not return a proper error code.
Or it would be better to use an SFTP v5 or higher.

As I said before: we see that some error is thrown from rename command, but not all of them justify the file removal.

[der-markus]

But wouldn't it be more straight forward, when removing the file ain't intended, at least to clean up the .writing file?

[artembilan]

Sorry, the English is not my first language, and I cannot compile your sentence in my head.

If you talk about removing a temporary .writing file when we fail to rename it to the target one, then yes that is what we can think about.

Otherwise, please, clarify.