Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification

[brookd]

This is on a CF9 server on Windows 2008

We do a lot of writing to and then including from, the VFS (Virtual File System). Very intermittently, we get these errors when attempting to write to the RAM disk. Retrying the failed WRITE operation usually succeeds.

The filename is always a unique file name created with createUUID() in the filename.

I have read that this stack trace error is related to deleting items from an array while accessing it at the same time. Do I need to lock all access to the VFS? I feel like that would really hurt performance.  Should I clean up the temporary files created and written to the VFS at the end of the request, or maybe I should schedule it to run daily so there is less contention for those resources?

java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) at java.util.AbstractList$Itr.next(AbstractList.java:343) at org.apache.commons.vfs.provider.ram.RamFileSystem.listChildren(RamFileSystem.java:100) at org.apache.commons.vfs.provider.ram.RamFileObject.doListChildren(RamFileObject.java:81) at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:557) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(C:\inetpub\wwwroot\LF\web\health\monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReactorCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorCoreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Any help appreciated,  the VFS is really not working as well as I hoped it would.

Brook

[BKBK]

Like you, I suspect this is caused by race conditions. You could solve it with a design like:

<!--- Use a suitable value for the timeout --->

<cflock name="someUniqueNameDependingOnTheIndividualFileName" type="exclusive" timeout="2">

<!--- Code that (writes to the file / reads from the file ) --->

</cflock>

<cflock name="someUniqueNameDependingOnTheIndividualFileName" type="exclusive" timeout="2">

<!--- Code that (reads from the file / writes to the file ) --->

</cflock>

If the problem is indeed caused by race conditions, then you will have little choice but to use a lock. I don't expect the lock to hurt performance. If a process waited at the lock, it would only mean that the lock was necessary.

[brookd]

Ok thanks again for the ideas! I don't understand why a lock would be needed if the file name is unique, but... I guess I may have to try this method and see if it helps.

[brookd]

Whats weird is using the VFS seems problematic. I am wondering if these problems are related to the use of virtual directories within the VFS. I only just started using virtual sub directories to organize the temp files and be able to more easily delete them based on where they were stored (ie. temp + more perm files). But check this out:

I have a remote monitoring service that checks server health and also checks VFS space remaining by running this code (monitor.cfm)

<cfset vfsmetadata =  getVFSMetadata('ram')>

<cfdump var="#vfsmetadata#" label="Virtual File System">

<cfif vfsmetadata.free lt 10000000>

Alert me.....

</cfif>

The dump is for me if I want to visually look at remaining space

Intermittently throughout the day I get this error (Weird I get this error just from calling getVFSMetadata()).

---------------------

An error occurred when performing a file operation listFiles on file /widcardparsingtemplates.

org.apache.commons.vfs.FileSystemException: Could not list the contents of folder "ram:///widcardparsingtemplates". at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:561) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:131) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReactorCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorCoreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) Caused by: java.lang.ArrayIndexOutOfBoundsException: 2114 at

org.apache.commons.vfs.FileSystemException: Could not list the contents of folder "ram:///widcardparsingtemplates". at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:561) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:131) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(C:\inetpub\wwwroot\LF\web\health\monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReactorCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorCoreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) Caused by: java.lang.ArrayIndexOutOfBoundsException: 2114 at

[BKBK]

Might the misspelling widcardparsingtemplates have something to do with it?

[brookd]

No,its not related (its misspelled with consistency )

It may be to early to say for sure but locking all access to VFS may have resolved this problem... I'll  write back in a few days if so...