ColdFusion (2021 release) Update 11 Stops coldfusion-out.log

[Ali Sozkesen]

I use CF 2021 release for a long time. After installing the Update 11, when the coldfusion-out.log reaches a certain size, it's archieved as coldfusion-out.1.log but writing logs to default file stops. 

All services running properly, all other logs works just fine but out log stops writing untill restarting coldfusion. 

As far as I know, I've tested removing cflogging.jar, changing logging directory etc. 

It's working on Ubuntu 23.04 (GNU/Linux 6.2.0-36-generic x86_64)

Anyone facing the same issue or having a solution for that problem?

Any help would be appreciated..

[Charlie Arehart]

I've not heard of any problem of that sort caused by the update. I'd question any of many other things:

• Since the error happens when the log fills (and the archival DOES happen), what is shown in any of the cf logs at that time? 
• Have you confirmed there are 0 fatalerrors and nonfatalerrors in the latest update log in the hf-updates folder? 
• Since you refer to manipulating the logging jars, have you been doing that BEFORE the update? There's no reason to still be doing that. Log4j issues were solved in past updates
• While you may feel this is new since the cf update, do you know when cf had been restarted BEFORE that? It's tracked in the cf logs. If it may have been weeks or months (or even just days) before, note that something could have changed to impact this...which WOULD have started happening if you'd simply restarted cf without the update.

These are all just possibilities to consider. Keep us posted on anything you may find. 

[Ali Sozkesen]

I've not heard of any problem of that sort caused by the update. I'd question any of many other things:

• Since the error happens when the log fills (and the archival DOES happen), what is shown in any of the cf logs at that time? 
  11/11/2023 19:38:26 is the moment of archival and init of new log file. 
  Server log:
  

  "Warning","http-nio-8500-exec-6","11/11/23","19:38:32","promoManagement","Thread: http-nio-8500-exec-6, processing template: /opt/ColdFusion2021/cfusion/wwwroot/promos/syncAll.cfm, completed in 512 seconds, exceeding the 30 seco>

  Exception log: 
  Nothing relevant the issue, or at the specific moment. 
  coldfusion-error.log
  Nothing relevant the issue, or at the specific moment. 
• Have you confirmed there are 0 fatalerrors and nonfatalerrors in the latest update log in the hf-updates folder? 
  

  cfusion/hf-updates/hf-2021-00011-330247 > Adobe_ColdFusion_2021_Update_11_Install_11_08_2023_23_57_34.log
  

  1571 Successes

  0 Warnings

  0 NonFatalErrors

  0 FatalErrors

• Since you refer to manipulating the logging jars, have you been doing that BEFORE the update? There's no reason to still be doing that. Log4j issues were solved in past updates
  I did after update & facing the problem. I get back the file, restarted coldfusion several times. 
• While you may feel this is new since the cf update, do you know when cf had been restarted BEFORE that? It's tracked in the cf logs. If it may have been weeks or months (or even just days) before, note that something could have changed to impact this...which WOULD have started happening if you'd simply restarted cf without the update.
  It's nearly restarted weekly, never faced the issue. 

These are all just possibilities to consider. Keep us posted on anything you may find. 

By @Charlie Arehart

[Charlie Arehart]

I'll tap out, then. 

[Sireex]

I have the same issue with update 11.

Update logs shows no errors:

Installation: Successful.

1612 Successes
0 Warnings
0 NonFatalErrors
0 FatalErrors

coldfusion-out.1.log seems normal, but after that coldfusion-out.log has only few lines and hasn't been updated in a week.

Running jdk-11.0.16.1

[Sireex]

We have an identical test server, except this instance is using  jdk-11.0.19.

I did some testing with archiving current coldfusion-out.log and got these errors in the log file (coldfusion-out):

ERROR StatusConsoleListener Unable to delete file C:\ColdFusion2021\cfusion\logs\coldfusion-out.3.log: java.nio.file.FileSystemException C:\ColdFusion2021\cfusion\logs\coldfusion-out.3.log: The process cannot access the file because it is being used by another process.

ERROR StatusConsoleListener Unable to copy file C:\ColdFusion2021\cfusion\logs\coldfusion-out.2.log to C:\ColdFusion2021\cfusion\logs\coldfusion-out.3.log: java.nio.file.FileSystemException C:\ColdFusion2021\cfusion\logs\coldfusion-out.3.log: The process cannot access the file because it is being used by another process.

[Ali Sozkesen]

I've shutdown CF server, deleted logs directory, restarted it. 

logs directory and log files created again.

But still the same. When coldfusion-out.log reaches 20.4 MB, it is archived like coldfusion-out.1.log and creates a new coldfusion-out.log with zero content. 

All other logs are working perfect. 

[Sireex]

Same here, we have multiple instances running CF2021 update 11, and after update 11 all instances have stopped writing to new coldfusion-out.log after 20mb limit.

[Ali Sozkesen]

As a temporary solution I modified neo-logging.xml / maxOutLogSize from 20 to 2000 which changes 20MB limit for coldfusion-out.log to 2 GB. (requires CF to be restarted)

Unfortunately, logs are crucial for my production and I'll be keeping an eye on the log file size until a fix releases. Hopefully!

[neochad]

I can confirm the same problem in ColdFusion 2023

[Ali Sozkesen]

Update 12 is released today. Not seems relevant with the issue, seems mostly about security, but it's mentioned that "This update also removes the cf-logging.jar file and creates its backup in the updates folder."

I'm adding this comment for informational purposes. I didn't test it in the wild yet. 

[Charlie Arehart]

FWIW, I've not seen this in any of my own servers (cf2021 or 2023, with either of the two latest updates). I've also not heard of anyone else reporting this beyond you all here, including my clients (several dozen on those updates) nor in the community.

That's not denying it's happening, but instead suggesting that there may be something unique to your setups (collectively) to explain things. It will be interesting to hear the ultimate resolution. 

[neochad]

I have been going back and forth with Adobe CF support since I also encountered this problem after upgrading to 2023. They were able to reproduce it on their end and have esclated to product team. Thank you for making this post, I referenced this thread in my communication with them to show it was not a unique issue to our configuration.

[Ali Sozkesen]

You are welcome. That's very kind of you. 

I've allocated 20GB of out log space, by updating neo-logging xml and hopefully waiting for the permanent solution.  

[dwaynek27338072]

Did you find a solution for this?  We are having the same issue for all our Coldfusion 2021 instances with hotfix 12.  We are using Windows server OS running on AWS EC2 instances.  

[dwaynek27338072]

Sorry, forgot to post what we are seeing in the ColdFusion Exception.log:
"Error","scheduler-4","01/16/24","16:19:07","CMPRO","ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@685ebf04] unable to create manager for [L:\coldfusion\logs\session.timeout.log] with data

[carl type3]

Tap into. Seen the same on some of the CF2021 hosts I touch. Not got a solution to add.

[Aaron Trice]

We're seeing the same behavior in our environment. However, we're seeing all logs stop writing, not just coldfusion-out.

[dwaynek27338072]

It's the same with us for all logs when they reach time/size for rollover.

[Charlie Arehart]

To everyone in the thread, I want to repeat that whatever this problem is that you all are having, there's GOT to be some common denominator among you to explain it, other than simply updating cf 2021 or 2023 beyond the updates released in October (11 and 5, respectively). I say this because I've not seen the problem in any of several dozens of servers I help several dozen folks with, on all manner of OS versions, deployments, and architectures--all on those updates or the next. 

I realize neochad reported it to Adobe and they could reproduce it. They must have hit upon whatever the exact situation that uniquely leads to this problem. What is it? Did they tell you? It doesn't seem we (any of us here) know yet. But that's what we need to determine--at least until Adobe comes out with a fix--if people want to solve it in the meantime, other than cranking up that max file size as a workaround.

It may be related to some config change made (that not everyone makes). It may be about some particular installer used (there have been different ones for each of those two cf versions). It may have something to do with a particular java version, or how it was implemented. It could be related to running the lockdown tool, or some manual lockdown step some have done. I could go on. Or it could be some combination of the above, or of course some other thing/s. I'm just throwing out ideas.

Again, if anyone connects the dots, it will be great to hear.

I don't suppose any of you have yet opened a tracker ticket (tracker.adobe.com). If you may have done it, or will, you could share that here and the others could vote it up--and they would then get notified if Adobe marks it fixed. In that case, Adobe may offer a hotfix there--before any cf update includes it.

As always, just trying to help. No offense intended toward anyone. 

[Aaron Trice]

I opened a tracker yesterday https://tracker.adobe.com/#/view/CF-4220471I. Also have a support case open, not much traction on that front yet though. I'll update this thread if we find the root cause.

[Aaron Trice]

Support provided a new bootstrap.jar for us to try. That worked on some of our servers, but not all of them. Waiting for more feedback from them on what was changed in the version they provided, and any insight on what may be causing it to work on some servers but not all.

[Aaron Trice]

The logging eventually stopped again on our server that was working with the new bootstrap file. Support had us remove and reinstall the hotfix after re-downloading it from the web. He was adamant nothing had changed, but wanted me to re-download it. I did that, and so far logging has been working on one of our servers. We're monitoring for a few days to see if it continues. 

I also noted that the Tech note for HF12 now states if you are updating from HF10 or earlier you must recreate the connector. Support said this was not needed, but I went ahead and did it anyway as we were updating from HF9. 

[Aaron Trice]

The fix seems to be remove/reinstall the HF. That has worked on two of our servers, we'll be rolling out to more this week. I checked the hash of the two HF files (my original download and the new one support had me download) and they are the same. Support indicated that the HF must not have completely installed when we did the original deploy. I asked him how we can know if it was completely installed or not when the HF install log showed 100% success. I'm waiting for an answer back on that.

[dkeith@psasys]

Aaron, Thank you for keeping up on this.  We've had the same logging issue.  Not only did I have to re-install HF 11 and HF 12, I had to delete our application sandbox from the CF Security Sandbox page before the logging starting working again.  So there is a new permissions requirement for logging that our application is now needing.  We still need an application sandbox but at least it's a way forward.