ColdFusion (2021 release) Update 11 Stops coldfusion-out.log

[Ali Sozkesen]

I use CF 2021 release for a long time. After installing the Update 11, when the coldfusion-out.log reaches a certain size, it's archieved as coldfusion-out.1.log but writing logs to default file stops. 

All services running properly, all other logs works just fine but out log stops writing untill restarting coldfusion. 

As far as I know, I've tested removing cflogging.jar, changing logging directory etc. 

It's working on Ubuntu 23.04 (GNU/Linux 6.2.0-36-generic x86_64)

Anyone facing the same issue or having a solution for that problem?

Any help would be appreciated..

[Aaron Trice]

Following up on this thread, Even on machines I have reinstalled each patch, the logging eventually stops once the CF-Error or CF-out log hits it's max size. We don't use the sandbox in our environment. I had a call with support yesterday and they indicated that the log4j files don't seem to be getting updated correctly by the HF installer, even though the logs show 100% success. They are going to be uploading some files for me to test on a machine.

[Ali Sozkesen]

Really appreciate your focus on the matter.

I still have 20 GB of logs on a single machine. 

This evaluated very interesting, took so much time to be fixed. Really that would be something much more critical. 

Adobe doesn't hear community and fix that kind of bug in months. Kind'a shame. 

[Aaron Trice]

Which log(s) did you set to 20GB? And How? I've tried changing the settings on the logging page in the Admin but don't see any difference. 

[Ali Sozkesen]

Yeah it didn't work as well so I dig into some more documentation. 

This is how I found the work around:

https://community.adobe.com/t5/coldfusion-discussions/coldfusion-2021-release-update-11-stops-coldfu...

[Tim2842955474bi]

We're having the same problem across the board on all 32 of our CF2021 servers, and also in our local dev environments (each developer has a setup). We've tried running the updater from the command line and through the CFAdmin (which is notoriously sketchy in my experience, frequently trashing CF). It makes no difference.

We do have a ticket open with Adobe and have been sending back and forth logs over a month, no fix yet. I'm tempted to roll back to update 10. This is a huge impediment.

[Tim2842955474bi]

BTW like some others mentioned, it's not just Coldfusion-out that stops, ALL of our logging stops. 

[Aaron Trice]

Glad I'm not the only one. I'm still working this with support. No resolution yet. My Case is ColdFusion case# 111193 if you'd like to share it with your engineer. They're giving me the impression I'm the only person that's having this issue. 

What update were you on before loading HF10? We went from HF9 to 12 and that's when this started for us.

[Tim2842955474bi]

HF 10 was fine for us. We had a delay in upgrading to 11/12 because of the struct key casing change that 11 introduced. We eventually got a patch from Adobe to maintain backwards compatiblity for that. So we went straight to 12 once that patch was working. After we found the logging issue, further testing confirmed that happens in 11 as well as 12. And of course it's still here in 13. One of our engineers can't get logging to work at all anymore on his local environment, whereas elsewhere we see it work for a day after a CF restart.

I did forward this thread to my Adobe support rep as I was surprised it hadn't come up previously. 

[Aaron Trice]

Got it. Thanks! That's the same behavior we're seeing. After a restart of CF, logging resumes for a period of time then stops. It seems that when a log is supposed to roll over, it just stops logging altogether. But we can't pinpoint a single log or cause. They've given me several patches to try, but none have worked so far.

[Priyank Shrivastava.]

Hi All, 

We are currently debugging the issue at our end and hopefully provide a solution soon that will restore the logging functionally back to normal. Just to let you know, it is not impacting all the users but there are some where we heard this. Unfortunately, we are not able to replicate the issue in our environment and which is why it is taking time to resolve this quickly. 

[Ali Sozkesen]

Actually if replicating the issue is a problem on Adobe's side, I think I can provide some help. 

• Can create a snapshot of my server on Digital Ocean
• Clear critical data (files, logs and mysql) 
• transfer ownership of it to someone from Adobe team on DO. 

As this one took so much time to find the root cause, and impacting people with hundreds of server implementations, I can invest 2 hours to help to point out the exact problem.  

[Priyank Shrivastava.]

@Ali Sozkesen  Can you please DM me and I will share my email ID with you? So we can take this offline and share this with engineering. 

[BKBK]

@Ali Sozkesen  Can you please DM me and I will share my email ID with you? So we can take this offline and share this with engineering. 

By @Priyank Shrivastava.

Hi @Priyank Shrivastava. , could your colleagues in engineering please look into my earlier suggestion:

@BKBK: "I suspect that this has to do with the recent changes in log4j. So ... review the settings within /lib/log4j.properties,  /lib/log4j2.properties and /lib/logging.properties, as well as the required JVM flags."

[BKBK]

The Adobe team should, in any case, do a review of the changes in log4j (implementation, configuration and settings) in Update 13 of ColdFusion 2021. 

[Mike - Media3]

I ran into the same issue where the ColdFusion-out.log file stopped being written to. I stopped the service, cleared the Felix-cache, and started it back up, and that resolved it. Hope this helps! 

[Tim2842955474bi]

Is that still working for you? It didn't help on our end. Logs still stop within a day. We do have a ticket with Adobe, they are working on it.

[Aaron Trice]

This hasn't worked for us. We're still working with support. We found this week that when this issue occurs, the appenders are stopped. They provided an update, new bootstrap file, and a cfm to dump the logging status when the issue comes up. We uploaded logs they requested Friday for them to review. 

[BKBK]

@Ali Sozkesen originally reported this issue for Update 11.  The current version is ColdFusion 2021 Update 16.

So, my suggestion is to update ColdFusion 2021 to the current version. After that, check whether the issue still exists.

[Aaron Trice]

We just updated to HF15 now and the issue is still occurring. Per the support engineers we are working with, this hasn't been addressed in HF16 either. 

[Sireex]

I can confirm this. We are running HF16 in production, and the problem remains.

[BKBK]

Hi @Aaron Trice and @Sireex , thanks for confirming that the issue still exists on updates 15 and 16. That leads me then to believe that the log4j changes in CF2021 Update 11 / CF2023 Update 5 resulted in a problem with logging in ColdFusion.

My suggestion to the Adobe ColdFusion Team would be to investigate why the standard logs ( exception, coldFusion-out, coldfusion-error, etc. ) no longer rotate when the maximum size is attained. For example, we expect that, when coldfusion-out.log reaches the maximum size, ColdFusion will archive it as coldfusion-out.1.log, then automatically create a new coldfusion-out.log file. Apparently, that is no longer happening since CF2021 Update 11 / CF2023 Update 5.

Would it help if, for ColdFusion's standard log files, coldfusion-out, coldfusion-error, etc., settings similar to the following were added to /lib/log4j.properties or to /lib/log4j2.properties?

(*** represents an appropriate name)

appender.***logfile.type = RollingFile
appender.***logfile.name = ***logfile
appender.***logfile.fileName = ${basepath}/***.log
appender.***logfile.filePattern = ${basepath}/***.%i.log
appender.***logfile.createOnDemand = true
appender.***logfile.layout.type = PatternLayout
appender.***logfile.layout.pattern = %d{MM/dd HH:mm:ss} [%t] *** %-5p - %m%n%n
appender.***logfile.policies.type = Policies
appender.***logfile.policies.size.type = SizeBasedTriggeringPolicy
appender.***logfile.policies.size.size=5000KB
appender.***logfile.strategy.type = DefaultRolloverStrategy
appender.***logfile.strategy.max = 5

[default8vn6shtsmayn]

This is not exactly what we see.
At startup, we see the coldfusion-out.log file created. When it fills up, it is renamed coldfusion-out.1.log and a new coldfusion-out file is created. So the first log rotation seems to happen. But the new coldfusion-out.log file is empty, and remains so until the server is restarted.

We are running CF2021 Update 16 on a non-CF Hardening guide server, Windows Server 2019 Version 1809.
I'll have to check the JRE version we are using.

[BKBK]

Thanks for clarifying that, @default8vn6shtsmayn . As a result, I would modify my suggestion to the Adobe ColdFusion Team.

I would suggest to the team to investigate why, after log rotation, ColdFusion ceases to write  to the newly created log files, until ColdFusion is restarted. A hypothesis: it could be that, after coldfusion-out.log fills up and is renamed to coldfusion-out.1.log and after the new coldfusion-out.log file is created, the outputstream to coldfusion-out.log is not triggered.

I suspect that this has to do with the recent changes in log4j. So I would suggest to the ColadFusion Team to review the settings within /lib/log4j.properties,  /lib/log4j2.properties and /lib/logging.properties, as well as the required JVM flags.

[Tim2842955474bi]

Just wanted to note here that all this time later, we are still having this problem. Adobe has been looking into it but to no avail. Clean installs of CF 2021 have the same issue. Very frustrating. Our CF2023 test servers are still logging so far, I'm considering just migrating over to that, I don't have confidence at this point in any resolution for CF2021. Hopefully CF2023 doesn't have the same problem.

[Aaron Trice]

We too are still seeing this issue. We were just discussing internally doing a clean install since we've made no progress with support. Out of curoistiy, did you perform a clean windows install as well? Or just CF? This has been going on for over a year for us.