zero-day exploit affecting the popular Apache Log4j utility (CVE-2021-44228)

[Jdsplicer]

Does anyone know if the zero-day exploit affecting the popular Apache Log4j utility (CVE-2021-44228) that was announced on 12/9/2021 will affect ColdFusion version 10 and 2018?

[Miguel-F]

Never mind my comment. I was confused.

[Jdsplicer]

What if we are only at update 10 on CF2018 and have log4j ver 2.9.0. Will updating to 2.15.0 causes issues without being on update 11 or 12? I know we need to apply the updates and it is in the works, but have to go thru change request and other steps with different support teams before we can just apply updates. TIA

[NicoTexas]

You can do as described earlier and replace 2.9.0 with 2.15.0.
I tested locally on an update 10, and logs seem to work as expected.
Of course, I can not guaratntee this is not going to affect your particular instance.
You will need to test on your own test environments before deploying this fix to production.

[Ripley Casdorph]

So after careful checking I grabbed one of the wrong jar files and that was causing the issue.

The three links listed above all work and will give you correct jars to resolve issue.

Process:

1. Stop CF service

2. Paste in 3 new Jar files to \ColdFusion2018\cfusion\lib

3. Remove 3 old jar files from Lib folder \ColdFusion2018\cfusion\lib

4. Update the log4j.properities files changing m% to %m{nolookups} in the same lib folder

5. Update JVM config to include  -Dlog4j2.formatMsgNoLookups=true in the arguments

6. Start CF and you are good to go!

Thanks all for the help, you are awesome!

[jstratton77]

This is exactly the same situation we are in, we had CF2018 update 10 live.  Will look to apply to update 12, but need to make sure there are no breaking changes first.  But, your use case is exactly the same as ours when we patched on Friday night.

[Jdsplicer]

Thanks NicoTexas and jstratton77. I am in the process of doing this in our DEV environment now and then will have to test some apps and check log files to make sure everything is ok. Good to know someone else already went this route. I assume not issues cropped up on you at this point?

[jstratton77]

Nope, no issues at all.  Today would have been our first heavy day since applying on Friday night.

[T.Brown.10]

FYI this morning I attempted an update from Update 10 to 12 and had a failure which resulted in 403 errors that despite following all the troubleshooting steps online we were unable to recover from and required a restore from a VM backup to get back to level 10 we have since just gone with the library update.

We are on an IIS server with Coldfusion and have run the lock down.  I am not sure if you will have the same issue of course 

[s_sykes]

T.Brown,

Those that ran the lockdown tool appear to have this problem.  Disappointing that Adobe has not fixed it or put this info prominently on the update page.  It's a simple fix if you only know about it!

403 Permission Error with Tomcat

https://community.adobe.com/t5/coldfusion-discussions/coldfusion-2018-update-12-403-forbidden-error/...

Once you apply the hotfix, make sure to re-create the connector. Becase there is a new property added after update 7. In case, it is not matching the secret key, it will throw 403 error. 

If you have already done the connector and still getting the error, then check the server.xml(\ColdFusion2018\{instance}\runtime\conf) and then search for "secret". In the same line if you have both "secret" and "requiredsecret", please remove the requiredsecret entry and restart the server. It will fix the issue.

[T.Brown.10]

Yeah so we tried this and what occured was that instead of getting the 403 error IIS started returning the actual coldfusion code on the page instead of the HTML.   That really scared the person working on the issue with me 🙂

In the end we had a backupfrom 9pm the night before and nothing else had changed on the page to so we went that and will plan the update going forward 

[FishDev]

I have also noticed that there is a 1.2.17 log4j instance in the jetty folder (ColdFusion20XX\cfusion\jetty\lib\ext). Has anyone done anything to mitigate this?

[jeffh65754959]

There is also a log4j-1.2.15.jar in the instance/lib directory that was not deleted once the log4j files went to 2.x in later releases.  I have contacted Adobe asking if upgrading to 2.15.0 we are safe and are not voiding our warranty on the software.  I was told that Adobe security is still analysing the impact and they will make official announcement.

[michaelm87459285]

per apache these versions are not vulnerable: Versions Affected: all log4j-core versions between 2.0-beta9 and 2.14.1

[FishDev]

While that is technically true it may not be the case, and remains unclear as I have not seen any official word on if it is impacted or not:

See this: Log4j – Apache Log4j Security Vulnerabilities

In particular:

Please note that Log4j 1.x has reached end of life and is no longer supported. Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed.

[NicoTexas]

That's a good point. Why is there both versions in the classpath, this is not great and hard to justify.
Would love to understand what is actually used by the the default loggers. Also, why it takes so long to understand the possible impact?

[jeffh65754959]

My concern was that the log4j files include the 1.x file and it was not removed when the hotfix installed the 2.x files.  This is our list of files after HF 10 was applied.  

• log4j-1.2.15
• log4j-api-2.9.0
• log4j-core-2.9.0
• log4j-to-slf4j-2.9.1

[jhansen-cf]

@Ripley Casdorph When it comes to "4. Update the log4j.properities files changing m% to %m{nolookups} in the same lib folder" please elaborate...

In my log4j.properities I see a number of ConversionPattern values like...

%d{MM/dd HH:mm:ss} [%t] %-5p %m%n
This should be updated to read...

%d{MM/dd HH:mm:ss} [%t] %-5p %m{nolookups}%n

Correct?

[Ripley Casdorph]

In our properties file I see 8 places to update %m.

Most now look like this:

 %m{nolookups}%n

or

%m{nolookups}%n%n

and I think the line you are referring to is 58 and it reads.

%d{MM/dd HH:mm:ss} [%t] %-5p %m{nolookups}%n

Can I confirm that it is working? No.  I can confirm that it didn't break anything on our server and appears to be working and we are getting logs. When I had the wrong jar in there the server wouldn't run, so as long as the new jar works, I beleive we are golden!

[JuergenW]

Version 2.16 of log4j has been released:

https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.16.0/log4j-api-2.16.0.jar

https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.16.0/log4j-core-2.16.0.jar

https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-to-slf4j/2.16.0/log4j-to-slf4j-2.16.0....

Changes: https://logging.apache.org/log4j/2.x/changes-report.html#a2.16.0

[jeffh65754959]

Looking at the release notes, it turns off JNDI and requires log4j2.enableJndi to be set to true to allow it and turns off support for message lookups.

[32132132164654654]

Has anybody been able to reproduce a POC of this issue on Coldfusion?

[Ripley Casdorph]

I have seen discussion from several people that they have been unable to prove CF can  actually be compromised with this hack, but that doesn't mean you shouldn't patch it anyway. 

The steps for fix are easy, take about 5 minutes total, and a simple restart of  CF service allows for 40 seconds of down-time.

No reason not to fix for the peace of mind. 

[gualtiero sappa]

Hi all,

we contacted Adobe Product Security and Incident Response and received this reply:

Adobe is investigating potential impact and taking action, including updating affected systems to the latest versions of Apache log4j 2 recommended by the Apache Software Foundation.

The investigation is ongoing but, to date, Adobe has discovered no indication to suggest customer data has been impacted as a result of this issue.

ColdFusion plans to release a patch (version(s) 2021 & 2018) for this log4j vulnerability to customers on 12/17/2021. 

In the meantime, we recommend ColdFusion customers apply the following workarounds/mitigations steps until this patch has been released:

Workaround/Mitigation steps:

CF2021:

CF 2021 ships with log4j 2.13.3 and 1.2 versions. The former is impacted by this vulnerability while the latter is not. 

Steps:

1. Stop the server
2. Navigate to <cf_root>\<Instance_name>\bin directory
3. Open jvm.config file and add -Dlog4j2.formatMsgNoLookups=true argument in java.args section and save
4. If using any third-party libraries that use log4j2, and hence vulnerable, search for log4j-core in <cf_root> directory. If log4j2 version (<= 2.10 and >=2.0-beta9)  is found, remove the JndiLookup class from the classpath like below otherwise skip this step.

• If the operating systems is Windows , then unzip the log4j-core-2.x.jar file and remove the class from path : org/apache/logging/log4j/core/lookup/JndiLookup.class and zip the log4j-core-2.x.jar. X is the version number you found in the folder.
• If the operating systems is non-windows, then remove the JndiLookup class from the classpath : "zip -q -d log4j-core-2.x.jar org/apache/logging/log4j/core/lookup/JndiLookup.class". X is the version number you found in the folder.

5. Start the Instance

6. Repeat this for all the instances.

CF2018:

CF 2018 ships with log4j 2.13.3 and/or 2.9.0, and log4j 1.2. The former is impacted by this vulnerability while the latter (i.e., v1.2) is not impacted. 

Steps:

1. Stop the server
2. Navigate to <cf_root>\<Instance_name>\bin directory
3. Open jvm.config file and add -Dlog4j2.formatMsgNoLookups=true argument in java.args section and save.
4. Navigate to <cf_root >>\<Instance_name>\lib directory. 
5. If you find log4j-core-2.9.0.jar, move the file to a temporary location
6. Copy the patched log4j-core-2.9.0.jar file with JNDILookUp class removed from here
7. If using any third-party libraries that use log4j2, and hence vulnerable, search for log4j-core in <cf_root> directory. If log4j2 version (<= 2.10 and >=2.0-beta9) is found, remove the JndiLookup class from the classpath like below otherwise skip this step

• If the operating systems is Windows , then unzip the log4j-core-2.x.jar file and remove the class from path : org/apache/logging/log4j/core/lookup/JndiLookup.class and zip the log4j-core-2.x.jar. X is the version number you found in the folder.
• If the operating systems is non-windows, then remove the JndiLookup class from the classpath : "zip -q -d log4j-core-2.x.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ". X is the version number you found in the folder.

8. Start the Instance

9. You can now delete log4j-core-2.9.0.jar from the temporary location

10. Repeat this for all the instances.

PMT 2021:

PMT 2021 ships with log4j 2.11.1 and log4j 2.3. Both versions are impacted. 

Steps:

1. Stop the PMT and datastore services
2. Navigate to <PMT_Home>\datastore\config directory
3. Open jvm.options file, add -Dlog4j2.formatMsgNoLookups=true argument in #log4j2 section and save.
4. Navigate to <PMT_Home>\lib directory. 
5. Move the file log4j-core-2.3.jar to a temporary location
6. Copy the patched log4j-core-2.3.jar file with JNDILookUp class removed from here
7. Start the datastore and PMT services
8. You can now delete log4j-core-2.3.jar from the temporary location

PMT 2018:

PMT 2018 ships with log4j 2.9.1 and log4j 2.3. Both versions are impacted. 

Steps:

1. Stop the PMT and datastore services
2. Navigate to <PMT_Home>\datastore\lib directory
3. Move the file log4j-core-2.9.1.jar to a temporary location
4. Copy the patched log4j-core-2.9.1.jar file with JNDILookUp class removed from here
5. Navigate to <PMT_Home>\lib directory 
6. Move the file log4j-core-2.3.jar to a temporary location
7. Copy the patched log4j-core-2.3.jar file with JNDILookUp class removed from here
8. Start the datastore and PMT services
9. You can now delete log4j-core-2.3.jar and log4j-core-2.9.1 from the temporary location

 APIM 2021 and APIM 2018:

APIM 2021 and 2018 ship with log4j 2.3. This version is impacted.

Steps:

1. Stop the APIM server (<APIM_Home>\bin) and Analytics (<APIM_Home>database\analytics\bin) service.
2. Navigate to <APIM_Home>\lib directory. 
3. Move the file log4j-core-2.3.jar to a temporary location
4. Copy the patched log4j-core-2.3.jar file with JNDILookUp class removed from here
5. Start the Analytics service and the APIM server.
6. You can now delete log4j-core-2.3.jar from the temporary location

Note – For more information, we recommend users to refer to the below post made by Pete Freitag on the log4j issue:

https://www.petefreitag.com/item/923.cfm

[neochad]

So I am curious if this is functionally any different than doing the 2.15.0 replacement, is there anything about 2.15.0 that could cause issues? I have been running 2.15.0 now for about 12 hours and no issues have occured as far as I can tell.

[T.Brown.10]

My theory (and it's only a theory) that the 2.15.0 update accomplishs the same thing.  However if you are adobe it was quicker to push through a small patch to the exisiting libaray rather then to have to run regressions and unit tests on the new library version.   So there reamins a small risk doing the library update that you may find some issue that wasn't present in the older