Character entity usage

[jibinanto40792294]

Hello,

Using coldfusion 2016. have some queries related to character entity values.
Tried html equivalent character entity for replacing trademark and apostrphe special characters. Function used is given below.

private any function specialCharUpdate(strData){
// Ampersand
arguments.strData = replacenocase(strData, '&', '&##38;', 'All');
// ™ trademark
// arguments.strData = replace(strData, '™', '&##8482;', 'All');
arguments.strData = replacenocase(strData, '™', ' &##153;', 'All');
// Apostrophe
arguments.strData = replacenocase(strData, '''', '&##39;', 'All');
arguments.strData = replacenocase(strData, '’', '&##39;', 'All');
// Circle
arguments.strData = replacenocase(strData, '®', '&##174;', 'All');
// Copyright
arguments.strData = replacenocase(strData, '©', '&##169;', 'All');
return arguments.strData;
}

1) The apostrphe from keyboard ' is getting replaced with '
But another kind of apostrphe ’ is not getting replaced with either ' or «This kind of '’' weird character is received

2) Trade mark symbol ™ is not getting updated with ™ or ™Instead like this 'â„¢'

Q:Can anybody suggest the right way to replace this?

3) Data insertion happens as ? on production environment where as in development environment it gets inserted additional symbol along with question mark symbol.
Both are using same codeset and sam cf version.
Q: What are the chance for different behaviour?

3) is it possible to give array as paramter for replacing
eg: array first parameter contain special data($search) and second array parameter contain code to replace($replace) like in php
str_replace($search, $replace, $description);

[BKBK]

Two things:

1.  Make sure your ColdFusion server has Unicode encoding.
   To do so, open \bin\jvm.config in a text editor.(Back it up first).
   Does its java.args property include the flag -Dfile.encoding=UTF-8 ?
   If yes, then OK.
   If no, then include the flag, save the file and restart ColdFusion.
   
   
2.  You can find the character entities you need at https://www.toptal.com/designers/htmlarrows/symbols/.
   Use the search function on the site. What you will be looking for are the HTML Code and HTML Entity.

I ran the following test using the values from the web site:

<cfset testString = "Curly apostrophes: curling left &##8216; and curling right &##8217; <br>
Apostrophe: &##39; <br>
Circle: &##9900; <br>
Trademark: &trade; (from HTML Entity) or &##8482; (from HTML Code)<br>
Registered Trademark:  &reg; (from HTML Entity) or &##174; (from HTML Code) <br>
Copyright: &copy; (from HTML Entity) or &##169; (from HTML Code)">

<cfoutput>#testString#</cfoutput>

The result was:

[BKBK · Accepted solution]

Did you follow the suggestions? Did that help?

[jibinanto40792294]

Yes, you are correct

Adding jvm settings and restarting the cf server worked.
Reading a txt file containing around 1,60,000 records and it contain less than 1000 records with special characters.
When we read the big size file, the special character is not getting replaced .
It is working fine with small small size file with few records.

[BKBK]

I am glad to hear that it worked. 

File-size shouldn't matter. If the replacement worked on a small file, it should also work on a large file. In any case, assuming normal conditions.

How large is the largest file (in MB)? Could you share your JVM settings with the forum?

[jibinanto40792294]

The txt file with special character is around size 120 MB. JVM information below(Using Coldfusion 16)

-server -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:MaxMetaspaceSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file={application.home}/lib/logging.properties  -Dfile.encoding=UTF-8

Was looking earlier for a fix without making any server changes since cf need to restart on live after the jvm setting modification.Seems, permanent fix is to update the settings itself. Thanks a lot for the guidance.

It is taken from its vendor ftp location which contain product related information. Product title and description is having special character data.

The functionality doing is reading it line by line ,checking for any special character data ,replacing it with html character entity value, and updating it to database to cleanup the existing data.

Just tried by copying it fully to another text file manually using sublime editor and saving in a new txt file. The new file seems to be working fine.

Checked both original and new file created properties, both are in .txt format, no idea why does the original file is not saving the special characterequivalent value to db.

Another difference noticed is when we run with original file without making jvm settings unicode modification,the character saved as '?' to database. On the other hand, if we create a small sample .txt file with special character and run, it save some symbols along with '?'.
Guess the original file was created in any special manner.

[BKBK]

Thanks for the update.  What are your settings for Xmx and Xms? (Which you can easily verify on the Java page in the ColdFusion Administrator.)

Suggestion: delete the flag 

-XX:MaxMetaspaceSize=192m 

therebyletting ColdFusion itself decide which value to use. Remember to restart ColdFusion after that.

[jibinanto40792294]

The Minimum JVM Heap Size is 256 MB and Maximum JVM Heap Size is 2048 MB.

The default figures were modified when some metaspace related error was occurred before.

[BKBK]

Thanks for the information.

Besides removing the MaxMetaspaceSize flag, I would suggest setting the Minimum JVM Heap Size to the same value as the Maximum JVM Heap Size.

That is, 

Minimum JVM Heap Size = 2048 MB

Maximum JVM Heap Size = 2048 MB, 

Example reference: https://developer.jboss.org/thread/149559