Display accented characters MX7

Report · May 22, 2007

I'm in the process of transferring a site from CF5 to MX7 (running on Solaris 10) and have hit a problem with the display of accented characters, which appear as little boxes.

Have tried
<CFCONTENT type="text/html; charset=utf-8">
<CFPROCESSINGDIRECTIVE pageencoding="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

All made no difference.

I thought this may be to do with retrieval of data form the database (Oracle 8.1.7), however the boxes also appear from hardcoded text within the template e.g.

<CFIF session.lang is 'ENG'>
<A HREF="frame.cfm?lang=FRE" target="_top">Français</A><BR>
<A HREF="frame.cfm?lang=SPA" target="_top">Español</A><BR>

a square box appears instead of the ç in Français and instead of the ñ in Español

Any assistance would be extremely welcome !

Report · May 22, 2007

Question: are you storing the characters in the DB as html entities or the actual characters?
http://www.w3schools.com/tags/ref_entities.asp

The url above opens the HTML entities list (it's for the ISO 8859-1 character set) but, if you scroll down, you will see the entries for the n/tilde and c/cedilla:
ñ small n, tilde ñ ñ
ç small c, cedilla ç ç

This might not get you where you need to be but I wanted to understand a little more about how you were storing the character data in the DB itself.

Cheers,
Craig

Report · May 22, 2007

> I thought this may be to do with retrieval of data form the database (Oracle
> 8.1.7), however the boxes also appear from hardcoded text within the template
> e.g.

If you have the characters hard-coded in the template, then all you
definitely need the <cfprocessingdirective> tag on the template too (on the
actual template itself, not in Application.cfm or another "upstream"
template, it needs to be in every template that has UTF-8 data in it). It
also needs to be within the first 10 (?) lines of code on the template, by
some account that I read somewhere.

However that tag has no impact other than the text in the template file
itself (ie: it's not part of the solution for DB-oriented issues).

If the stuff is coming from the DB, all you should need to do is to tell
the browser that it's UTF-8, as per the <meta /> tag you mention.

Are you sure the data in the DB *is* UTF-8, and not some other encoding
scheme? How is the stuff getting into the DB?

Notes:
- If you are passing UTF-8 stuff around on the URL or via forms, then you
need to use setEncoding() to make that work.
- And if you're writing to file (with <CFFILE>), you need to use the
CHARSET attribute of that.

--
Adam

Report · May 22, 2007

Grynch108 wrote:
> I'm in the process of transferring a site from CF5 to MX7 (running on Solaris
> 10) and have hit a problem with the display of accented characters, which
> appear as little boxes.

cf5 didn't know encoding from a hole in the ground. cf6/7 is all about unicode
(utf-8). since it's boxes rather than question marks it appears to be a simple
case of the wrong encoding (question marks would mean your data's garbaged).

what encoding did you use to create the cf pages? what encoding do you want to
use for your cf pages? if you *never* intend on using other langauges (CJK,
arabic, etc) then you might consider simply setting the server to iso-8859-1,
find & edit the cf_install_root/lib/neo-runtime.xml file changing UTF-8 in the
following xml var to ISO-8859-1 or whatever encoding you intend to use.

<var name='defaultCharset'><string>UTF-8</string></var>

Report · May 23, 2007

Changing the neo-runtime.xml defaultCharset to ISO-8859-1 has fixed the problem with regards hardcoded text. However the data retrieved from the database stiil reamins, though instead of boxes, all accented characters are now y's .

Found a technote on the adobe site (tn_19217) as follows:

After upgrading to ColdFusion MX, which uses JDBC drivers for database access, ordinary 8-bit CHARACTER data is converted per the (database) default character set and returned to ColdFusion in Unicode. This conversion can corrupt double byte data, causing incorrect query results (including non-printable characters, etc.). This data corruption is caused by ColdFusion 5 storing the data initially in a non-standard way. This problem does not occur if the data is both inserted and retrieved with ColdFusion MX.

Macromedia is aware of this problem, and has logged issue 54399 against ColdFusion 5 for this behavior.

However this was last udated 2 years ago.

Their workaround is to retrieve the data from the database using a ColdFusion 5 installation, and reinsert it using ColdFusion MX. This reinsertion of the data with ColdFusion MX will cause the data to be stored in standard Unicode format, so that no further conversion should be necessary. This is no good to me as i will still have two sites usnig CF5 to access and maintain the same database.

I have started looking into Connection string settings for the JDBC but am unsure about this area.

Have tried setting JVM argument -Dfile.encoding=ISO-8859-1 but made no difference