Certain glyphs not appearing in browser

[ljensen1]

I have a web-based page that allows a user to paste information into a text field. This data is sent to FileMaker 13 using FX.php.  When returned, so the user can preview what was entered, some of the glyphs appear as empty boxes in Internet Explorer, or number-filled boxes in Firefox.  This includes various quotes, apostrophes, emdashes, etc.  I don't recall this problem before we moved up to FM13.  The data looks fine within FileMaker itself.  The web page uses a font family of Verdana, Arial, Helvetic, san-serif.  I've tried creating the text in a variety of different fonts before pasting into the web page, and have the same result each time.

 

Right now I have a php function that replaces html entity names so the characters will appear correctly, but that isn't a great solution because I'm always finding more problem glyphs.  Does anyone have any suggestions?

 

Below are some the html entities that give me problems:

򢀔  (I replace with —)  //emdash

򢀜  (I replace with “) // left quote
򢀝  (I replace with ”) // right quote
򢀙  (I replace with ’) // apostrophe
򢀓  (I replace with —) //emdash
�  (I replace with —) //emdash
򢀘 (I replace with ’) // apostrophe

[rivet]

I believe you can have an auto calc on the field in FMP to clean up the data on commit  ( after FXphp submission )

http://www.fmdiff.com/fm/cleantext.html?session=1670e9be06a267b98b7f955ccfcd4f39

[ljensen1]

Thank you.  I will try that.  Sorry I didn't get back to this sooner - I get caught up in another project.

[ljensen1]

Unfortunately that did not work.  I still get "funny" characters when viewing the inputted data through the web (Firefox, in this case).  Em-dashes and smart quotes appear as a small box with 6 digits of some sort inside of the box - three on the top and three on the bottom.

[webko]

You have set the encoding of the actual web page and enclosed form to UTF-8, for starters?

 

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>

 

This should prevent the MS (mainly) issues...

 

You can also perform a PHP substitute on the way through:

 

http://alanwhipple.com/2011/06/04/php-clean-encoding-issues-smart-curly-quotes-em-dashes/

[ljensen1]

Yes, that is exactly what I have on every web page:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

 

I tried that function, but still have the same issue. The latest problem, for instance, is a dash showing as the funny box character in the browser, and then showing as "&#663571;” by a pdf created through fdpf.  My solution is to add another substitution to my php function and change "&#663571;” to "&#8212;" before it hits the browser. 

 

This is not a good solution because I only find these problems as users complain.

[ljensen1]

I have further information that may be helpful:

This is the problem scenario:

1) User pastes something into a web form (probably from Word)

2) Web page sends data to FM13 database, using FX.php.

3) Data looks fine in FM13- no problem with emdashes, smart quotes, etc.

4) FM13 database sends data back to web page using FX.php (this is where the problem occurs).

5) On web page, user sees boxes instead of certain characters.

 

However, if I leave FM13 out of the loop and do the following, the web page displays the characters without a problem:

1) User pastes something into a web form (probably from Word).

2) Data is passed to a new web page using $_POST

3) Characters display properly.

 

So the problem seems to lie in how FileMaker is returning these characters to the browser.

[ljensen1]

More information.

One of the characters that comes across as a box is "&#663571;" (at least that is how it is rendered when sent to fpdf).  The user entered this as an emdash (—) in text pasted from Microsoft Word.  When FileMaker sends the data back to the web, the emdash becomes "&#663571;".  The same thing occurs if the Microsoft Word text is pasted directly into the database rather than entered through the web.

 

When I look up "&#663571;" here:  "http://www.fileformat.info/info/unicode/char/a2013/index.htm, this is said to be NOT a valid unicode character.  Why is FileMaker returning data like this?  Or is it php or XML (I'm using FX.php)?

[ljensen1]

I got an answer off the forum and thought I would add it here in case it helps anyone.  I just needed to use the encoding function in FX.php: $find_lang->setCharacterEncoding( 'UTF-8' );  to make sure the data going into and out of FileMaker is UTF-8.  That function wasn't available years ago when I started using FX.php, and I didn't notice it had been added.  Glad it ended up to be something easy.

 

I also had to switch from fpdf to tcpdf so I could use a unicode font and the characters would show up correctly on the pdf's.  I had actually known I would have to do that, but I wanted to get the FileMaker part straightened out first.

[webko]

Nice one... glad you found an answer to this...