Programming

Hi All,

As Neo says I have been spending a bit of time on this migration integrity issue.

The irritating "Thingy" (white diamond with question mark in the middle) is officially the Unicode symbol called "Replacement character". The character set inserts this as a placeholder for a character that it doesn't understand. IMHO, the issue here is simply that the migration script (or whatever process) SHOULD understand all the characters on our old site. Yes, we already have "Replacement characters" on the old site switch probably emanated from a long ago upgrade from ascii to Unicode, or from Unicode version x to Unicode version y. As Neo says, replacement character symbols in our old site must be ignored because there's nothing we can do about them now apart from manually edit them out as time goes on.

However, I believe that the currently used (Discourse provided??) process is stuffed because it doesn't understand some of the perfectly correct text on our old site. It even screws up a thread title on the old site containing the replacement character symbol - look at this......

Post migration
How to grep i?1/2 symbol? - Shell Programming and Scripting - UNIX.COM Community

Pre migration
How to grep � symbol?

So the process doesn't even understand it's own Unicode character set!!!!

So FWIW, I've come to the conclusion that trying to modify our old dB is futile as the process will probably find something else to screw up.

Indeed, if you follow the first link I posted on this thread further back, others are having the same issue.

That's my update thus far. I'll report back again as my investigation continues.

EDIT: Replacement character symbol is U+FFFD

What are you talking about?

The migrated versus is the same as the original version.

That is exactly how it should be.

This has nothing at all to do with migration, discourse or translating encoding.

Migrating a post where the encoding has already been replaced in the original DB as "a question replacement char" will keep the same "question replacement char" .

Indeed, are you working too hard?

These example posts you just posted seem "perfect" to me. The migration is just like the original. The original has already replaced unknown encoding (you do not know if the original encoding was unicode or not, that information is gone from existence in the original post and has been replaced with the "wft" encoding symbol.

Whoa, hang on a minute........

I'm talking about the thread title. Is your migrated thread title reading correctly? If so, that's news.

My thread title shown is screwed.

Yes, the thread title in the migrated post is different because there is no migration script running against titles.

That's not a big deal, it's a total outlier.

We are not processing "thread titles" in the migration (on migration scripts do) because the migration scripts are not designed to process any titles at all.

All processing is for the posts only.

Let's don't go chasing down outlier rabbit holes where we are not even concerned about.

It takes less time to edit an outlier like that than to discuss it, LOL.

Obviously, is a thread title has a char which is not encoded properly, then it will have issues when migrated. That is like some 0.00001 kinda outlier.

It's good you found it so we can edit it later; but it's nothing to be concerned about.

So, to be clear... the migration script does not do any processing on titles. Titles are expected to be written in "basic normal charsets" and when they are not, it is quite the outlier case.

And look at this.............

Old site
find . -name �*.*� | xargs grep "help"

New site
find . -name “*.*” | xargs grep "help" - UNIX for Dummies Questions & Answers - UNIX.COM Community

The migration has taken out the replacement character placeholders and put in double quotes which, I wouldn't mind betting, is what was originally there.

What? Anybody got a good working crystal ball?

Let me check the original DB and post back..

Hold on.

Yes, sure, I need time to think about this too. A lot of head scratching coming on.