Programming

04-28-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

FYI existing old mysql dB

Code:

mysql> SELECT count(postid)  from post where pagetext like '%â€œ%';
+---------------+
| count(postid) |
+---------------+
|            66 |
+---------------+
1 row in set (1.64 sec)

mysql> SELECT count(postid)  from post where pagetext like  '%â€ %';
+---------------+
| count(postid) |
+---------------+
|            14 |
+---------------+
1 row in set (1.68 sec)


mysql> SELECT count(postid)  from post where pagetext like '%â€�%';                                                                                                                    
+---------------+
| count(postid) |
+---------------+
|            45 |
+---------------+
1 row in set (1.66 sec)

mysql> SELECT count(postid)  from post where pagetext like '%â€™%';                                                                                                                    
+---------------+
| count(postid) |
+---------------+
|           165 |
+---------------+
1 row in set (1.63 sec)

mysql> SELECT count(postid)  from post where pagetext like '%â€˜%';                                                                                                                    
+---------------+
| count(postid) |
+---------------+
|            38 |
+---------------+
1 row in set (1.69 sec)


mysql> SELECT count(postid)  from post where pagetext like '%â€¢%';
+---------------+
| count(postid) |
+---------------+
|             4 |
+---------------+
1 row in set (1.70 sec)

mysql> SELECT count(postid)  from post where pagetext like '%â€¦%';
+---------------+
| count(postid) |
+---------------+
|            23 |
+---------------+
1 row in set (1.68 sec)

Now that SELECT shows some goodies, maybe UPDATE on main DB ?

Neo

04-28-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Like this:

Issue with Keyboard or Char Encoding During Migration-screen-shot-2020-04-28-124301-pmjpg

Neo

04-28-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

This spam page (on new site) was full of those "odd diamond things"...

Issue with Keyboard or Char Encoding During Migration-screen-shot-2020-04-28-15521-pmjpg

Code:

https://community.unix.com/t/infraction-for-solosx-spammed-advertisements/271573

... but I got rid of them on old site:

Issue with Keyboard or Char Encoding During Migration-screen-shot-2020-04-28-15705-pmjpg

Code:

https://www.unix.com/user-infractions/142431-infraction-solosx-spammed-advertisements.html

Using this mysql code in old forums against mysql DB (for some reason, had to repeat, but that is OK):

Issue with Keyboard or Char Encoding During Migration-screen-shot-2020-04-28-15933-pmjpg

Will run against staging mysql soon....

This User Gave Thanks to Neo For This Post:

Neo

View Public Profile for hicksd8

04-28-2020

Moderator

2,327, 710

Join Date: Feb 2012

Last Activity: 3 May 2020, 3:12 AM EDT

Location: Devon, UK

Posts: 2,327

Thanks Given: 442

Thanked 710 Times in 578 Posts

Hmmmmmmmmmmmmmm.......

Why do i keep getting apostrophe's converted to ? when i import - beets

Unicode apostrophe standardization - Style - MetaBrainz Community Discourse

Last edited by hicksd8; 04-28-2020 at 04:23 AM..

hicksd8

Find all posts by hicksd8

04-28-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Quote:

Originally Posted by hicksd8

Hmmmmmmmmmmmmmm.......

Why do i keep getting apostrophe's converted to ? when i import - beets

Unicode apostrophe standardization - Style - MetaBrainz Community Discourse

This is all good... but I want to focus bottom up... only on the specific chars causing a problem in our DB.

That is what I am doing now ... finding the exact offending char and then finding the correct transform to cleanse it.

Please hold off on posting links unless the link contain solutions for the exact char we are having issues with (let's stick to bottoms up approach, not top down, for now).

I have all the chars we have found so far covered, so hold off (on these funny chars) until I get the various staging DBs synced.

Thanks.

Right now I have all the transforms I need based on what we have found so far. We can search for more in the next round. In other words, I know what the problem is. What we need is to find them and then fix them, from a bottoms up approach because I am not going to run any code which "transforms" problems we have not identified and tested. I do not want unintended consciences of running code and others transforms unless they solve a specific, clearly identified issue.

Will update soon.

Neo