Sponsored Content
Top Forums Programming Issue with Keyboard or Char Encoding During Migration Post 303046223 by Neo on Tuesday 28th of April 2020 04:42:58 AM
Old 04-28-2020
While I am doing another test run, let me try to explain this better.

Our DB is nearly 15 years old.

People have copy-and-paste any kinds of encoding into the database. That stuff may have or may not have been transform to the encoding of the DB. In addition, over the years, the coding of the DB has changed. It was not UNICODE in the beginning.

The same is true for keyboards. People type from all kinds of keyboard over the years. Sometimes this adds to the problem of encoding, but generally it is from copy-and-post, from what I have seen. Many people like to write their post on their desktop editor and copy and paste that into the forums.

So, running any generic encoding translation will not work for all encodings. If it was, this problem would have already been solved. Sometimes UNICODE does not work because there are encoded chars with are not part of UNICODE.

It's not a theory. It is a fact of years of having a busy forum with people all over the world copy-and-pasting their locally encoded text into our DB. Sometime we get lucky and the encoding works.

All we can do, is identify it and squash it, or ignore it.

It's not critical either, because I can fix it after migration directly in the DB, as I have been doing today. But the best place to fix it is in the legacy mysql DB when possible but it is also doable if information was not lost in migration from mysql to postgres to do it in postgres.

This is why I am kinda begging everyone to help test. I can write the code to fix the issue if I clearly see the issues. There are one million posts. The more people take a look, the more it helps.

Sorry to be begging... LOL. I have been working on this for months. My wife is starting to feel like she has no husband; which I can understand why.

But I wanted everyone to understand why I have asked for this help.

This is exactly what I need..... (image from first post on this test)


Issue with Keyboard or Char Encoding During Migration-screen-shot-2020-04-28-31655-pmjpg


-------------------------

Honestly, so far people have provided me a total of about 3 or 4 links only where this encoding issue comes up and most of those are in non-public spam archives.

I don't want to be spending my time chasing outliers in two decades of encoding. Either there are issues or not. I am not going to spend my entire life working on chasing unimportant encoding issues to try to make a migration which s 99.99% perfect to 99.9999% perfect. It's not a good use of our time.

So, please provide details accounts of any remain encoding issues with links to the original and the migrated version.

Thanks.
 

7 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how2 get single char from keyboard w/o enter

I am writing a bash shell menu and would like to get a char immediately after a key is pressed. This script does not work but should give you an idea of what I am trying to do.... Thanks for the help #! /bin/bash ANSWER="" echo -en "Choose item...\n" until do $ANSWER = $STDIN ... (2 Replies)
Discussion started by: jwzumwalt
2 Replies

2. Shell Programming and Scripting

Encoding of a text issue

I created one file on windows system and is visible as : TestTable,INSERT,večilnin1ईगल受害者是第,2010-02-02 10:10:10.612447,137277,ईगल受害者是第večilnin!@#$%^&*()_+=-{}] But when send this file to unix system, the file is visible as : TestTable,INSERT,žvečilnin1ई-ल -害...是第,2010-02-02 ... (4 Replies)
Discussion started by: Shaishav Shah
4 Replies

3. Solaris

Solaris 10 p2v migration issue

Hi All, We need to move Physical Solaris 10 system to Virtual Solaris 10(p2v). Both the servers having Solaris 10(Generic_147440-25) means physical server which we are going to move is having Solaris 10 and this physical server will be converted as a virtualserver on another physical server... (9 Replies)
Discussion started by: sb200
9 Replies

4. UNIX for Dummies Questions & Answers

Strange Keyboard and Mouse Issue

Hello All, PC: CuBox-i (*i.MX6) Mini-PC OS: openSUSE 13.1 (Bottle) (armv7hl) Kernel: 3.14.14-cubox-i # uname -a Linux CuBox-HQ 3.14.14-cubox-i #1 SMP Sat Sep 13 03:48:24 UTC 2014 armv7l armv7l armv7l GNU/LinuxSo I've been having this random issue happen on this PC where a few strange... (12 Replies)
Discussion started by: mrm5102
12 Replies

5. AIX

AIX Migration issue with EMC ODM sets

Hi Experts , I want to start migrating our AIX 6.1 to AIX 7.1 . I am planning to use alt_disk_migration . Chris gibson has awesome documentation in the internet. However I am running into an issue with EMC odm filesets . So my current OS is AIX 6.1. and I have this : lslpp -l | grep EMC ... (7 Replies)
Discussion started by: JME2015
7 Replies

6. Shell Programming and Scripting

AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue

Greetings Experts, We are migrating from AIX to RHEL Linux. I have created a script to verify and report the NULLs and SPACEs in the key columns and duplicates on key combination of "|" delimited set of big files. Following is the code that was successfully running in AIX. awk -F "|" 'BEGIN {... (5 Replies)
Discussion started by: chill3chee
5 Replies

7. Solaris

View file encoding then change encoding.

Hi all!! Im using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . Im expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies
quota(2)							System Calls Manual							  quota(2)

Name
       quota - manipulate disk quotas

Syntax
       #include <sys/types.h>
       #include <sys/param.h>
       #include <sys/quota.h>

       quota(cmd, uid, arg, addr)
       int cmd, uid, arg;
       caddr_t addr;

Description
       The  call manipulates disk quotas for file systems that have had quotas enabled with The cmd parameter indicates a command in the following
       list that is applied to the user ID uid.  The arg parameter is a command specific argument and addr is the address of an optional,  command
       specific  data  structure,  which  is copied in or out of the system.  The interpretation of arg and addr is given with each command in the
       list that follows:

       Q_SETDLIM
	      Set disk quota limits and current usage for the user with ID uid.  The arg parameter is a major-minor device indicating a particular
	      file  system.   The addr parameter is a pointer to a struct dqblk structure, defined in <sys/quota.h>.  Only the superuser can issue
	      this call.

       Q_GETDLIM
	      Get disk quota limits and current use for the user with ID uid.  The remaining parameters are identical  to  the	Q_SETDLIM  command
	      parameters.

       Q_SETDUSE
	      Set  disk  use limits for the user with ID uid.  The arg parameter is a major-minor device indicating a particular file system.  The
	      addr is a pointer to a struct dqusage structure, defined in <sys/quota.h>.  Only the superuser can issue this call.

       Q_SYNC Update the on-disk copy of quota uses.  The uid, arg, and addr parameters are ignored.

       Q_SETUID
	      Change the calling process's quota limits to those of the user with ID uid.  The arg and addr  parameters  are  ignored.	 Only  the
	      superuser can issue this call.

       Q_SETWARN
	      Alter  the disk usage warning limits for the user with ID uid.  The arg is a major-minor device indicating a particular file system.
	      The addr parameter is a pointer to a struct dqwarn structure, which is defined in <sys/quota.h>.	Only the superuse can  issue  this
	      call.

       Q_DOWARN
	      Warn  the user with user ID uid about excessive disk use.  This call causes the system to check its current disk use information and
	      print a message on the terminal of the caller for each file system on which the user is over quota.  If the arg parameter is  speci-
	      fied  as	NODEV,	all  file systems that have disk quotas are checked.  Otherwise, arg indicates a specific major-minor device to be
	      checked.	Only the superuser can issue this call.

Return Values
       A successful call returns 0 and, possibly, more information specific to the command specified in the cmd parameter; when an  error  occurs,
       the value -1 is returned and the global variable errno is set to indicate the reason.

Diagnostics
       A quota call fails when one of the following occurs:

       [EINVAL]       The kernel has not been compiled with the QUOTA option.

       [EINVAL]       The cmd parameter is invalid.

       [ESRCH]	      No disk quota is found for the indicated user.

       [EPERM]	      Only the superuser can issue the call and the caller is not the superuser.

       [ENODEV]       The arg parameter is being interpreted as a major-minor device, and it indicates an unmounted file system.

       [EFAULT]       An invalid addr parameter is supplied; the associated structure could not be copied in or out of the kernel.

       [EUSERS]       The quota table is full.

See Also
       "Disk Quotas in a UNIX Environment", ULTRIX Supplementary Documents, Volume 3:System Manager

																	  quota(2)
All times are GMT -4. The time now is 05:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy