UNIX for Dummies Questions & Answers

If you want to process REs for characters that are not in the portable character set defined by the C and POSIX standards, you have to be sure that the characters you're looking for exist in your current locale. The character '¿' is not in the portable character set defined by C and POSIX. You probably want a Spanish or Portuguese locale if that character is present in your data. You haven't said anything about what system you're using, but I probably can't help here. You need to talk to your system administrator to find out what locales have been installed on your system that support the languages and character sets you're using in your data. (Note that there is nothing wrong with the awk script, the problem is the environment in which it is being run.) To find out what locale you're using, run the command:locale. If the setting of LC_COLLATE reported is something like LC_COLLATE="en_US.UTF-8", '¿' is probably not a collating element in your current locale.

Note also that I am not offering to localize the error messages from this awk script for you to produce diagnostic messages in anything other than English.

Thanks Don!

My system Locale :

Code:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Is there a way to override locage LC_COLLATE in the script/my session only to support all international languages as my source data contains multiple languages?

Appreciate all your help.
Thanks,

There is no locale that applies to local conventions around the world (because by definition local conventions such as the spelling of calendar months varies from location to location), but I suppose it is theoretically possible to have an incomplete locale that would have character type and collation tables that contained all of the characters in UTF-8. Assuming that your system administrator had created and installed such an incomplete locale on your system as utf8.UTF-8, you could then invoke your script by using the command:

Code:

LANG=en_US.UTF-8 LC_COLLATE=utf8.UTF-8 LC_CtYPE=utf8.UTF-8 substitute [options] input_data_file...

instead of the command:

Code:

substitute [options] input_data_file...

Note that the setting of LC_COLLATE will affect everything in your script that compares characters such as the sort utility (other than just testing that two strings are the same or that two strings are different) and LC_CTYPE will affect everything in your script that tries to determine whether a character is in a certain class (such as uppercase letter, lowercase letter, alphanumeric, ...) so setting up an LC_CTYPE category for this locale to work correctly when used with other categories in other locales might not be a trivial task.

Good luck,
Don

Don,

Thanks for your inputs.
One help on 'Equals' rule, it should replace the value only when the source field value is matching with the rules field value. Sorry, earlier I gave wrong output example for Equals.

Input:

Code:

field1|field2|field3
rc11|rc12|rec1
rc$$$21|rc#21|rec2
x|y|rec3
xx11|yy11|rec$4
rc41|r!42|rec5

rules:

Code:

field|search|condition|replace
field1|x|Equals|rc
field1|$|Contains|
field2|#|Contains|
field2|!|Contains|c
field2|y|Equals|rc
fieldx|{{|Equals|

Output:

Code:

field1|field2|field3
rc11|rc12|rec1
rc21|rc21|rec2
rc|rc|rec3
xx11|yy11|rec$4        --> not altered due to equality rules.
rc41|rc42|rec5

Could you please suggest the changes for equals condition to replace whole field value when matched.

Thank you!

You are a little vague on what is supposed to happen. If what you mean is that a rule with a "condition" field value that is "Equals" should only be applied when the "search" field value matches the entire contents of the field specified by the "field" field value then change the following line in the script I gave you:

Code:

  ruleC[rc] = $3  # Save condition (even though it is not used

to:

Code:

  if($3 == "Equals") ruleS[rc] = "^" ruleS[rc] "$"

Don,
Appreciate all your help!
Is there a way to improve performance of this script? Currently it's taking 1hr to process 200,000 source records with around 1000 records in rules file.

Thank you!

There are lots of ways to improve the performance of this script. Most of them involve reducing the tremendous flexibility that this script provides.