Visit Our UNIX and Linux User Community

Find records with specific characters in 2 nd field

Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Find records with specific characters in 2 nd field
# 1  
Old 08-30-2018
Find records with specific characters in 2 nd field

Hi ,
I have a requirement to read a file ( 5 fields , ~ delimited) and find the records which contain anything other than Alphabets, Numbers , comma ,space and dot . ie a-z and A-Z and 0-9 and . and " " and , in 2nd field. Once I do that i would want the result to have field1|<flag>

flag can be Y or N .

N - If 2nd field doesnt have anything other above mentioned characters.
Else Y .

I am able to achieve this using below code by reading line by line . Please note second field is "address".

 rm -f ca_sc_flag.txt 
while read rec

cust_id=`echo $rec | cut -d'~' -f1`
addr=`echo $rec | cut -d'~' -f2`

addr_rem=`echo ${addr}|tr -d 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789,. '`

if [ -z "${addr_rem}" ]; then
    echo "$cust_id|$sc" >> ca_sc_flag.txt   

     echo "$cust_id|$sc" >> ca_sc_flag.txt   

done < ca.txt

The issue is it is very ineffective and takes almost 30 mins for 100K records. Can I improve it by using better logic. May be by avoiding reading line by line.

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 08-30-2018 at 06:40 PM.. Reason: Changed HTML to CODE tags.
# 2  
Old 08-30-2018
Don't use shell loops to process that large text files. Try text tools like perl, awk, or other. awk example (untested, as I am on a (yecc) windows laptop):

awk -F"~" '
{print $1 "|" ($2 ~ /[^A-Za-z0-9., ]/)?"Y":"N"
' ca.txt

Last edited by RudiC; 08-31-2018 at 05:16 AM..
This User Gave Thanks to RudiC For This Post:
# 3  
Old 08-31-2018
Thanks for the input Rudy.
It did point in the right direction. Not sure why the ?Yes:No didnt work. I changed the code to use if else loop and it worked.

awk -F"|" '{ if ($2 ~ /[^0-9a-zA-Z,. ]/) print $0,"|Has other chars";else print $0,"|Only Valid Chars"}' test.txt

Previous Thread | Next Thread
Test Your Knowledge in Computers #208
Difficulty: Medium
Open Shortest Path First (OSPF) was designed as an exterior gateway protocol (EGP) for use in an autonomous systems such as a local area network (LAN).
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Calling specific characters from a find variable

I'm trying to do something like this: find . -name blablabla -exec ln -s ./"{:53:14} blablabla" \; The idea is find blablabla and create a symbolic link to it using part of it's path and then it's name, "blablabla." I just don't know if I can call characters out of a find variable. ... (16 Replies)
Discussion started by: scribling
16 Replies

2. Shell Programming and Scripting

Find the difference in specific field

Hi All, Seeking for your assistance to get the difference of field1 and field2 and output all the records if there's a difference. please see below scenario. file1.txt 250|UPTREND FASHION DESIGN,CORP.|2016-04-04 09:36:13.991257 74|MAINSTREAM BUSINESS INC.|2016-04-04 09:36:13.991257... (1 Reply)
Discussion started by: znesotomayor
1 Replies

3. UNIX for Dummies Questions & Answers

How to replace and remove few junk characters from a specific field?

I would like to remove all characters starting with "%" and ending with ")" in the 4th field - please help!! 1412007819.864 /device/services/heartbeatxx 204 0.547%!i(int=0) 0.434 0.112 1412007819.866 /device/services/heartbeatxx 204 0.547%!i(int=1) 0.423 0.123... (10 Replies)
Discussion started by: snemuk14
10 Replies

4. Shell Programming and Scripting

Find and replace with 0 for characters in a specific position

Need command for position based replace: I need a command to replace with 0 for characters in the positions 11 to 20 to all the lines starts with 6 in a file. For example the file ABC.txt has: abcdefghijklmnopqrstuvwxyz 6abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz... (4 Replies)
Discussion started by: thangabalu
4 Replies

5. Shell Programming and Scripting

Can't figure out how to find specific characters in specific columns

I am trying to find a specific set of characters in a long file. I only want to find the characters in column 265 for 4 bytes. Is there a search for that? I tried cut but couldn't get it to work. Ex. I want to find '9999' in column 265 for 4 bytes. If it is in there, I want it to print... (12 Replies)
Discussion started by: Drenhead
12 Replies

6. Shell Programming and Scripting

[Solved] Counting specific characters within each field

Hello, I have a file like following: ALB_13554 1 1 1 ALB_13554 1 2 1 ALB_18544 2 0 2 ALB_18544 1 0 1 This is a sample of my file, my real file has 441845 number of fields. What I want to do is to calculate the number of 1 and 2 in each column using AWK, so, the output file looks like... (5 Replies)
Discussion started by: Homa
5 Replies

7. UNIX for Dummies Questions & Answers

Grep specific records from a file of records that are separated by an empty line

Hi everyone. I am a newbie to Linux stuff. I have this kind of problem which couldn't solve alone. I have a text file with records separated by empty lines like this: ID: 20 Name: X Age: 19 ID: 21 Name: Z ID: 22 Email: Name: Y Age: 19 I want to grep records that... (4 Replies)
Discussion started by: Atrisa
4 Replies

8. Shell Programming and Scripting

[Solved] Find Specific records from file and add totals into variables

Hi Eveyone, I am working on one shell script to find the specific records from data file and add the totals into variables and print them. you can find the sample data file below for more clarification. Sample Data File: PXSTYL00__20090803USA CHCART00__20090803IND... (7 Replies)
Discussion started by: veeru
7 Replies

9. Shell Programming and Scripting

count characters in specific records

I have a text file which represents a http flow like this: HTTP/1.1 200 OK Date: Fri, 23 Jan 2009 17:16:24 GMT Server: Apache Last-Modified: Fri, 23 Jan 2009 17:08:03 GMT Accept-Ranges: bytes Cache-Control: max-age=540 Expires: Fri, 23 Jan 2009 17:21:31 GMT Vary: Accept-Encoding ... (1 Reply)
Discussion started by: littleboyblu
1 Replies

10. HP-UX

extract field of characters after a specific pattern - using UNIX shell script

Hello, Below is my input file's content ( in HP-UX platform ): ABCD120672-B21 1 ABCD142257-002 1 ABCD142257-003 1 ABCD142257-006 1 From the above, I just want to get the field of 13 characters that comes after 'ABCD' i.e '120672-B21'... . Could... (2 Replies)
Discussion started by: jansat
2 Replies

Featured Tech Videos