Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

To all the awk experts out there!

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting To all the awk experts out there!
# 1  
Old 06-10-2013
To all the awk experts out there!
Hello All,

I recently encountered this difficulty in processing a File.

Input File has millions of records with fields like below
Code:
ID1,ID2,DATE,FLAG,VAL
123,432,0604,1,-0.5
123,432,0604,22,0.5
123,433,0604,1,-0.54
123,433,0604,22,6.77
123,543,0605,22,0.94

To put this simply, I will have to eliminate a couple of 1 and 22 with rest of the fields being the same.

What actually I did was,

Code:
awk -F"," '{count[$0]++} END { for (i in count) print $0, i}' input | awk '$5 < 0 '{$5=$5*-1;} {print $1,$2,$3,$5,$4,$6}'> temp1

==> this gives me lines with their counts
I will sort it and do this
Code:
awk -F"," '{count[$1" "$2" "$3" "$4]++} END { for (i in count) print i}' temp1 > temp2

Then by reading patterns from temp2, I will have to eliminate using another awk statement! I am working on it.

But I feel that this is going to take me too long when file size increases.

Need some help here Smilie
Last edited by Scott; 06-11-2013 at 07:39 AM.. Reason: Please use a descriptive subject title...
# 2  
Old 06-10-2013
Could you show a sample of the output that you would like to achieve?
# 3  
Old 06-10-2013
INPUT:
Code:
ID1,ID2,DATE,FLAG,VAL
123,432,0604,1,-0.5 -->
123,432,0604,22,0.5 --> Same fields except 4th.. so these two will be deleted
123,433,0604,1,-0.54
123,433,0604,22,6.77
123,543,0605,22,0.94

OUTPUT:
Code:
123,433,0604,1,-0.54
123,433,0604,22,6.77
123,543,0605,22,0.94

# 4  
Old 06-10-2013
Quote:
Originally Posted by PikK45
INPUT:
Code:
ID1,ID2,DATE,FLAG,VAL
123,432,0604,1,-0.5 -->
123,432,0604,22,0.5 --> Same fields except 4th.. so these two will be deleted
. . .

Sorry - these are not the same: field $5 is positive in one instance and negative in the other. Are they to be added and deleted if 0?
# 5  
Old 06-10-2013
Do you mean this? Make field 5 absolute and print unless fields 1,2,3,5 are duplicate
Code:
awk 'BEGIN {FS=OFS=","} $5<0 {$5=-$5} A[$1 FS $2 FS $3 FS $5]++==0' input

Otherwise please give required output!
# 6  
Old 06-10-2013
All,

Thanks for looking at this!!! Yes below are the conditions
  • Process should take the absolute value of 5th field
  • Value 1 in 4th field will always make the 5th field negative
  • Value 22 in the 4th field always make the 5th field positive
  • Final output should cancel the matching values for 1 and 22 [Rest of the fields to be taken]

Hope this is clear! Smilie
# 7  
Old 06-11-2013
Not clear: does "cancel" mean 1. set field 4 to null, or 2. not display the 2nd matching line, or 3. not display any of the matching lines?
Please give an example with a 10 lines input and expected output!
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Couple of easy questions for experts on awk/sed

Hello Experts.. I have 3-4 C codes with Oracle SQL statements embedded. All the SQL statements starts with EXEC SQL keyword and ends with ;. I want to extract all the SQL statements out of these codes. I did awk '/^EXEC SQL/,/\;/' inputFile (I use this on all of the codes individually). That... (2 Replies)
Discussion started by: juzz4fun
2 Replies

2. Shell Programming and Scripting

awk experts please help

I have a log file of 60 MB with 20k records which contains data like below. this contains some data so removed 2891358271000020, 2012-12-02 23:16:17 , 2012-12-02 23:16:17 , 378015123, 2012-12-02 23:16:19 , 2012-12-02 23:16:19 , (15 Replies)
Discussion started by: mirwasim
15 Replies

3. UNIX for Advanced & Expert Users

Experts!! please help me

Hi experts, Please help me on the below: how to write a shell script to search and delete files on windows server. -script runs on unix box -it should search for specific files on windows server and delete them periodically. (1 Reply)
Discussion started by: chpradeepch
1 Replies

4. Shell Programming and Scripting

awk question for experts

Hi guys I am trying to perform a substitution using 'awk' command, but it fails. I work in ksh. Here is my code: $ line="F 18:30 10 23:00 ts1632back" $ n="ts1632back" $ m="18:45" $ echo ${line} | nawk -v a=$n -v b=$m '{if ($5==a) $2=m; print }' F 10 23:00 ts1632back $It should've... (2 Replies)
Discussion started by: aoussenko
2 Replies

5. UNIX for Advanced & Expert Users

for experts

Hi i'm working with mpi programs every thing ok but i need after i compile the mpi to calc the area of the rectangle for example (not my program) accept two arguments from the command line: mpirun -n 4 myprog 24 100 here 24 and 100 two arguments i'll pass them to the program how can... (1 Reply)
Discussion started by: Scotch
1 Replies

6. Solaris

Experts !!! Please advise

Hi, I work on sun Solaris. Am hosting few web services on my server which are accessed over the internet. Now to check whether the web service is responding or not, i first have to log in to the web service URL. If it doesn't respond there, i come back to my server box and restart the service... (4 Replies)
Discussion started by: sting672744
4 Replies

7. UNIX for Dummies Questions & Answers

Perl Experts - Need your help

Hi All, I am using ingres in perl select count(*) rec from user_tables where table_name = 'abc'; I want to use the alias variable 'rec' and check the value if >0 insert values else create table. How can I do this without using hash variables in perl. Kindly help me in this regard. Much... (1 Reply)
Discussion started by: karthickrn
1 Replies

8. UNIX for Dummies Questions & Answers

Looking for Advice from Experts

Where to start... I am a system administrator who didn't think he would ever be one. My first work was on Window 2000 as a tech(hardware, installs, stuff like that). Then we got Macs (I work in photography and Videography). The I was sent to Mac cert school. Set up a Mac Xserve and about 100... (2 Replies)
Discussion started by: Squidy P
2 Replies

9. Shell Programming and Scripting

Need help - from awk, sed experts

Hi , I have a file as below. Contents of the file are -------------------- aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa bbbbbbbbbbb ccccccccccc ddddddddddd aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa aaaaaaaaaaa bbbbbbbbbbb (4 Replies)
Discussion started by: Srini75
4 Replies
Login or Register to Ask a Question