Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk parsing file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk parsing file
# 1  
Old 01-15-2018
RedHat awk parsing file
Looking to parse a file to remove the duplicates and get only few fields and uniq rows.



input.tx
Code:
    Loc (TC) ID   : ssfad_fs
    Serial         : PIC002340098    
    
-----------------------------------------------------------------------------------------------------------------------
                                                                          Total
                                           Pic                                      Comp
ID   Name                        Ver  LOC       Time                      (Size)   (Size)         Complete Time
----------------------------------- ---- ----- ------------------------ ---------- ---------- ------------------------
FAD00 TINT_PAP_1516048511           0 .X... Mon Jan 15 20:35:14 2018          0          0                       NA
      sfad_jpg                      0 ..... Mon Jan 15 20:24:02 2018          0          0 Wed Jan 17 20:24:01 2018
      sfad_jpg                      1 ..... Mon Jan 15 16:24:02 2018          0          0 Wed Jan 17 16:24:02 2018
      sfad_jpg                      2 ..... Mon Jan 15 12:24:03 2018          0          0 Wed Jan 17 12:24:02 2018
      sfad_jpg                      3 ..... Mon Jan 15 08:24:02 2018          0          0 Wed Jan 17 08:24:02 2018
      sfad_jpg                      4 ..... Mon Jan 15 04:24:02 2018          0          0 Wed Jan 17 04:24:02 2018
      sfad_jpg                      5 ..... Mon Jan 15 00:24:03 2018          0          0 Wed Jan 17 00:24:02 2018
      sfad_jpg                      6 ..... Sun Jan 14 20:24:03 2018          0          0 Tue Jan 16 20:24:02 2018
      sfad_jpg                      7 ..... Sun Jan 14 16:24:03 2018          0          0 Tue Jan 16 16:24:02 2018
      sfad_jpg                      8 ..... Sun Jan 14 12:24:02 2018          0          0 Tue Jan 16 12:24:02 2018
      sfad_jpg                      9 ..... Sun Jan 14 08:24:03 2018          0          0 Tue Jan 16 08:24:02 2018
      sfad_jpg                     10 ..... Sun Jan 14 04:24:02 2018          0          0 Tue Jan 16 04:24:01 2018
      sfad_jpg                     11 ..... Sun Jan 14 00:24:02 2018          0          0 Tue Jan 16 00:24:02 2018
      sfad_jpg                     12 ..... Tue Oct 10 21:00:47 2017      65550      49153                       NA
BAC76 TINT_PAP_1516048511           0 .X... Mon Jan 15 20:35:14 2018          0          0                       NA
      sfad_jpg                      0 ..... Mon Jan 15 20:24:02 2018          0          0 Wed Jan 17 20:24:01 2018
      sfad_jpg                      1 ..... Mon Jan 15 16:24:02 2018          0          0 Wed Jan 17 16:24:02 2018
      sfad_jpg                      2 ..... Mon Jan 15 12:24:03 2018          0          0 Wed Jan 17 12:24:02 2018
      sfad_jpg                      3 ..... Mon Jan 15 08:24:02 2018          0          0 Wed Jan 17 08:24:02 2018
      sfad_jpg                      4 ..... Mon Jan 15 04:24:02 2018          0          0 Wed Jan 17 04:24:02 2018
      sfad_jpg                      5 ..... Mon Jan 15 00:24:03 2018          0          0 Wed Jan 17 00:24:02 2018
      sfad_jpg                      6 ..... Sun Jan 14 20:24:03 2018          0          0 Tue Jan 16 20:24:02 2018
      sfad_jpg                      7 ..... Sun Jan 14 16:24:03 2018          0          0 Tue Jan 16 16:24:02 2018
      sfad_jpg                      8 ..... Sun Jan 14 12:24:02 2018          0          0 Tue Jan 16 12:24:02 2018
      sfad_jpg                      9 ..... Sun Jan 14 08:24:03 2018          0          0 Tue Jan 16 08:24:02 2018
      sfad_jpg                     10 ..... Sun Jan 14 04:24:02 2018          0          0 Tue Jan 16 04:24:01 2018
      sfad_jpg                     11 ..... Sun Jan 14 00:24:02 2018          0          0 Tue Jan 16 00:24:02 2018
      sfad_jpg                     12 ..... Tue Oct 10 21:00:47 2017      65550      49153                       NA
      sfad_sf_test                  0 ..... Fri Aug 11 19:14:27 2017      65550      49153                       NA

Expecting output
Code:
Loc ID      Img Name              Ver.   Time                    Complete Time
----------     -------------            -----   ------------------   ----------------
ssfad_fs  TINT_PAP_1516048511    0  Mon Jan 15 20:35:13 2018     NA
ssfad_fs  sfad_jpg                  1  Mon Jan 15 16:24:02 2018   Wed Jan 17 16:24:01 2018
ssfad_fs  sfad_jpg                  2  Mon Jan 15 12:24:02 2018   Wed Jan 17 12:24:02 2018
ssfad_fs  sfad_jpg                  3  Mon Jan 15 08:24:02 2018   Wed Jan 17 08:24:01 2018
ssfad_fs  sfad_jpg                  4  Mon Jan 15 04:24:02 2018   Wed Jan 17 04:24:01 2018

I am using following awk command

Code:
cat input.txt | head -n-12 |cut -c6- |awk -v LOC="ssfad_fs" 'NR>9;$0=SG $0; {if ($1 ~ /^[\w-]+$/ && $2 ~ /0-100/);{$4=$10=$11="";print}}' |sort -unk 2,2

But when I use following code I don't get desired result.
Code:
cat input.txt | head -n-12 |cut -c6- | awk -v SG="ssfad_fs" 'NR>9;$0=SG $0 {$4=$10=$11=""}; {print $0}'

ssfad_fs sfad_jpg 9  Sun Jan 14 08:24:02 2018   Tue Jan 16 08:24:02 2018
 sfad_jpg                     10 ..... Sun Jan 14 04:24:01 2018          0          0 Tue Jan 16 04:24:01 2018
ssfad_fs sfad_jpg 10  Sun Jan 14 04:24:01 2018   Tue Jan 16 04:24:01 2018
 sfad_jpg                     11 ..... Sun Jan 14 00:24:02 2018          0          0 Tue Jan 16 00:24:01 2018
ssfad_fs sfad_jpg 11  Sun Jan 14 00:24:02 2018   Tue Jan 16 00:24:01 2018
 sfad_jpg                     12 ..... Tue Oct 10 21:00:46 2017      65550      49153                       NA
ssfad_fs sfad_jpg 12  Tue Oct 10 21:00:46 2017   NA
 ssfad_fs_test                      0 ..... Fri Aug 11 19:14:26 2017      65550      49153                       NA
ssfad_fs ssfad_fs_test 0  Fri Aug 11 19:14:26 2017   NA

Other options I tried doesn't come closer to what I am expecting. Your help/guidance would be appreciated.
Code:
awk 'NR>9;BEGIN {printf ("%s%n%s%s","Loc ID", "Img Name" "Ver#", "Time", "Complete Time")}{if ($1 ~ /^[\w-]+$/ && $2 ~ /0-100/);{$3=$9=$10="";print}

Last edited by RudiC; 01-16-2018 at 08:43 AM.. Reason: Added code tag.
# 2  
Old 01-16-2018
What defines a "duplicate" in your input file?
Why are some of the seconds in your expected output decremented by one, others not?
# 3  
Old 01-16-2018
Actually, I don't need ID field ( ID Field has hex values like FAD00, BAC76) I don't need those. I want only LOC ID which I can provide as a variable input as well ( awk -v LOC="ssfad_fs" . Also don't require LOC, Total Size, Comp Size fields in the output.

Regarding some of the seconds are decremented because when I generate file sometimes there is 1 second difference. Actually the I/P file I provided and O/P are from different files. That is why there is mismatch. Sorry for the confusion. Here is the O/P from same I/P file.
Code:
Loc ID	 Img Name               Ver.   Time                    Complete Time
ssfad_fs TINT_PAP_1516048511     0    Mon Jan 15 20:35:13 2018 NA
ssfad_fs sfad_jpg                1    Mon Jan 15 16:24:02 2018 Wed Jan 17 16:24:02 2018
ssfad_fs sfad_jpg                2    Mon Jan 15 12:24:03 2018 Wed Jan 17 12:24:02 2018
ssfad_fs sfad_jpg                3    Mon Jan 15 08:24:02 2018 Wed Jan 17 08:24:02 2018
ssfad_fs sfad_jpg                4    Mon Jan 15 04:24:02 2018 Wed Jan 17 04:24:02 2018
ssfad_fs sfad_jpg                5    Mon Jan 15 00:24:03 2018 Wed Jan 17 00:24:02 2018
ssfad_fs sfad_jpg                6    Sun Jan 14 20:24:03 2018 Tue Jan 16 20:24:02 2018
ssfad_fs sfad_jpg                7    Sun Jan 14 16:24:03 2018 Tue Jan 16 16:24:02 2018
ssfad_fs sfad_jpg                8    Sun Jan 14 12:24:02 2018 Tue Jan 16 12:24:02 2018
ssfad_fs sfad_jpg                9    Sun Jan 14 08:24:03 2018 Tue Jan 16 08:24:02 2018
ssfad_fs sfad_jpg               10    Sun Jan 14 04:24:02 2018 Tue Jan 16 04:24:01 2018
ssfad_fs sfad_jpg               11    Sun Jan 14 00:24:02 2018 Tue Jan 16 00:24:02 2018
ssfad_fs sfad_jpg               12    Tue Oct 10 21:00:47 2017 NA

# 4  
Old 01-16-2018
You still didn't tell people in here which lines to print and which to suppress.
# 5  
Old 01-16-2018
After Removing fields ID,LOC, Total Size, Comp Size fields from the input I want unique lines only.
# 6  
Old 01-16-2018
How far would
Code:
cut -c7-38,45-69,92- file | tail -n+9 | sort -uk2,2n
TINT_PAP_1516048511           0 Mon Jan 15 20:35:14 2018                       NA
sfad_jpg                      1 Mon Jan 15 16:24:02 2018 Wed Jan 17 16:24:02 2018
sfad_jpg                      2 Mon Jan 15 12:24:03 2018 Wed Jan 17 12:24:02 2018
sfad_jpg                      3 Mon Jan 15 08:24:02 2018 Wed Jan 17 08:24:02 2018
sfad_jpg                      4 Mon Jan 15 04:24:02 2018 Wed Jan 17 04:24:02 2018
sfad_jpg                      5 Mon Jan 15 00:24:03 2018 Wed Jan 17 00:24:02 2018
sfad_jpg                      6 Sun Jan 14 20:24:03 2018 Tue Jan 16 20:24:02 2018
sfad_jpg                      7 Sun Jan 14 16:24:03 2018 Tue Jan 16 16:24:02 2018
sfad_jpg                      8 Sun Jan 14 12:24:02 2018 Tue Jan 16 12:24:02 2018
sfad_jpg                      9 Sun Jan 14 08:24:03 2018 Tue Jan 16 08:24:02 2018
sfad_jpg                     10 Sun Jan 14 04:24:02 2018 Tue Jan 16 04:24:01 2018
sfad_jpg                     11 Sun Jan 14 00:24:02 2018 Tue Jan 16 00:24:02 2018
sfad_jpg                     12 Tue Oct 10 21:00:47 2017                       NA

get you?
# 7  
Old 01-16-2018
Hi RUDIC, Thanks for your response but would be better if we can append "Loc (TC) ID" which is "ssfad_fs" in this case. I would prefer to pass it as a variable so that I can use it in the script.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing a file name with awk

I have a command to print out the top 5 most cpu intensive commands: ps aux --sort=-%cpu --no-headers I would like to make a printout containing only the parsed file name. So, I want to turn this: $ ps aux --sort=-%cpu --no-headers user 24883 4.3 1.7 2010436 131440 ? Sl ... (2 Replies)
Discussion started by: bedtime
2 Replies

2. Shell Programming and Scripting

AWK? parsing /etc/passwd file.

Hello guys, please help me to make simple script for parsing passwd file. I have many passwd files from our servers, named server1.pass, server2.pass etc.. so for server in `ls *.pass` i need to print these rows: server1;root:!:0:0::/root:/usr/bin/ksh... (7 Replies)
Discussion started by: rubico
7 Replies

3. Shell Programming and Scripting

Help in file parsing with awk

Hi, I have a data set as shown below: 09e757fd,22727,2012-03-01,text1,text2,to 3fd0cae7,310,2012-03-01,text3,text4,to 3fd0cae7,310,2012-03-01,text3,text5,to 3fd0cae7,311,2012-03-01,text7,text10,cc 3fd0cae7,311,2012-03-01,text7,text11,to 3fd0cae7,312,2012-03-01,text8,text15,to... (3 Replies)
Discussion started by: shekhar2010us
3 Replies

4. Shell Programming and Scripting

Parsing a file with AWK

Input Group: Erecords: Copy: ADC R2: Replication volumes: Replication set: RSet 1 Replication size: 50.00GB SAN Info: 50.00GB DGC VRAID CX4-960 LUN 1040 (1040) 60,06,01,60,32,bb,21,00,84,a0,08,b1,da,ec,df,11... (2 Replies)
Discussion started by: greycells
2 Replies

5. UNIX for Advanced & Expert Users

Parsing through a file with awk/sed

I don't necessary have a problem, as I have a solution. It is just that there may be a better solution. GOAL: Part one: Parse data from a file using the "\" as a delimiter and extracting only the last delimiter. Part two: Parse same file and extract everything but the last delimited item. ... (8 Replies)
Discussion started by: OrangeYaGlad
8 Replies

6. Shell Programming and Scripting

awk/sed for parsing file

Hi All, I have a log file like this E Mon Oct 06 00:17:08 2008 xxx2 cm:10614 fm_pi2_svc_iptv_purchase.c:149 1:pin_deferred_act:10601:11:169:1223245028:16 pi2_op_svc_iptv_purchase error <location=PIN_ERRLOC_FM:5 class=PIN_ERRCLASS_SYSTEM_DETERMINATE:1... (10 Replies)
Discussion started by: subin_bala
10 Replies

7. Shell Programming and Scripting

Parsing a file (sed/awk?)

Hello people, newbie question. I'm trying to parse these type of file 1 "CAR " " C1 " " " 6 0 C1 2 "CAR " " O1A" " " 8 0 O1A 3 "CAR " " O1B" " " 8 -1 O1B 4 "CAR " " C2 " " " 6 0 C2 5 "CAR " " C3 " " " 6 ... (10 Replies)
Discussion started by: aristegui
10 Replies

8. Shell Programming and Scripting

awk and file parsing

Hi, I have a input file like this TH2TH2867Y NOW33332106Yo You Baby TH2TH3867Y NOW33332106No Way Out TH2TH9867Y NOW33332106Can't find it TJ2TJ2872N WOW33332017sure thing alas TJ2TJ3872N WOW33332017the sky rocks TJ2TJ4872N WOW33332017nothing else matters ... (4 Replies)
Discussion started by: devtakh
4 Replies

9. Shell Programming and Scripting

AWK and Magic with file parsing

Hi, I have a input file like this TH2TH2867Y NOW33332106Yo You Baby TH2TH3867Y NOW33332106No Way Out TH2TH9867Y NOW33332106Can't find it TJ2TJ2872N WOW33332017sure thing alas TJ2TJ3872N WOW33332017the sky rocks TJ2TJ4872N WOW33332017nothing else matters TJ2TJ5872N WOW33332017you know... (1 Reply)
Discussion started by: devtakh
1 Replies

10. Shell Programming and Scripting

parsing file through awk

hi, how can i achieve this in awk CON Controllers Department R abcuser usernamedesc1 R defuser usernamedesc2 R ... (13 Replies)
Discussion started by: bbeugie
13 Replies
Login or Register to Ask a Question