Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using awk to reformat file output

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using awk to reformat file output
# 1  
Old 08-14-2014
Using awk to reformat file output
Hi there. I need to reformat a large file. Here is a sample of the file.

Code:
 
NETIK0102_UCS_Boot_a,NETIK0102_UCS_Boot_b
5200 2438 70G
5200 2439 70G
NETIK0102_UCS_HBA0_a,NETIK0102_UCS_HBA1_b,NETIK0102_UCS_HBA2_a,NETIK0102_UCS_HBA3_b
2673 19D7 55G
2673 19C0 30G
2673 19F5 120G
SEIADWFMPRD1-a,SEIADWFMPRD1-b
2673 1992 8.43G
2673 1993 8.43G
2673 19B1 16.9G

The line separated by commas needs to be split and I need to print the strings in the first column...

Here is the way I need the data presented.


Code:
 
NETIK0102_UCS_Boot_a 5200 2438 70G
NETIK0102_UCS_Boot_a 5200 2439 70G
NETIK0102_UCS_Boot_b 5200 2438 70G
NETIK0102_UCS_Boot_b 5200 2439 70G
NETIK0102_UCS_HBA0_a 2673 19D7 55G
NETIK0102_UCS_HBA0_a 2673 19C0 30G
NETIK0102_UCS_HBA0_a 2673 19F5 120G
NETIK0102_UCS_HBA1_b 2673 19D7 55G
NETIK0102_UCS_HBA1_b 2673 19C0 30G
NETIK0102_UCS_HBA1_b 2673 19F5 120G
NETIK0102_UCS_HBA2_a 2673 19D7 55G
NETIK0102_UCS_HBA2_a 2673 19C0 30G
NETIK0102_UCS_HBA2_a 2673 19F5 120G
NETIK0102_UCS_HBA3_b 2673 19D7 55G
NETIK0102_UCS_HBA3_b 2673 19C0 30G
NETIK0102_UCS_HBA3_b 2673 19F5 120G
SEIADWFMPRD1-a 2673 1992 8.43G
SEIADWFMPRD1-a 2673 1993 8.43G
SEIADWFMPRD1-a 2673 19B1 16.9G
SEIADWFMPRD1-b 2673 1992 8.43G
SEIADWFMPRD1-b 2673 1993 8.43G
SEIADWFMPRD1-b 2673 19B1 16.9G

T
# 2  
Old 08-14-2014
Code:
awk '/,/ {rcrsv(); c=split($0, a, ","); n=0; delete b; getline} {b[++n] = $0}
func rcrsv() {
  for(i=1; i<=c; i++) {for(j=1; j<=n; j++) {print a[i], b[j]}}}
END {rcrsv()}' file

# 3  
Old 08-14-2014
Hello kieranfoley,
I have a few to questions pose in response first:-
  • Is this homework/assignment? There are specific forums for these.
  • What have you tried so far?
  • What output/errors do you get?
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.
# 4  
Old 08-14-2014
Hi rbatte1 thank you for your reply.

No this is not homework. I use awk/sed/grep scripts on occasion and at times I get caught in a rut when more complex scripting is required. For this particular thread I have been tasked with having to pull relevent info from a file is produced daily from our SAN storage enviroment. My background is unix, I use shell scripts more than anything else to filter out information when needs be.

This is a sample of the original file.

Code:
 
/clusters/cluster-1/exports/storage-views/NETIK0102_Boot:
Name                      Value
------------------------  -----------------------------------------------------------------------------
caw-enabled               true
controller-tag            -
initiators                [NETIK0102_UCS_Boot_a, NETIK0102_UCS_Boot_b]
operational-status        ok
port-name-enabled-status  [P00000000476043B3-A0-FC01,true,ok, P00000000476046CC-A0-FC00,true,ok,
                          P00000000477043B3-B0-FC00,true,ok, P00000000477046CC-B0-FC01,true,ok]
ports                     [P00000000476043B3-A0-FC01, P00000000476046CC-A0-FC00,
                          P00000000477043B3-B0-FC00, P00000000477046CC-B0-FC01]
virtual-volumes           [(0,device_Symm5200_2438_1_vol,VPD83T3:60001440000000106046cc5f567eb9b9,70G),
                          (1,device_Symm5200_2439_1_vol,VPD83T3:60001440000000106046cc5f567eb9bf,70G)]
write-same-16-enabled     true

/clusters/cluster-1/exports/storage-views/NETIK0102_Shared:
Name                      Value
------------------------  -------------------------------------------------------------------------------
caw-enabled               true
controller-tag            -
initiators                [NETIK0102_UCS_HBA0_a, NETIK0102_UCS_HBA1_b, NETIK0102_UCS_HBA2_a,
                          NETIK0102_UCS_HBA3_b]
operational-status        ok
port-name-enabled-status  [P00000000476043B8-A0-FC02,true,ok, P0000000047604458-A0-FC03,true,ok,
                          P00000000477043B8-B0-FC03,true,ok, P0000000047704458-B0-FC02,true,ok]
ports                     [P00000000476043B8-A0-FC02, P0000000047604458-A0-FC03,
                          P00000000477043B8-B0-FC03, P0000000047704458-B0-FC02]
virtual-volumes           [(0,device_Symm2363_19D7_1_vol,VPD83T3:60001440000000106046cc5f567eb7dc,55G),
                          (1,device_Symm2363_19C0_1_vol,VPD83T3:60001440000000106046cc5f567eb769,30G),
                          (2,device_Symm2363_19F5_1_vol,VPD83T3:60001440000000106046cc5f567eb872,120G),
                          (3,device_Symm2363_1A03_1_vol,VPD83T3:60001440000000106046cc5f567eb8ae,280G),
                          (4,device_Symm2363_19DC_1_vol,VPD83T3:60001440000000106046cc5f567eb7f5,66G),
                          (5,device_Symm2363_1A6C_1_vol,VPD83T3:60001440000000106046cc5f567eb93f,600G),
                          (6,device_Symm2363_1998_1_vol,VPD83T3:60001440000000106046cc5f567eb9ad,10G),
                          (7,device_Symm2363_19CE_1_vol,VPD83T3:60001440000000106046cc5f567eb9b3,40G),
                          (8,device_Symm2363_1A7E_1_vol,VPD83T3:60001440000000106046cc5f567eb9c5,1000G),
                          (9,device_Symm2363_19B9_1_vol,VPD83T3:60001440000000106046cc5f567eb9cb,20G),
                          (10,device_Symm2363_19BA_1_vol,VPD83T3:60001440000000106046cc5f567eb9d1,20G),
                          (11,device_Symm2363_19BB_1_vol,VPD83T3:60001440000000106046cc5f567eb9d7,20G),
                          (12,device_Symm2363_19BC_1_vol,VPD83T3:60001440000000106046cc5f567eb9dd,20G),
                          (13,device_Symm2363_1AA3_1_vol,VPD83T3:60001440000000106046cc5f567eb9e3,2.93T),
                          (14,device_Symm2363_1A94_1_vol,VPD83T3:60001440000000106046cc5f567eb9e9,750G),
                          (15,device_Symm2363_1A88_1_vol,VPD83T3:60001440000000106046cc5f567eb9ef,300G)]
write-same-16-enabled     true

From this file I have been able to use the following commands to pull out the releven information.

Code:
cat file | awk '/initiators/,/write-same-16-enabled/ {print $0}' \
 | grep -v operational-status \
 | grep -v P00000000 \
 | grep -v write-same-16-enabled \
 | grep -v '\[\]' \
 | sed 's/ //g' \
 | sed 's/initiators/initiators    /g' \
 | nawk '/a,$|b,$|A,$|B,$/ {printf "%s",substr($0, 1, length-1)",";next} 1' \
 | sed 's/initiators    \[//g;s/\]//g' | sed 's/virtual-volumes\[//g' \
 | nawk '{if(/^\(/) {FS=","; print$2,$4 }else {print$0}}' \
 | sed 's/device_Symm2363_/2673 /g' \
 | sed 's/device_Symm5200_/5200 /g' \
 | sed 's/_1_vol//g' \
 | sed 's/)//g'

The code above is not very efficient I know but it works. I will go back to cleaning it up once I get it all working the way I want it. But for the past few days I have struggled with trying to get the last bit regarding what I posted earlier.

I thank you for your help on this.

---------- Post updated at 01:34 PM ---------- Previous update was at 01:25 PM ----------

Hi SriniShoo,

I have tried you code but I am getting the following error. I tried using awk and nawk (I am running this on a Solaris box).

Code:
nawk: you can only delete array[element] at source line 1
 context is
        /,/ {rcrsv(); c=split($0, a, ","); n=0; delete >>>  b; <<<
nawk: syntax error at source line 1
nawk: illegal statement at source line 1

Last edited by rbatte1; 08-14-2014 at 09:33 AM.. Reason: Breaking up one huge long line to make it more readable
# 5  
Old 08-14-2014
I've taken the liberty to split your huge single line of code onto several to make it more readable.

There are lots of steps in this processing. Some grep commands in sequence could be combined, as could the sed commands.

Could you explain a bit more about the logic you are using to generate so many steps?

You could re-write the section:-
Code:
 | grep -v operational-status \
 | grep -v P00000000 \
 | grep -v write-same-16-enabled \
 | grep -v '\[\]' \

.... as
Code:
 | egrep -v "operational-status|P00000000|write-same-16-enabled|\[\]" \

# 6  
Old 08-14-2014
Yes thanks for cleaning up the greps. I have also cleaned up the sed substitutions somebit. The reason I have ended up with so many steps is because I have been doing this bit by bit. I run an awk command and sed to filter out piece by piece and I have being piping the commands together. Not very efficient I know. This is a little bit cleaner but not to your liking I'm sure. Thanks!

Code:
 
cat file | awk '/initiators/,/write-same-16-enabled/ {print $0}' \
| egrep -v "operational-status|P00000000|write-same-16-enabled|\[\]" \
| sed 's/ //g;s/initiators/initiators    /g' \
| nawk '/a,$|b,$|A,$|B,$/ {printf "%s",substr($0, 1, length-1)",";next} 1' \
| sed 's/initiators    \[//g;s/\]//g;s/virtual-volumes\[//g' \
| nawk '{if(/^\(/) {FS=","; print$2,$4 }else {print$0}}' \
| seds/device_Symm2363_/2673 /g;\
s/device_Symm5200_/5200 /g;s/_1_vol//g;s/)//g'

---------- Post updated at 02:41 PM ---------- Previous update was at 01:58 PM ----------

SriniShoo,

when I take out the "delete" it works. The way I want it. Thanks!!

Code:
 
awk '/,/ {rcrsv(); c=split($0, a, ","); n=0; getline} {b[++n] = $0}
func rcrsv() {
  for(i=1; i<=c; i++) {for(j=1; j<=n; j++) {print a[i], b[j]}}}
END {rcrsv()}' file

Last edited by rbatte1; 08-14-2014 at 10:17 AM.. Reason: Emboldened commands for clarity
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to reformat output if input file is empty, but not if file has data in it

The below awk improved bu @MadeInGermany, works great as long as the input file has data in it in the below format: input chrX 25031028 25031925 chrX:25031028-25031925 ARX 631 18 chrX 25031028 25031925 chrX:25031028-25031925 ARX 632 14... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk to reformat text file

Howdy. AWK beginner here. I need to reformat a text file in the following format: TTGS08-2014001 6018.00 143563.00 ... (2 Replies)
Discussion started by: c47v3770
2 Replies

3. Shell Programming and Scripting

Reformat awk output

I need to rearrange the output but i am unable to arrange it to match the format. In the output i need NAME=\"To in the column . Bash: #!/bin/bash cd /cygdrive/c/output/a cat *.txt > output.txt i=/cygdrive/c/output/a/output.csv #echo "NE_Name, Source, Destination, OSPF_AREA_ID"... (4 Replies)
Discussion started by: adgjmpt
4 Replies

4. Shell Programming and Scripting

Alter awk script to reformat output

Data: 0,mfrh_green_screen,1455432969,37540,/prod/test/system/sys/unikixmain.log,3.0M,mfrh_green_screen,3120660,0,36964--37540 0,mfrh_green_screen,1455433269,38100,/prod/test/system/sys/unikixmain.log,3.1M,mfrh_green_screen,3164223,0,37540--38100... (1 Reply)
Discussion started by: SkySmart
1 Replies

5. Shell Programming and Scripting

Use search pattern to reformat the output

I have below file listing ] ls -1 *.txt MISTradesReport_141105_d130240_VOLCKER_EMEA_LOANIQ_FEED_2013-12-24.txt MISTradesReport_141106_d130240_VOLCKER_NA_LOANIQ_FEED_2013-12-24.txt MISTradesReport_141107_d130240_VOLCKER_EMEA_CDS_CRDI_FEED_2013-12-24.txt... (4 Replies)
Discussion started by: krg.sati
4 Replies

6. Shell Programming and Scripting

awk reformat file

Hello: When I tried a perl-oneliner to re-format fasta file. infile.fasta >YAL069W-1.334 Putative promoter CCACACCACACCCACACACC ACACCACACCCACACACACA ACAGCCCTAATCTAACCC >YAL068C-7235.2170 Putative ABC sequence TACGAGAATAATTT ACGTAAATGAAGTT TATATATAAA >gi|31044174|gb|AY143560.1|... (15 Replies)
Discussion started by: yifangt
15 Replies

7. Shell Programming and Scripting

need awk or sed help to reformat output

We have the following output: server1_J00_data_20120711122243 server1_J00_igs_20120711122243 server1_J00_j2ee_20120711122243 server1_J00_sec_20120711122243 server1_J00_data_20120711131819 server1_J00_igs_20120711131819 server1_J00_j2ee_20120711131819 server2_J00_data_20120711122245... (10 Replies)
Discussion started by: ux4me
10 Replies

8. UNIX for Advanced & Expert Users

Script to reformat output

Hi colleagues, I have the followind script. db2 -x "select substr(TBSPACE,1,20) TABLESPACE from syscat.tables where tabschema = 'SCHEMA' and tabname like '%XXXX' group by TBSPACE order by TBSPACE" | awk '{print $1}' | while read tablespace do db2 "list tablespaces show detail" |grep -p -w... (5 Replies)
Discussion started by: systemoper
5 Replies

9. Shell Programming and Scripting

awk to reformat a text file

I am definitely not an expert with awk, and I want to reformat a text file like the following. This is probably a very easy one for an expert out there. I would like to keep the lines in the same order, but move the heading to only be listed once above the lines. This is what the text file... (7 Replies)
Discussion started by: linux4life
7 Replies

10. UNIX for Advanced & Expert Users

reformat ps output

I often use "ps -ef" command to list all running processes. Now i want to customize the output to show only 2 parts: CMD and UID as below: /bin/bash /usr/bin/run-parts /etc/cron.weekly root /usr/sbin/httpd apache /usr/sbin/httpd apache /usr/sbin/httpd apache I use ps -ef | awk '{print $8"... (3 Replies)
Discussion started by: fongthai
3 Replies
Login or Register to Ask a Question