Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Shell script change new format on the file.

Shell Programming and Scripting


Tags
solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 06-08-2017
dotran dotran is offline
Registered User
 
Join Date: Aug 2012
Last Activity: 9 June 2017, 12:27 AM EDT
Posts: 37
Thanks: 0
Thanked 0 Times in 0 Posts
Shell script change new format on the file.

Hi---Is there's way can write small shell script or perl script open "abc.txt" file and create new "new_abc.txt" file with format output below? Thanks

cat abc.txt

Code:
 
 ###########################Readme###############################
Contained with this README.TXT file are all of the 
file specs for your
directory abt.
###########################Readme###############################
 
Filename : SW_PP_CTRL_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 2019064
Unzipped Bytes : 11413730
Records : 95788
Record Length : 157
 
Filename : SW_PP_DATA_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 691778058
Unzipped Bytes : 8316153069
Records : 60400481
Record Length : 158
 
Filename : SW_PP_DEMO_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 26240709
Unzipped Bytes : 77053000
Records : 543250
Record Length : 227
 
Filename : SW_PP_PLANXREF_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 557904
Unzipped Bytes : 3061930
Records : 16262
Record Length : 310
 
Filename : SW_PP_PRODUCT_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 21375
Unzipped Bytes : 229431
Records : 1264
Record Length : 211
 
Filename : SW_PP_REASSIGN_20170505.txt.gz
Data Format : ASCII with carriage returns and linefeeds
Compression : GZIP
GZIP Bytes : 32681
Unzipped Bytes : 69399
Records : 802
Record Length : 130

Output: cat new_abc.txt

Code:
FILE_NAME,Filename|Data Format|Compression|GZIP Bytes|Unzipped Bytes|Records|Record Length
SW_PP_CTRL_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|2019064|11413730|95788|157
SW_PP_DATA_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|691778058|8316153069|60400481|158
SW_PP_DEMO_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|26240709|77053000|543250|227
SW_PP_PLANXREF_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|557904|3061930|16262|310
SW_PP_PRODUCT_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|21375|229431|1264|211
SW_PP_REASSIGN_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|32681|69399|802|130

---------- Post updated at 08:39 PM ---------- Previous update was at 05:33 PM ----------

I have this perl code and when run see extra "|" at the end of each line. And not sure how to code remove it or can write shell script easier.....can someone help? Thanks

Code:
 
 #!/usr/bin/perl 
 my $filename = 'abc.txt';
open(my $fh, '<:encoding(UTF-8)', $filename)  or die "Could not open file '$filename' $!";
print "FILE_NAME,Filename|Data Format|Compression|GZIP Bytes|Unzipped Bytes|Records|Record Length";
while (my $row = <$fh>) {
  chomp $row;
  my ($label, $value) = split /: /, $row;
  if ($row eq '') {
    print "\n";
  }
  else
  {
     print "$value|";
  }
}

/test1> ./test.pl

Code:
FILE_NAME,Filename|Data Format|Compression|GZIP Bytes|Unzipped Bytes|Records|Record Length|||||
SW_PP_CTRL_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|2019064|11413730|95788|157|
SW_PP_DATA_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|691778058|8316153069|60400481|158|
SW_PP_DEMO_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|26240709|77053000|543250|227|
SW_PP_PLANXREF_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|557904|3061930|16262|310|
SW_PP_PRODUCT_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|21375|229431|1264|211|
SW_PP_REASSIGN_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|32681|69399|802|130|


Last edited by dotran; 06-08-2017 at 06:48 PM..
Sponsored Links
    #2  
Old Unix and Linux 06-08-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 October 2017, 10:18 PM EDT
Location: San Jose, CA, USA
Posts: 10,563
Thanks: 555
Thanked 3,699 Times in 3,156 Posts
Except for the strange field 1 header in your output, the following awk script seems to produce the output you requested:

Code:
awk -F ' : ' '
NF == 2 {
	if(h < 7)
		printf("%s%s", $1, (++h == 7) ? ORS : OFS)
	o = f++ ? (o OFS $2) : $2
	if(f == 7) {
		print o
		o = ""
		f = 0
	}
}' OFS='|' file

which prints:

Code:
Filename|Data Format|Compression|GZIP Bytes|Unzipped Bytes|Records|Record Length
SW_PP_CTRL_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|2019064|11413730|95788|157
SW_PP_DATA_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|691778058|8316153069|60400481|158
SW_PP_DEMO_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|26240709|77053000|543250|227
SW_PP_PLANXREF_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|557904|3061930|16262|310
SW_PP_PRODUCT_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|21375|229431|1264|211
SW_PP_REASSIGN_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|32681|69399|802|130

This could have been done much more simply with:

Code:
awk -F ' : |\n' '
NF == 14 {
	if(!h) {h = 1
		print $1, $3, $5, $7, $9, $11, $13
	}
	print $2, $4, $6, $8, $10, $12, $14
}' OFS='|' RS='' file2

if the blank lines in your input file had been empty lines instead (i.e., no <space> characters on the blank lines between records).

If you want to try either of these on a Solaris/SunOS system, change awk to nawk or /usr/xpg4/bin/awk.
Sponsored Links
    #3  
Old Unix and Linux 06-08-2017
dotran dotran is offline
Registered User
 
Join Date: Aug 2012
Last Activity: 9 June 2017, 12:27 AM EDT
Posts: 37
Thanks: 0
Thanked 0 Times in 0 Posts
Thank you for your help Mr.Don. I run the code and output 1 line header should no space and require extra 'FILE_NAME,' like below. Could you please help out again?


Code:
FILE_NAME,Filename|Data Format|Compression|GZIP Bytes|Unzipped Bytes|Records|Record Length

/test :./abc1


Code:
 
 Filename        | Data Format    | Compression    | GZIP Bytes     | Unzipped Bytes | Records        | Record Length  
SW_PP_CTRL_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|2019064|11413730|95788|157
SW_PP_DATA_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|691778058|8316153069|60400481|158
SW_PP_DEMO_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|26240709|77053000|543250|227
SW_PP_PLANXREF_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|557904|3061930|16262|310
SW_PP_PRODUCT_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|21375|229431|1264|211
SW_PP_REASSIGN_20170505.txt.gz|ASCII with carriage returns and linefeeds|GZIP|32681|69399|802|130

    #4  
Old Unix and Linux 06-08-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 October 2017, 10:18 PM EDT
Location: San Jose, CA, USA
Posts: 10,563
Thanks: 555
Thanked 3,699 Times in 3,156 Posts
If you copied the code I gave you and you executed that code as given, the output you showed us is not the output that would have been produced unless your sample input file format is significantly different from the input file you used when you ran my script.

With a very simple script like the one I suggested, it should be easy for you to modify it to print a constant header line that doesn't try to use the field headings found in the data being read. Why don't you try modifying the code I suggested and let us know where you run into problems if you can't make it work?
Sponsored Links
    #5  
Old Unix and Linux 06-09-2017
dotran dotran is offline
Registered User
 
Join Date: Aug 2012
Last Activity: 9 June 2017, 12:27 AM EDT
Posts: 37
Thanks: 0
Thanked 0 Times in 0 Posts
Thank you mr.Don. Yes...the input had space.....anyway. I got it work.....thank you for your input.....Linux


Code:
 
 Filename         : SW_PP_PRODUCT_20170505.txt.gz
  Data Format     : ASCII with carriage returns and linefeeds
  Compression     : GZIP
  GZIP Bytes      : 21375
  Unzipped Bytes  : 229431
  Records         : 1264
  Record Length   : 211

Sponsored Links
    #6  
Old Unix and Linux 06-09-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 October 2017, 10:18 PM EDT
Location: San Jose, CA, USA
Posts: 10,563
Thanks: 555
Thanked 3,699 Times in 3,156 Posts
You're welcome. Would you like to show us what you did so others reading your thread can learn from your experience?
Sponsored Links
    #7  
Old Unix and Linux 06-09-2017
rovf rovf is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 12 September 2017, 2:10 AM EDT
Posts: 171
Thanks: 24
Thanked 31 Times in 27 Posts
(1) Your input file has a "readme" block at the start. You process it as if it were a normal line, which results in the vertical bars added to your header line.

(2) Instead of
Code:
if ($row = '')

, I would reverse the test and ask, whether a line contains a colon - and only then split it:


Code:
if ($row =~ /:/)
   # .... split
else
   # .... process other lines
end

Actually, you can even get rid of the split by


Code:
if ($row =~ /:(.*)/)
  # ..... The part after the colon is now stored in $1
else
  # .... process non-colon lines
end



(3) Note that, if you find a value, you always print it as "$value|". This means that every line has a vertical bar at the end.

(4) If your input has a sequence of more than one lines without a colon, you would also produce the same number of empty lines in the output.

Last edited by rovf; 06-09-2017 at 12:39 AM.. Reason: Improving solution
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Change date format in shell script shivasaini Shell Programming and Scripting 1 01-25-2013 09:50 AM
How to check file name format using shell script? Poonamol Shell Programming and Scripting 7 10-13-2010 07:20 AM
format file using shell script balajim Solaris 2 10-23-2008 09:18 AM
format file using shell script balajim Solaris 1 10-22-2008 03:30 PM
script to change the date format in a file shehzad_m Shell Programming and Scripting 15 06-11-2008 10:30 AM



All times are GMT -4. The time now is 12:08 AM.