Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Problems with Sed/awk/grep and line endings
Top Forums Shell Programming and Scripting Problems with Sed/awk/grep and line endings Post 302425807 by benwiggy on Sunday 30th of May 2010 11:49:32 AM
Old 05-30-2010
Problems with Sed/awk/grep and line endings
Hello
I have created the following script, which is designed to manipulate a text document:
Code:
#!/bin/sh
# Get 3 lines, (last of which is "Quantity"); adjust order; put all three on one line with tabs.
FILENAME=~/Desktop/email.txt
LIST=$(grep -B2 "Quantity" ${FILENAME} |awk 'BEGIN { FS = "\n"; RS = "--"; } 
{ if ($4 != "")
 printf ( $4 "\t" $2 "\t" $3 "\n") 
 else { printf ($3 "\t" $1 "\t" $2 "\n") }
 } 
 END { }');

# Remove "Quantity :" and "Price : ".
LIST=$(echo -e $LIST |sed s/"Quantity: "//g);
LIST=$(echo -e $LIST |sed s/"Price: "//g);
echo -e $LIST > email2.txt;

#Remove asterisks; get range of text between strings; remove up to colon on each line.
ADDRESS=$(cat $FILENAME |sed s/\*//g);
ADDRESS=$(echo -e $ADDRESS |sed -n '/BILLING DETAILS/,/DELIVERY DETAILS/p');
ADDRESS=$(echo -e $ADDRESS |awk -F: '{ printf $2 "\n" }');
echo -e $ADDRESS >> email2.txt;

I have a number of problems with it.
1. The \t tabs and \n newlines in the printf section of awk don't get written to the file, only spaces.
2. The last three commands doesn't seem to work in the script, though they seem to work individually on the command line. Again, echoing to the Terminal displays linefeeds, but echoing to the file in the script does not produce line feeds, just spaces.
3. Some of the text being processed is a few lines of "***********". When these lines are present, I end up with a directory listing in my final output file.

Can anyone explain why these problems are happening and how to stop them? Thanks.

The script processes two parts of a text file in two different ways. The second half (between two strings) should just be written to a new file without everything up to an including a colon on each line.
The first half takes three lines (the 3rd of which starts "Quantity: "), rearranges their order and then removes some text. I seem to have an off-by-one error for the first item, which is why there's an if..then.

Hope this makes sense! (Oh yes, I'm running OS X 10.6.3)
Last edited by benwiggy; 05-30-2010 at 01:16 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Vi line endings conversions

I was reading these 2 articles. Why does the wikia one think :e ++ff=dos? Or am I just misunderstanding it? :e ++ff=unix :e ++ff=dos File format - Vim Tips Wiki Managing/Munging Line-Endings with Vi/Vim | Jeet Sukumaran (1 Reply)
Discussion started by: cokedude
1 Replies

2. UNIX for Advanced & Expert Users

line endings help of non-ASCII files

When you are dealing with ASCII files it easy to check on line endings type. You can just use the file command. You are not always lucky enough to be dealing with ASCII files. So in the cases that you don't have ASCII files how can you check what type of line endings you have? Please list all... (5 Replies)
Discussion started by: cokedude
5 Replies

3. UNIX for Advanced & Expert Users

vimrc help with line endings

I was reading this and thought I could put this in my vimrc and it would convert the line endings to unix. Am I doing something wrong or am I missing something? set ff=unixManaging/Munging Line-Endings with Vi/Vim | Jeet Sukumaran I used this command and it confirms that my global option is... (2 Replies)
Discussion started by: cokedude
2 Replies

4. Shell Programming and Scripting

sed command to grep multiple pattern present in single line and delete that line

here is what i want to achieve.. i have a file with below contents cat fileName blah blah blah . .DROP this REJECT that . --sport 7800 -j REJECT --reject-with icmp-port-unreachable --dport 7800 -j REJECT --reject-with icmp-port-unreachable . . . more blah blah blah --dport 3306... (14 Replies)
Discussion started by: vivek d r
14 Replies

5. Shell Programming and Scripting

Use less pipe for grep or awk sed to print the line not include xx yy zz

cat file |grep -v "xx" | grep -v "yy" |grep -v "zz" (3 Replies)
Discussion started by: yanglei_fage
3 Replies

6. Shell Programming and Scripting

Sendmail ignoring line endings

Mails from Sendmail are ignoring line endings, when I try to send email with attachment. I have tried to specify the font in the html but line endings are still ignored. I also tried unix2dos, still no luck. #!/usr/bin/ksh ###Send Email MAILTO=`cat mail2.list | tr -s '\n' ','` SUBJECT="bla bla... (3 Replies)
Discussion started by: aydj
3 Replies

7. Shell Programming and Scripting

awk or sed or grep filter a line and/or between strings

Hi, I have multiple files on a directory with the following content: blahblah blahblah hostname server1 blahblah blahblah ---BEGIN--- aaa bbb ccc ddd ---END--- blahblah blahblah blahblah I would like to filter all the files with awk or sed or something else so I can get below... (6 Replies)
Discussion started by: bayupw
6 Replies

8. Shell Programming and Scripting

Sed/grep: check if line exists, if not add line?

Hello, I'm trying to figure out how to speed up the following as I want to use multiple commands to search thousands of files. is there a way to speed things up? Example I want to search a bunch of files for a specific line, if this line already exists do nothing, if it doesn't exist add it... (4 Replies)
Discussion started by: f77hack
4 Replies

9. Shell Programming and Scripting

sed or awk grep, that will only get the line with more characters.

Is there a command for sed and awk that will only sort the line with more characters? #cat file 123 12345 12 asdgjljhhho bac ss Output: asdgjljhhho #cat file2 11.2 12345.00 21.222 12345678.10 (2 Replies)
Discussion started by: invinzin21
2 Replies

10. UNIX for Beginners Questions & Answers

Tip to remove line endings and spaces on a pre-formatted text file?

Hi, At the moment, using Notepad++ to do a search and replace, manually section by section which is real painful. Yeah, so copying each section of the line of text and putting into a file and then search and replace, need at least 3-operations in Notepad++. Here's hoping I will be able to... (1 Reply)
Discussion started by: newbie_01
1 Replies

LEARN ABOUT DEBIAN

prunehistory

PRUNEHISTORY(8) 					      System Manager's Manual						   PRUNEHISTORY(8)

NAME

       prunehistory - remove file names from Usenet history file

SYNOPSIS

       prunehistory [ -f filename ] [ -p ] [ input ]

DESCRIPTION

       Prunehistory  modifies  the  history(5)	text file to ``remove'' a set of filenames from it.  The filenames are removed by overwriting them
       with spaces, so that the size and position of any following entries does not change.

       Prunehistory reads the named input file, or standard input if no file is given.	The input is taken as a set of	lines.	 Blank	lines  and
       lines  starting	with a number sign (``#'') are ignored.  All other lines are should consist of a Message-ID followed by zero or more file-
       names.

       The Messge-ID is used as the dbz(3) key to get an offset into the text file.  If no filenames are mentioned on the  input  line,  then  all
       filenames  in the text are ``removed.''	If any filenames are mentioned, they are converted into the history file notation.  If they appear
       in the line for the specified Message-ID then they are removed.

       Since innd(8) only appends to the text file, prunehistory does not need to have any interaction with it.

OPTIONS

       -p     Prunehistory will normally complain about lines that do not follow the correct format.  If the ``-p'' flag is used, then the program
	      will silently print any invalid lines on its standard output.  (Blank lines and comment lines are also passed through.)  This can be
	      useful when prunehistory is used as a filter for other programs such as reap.

       -f     The default name of the history file is /var/lib/news/history; to specify a different name, use the ``-f'' flag.

EXAMPLES

       It is a good idea to delete purged entries and rebuild the dbz database every so often by using a script like the following:

	      ctlinnd throttle "Rebuilding history database"
	      cd /var/lib/news
	      awk 'NF > 2 {
		   printf "%s	%s	%s", $1, $2, $3;
		   for (i = 4; i <= NF; i++)
			printf " %s", $i;
		   print "
";
	      }' <history >history.n
	      if makehistory -r -f history.n ; then
		  mv history.n history
		  mv history.n.pag history.pag
		  mv history.n.dir history.dir
	      else
		  echo 'Problem rebuilding history; old file not replaced'
	      fi
	      ctlinnd go "Rebuilding history database"

       Note that this keeps no record of expired articles.

HISTORY

       Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews.  This is revision 1.9, dated 1996/10/29.

SEE ALSO

       dbz(3), history(5), innd(8).

																   PRUNEHISTORY(8)
All times are GMT -4. The time now is 08:48 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy