Sponsored Content
Top Forums Shell Programming and Scripting Append next line to previous lines when NF is less than 0 Post 302898216 by Don Cragun on Sunday 20th of April 2014 02:03:31 PM
Old 04-20-2014
After reformatting your code so we can see the structure, getting rid of the subshell issue Scrutinizer mentioned, adding missing <dollar-sign> characters, changing <single-quote> characters to <double-quote> characters, and adding missing <double-quote> characters to get around syntax errors:
Code:
OUT_FILE=out
FLAG=0
while read CUR_LINE
do
        if [[ $FLAG -ne 0 ]]
        then
                if [[ `echo ${CUR_LINE} | awk -F "Ç" '{print NF -1}'` -le 0 ]]
                then
                        PREV_LINE="${PREV_LINE} ${CUR_LINE}"
                        NEW_LINE=`echo ${PREV_LINE} | tr -d '\n' | tr -d '^M'`
                        PREV_LINE="${NEW_LINE}"
                else
                        echo ${PREV_LINE} >> ${OUT_FILE}
                        PREV_LINE="${CUR_LINE}"
                fi
        else
                PREV_LINE="${CUR_LINE}"
                FLAG=1
        fi
done < filename
echo ${PREV_LINE} >> ${OUT_FILE}

we can see that this is grossly inefficient code. Having a while loop is not your problem, executing awk once for each of your 1.7 million input lines (except the 1st ) and tr twice for both empty lines and lines with only one field (especially since one of those invocations of tr is always a no-op) is going to be extremely slow.

Your code seems to be trying to remove <carriage-return> characters from your input (which you never mentioned were present before). And, we can't tell if you're trying to remove <carriage-return> or circumflex and upper-case M characters. (The above code removes all circumflex and upper-case M characters from your input.)

It also converts all sequences of one or more adjacent <space> and <tab> characters to a single <space> character (which again was not mentioned as a requirement until now). Is this intentional, or an accident? Or does your input contain no <tab> characters and no occurrences of multiple adjacent <space> characters?

It gets rid of backslash characters at the ends of input lines and joins lines that end with <backslash> characters no matter how many fields are on the joined lines. Is this intentional, or an accident? Or, are you sure that none of your input lines end with a <backslash> character just before a <newline> character?

And, depending on what shell you're using and what operating system you're using, any other <backslash> characters in your input could be deleted or converted to other characters by your uses of echo.

Please show us the code you are really using. Please also upload a SMALL sample input file (not more than 50 lines) that contains examples of all of the transformations that need to take place while removing characters, joining lines, and squeezing blanks, AND upload the desired output corresponding to that input. I explicitly say upload because we need to be sure that we will be able to see the difference between spaces and tabs in your desired input and output and see the <carriage-return> characters in your input.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Append line that does not contain pipe to it previous line

Hi All, I have a file which contains data as below When we see no pipe character in the line. append those lines to the previous line with pipe character till we get the next line with pipe character with ~(concat with ~) Input file looks like: 1080530944|001|john.l.bonner|Acknowledge|CN... (11 Replies)
Discussion started by: ainuddin
11 Replies

2. UNIX for Advanced & Expert Users

append the line with the previous if it not start with 1=

How to append the line with the previous if it not start with 1=. 1=ttt, 2=xxxxxx, 3=4545 44545, 4=66666, 1=ttt, 2=xxxxxx, 3=34434 3545, 4=66666, 5=ffffff 6=uuuuuuu, 7=ooooooo 1=ttt, 2=xxxxxx, 3=311343545, 4=66666 1=ttt, 2=xxxxxx, 5=XAXAXA, 7=FDFD (3 Replies)
Discussion started by: palsevlohit_123
3 Replies

3. Shell Programming and Scripting

Append each line to next previous line in a file

Hi all, Please help me in providing sample code to append the following 4 lines in one row. Input : A1/EXT "BAPBSC10/07B/00" 523 090530 0115 RXOCF-430 HY1711 1 EXTERNAL ALARM DOOR ALARM Output should be : A1/EXT "BAPBSC10/07B/00" 523 090530 0115 ... (8 Replies)
Discussion started by: sudhakaryadav
8 Replies

4. Shell Programming and Scripting

Append specific lines to a previous line based on sequential search criteria

I'll try explain this as best I can. Let me know if it is not clear. I have large text files that contain data as such: 143593502 09-08-20 09:02:13 xxxxxxxxxxx xxxxxxxxxxx 09-08-20 09:02:11 N line 1 test line 2 test line 3 test 143593503 09-08-20 09:02:13... (3 Replies)
Discussion started by: jesse
3 Replies

5. Shell Programming and Scripting

Append next line to previous line when one pattern not found

Hi, I need help for below scenario.I have a flat file which is having records seperated by delimiters which will represent each record for oracle table.My Control file will consider each line as one record for that table. Some of the lines are aligned in two/three lines so that records are... (4 Replies)
Discussion started by: kannansr621
4 Replies

6. Shell Programming and Scripting

Remove previous line if next & previous lines have same 4th character.

I want to remove commands having no output. In below text file. bash-3.2$ cat abc_do_it.txt grpg10so>show trunk group all status grpg11so>show trunk group all status grpg12so>show trunk group all status GCPKNYAIGT73IMO 1440 1345 0 0 94 0 0 INSERVICE 93% 0%... (4 Replies)
Discussion started by: Raza Ali
4 Replies

7. UNIX for Dummies Questions & Answers

How to remove fields space and append next line to previous line.?

awk 'BEGIN{FS = "Ç"} NR == 1 {p = $0; next} NF > 1 {print p; p = $0} NF <= 1 {p = (p " " $0)} END {print p}' input.txt > output.txt This is what the input data file looks like with broken lines Code: 29863 Ç890000000 Ç543209911 ÇCHNGOHG Ç000000001 Ç055 ... (4 Replies)
Discussion started by: cumeh1624
4 Replies

8. UNIX for Advanced & Expert Users

How to find a string in a line in UNIX file and delete that line and previous 3 lines ?

Hi , i have a file with data as below.This is same file. But actual file contains to many rows. i want to search for a string "Field 039 00" and delete that line and previous 3 lines in that file.. Can some body suggested me how can i do using either sed or awk command ? Field 004... (7 Replies)
Discussion started by: vadlamudy
7 Replies

9. Shell Programming and Scripting

Issue while append to previous line

Hi, I have data as below. 36578019,005-923887317,UNMDL,20151230,2C3CCAAG4GH135448,L,TX,20160108,62,"030916 PPT TX AFF RPRT VALID AFF IN PDP WLL FWD TO RYAN ON 031116 CB1619 ",, 36580219,611-923785453,FC,20151209,ZACCJABT9FPC19274,L,TX,20160108,83,,,... (4 Replies)
Discussion started by: JSKOBS
4 Replies

10. Shell Programming and Scripting

Remove new line starting with a numeric value and append it to the previous line

Hi, i have a file with multiple entries. After some tests with sed i managed to get the file output as follows: lsn=X-LINK-IN0,apc=661:0,state=avail,avail/links=1/1, 00,2110597,2094790,0,81,529,75649011,56435363, lsn=TM1ITP1-AM1ITP1-LS,apc=500:0,state=avail,avail/links=1/1,... (5 Replies)
Discussion started by: nms
5 Replies
UNIQ(1) 						      General Commands Manual							   UNIQ(1)

NAME
uniq - report repeated lines in a file SYNOPSIS
uniq [ -udc [ +n ] [ -n ] ] [ input [ output ] ] DESCRIPTION
Uniq reads the input file comparing adjacent lines. In the normal case, the second and succeeding copies of repeated lines are removed; the remainder is written on the output file. Note that repeated lines must be adjacent in order to be found; see sort(1). If the -u flag is used, just the lines that are not repeated in the original file are output. The -d option specifies that one copy of just the repeated lines is to be written. The normal mode output is the union of the -u and -d mode outputs. The -c option supersedes -u and -d and generates an output report in default style but with each line preceded by a count of the number of times it occurred. The n arguments specify skipping an initial portion of each line in the comparison: -n The first n fields together with any blanks before each are ignored. A field is defined as a string of non-space, non-tab charac- ters separated by tabs and spaces from its neighbors. +n The first n characters are ignored. Fields are skipped before characters. SEE ALSO
sort(1), comm(1) UNIQ(1)
All times are GMT -4. The time now is 09:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy