Sponsored Content
Top Forums Shell Programming and Scripting Append next line to previous lines when NF is less than 0 Post 302898208 by cumeh1624 on Sunday 20th of April 2014 11:31:23 AM
Old 04-20-2014
The command we had in the script that performs the next line append to previous line takes almost 4 hrs because of the while loop and for performance reason is why we are looking to use a faster command like awk command but your lines of code do not provide same data file count

The lines of code in the script that takes four hrs to complete has output data file count of 176060 more than what your lines of code will produce at 175044 data file count.

Below is what the lines of code that we had in the script looks like

Code:
FLAG=0
cat filename | while read CUR_LINE
do
if [[ $FLAG -ne 0 ]];then
If [[ `echo ${CUR_LINE} | awk -F "Ç" '{print NF -1}'` -le 0 ]];then
PREV_LINE="{PREV_LINE} ${CUR_LINE}"
NEW_LINE=`echo ${PREV_LINE} | tr -d '\n' | tr -d '^M'`
PREV_LINE=${NEW_LINE}
else
echo ${PREV_LINE} >> ${OUT_FILE}
PREV_LINE='${CUR_LINE}
fi
else
PREV_LINE=${CUR_LINE}
FLAG=1
fi
done
echo ${PREV_LINE} >> ${OUT_FILE}

what I'm looking is to reduce the 4 hrs completion time in which your line of code will do but the output file count is different and formatting is also different.

Please let me know if you have a suggestion to this issue.

Last edited by Scrutinizer; 04-20-2014 at 01:19 PM.. Reason: CODE tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Append line that does not contain pipe to it previous line

Hi All, I have a file which contains data as below When we see no pipe character in the line. append those lines to the previous line with pipe character till we get the next line with pipe character with ~(concat with ~) Input file looks like: 1080530944|001|john.l.bonner|Acknowledge|CN... (11 Replies)
Discussion started by: ainuddin
11 Replies

2. UNIX for Advanced & Expert Users

append the line with the previous if it not start with 1=

How to append the line with the previous if it not start with 1=. 1=ttt, 2=xxxxxx, 3=4545 44545, 4=66666, 1=ttt, 2=xxxxxx, 3=34434 3545, 4=66666, 5=ffffff 6=uuuuuuu, 7=ooooooo 1=ttt, 2=xxxxxx, 3=311343545, 4=66666 1=ttt, 2=xxxxxx, 5=XAXAXA, 7=FDFD (3 Replies)
Discussion started by: palsevlohit_123
3 Replies

3. Shell Programming and Scripting

Append each line to next previous line in a file

Hi all, Please help me in providing sample code to append the following 4 lines in one row. Input : A1/EXT "BAPBSC10/07B/00" 523 090530 0115 RXOCF-430 HY1711 1 EXTERNAL ALARM DOOR ALARM Output should be : A1/EXT "BAPBSC10/07B/00" 523 090530 0115 ... (8 Replies)
Discussion started by: sudhakaryadav
8 Replies

4. Shell Programming and Scripting

Append specific lines to a previous line based on sequential search criteria

I'll try explain this as best I can. Let me know if it is not clear. I have large text files that contain data as such: 143593502 09-08-20 09:02:13 xxxxxxxxxxx xxxxxxxxxxx 09-08-20 09:02:11 N line 1 test line 2 test line 3 test 143593503 09-08-20 09:02:13... (3 Replies)
Discussion started by: jesse
3 Replies

5. Shell Programming and Scripting

Append next line to previous line when one pattern not found

Hi, I need help for below scenario.I have a flat file which is having records seperated by delimiters which will represent each record for oracle table.My Control file will consider each line as one record for that table. Some of the lines are aligned in two/three lines so that records are... (4 Replies)
Discussion started by: kannansr621
4 Replies

6. Shell Programming and Scripting

Remove previous line if next & previous lines have same 4th character.

I want to remove commands having no output. In below text file. bash-3.2$ cat abc_do_it.txt grpg10so>show trunk group all status grpg11so>show trunk group all status grpg12so>show trunk group all status GCPKNYAIGT73IMO 1440 1345 0 0 94 0 0 INSERVICE 93% 0%... (4 Replies)
Discussion started by: Raza Ali
4 Replies

7. UNIX for Dummies Questions & Answers

How to remove fields space and append next line to previous line.?

awk 'BEGIN{FS = "Ç"} NR == 1 {p = $0; next} NF > 1 {print p; p = $0} NF <= 1 {p = (p " " $0)} END {print p}' input.txt > output.txt This is what the input data file looks like with broken lines Code: 29863 Ç890000000 Ç543209911 ÇCHNGOHG Ç000000001 Ç055 ... (4 Replies)
Discussion started by: cumeh1624
4 Replies

8. UNIX for Advanced & Expert Users

How to find a string in a line in UNIX file and delete that line and previous 3 lines ?

Hi , i have a file with data as below.This is same file. But actual file contains to many rows. i want to search for a string "Field 039 00" and delete that line and previous 3 lines in that file.. Can some body suggested me how can i do using either sed or awk command ? Field 004... (7 Replies)
Discussion started by: vadlamudy
7 Replies

9. Shell Programming and Scripting

Issue while append to previous line

Hi, I have data as below. 36578019,005-923887317,UNMDL,20151230,2C3CCAAG4GH135448,L,TX,20160108,62,"030916 PPT TX AFF RPRT VALID AFF IN PDP WLL FWD TO RYAN ON 031116 CB1619 ",, 36580219,611-923785453,FC,20151209,ZACCJABT9FPC19274,L,TX,20160108,83,,,... (4 Replies)
Discussion started by: JSKOBS
4 Replies

10. Shell Programming and Scripting

Remove new line starting with a numeric value and append it to the previous line

Hi, i have a file with multiple entries. After some tests with sed i managed to get the file output as follows: lsn=X-LINK-IN0,apc=661:0,state=avail,avail/links=1/1, 00,2110597,2094790,0,81,529,75649011,56435363, lsn=TM1ITP1-AM1ITP1-LS,apc=500:0,state=avail,avail/links=1/1,... (5 Replies)
Discussion started by: nms
5 Replies
rl(1)								   User Commands							     rl(1)

NAME
rl - Randomize Lines. SYNOPSIS
rl [OPTION]... [FILE]... DESCRIPTION
rl reads lines from a input file or stdin, randomizes the lines and outputs a specified number of lines. It does this with only a single pass over the input while trying to use as little memory as possible. -c, --count=N Select the number of lines to be returned in the output. If this argument is omitted all the lines in the file will be returned in random order. If the input contains less lines than specified and the --reselect option below is not specified a warning is printed and all lines are returned in random order. -r, --reselect When using this option a single line may be selected multiple times. The default behaviour is that any input line will only be selected once. This option makes it possible to specify a --count option with more lines than the file actually holds. -o, --output=FILE Send randomized lines to FILE instead of stdout. -d, --delimiter=DELIM Use specified character as a "line" delimiter instead of the newline character. -0, --null Input lines are terminated by a null character. This option is useful to process the output of the GNU find -print0 option. -n, --line-number Output lines are numbered with the line number from the input file. -q, --quiet, --silent Be quiet about any errors or warnings. -h, --help Show short summary of options. -v, --version Show version of program. EXAMPLES
Some simple demonstrations of how rl can help you do everyday tasks. Play a random sound after 4 minutes (perfect for toast): sleep 240 ; play `find /sounds -name '*.au' -print | rl --count=1` Play the 15 most recent .mp3 files in random order. ls -c *.mp3 | head -n 15 | rl | xargs --delimiter=' ' play Roll a dice: seq 6 | rl --count 2 Roll a dice 1000 times and see which number comes up more often: seq 6 | rl --reselect --count 1000 | sort | uniq -c | sort -n Shuffle the words of a sentence: echo -n "The rain in Spain stays mainly in the plain." | rl --delimiter=' ';echo Find all movies and play them in random order. find . -name '*.avi' -print0 | rl -0 | xargs -n 1 -0 mplayer Because -0 is used filenames with spaces (even newlines and other unusual characters) in them work. BUGS
The program currently does not have very smart memory management. If you feed it huge files and expect it to fully randomize all lines it will completely read the file in memory. If you specify the --count option it will only use the memory required for storing the specified number of lines. Improvements on this area are on the TODO list. The program uses the rand() system random function. This function returns a number between 0 and RAND_MAX, which may not be very large on some systems. This will result in non-random results for files containing more lines than RAND_MAX. Note that if you specify multiple input files they are randomized per file. This is a different result from when you cat all the files and pipe the result into rl. COPYRIGHT
Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 Arthur de Jong. This is free software; see the license for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Version 0.2.7 Jul 2008 rl(1)
All times are GMT -4. The time now is 04:37 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy