Read multiple lines at a time from file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read multiple lines at a time from file
# 1  
Old 07-03-2013
Read multiple lines at a time from file

Hello All,

I have a file like below....
HTML Code:
dn: cn=user1,ou=org,o=org
cn=user1
uid=user1

cn=user2,ou=org,o=org
cn=user2
uid=user2

cn=user3,ou=org,o=org
cn=user3
cn=user33
uid=user3

cn=user4,ou=org,o=org
cn=user4
uid=user4
I want to read lines until first blank line in first attempt put those lines into seperate variable and then do something and then read line from first empty line to second empty line, put those lines into seperate variable and then do something...etc. I want to this to the end of the file. It could be 3-5 lines. How can I do this. Please assist. Thanks

Last edited by s_linux; 07-03-2013 at 11:55 AM..
# 2  
Old 07-03-2013
A tricky question indeed. It depends on what you are then going to go on and do, but if you are on AIX, there is a -p flag on grep to get you a paragraph. I'm not sure how many variants have it available through.

If the file is not large, you could:-
Code:
#!/bin/ksh
while read line
do
   if [ "$line" = "" ]
   then
      <handle new section>
   else
      <continue processing section>
   fi
done < file

It will be slow for larger files, but as an alternate you could split the file into multiple smaller ones based on the blank line, then process them in turn:-
Code:
#!/bin/ksh

mkdir $$
cd $$
typeset -Z12 i=0

linecnt=`grep -c file`
((linecnt=$linecnt+1))
csplit -n 6 ../file "/^$/" {$linecnt}

Then, loop through the files generated. You may need to consider the length of the file name counter (-n 6 in my example) and the red double dots are only if the file is in the current directory. You may have to adjust that to suit too.

You can then (because you are in a new directory with only your data in it) run a loop to process the files as you wish. I would recommend against a for file in * construct if you expect a lot of files as this may exceed the command line limit. You have the counter, so probably best to use that:-

Code:
typeset -Z12 file_num=0
while [ $file_num -le $linecnt ]
do
   ((file_num=$file_num+1))
   ....whatever....
done


If you are looking for just certain patterns, you could get a list of file names with a grep from this point and work through them too.



I hope that this helps.



Robin
Liverpool/Blackburn
UK
# 3  
Old 07-03-2013
The first version seems the most sensible and efficient, really.
# 4  
Old 07-03-2013
try also:
Code:
#!/bin/ksh
set +A arr
c1=0
awk '$1=$1' RS="" infile | while read rec
do
  arr[$c1]="$rec"
  (( c1 = c1 + 1 ))
done
c2=0
while [ $c2 -lt $c1 ]
do
   echo "${arr[$c2]}"
   (( c2 = c2 + 1 ))
done

# 5  
Old 07-03-2013
Code:
$ awk -v RS='\n\n' -v ORS='\n\n' ' { print NR":" $0 } ' file
1:dn: cn=user1,ou=org,o=org
cn=user1
uid=user1

2:cn=user2,ou=org,o=org
cn=user2
uid=user2

3:cn=user3,ou=org,o=org
cn=user3
cn=user33
uid=user3

4:cn=user4,ou=org,o=org
cn=user4
uid=user4

# 6  
Old 07-03-2013
@anbu: you can do this more reliably, using RS= instead of RS='\n\n'. Also, standard awk can only use a single character for RS.
These 2 Users Gave Thanks to Scrutinizer For This Post:
# 7  
Old 07-08-2013
Thanks All for your replies...
I started putting something together from first version of the code from "rbatte1" and it worked except that i had few issues with the loop and code that i put in to do something but all those resolved now and it took about 35mins to finish up some compares with 155k users file. By the way, I'm on RHEL and using bash script. Anyways thanks all for your support. Really helped.
This User Gave Thanks to s_linux For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to read multiple files at same time through UNIX scripting?

How to read multiple files at simultaneously? (1 Reply)
Discussion started by: Priyanka_M
1 Replies

2. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

3. Shell Programming and Scripting

Read two lines at time from a file

Hello community, what I need to do is read 2 rows at time from a file. I have this simple solution: File to read: LINE1 LINE2 LINE3 LINE4 LINE5 LINE6 LINE7 LINE8Read routine:#!/bin/ksh sed '1,3d' /out.txt | while read line; do read line2 echo $line $line2 doneResult:LINE1... (5 Replies)
Discussion started by: Lord Spectre
5 Replies

4. Shell Programming and Scripting

read the lines of multiple files

I am trying to create a script which will read 2 files and use the lines of file 1 for each line on file 2. here's my sample code cat $SBox | while read line do cat $Date | while read line do $SCRIPTEXE <line from first file> $2 <line from 2nd file> ... (12 Replies)
Discussion started by: khestoi
12 Replies

5. Shell Programming and Scripting

read one line file and separate into multiple lines

I have one long line text with semicolon used as separator between values in that line. Now, I want to separate the line into multiple line right after every 29th field. example input line: ... (1 Reply)
Discussion started by: erlanq
1 Replies

6. Shell Programming and Scripting

read one line file and separate into multiple lines

I have one long line text with semicolon used as separator between values in that line. Now, I want to separate the line into multiple line right after every 29th field. example input line: ... (2 Replies)
Discussion started by: erlanq
2 Replies

7. Shell Programming and Scripting

How to read a multiple lines from a file n executing them?

Hi all, I am just trying to read the contents of a file. basically this file has a list of dat files. then i want to access these dat files n execute a script on them one by one using a loop. i hav e written like this ls -l | cut -c 58-88 > file1.txt while do arr1="$( sed -n '1p'... (7 Replies)
Discussion started by: navjyotisonu5
7 Replies

8. Shell Programming and Scripting

Read multiple arguments in for loop each time

Hi, Guys I am new to shell programming and just get stuck with one simple question. please kindly help. According to the tutorial here, we can do something like for NODE in "ABC 10" "EFG 20" do set -- $NODE echo "letter is $1, number is $2" done And the result will... (3 Replies)
Discussion started by: yuanli
3 Replies

9. Shell Programming and Scripting

How to read and compare multiple fields in a column at the same time

Hi, Currently I am coding up a nasty way of reading file input using *cat* rather than *read*. My text input looks like TextA 100 TextB 110 TextC 120 Currently I am using cat |while read line to read the first column and second column fields. cat foo.txt|while read line do ... (1 Reply)
Discussion started by: ahjiefreak
1 Replies

10. Shell Programming and Scripting

read and match multiple lines in perl

Could any one tell me how to read and match multiple lines in perl? Did this code below still work in this situation? while (<FILE>) { if (/ /) { } } Thanks a lot! (5 Replies)
Discussion started by: zx1106
5 Replies
Login or Register to Ask a Question