Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parse large file on line count (random lines)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse large file on line count (random lines)
# 1  
Old 11-01-2011
Parse large file on line count (random lines)
I have a file that needs to be parsed into multiple files every time there line contains a number 1. the problem i face is the lines are random and the file size is random. an example is that on line 4, 65, 187, 202 & 209 are number 1's so there has to be file breaks between all those to create 4 new files. I can find the number 1's using sed and i can create 1 file based on the first line count but not the continuing ones.

Does anyone know how to parse a file based on random lines

Thanks ahead of time
# 2  
Old 11-01-2011
Do you have some example input/output?
# 3  
Old 11-02-2011
I do, its a bit long but this should give the right idea

Code:
 MM  PUT BATCH LETTER WRITER LETTERHEAD
 MM  AND CARRIAGE TAPE # Z2551Z11 ON PRINTER FOR
 MM  CLIENTXXXXXXXXX
 MM  SET CHANNEL # 1 FIRST PRINT LINE.
 MMH SET PRINT POSITION # 1 ONE HALF INCH FROM LEFT EDGE OF FORM.
1



 October 11, 2011



 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxx, XX XXXXX



 RE CCCCCCCCC No: XXXXXXX
 Past Due Since: 00-00-00
 Total Amount Past Due: $.00

 Aviso importante para las personas que hablan espanol: Esta
 notificacion es de suma importancia.  Puede afectar su derecho
 a continuar viviendo en su casa.  Si no entiende el continido

1



 October 11, 2011



 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Xxxxxxxxxxxxxxxxxxxxx, XX XXXXX



 RE CCCCCCCCC No: XXXXXXX
 Past Due Since: 00-00-00
 Total Amount Past Due: $.00

 Aviso importante para las personas que hablan espanol: Esta
 notificacion es de suma importancia.  Puede afectar su derecho
 a continuar viviendo en su casa.  Si no entiende el continido
 notificacion es de suma importancia.  Puede afectar su derecho
 a continuar viviendo en su casa.  Si no entiende el continido
 notificacion es de suma importancia.  Puede afectar su derecho
 a continuar viviendo en su casa.  Si no entiende el continido

1

As you can see the number 1's are at lines 6, 31 and 60 so i would need 2 files created, one from like 6 to 31 and 1 from 31 to 60 and so on
# 4  
Old 11-02-2011
With what filenames?
# 5  
Old 11-02-2011
Any file name would be fine, if it creates the files from there i can do what ever with them, i just am having the issue with the create part. Something in numeric order or random is fine.
# 6  
Old 11-02-2011
Code:
currfile=1
while read line
do
   if [ "$line" = "1" ]
   then
      (( currfile++ ))
   else
      echo "$line" >> ${currfile}.txt
   fi
done < infile.txt

This User Gave Thanks to CarloM For This Post:
darbs121
# 7  
Old 11-02-2011
MySQL
Thank You! Worked like a champ!!!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parse for 2 numbers in large single line

Hi All, I am writing a script in which I need to gather 2 numbers for 'total' and 'successful'. The goal is to compare the two numbers and if they are not equal, rerun the task until all are successful. I'm thinking the best way will be with awk or sed, but I really don't know where to begin... (8 Replies)
Discussion started by: hburnswell
8 Replies

2. Shell Programming and Scripting

Want to count the number of lines after the first line

hi, How can i count the number of lines after the first line in a flat file in unix? Say i have a flat file with a header like: Student Name Student ID .... Tnx (7 Replies)
Discussion started by: reignangel2003
7 Replies

3. Shell Programming and Scripting

Random shuffle of lines of a TXT file

Hello friends, I have a TXT file with 300 lines in it. I need to shuffle all the lines (randomly) so that they get into different order. Can anyone pls provide easy way, if any? I got it done by doing this below but I see it very lengthy/inefficient way. call random function to generate... (2 Replies)
Discussion started by: prvnrk
2 Replies

4. UNIX for Advanced & Expert Users

Count number of lines between a pattern in a large file

1000CUS E Y4NYRETAIL 10010004HELIOPOLIS 110000500022360591000056XX EG 1101DEBY XXAD ZSSKY TSSROS 1102HANYNNYY@HOTMAIL.COM 210030/05/201301/06/2013AED 3100 OPE 3100 CLO 3100 The 1000CUS E Y NYCORPORATE 10010004HELIOPOLIS 110000500025270504550203XX EG 1101XXXQ FOR CTING AND... (1 Reply)
Discussion started by: john2022
1 Replies

5. Shell Programming and Scripting

Count lines separated by new line

Hi guys, I have a file which has random records like mentioned below emcpower28a pci@3,03 (disk physical name) pci@3,04 emcpower9a pci@1,03 pci@2,03 pci@3,01 pci@4,03 there could be any number of disk names for any LUN (emc...) So, I want a solution to count disk names for its... (12 Replies)
Discussion started by: prashant2507198
12 Replies

6. Shell Programming and Scripting

Parse file from remote server to calculate count of string existence in that file

Hi I need to parse the file of same name which exist on different servers and calculate the count of string existed in both files. Say a file abc.log exist on 2 servers. I want to search for string "test" on both files and calculate the total count of search string's existence. For... (6 Replies)
Discussion started by: poweroflinux
6 Replies

7. Shell Programming and Scripting

How to Pick Random records from a large file

Hi, I have a huge file say with 2000000 records. The file has 42 fields. I would like to pick randomly 1000 records from this huge file. Can anyone help me how to do this? (1 Reply)
Discussion started by: ajithshankar@ho
1 Replies

8. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

9. Shell Programming and Scripting

Parse and count lines

I have a data file in the following format (refer to input file) with multiple lines containing some information. I need an output file to loop thorough the input file with summarized information as seen below (refer to output file) ‘Date Time' and ‘Beta Id' input file values should be concatenated... (7 Replies)
Discussion started by: shekharaj
7 Replies

10. Shell Programming and Scripting

Count specific character(s) very large file

I'm trying to count the number of 2 specific characters in a very large file. I'd like to avoid using gsub because its taking too long. I was thinking something like: awk '-F' { t += NF - 1 } END {print t}' infile > outfile which isn't working Any ideas would be great. (3 Replies)
Discussion started by: dcfargo
3 Replies
Login or Register to Ask a Question