Breaking the files as 10k recs. per file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Breaking the files as 10k recs. per file
# 1  
Old 03-22-2010
Bug Breaking the files as 10k recs. per file

Hi,
I have a code as given below
Code:
Set -A _Category="A\
                          B\
                          C"
for _cat in ${_Category}
  do
    sed -e "s:<TABLE_NAME>:${_cat}:g" \
          -e "s:<date>:${_dt}:g" \
          ${_home}/skl/sq1.sql >> ${_dest}/del_${_dt}.sql
    fi
    _seq_cnt=`wc -l ${_dest}/seqlist_${_dt}.lst|awk '{print $1}'`
    if [[ ${_seq_cnt} -gt 999 ]] ; then
         _start=1
         _end=999
         while [[ ${_seq_cnt} -gt 0 ]]
         do
           cat ${_dest}/seqlist_${_dt}.lst|sed  -n "${_start},${_end}p"|awk '{if (NR==1 && substr($0,1,1)==",") {print sub
str($0,2)} else {print $0}}' >> ${_dest}/del_${_dt}.sql
           fi
            ((_seq_cnt =_seq_cnt-999))
            ((_start=_start+999))
            ((_end=_end+999))
            if [[ ${_seq_cnt} -gt 0 ]] ; then
                echo ")"  >> ${_dest}/del_${_dt}.sql
                echo " or  DW_AUTHOP_SEQ_NBR in (" >> ${_dest}/del_${_dt}.sql
             fi
            fi
         done
 echo "));" >> ${_dest}/del_${_dt}.sql

The above code forms a sql query with a field value(seq. nbr) taken from a text file "999" at a time and put it as a seperate condition.
Here my requirement is , each file should only have 10000 such seq. nbrs, and the file should be created like
${_dest}/del_${_dt}.1..sql
${_dest}/del_${_dt}.2..sql
${_dest}/del_${_dt}.n..sql until all the seq. nbrs. given in the text(${_dest}/seqlist_${_dt}.lst) file has been included in the condition.

Can anyone help me achieve this.
# 2  
Old 03-22-2010
Without trying to fully understand your logic, but
Code:
split -l

command at unix level would not work for you?
The split command breaks a file into pieces, and the -l option allows a user to specify number of lines.
# 3  
Old 03-22-2010
Hi,
Split is to split an already written file.
Here in the script, I am forming something and writing it into a file. My question is to how to form a file appending only 10000 seq. nos. per file?
# 4  
Old 03-22-2010
In theory, you'd pipe it through awk or something that tracks line or record numbers. However, the logic described in writing a query in chunks of 999 eludes me. Wouldn't you be spooling output from a query's resultset into an output file? At least there most rdbms systems offer a rowlimit or similar control. The method described seems a little upside down...
# 5  
Old 03-23-2010
Quote:
Originally Posted by mr_manii
Hi,
Split is to split an already written file.
Here in the script, I am forming something and writing it into a file. My question is to how to form a file appending only 10000 seq. nos. per file?
Split is to split already written files -- No, you can pipe the input and get output in different files.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Breaking large file into small files

Dear all, I have huge txt file with the input files for some setup_code. However for running my setup_code, I require txt files with maximum of 1000 input files Please help me in suggesting way to break down this big txt file to small txt file of 1000 entries only. thanks and Greetings, Emily (12 Replies)
Discussion started by: emily
12 Replies

2. Shell Programming and Scripting

Why sum of recs in awk don't match total rec count?

I'm using awk to determine if a field starting in position 604 for length of 10 is not equal to ALL spaces. It's searching several files which are in the current directory. The below awk indicates that there are 84 records on all files where this field IS NOT equal to ALL spaces ( there are 10... (2 Replies)
Discussion started by: mjf
2 Replies

3. UNIX for Dummies Questions & Answers

Breaking a fasta formatted file into multiple files containing each gene separately

Hey, I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time: for i in *.rtf.out do awk '/^>/{f=++d".fasta"} {print > $i.out}' $i done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies

4. Shell Programming and Scripting

FASTEST way to loop a script 10k times

Is there any FASTEST way to loop a script 10k times my script works likes this c-randomnumbers-script -i input1.bed -g g19 -e DB >> output1 I need to run this 10k times by using consecutive outputs to get my final output i.e, output10000 c-random-script -i input1.bed -g g19 -e DB >>... (5 Replies)
Discussion started by: quincyjones
5 Replies

5. UNIX for Dummies Questions & Answers

Breaking up a text file into lines

Hi, I have a space delimited text file that looks like the following: BUD31 YRI 2e-06:CXorf15 YRI 3e-06:CREB1 YRI 4e-06 FLJ21438 CEU 3e-07:ETS1 CEU 8e-07:FGD3 CEU 2e-06 I want to modify the text file so that everytime there is a ":", a new line is introduced so that the document looks... (3 Replies)
Discussion started by: evelibertine
3 Replies

6. Shell Programming and Scripting

File breaking

Hey, I have to take one CSV file and break into more files. Let's I have a file prices.csv and the data in the file like 1,12345 1,34567 1,23456 2,67890 2,77720 2,44556 2,55668 10,44996 based on the first column, I want to create files. in this example 1 is repeated three times... (12 Replies)
Discussion started by: bond2222
12 Replies

7. Shell Programming and Scripting

Breaking a file into three new files, character by character

I am new to shell scripting, and need a script to randomly distribute each character from a file into one of three new files. I also need each character to maintain it's position from the original file in the new file (such that if a character is written to File 1, Files 2 and 3 have spaces... (10 Replies)
Discussion started by: foxcastle
10 Replies

8. Shell Programming and Scripting

Breaking one file into many files based on first column?

Hi, I have a file that looks like this (tab deliminited). MAT1 YKR2 3 MAT1 YMR1 2 MAT1 YFG2 2 MAT2 YLM4 4 MAT2 YHL2 1 BAR1 YKR2 3 BAR1 YFR1 4 BAR1 YMR1 1 What I want to do is break this file down into multiple files. So the result will look like this: File 1... (2 Replies)
Discussion started by: kylle345
2 Replies

9. Shell Programming and Scripting

Breaking up a file

Hi, I have a file that looks like this - lets call it fileA >hhm2 IIIIIIIIILLLLLLLMMMMMMMMMNNNNNNNNNNGGGGGGHHHHHHHH >hhm4 OOOOOKKKKKKKKMMMMMHHHHHLLLLLLLLWWWWWWWWWWW >hhm9 OOOOOOOIIIIIIIIIKKKKKKKKKMMMMMHHHHHHHHHHHLLLLLLLLLL So the file is pretty straight forward. The name is indicated... (2 Replies)
Discussion started by: phil_heath
2 Replies

10. UNIX for Dummies Questions & Answers

how to cut selected 10k lines continuosly

I have a source file having 20lacs lines i want to cut first 1 lac lines from source file and redirect to other file (i.e after redirecting the output to other file the source file should have only 19 lacs count ). Pls help me in getting solution. Thanks in advance. (9 Replies)
Discussion started by: vamshi
9 Replies
Login or Register to Ask a Question