Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Splitting the files via shell script
Top Forums Shell Programming and Scripting Splitting the files via shell script Post 302437402 by Maya_Pillai on Thursday 15th of July 2010 03:00:56 AM
Old 07-15-2010
Splitting the files via shell script
Hi all,
We have 102 flat files created by Informatica from 102 tables. These 102 files contain pharmcy details.
There are a total of 450 pharmcyids.The naming convention for the flat file is ODS_<TABLE NAME>_yyyymmdd_timestamp.dat.
Each flat file may contain data for 450 pharmacies which is identified by pharmacy ids.
The requirement is to split each of the flat files to 450 pharmcyfiles(102*450).
1)The flat file name is dynamic which changes with date and time evry day
2)The flat file may or may not contain data for some pharmacies. At the maximum, it may conatin 450 pharmcy ids.
3)The output file name is dynamic with the format <pharmacyid>_TABLENAME_yyyymmdd_timestamp.
4)The location of pharmcyid in each input flatfile varies according to the source table structure.

eg:-The input flatfile - ODS_ADT_20100731_000001.dat contains 5 rows
pharmacyid description objectid reasoncode
A123 pharmacy1 101 null
b123 pharmacy2 102 null
C123 pharmacy3 103 null
A123 pharmacy1 104 null
B123 pharmcy 105 null

These data need to be split into 3 output flat files(as there as 3 distinct pharmacy ids)
namely A123_ADT_20100731_000002, B123_ADT_20100731_000002, c123_ADT_20100731_000002.

How can we generate a generic script to handle 102 input files (each 102 file may generate 450 pharmacy files )?

Any help on this is appreciated.

Thanks
Maya
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

splitting the files

Hi, I'am using HP-UX.I have a input file which has 102 drop statements in it.I'am using csplit to split the files.The upper limit is 99 only.I'am using the -n 102 option.It says "suffix size not vaild".Any suggestions how to do it using csplit? Thanx in advance, sounder. (1 Reply)
Discussion started by: sounder123
1 Replies

2. UNIX for Advanced & Expert Users

splitting the files

Hi, How can i split the big file by the lines?. For eg. I wanna split the file from the line 140 to 1700. (9 Replies)
Discussion started by: sharif
9 Replies

3. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

4. UNIX for Dummies Questions & Answers

splitting shell window

hi is it possible to split the shell window in two or more parts like split command vi? i am using putty as telnet client. (2 Replies)
Discussion started by: ilayans
2 Replies

5. Shell Programming and Scripting

Help with shell script - splitting

Hi, I need to split the file lines in below format as. Input file : Sample.txt <Rule expression="DeliverToCompID IS NULL" invert="true"> <Rule field="PossDupFlag" value="Y" > <Rule expression="OrdStatus = '2' AND OrigClOrdID IS NULL"> Output... (5 Replies)
Discussion started by: manosubsulo
5 Replies

6. Shell Programming and Scripting

Splitting files

Hello all I have a file which has around 80 million records, I want to split it to 12 equal files, I tried using the split command but it is allowing me to split according to number of lines or by size. Is there a way i can split the file into 12 files without worrying about the number of lines... (7 Replies)
Discussion started by: Sri3001
7 Replies

7. Shell Programming and Scripting

Splitting files into 100 files with field value

I want a script to split my file upon the last field (15) As file A,b,c,.......,01 C,v,n,.......,02 C,r,v,........,01 F,s,a,........,03 X,y,d,........,99 To make output 01.txt A,b,c,.......,01 C,r,v,........,01 02.txt C,v,n,.......,02 (12 Replies)
Discussion started by: teefa
12 Replies

8. Shell Programming and Scripting

Splitting a file and creating new files using Perl script

Hi All, I am new to Scripting language. I want to split a file and create several subfiles using Perl script. Example : File format : Sourcename ID Date Nbr SU IMYFDJ 9/17/2012 5552159976555 SU BWZMIG 9/14/2012 1952257857887 AR PEHQDF 11/26/2012 ... (13 Replies)
Discussion started by: Deepak9870
13 Replies

9. Shell Programming and Scripting

Script for splitting file of records into multiple files

Hello I have a file of following format HDR 1234 abc qwerty abc def ghi jkl HDR 4567 xyz qwerty abc def ghi jkl HDR 890 mno qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl -Need to split this into multiple files based on tag... (8 Replies)
Discussion started by: wincrazy
8 Replies

10. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

LEARN ABOUT DEBIAN

bp_bioflat_index

BP_BIOFLAT_INDEX(1p)					User Contributed Perl Documentation				      BP_BIOFLAT_INDEX(1p)

NAME

       bioflat_index.pl - index sequence files using Bio::DB::Flat

DESCRIPTION

	Create or update a biological sequence database indexed with the
	Bio::DB::Flat indexing scheme.	The arguments are a list of flat files
	containing the sequence information to be indexed.

USAGE

	bioflat_index.pl <options> file1 file2 file3...

	Options:

	    --create		  Create or reinitialize the index.  If not specified,
				  the index must already exist.

	    --format   <format>   The format of the sequence files.  Must be one
				  of "genbank", "swissprot", "embl" or "fasta".

	    --location <path>	  Path to the directory in which the index files
				  are stored.

	    --dbname <name>	  The symbolic name of the database to be created.

	    --indextype <type>	  Type of index to create.  Either "bdb" or "flat".
				  "binarysearch" is the same as "flat".

       Options can be abbreviated.  For example, use -i for --indextype.

       The following environment variables will be used as defaults if the corresponding options are not provided:

	    OBDA_FORMAT      format of sequence file
	    OBDA_LOCATION    path to directory in which index files are stored
	    OBDA_DBNAME      name of database
	    OBDA_INDEX	     type of index to create

perl v5.14.2							    2012-03-02						      BP_BIOFLAT_INDEX(1p)
All times are GMT -4. The time now is 07:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy