|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Creating file from an existing file using CUT, is it the best option?
Post 302526054 by abhishekakaomi on Monday 30th of May 2011 07:01:51 AM
05-30-2011
Creating file from an existing file using CUT, is it the best option?
Dear All,
I have a requirement in which i have to load a file placed in FTP location onto my database. The process i'll follow is as below:
1) Get the files using FTP.
2) Create the desired load files as i have to load only 19 fields out of the 104 available in the file.
The fields i require can only be distinguished using their position. E.g 1-3 is one value, 5th field is another, 16-33 is another and so on for 19 fields.
I'm using CUT command to create a new file which will have "|" as delimiter. So while cutting the fields i'm appending the delimiter.
3) The file created above will then be used to load into database, pipe "|" being the delimiter using SQLLDR utility.
My concern is the step no. 2), as even for a file of length 10000 (each line 3159), it takes around 20 minutes
. And i have to process around >7500000 records per day.
Is there any other method to achieve wht i want in a much shorter time
?
I'm new to Unix, so please point out if you see any area for improvement.
Unix box is : SunOS 5.8
Thanks!!
I have a requirement in which i have to load a file placed in FTP location onto my database. The process i'll follow is as below:
1) Get the files using FTP.
2) Create the desired load files as i have to load only 19 fields out of the 104 available in the file.
The fields i require can only be distinguished using their position. E.g 1-3 is one value, 5th field is another, 16-33 is another and so on for 19 fields.
I'm using CUT command to create a new file which will have "|" as delimiter. So while cutting the fields i'm appending the delimiter.
3) The file created above will then be used to load into database, pipe "|" being the delimiter using SQLLDR utility.
My concern is the step no. 2), as even for a file of length 10000 (each line 3159), it takes around 20 minutes
. And i have to process around >7500000 records per day.Is there any other method to achieve wht i want in a much shorter time
?I'm new to Unix, so please point out if you see any area for improvement.
Unix box is : SunOS 5.8
Thanks!!
Last edited by abhishekakaomi; 05-30-2011 at 08:03 AM.. Reason: Updating
|
|
7 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi I am Newbie to Unix.Appreciate Help from forum
user would loada b.Csv File(Below example) in /data/m/ directory.Program need to read the b.csc to extract certain column and create a new file /data/d/ directory as csv file with new name.
User File Format
1232,samshouston,12345... (3 Replies)
Discussion started by: skywayterrace
3 Replies
2. Shell Programming and Scripting
With the help of this forum, I have a script with the following output:
chr7 27104483 27105154
chr7 27106872 27110789
chr7 27111956 27112830
chr7 27114388 27125180
chr7 27126966 27131260
chr7 27135440 27137796
which was created by the following script:
awk '1 == NR || $NF >= 1000 {... (6 Replies)
Discussion started by: awknerd
6 Replies
3. Shell Programming and Scripting
I want to look into a folder to see if there are any folders within it. If there are, I need to check inside each folder to see if it contains a .pdf file
So
If /myserver/myfolder/
contains a folder AND that folder conatins a .pdf file
do X
Else
do Z
I may have multiple folders and... (4 Replies)
Discussion started by: crowman
4 Replies
4. Shell Programming and Scripting
Hello Guys ,
I need an another help regarding the below problem.
I want to create a new file based on the existing file ,where two columns will be changed according to user input .(say column 4 and column 5)
Please let me know how to proceed with
Thanks (3 Replies)
Discussion started by: Pratik4891
3 Replies
5. Shell Programming and Scripting
Hi All,
I am very new to shell scripting .
In my current task i want to create .sh file that will rename the existing file with appending _bu in it. And then copy new file .
e.g if i have file linuxFirst.java then i want to rename it to linuxFirst_bu.java ..Then want replace with latest... (1 Reply)
Discussion started by: maheshkaranjkar
1 Replies
6. Shell Programming and Scripting
I have a text file with contents like this:
a,b,c,
d~e,f,g,h~i,j
,k,l,m~n,o,p,q~
I need to convert this file into this format unix shell script commands:
a,b,c,d~
e,f,g,h~
i,j,k,l,m~
n,o,p,q~
as you may have noticed, I need to retain the ~ signs at the end.
Any help is greatly... (3 Replies)
Discussion started by: harsha1238
3 Replies
7. UNIX for Advanced & Expert Users
Hi
I need some help with a task, i am an absolute newbie to any form of shell scripting and request guidance.
I have been building a proxy server using the apache mod proxy currently my solution is working , but i need to automate the process , suppose if any changes need to be made on... (0 Replies)
Discussion started by: satej
0 Replies
LEARN ABOUT REDHAT
cut
CUT(1) FSF CUT(1) NAME
cut - remove sections from each line of files SYNOPSIS
cut [OPTION]... [FILE]... DESCRIPTION
Print selected parts of lines from each FILE to standard output. Mandatory arguments to long options are mandatory for short options too. -b, --bytes=LIST output only these bytes -c, --characters=LIST output only these characters -d, --delimiter=DELIM use DELIM instead of TAB for field delimiter -f, --fields=LIST output only these fields; also print any line that contains no delimiter character, unless the -s option is specified -n with -b: don't split multibyte characters -s, --only-delimited do not print lines not containing delimiters --output-delimiter=STRING use STRING as the output delimiter the default is to use the input delimiter --help display this help and exit --version output version information and exit Use one, and only one of -b, -c or -f. Each LIST is made up of one range, or many ranges separated by commas. Each range is one of: N N'th byte, character or field, counted from 1 N- from N'th byte, character or field, to end of line N-M from N'th to M'th (included) byte, character or field -M from first to M'th (included) byte, character or field With no FILE, or when FILE is -, read standard input. AUTHOR
Written by David Ihnat, David MacKenzie, and Jim Meyering. REPORTING BUGS
Report bugs to <bug-coreutils@gnu.org>. COPYRIGHT
Copyright (C) 2002 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU- LAR PURPOSE. SEE ALSO
The full documentation for cut is maintained as a Texinfo manual. If the info and cut programs are properly installed at your site, the command info cut should give you access to the complete manual. cut (coreutils) 4.5.3 February 2003 CUT(1)