Thanks for the reply. Below are the information you asked for :
I'm running Linux kernel 2.6.39.
The script I'm trying to write is for bash. I'm comfortable working with Awk/sed etc.
The actual table that I'm querying is something like below
There are about 10K rows in this table, and only the URL column needs to be dumped into a file and sent to an FTP server. There should be a check if the file is more than 10MB in size, and if it is, the file needs to be splitted into smaller files, each of size 10MB or lower.
The structure of the output file will be just something simple like this:
In case the size exceeds 10MB, splitting this file will be easy with the SPLIT function, based on the size limit. But the problem is that each URL XML is extremely large, and the output file has got it's limit on the length of a single line, so the output file is being generated as something like below
I found it tricky to split this file, since the SPLIT function won't understand XML tags.
I have been successfully splitting file with sample data in my table where the XML length is much smaller. As in the above example, my script works perfectly. However, it's the Production data that's causing the problem.
Quote:
Originally Posted by Chubler_XL
how about this:
Just be careful awk and many other unix utilities have limits on the length of a single line you may be better off putting a newline character after each </URL>
---------- Post updated at 10:17 AM ---------- Previous update was at 10:06 AM ----------
Depending on your OS the stat command I used above may not be available. A much more portable (but possible less efficient) version would be:
Thanks a lot Chubler_XL. I'll surely try out your idea. It looks good to me. I've found that in my version of Linux, the STAT command is available. I'll try this out and let you know.
Hello gurus,
I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files.
e.g. my data is like:
Row_Num,... (6 Replies)
I need to write a shell script for below scenario
My input file has data in format:
qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43
qwerty0101CFG 12345... (19 Replies)
Hi Experts,
I have to split huge file based on the pattern to create smaller files. The pattern which is expected in the file is:
Master.....
First...
second....
second...
third..
third...
Master...
First..
second...
third...
Master...
First...
second..
second..
second..... (2 Replies)
I am trying to update an older program on a small cluster. It uses individual files to send jobs to each node. However the newer database comes as one large file, containing over 10,000 records. I therefore need to split this file. It looks like this:
HMMER3/b
NAME 1-cysPrx_C
ACC ... (2 Replies)
HI All,
I have to split a xml file into multiple xml files and append it in another .xml file. for example below is a sample xml and using shell script i have to split it into three xml files and append all the three xmls in a .xml file. Can some one help plz.
eg:
<?xml version="1.0"?>... (4 Replies)
I will simplify the explaination a bit, I need to parse through a 87m file -
I have a single text file in the form of :
<NAME>house........
SOMETEXT
SOMETEXT
SOMETEXT
.
.
.
.
</script>
MORETEXT
MORETEXT
.
.
. (6 Replies)
Hello All ,
Please help me with below requirement
I want to split a xml file based on tag.here is the file format
<data-set>
some-information
</data-set>
<data-set1>
some-information
</data-set1>
<data-set2>
some-information
</data-set2>
I want to split the above file into 3... (5 Replies)
Hi Everyone,
I'm new here and I was checking this old post:
/shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html
(cannot paste link because of lack of points)
I need to do something like this but understand very little of perl.
I also check... (4 Replies)
Hi,
I'm having a xml file with multiple xml header. so i want to split the file into multiple files.
Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix.
eg :
<?xml version="1.0" encoding="UTF-8"?>
<ml:individual... (3 Replies)