07-09-2009
Can I split a 10GB file into 1 GB sizes using my repeating data pattern
I'm not a unix guy so excuses my ignorance... I'm the database ETL guy.
I'm trying to be proactive and devise a plan B for a ETL process where I expect a file 10X larger than what I process daily for a recast job. The ETL may handle it but I just don't know.
This file may need to be split and we don't want to lose related data. I assume it would be easier to do it at the unix level rather than the etl tool providing there are no limitations to file sizes with the unix commands.
The file will most likely be 10GB +- a few GB. It is unknown at this time
The basic file format is as follows with the first 3 characters being the record type (100,401,404,410,411)
The file must be split into segments equal to a daily run approximately 1gb in size and it has to occur just before a 100 record as all the rows that follow a 100 belong together.
1001104vvbvnbvd
4011104ghghghgh
404111kjdkfjkdf
404111kjdkfjkdf
404111kjdkfjkdf
404111kjdkfjkdf
4103445kkjkljlk
4103445kkjkljlk
4113445kkjkljlk
4043445kkjkljlk
10011ffgfgg1250
4011104fffhghgh
404111kjddfjkdf
404111kjdkrtrdf
etc...
thanks in advance. I think we use HP-UX
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this.
For example:
split -l 3000000 filename.txt
This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies
2. Shell Programming and Scripting
Dear all,
I have a large file which is composed of 8000 frames, what i would like to do is split the file into 8000 single files names file.pdb.1, file.pdb.2 etc etc
each frame in the large file is seperated by a "ENDMDL" flag so my thinking is to use this flag a a point to split the files... (4 Replies)
Discussion started by: Mish_99
4 Replies
3. Shell Programming and Scripting
I want a shell script that will traverse a file system starting at specific path.
And look at all file names for repeating sequences of and remove them from the file name.
The portion of the name that gets removed has to be a repeating sequence of the same characters.
So the script would... (3 Replies)
Discussion started by: z399y
3 Replies
4. Shell Programming and Scripting
Hello!
Have some problem with extract files from saved session.
File contains any kind of special/printable characters.
DATA NumberA DATA
DATA Begin
DATA1.1
DATA1.2 NumberB1 DATA1.3
DATA1.4
End DATA
DATA
DATA Begin
DATA2.1
DATA2.2 NumberB2 DATA2.3
DATA2.4
End DATA
DATA
... (4 Replies)
Discussion started by: vvild
4 Replies
5. UNIX for Dummies Questions & Answers
I want to extract the last rows of a data file, similar to that one below:
C1 xxx
C2 rrr
C3 ttt
....
Cn-1 hhh
Cn bbb
C1 yyy
C2 sss
C3 uuu
...
Cn-1 iii
Cn ccc
...
I just want to extract the final rows between C1 and Cn at each data file. n is not a constant,... (2 Replies)
Discussion started by: natasha
2 Replies
6. Shell Programming and Scripting
Hi,
I have an sqlplus output file using the character ';' as a delimiter and I would like to replace the fields without datas (i.e delimited by ';;') by ';0;'
Example: my sqlplus output:
11;22;33;44;;;77;;
What I would like to have:
11;22;33;44;0;0;77;0;
Thanks in advance for your... (2 Replies)
Discussion started by: popesk
2 Replies
7. Solaris
Hi Team
I have one 10 Gb log file
I want to split it into say 10 of 1-1Gb file
pls share ur experiences how to do this?
Thanks in advance, (3 Replies)
Discussion started by: zimmyyash
3 Replies
8. Shell Programming and Scripting
Hi ,
I have huge files around 400 mb, which has clob data and have diffeent scenarios:
I am trying to pass scenario number as parameter and and get required modified file based on the scenario number and criteria.
Scenario 1:
file name : scenario_1.txt
... (2 Replies)
Discussion started by: sol_nov
2 Replies
9. Shell Programming and Scripting
I need to send email to receipient in each block of data in a file which has the sender address under TO and just send that block of data where it ends as COMPANY.
I tried to work this out by getting line numbers of the string HELLO but unable to grab the next block of data to send the next... (5 Replies)
Discussion started by: loggedout
5 Replies
10. UNIX for Advanced & Expert Users
Hello All,
I have records in a file in a pattern A,B,B,B,B,K,A,B,B,K
Is there any command or simple logic I can pull out records into multiple files based on A record? I want output as
File1: A,B,B,B,B,K
File2: A,B,B,K (9 Replies)
Discussion started by: deal1dealer
9 Replies
LEARN ABOUT DEBIAN
rplay.hosts
RPLAY.HOSTS(5) File Formats Manual RPLAY.HOSTS(5)
NAME
rplay.hosts - rplay host authentication database
SYNOPSIS
/usr/local/etc/rplay.hosts
DESCRIPTION
The rplay.hosts file contains a list of hosts and access permissions which rplayd uses to validate incoming connections. Each line is of
the form:
hostname[:permission]
where
hostname is the name of a remote host or a host's IP address. Wildcards can be used within IP addresses to match multiple hosts.
The wildcard character is "*".
permission is an optional field containing any combination of the following characters:
r the host can read sounds.
w the host can write sounds.
x the host can play, stop, pause, and continue sounds. This is normally called execute permission.
m the host can monitor the audio stream written by rplayd to the audio device.
"rx" permissions are used when no permissions are specified.
EXAMPLE
#
# All hosts have read access:
#
*:r
#
# Trusted hosts:
#
nice-guy.sdsu.edu:rwx
friend.sdsu.edu:rwx
amigo.sdsu.edu:rwx
130.191.224.224:rwx
#
# Hosts which can read and execute:
#
foo.bar.com:rx
130.190.*:rx
146.244.234.*:rx
using.default.perms.edu
#
# Hosts which can monitor the audio stream
#
nsa.sdsu.edu:m
FILES
/usr/local/etc/rplay.hosts
SEE ALSO
rplayd(1)
BUGS
The permissions for an "*" entry apply to all matching hosts. The order of this file does not matter. Specific hosts can have access dis-
abled with entries like:
bad.guy.edu:
however, any matching "*" entries will still apply.
12/21/97 RPLAY.HOSTS(5)