|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
efficiently split a 2GB text file into two
Post 302140710 by ihot on Monday 15th of October 2007 03:56:35 PM
10-15-2007
efficiently split a 2GB text file into two
Can an expert kindly write an efficient Linux ksh script that will split a large 2 GB text file into two?
Here is a couple of sample record from that text file:
"field1","field2","field3",11,22,33,44
"TG","field2b","field3b",1,2,3,4
The above rows are delimited by commas.
This script is to search the first field for the word "TG". If that row is present, it is to load that row to a TG.txt file. If this first field is not "TG", then it is to load that row into a NoTG.txt file.
So the result is a new TG.txt with the following row:
"TG","field2b","field3b",1,2,3,4
and a new NoTG.txt with the following row:
"field1","field2","field3",11,22,33,44
Thanks in advance. This forum rocks - with lots of helpful heroes!!
Here is a couple of sample record from that text file:
"field1","field2","field3",11,22,33,44
"TG","field2b","field3b",1,2,3,4
The above rows are delimited by commas.
This script is to search the first field for the word "TG". If that row is present, it is to load that row to a TG.txt file. If this first field is not "TG", then it is to load that row into a NoTG.txt file.
So the result is a new TG.txt with the following row:
"TG","field2b","field3b",1,2,3,4
and a new NoTG.txt with the following row:
"field1","field2","field3",11,22,33,44
Thanks in advance. This forum rocks - with lots of helpful heroes!!
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello!
Firts of all, I'm sorry for my English.
My problem:
I have text file with few Form Feed symbols (FF, ASCII code =12) inside (for example - some report, consists of some pages for
printing).
I want to split this text by pages - each page (until FF symbol)
in single file.
I... (2 Replies)
Discussion started by: ranri
2 Replies
2. Shell Programming and Scripting
Hi all
I have written Perl script to swap the strings in the second a third column from a text file.
My input file format is :
the|empty|the|det lake|empty|lake|conj_and was|empty|was|auxpass drained|empty|drained|conj_and birds|empty|bird|s|nn
The expected output file format is... (11 Replies)
Discussion started by: my_Perl
11 Replies
3. UNIX for Dummies Questions & Answers
Hi,
Often when I use echo statements in scripts I echo a line of #'s above and below. For example:
echo #####
echo hello world
echo #####
However, I generally have a series of about 75 #'s. For example:
echo #(x 75)
echo hello world
echo #(X 75)
While this helps to delineate... (7 Replies)
Discussion started by: msb65
7 Replies
4. Shell Programming and Scripting
Hi
I am using shell script where I am calling SQLPLUS and executing one PL/SQL block.
This PL/SQL block generates the spool file for example splfile.txt.
After successful generation of spool file I use nawk to split this file into 2 different files. Till here no issues.
nawk... (1 Reply)
Discussion started by: shekharjchandra
1 Replies
5. Shell Programming and Scripting
Hi all,
I am very new to shell scripting and some help is greatly appreciated.
I have 10 column based text files, i would like to split each of them into 6 files ; the 1st one having columns 1, 2 ,3,4 , the second one having columns 1,2,8,9 etc.
Is there a way I could get 60 files out my... (3 Replies)
Discussion started by: shreymuk
3 Replies
6. Shell Programming and Scripting
chr1 412573 . A C 2758.77 . AC=2;AF=1.00;AN=2;DP=71;Dels=0.00;FS=0.000;HaplotypeScore=2.8822;MLEAC=2;MLEAF=1.00;MQ=58.36;MQ0=0;QD=38.86;resource.EFF=INTERGENIC(MODIFIER||||||||) GT:AD:DP:GQ:PL 1/1:0,71:71:99:2787,214,0 GATKSAM
chr1 602567 rs21953190 A ... (9 Replies)
Discussion started by: mehar
9 Replies
7. Shell Programming and Scripting
I have a text file with entries like
1186
5556
90844
7873
7722
12
7890.6
78.52
6679
3455
9867
1127
5642
..N so many records like this.
I want to split this file into multiple files like cluster1.txt, cluster2.txt, cluster3.txt, ..... clusterN.txt. (4 Replies)
Discussion started by: sammy777
4 Replies
8. Shell Programming and Scripting
Hi,
I have a text file (attached the sample). I have also, attached the way the way the files need to be split.
We get this file, that will either have 24 Jurisdictions, or will miss some and retain some.
Like in the attached sample file, there are only Jurisdictions 03,11,14,15, 20 and 30.... (3 Replies)
Discussion started by: ebsus
3 Replies
9. Shell Programming and Scripting
Hi, all.
I have an input file. I would like to generate 3 types of output files.
Input:
LG10_PM_map_19_LEnd_1000560
LG10_PM_map_6-1_27101856
LG10_PM_map_71_REnd_20597718
LG12_PM_map_5_chr_118419232
LG13_PM_map_121_24341052
LG14_PM_1a_456799
LG1_MM_scf_5a_opt_abc_9029993
... (5 Replies)
Discussion started by: huiyee1
5 Replies
10. Shell Programming and Scripting
OS : RHEL 7.3
I have a file like below. I want to move (cut and paste) the first 7 lines of file1 to another file (file2).
How can I do this ? In my real life scenario, I will be moving first 12 millions lines of file1 to file2
$ cat file1.txt
7369|SMITH |CLERK | ... (5 Replies)
Discussion started by: kraljic
5 Replies
LEARN ABOUT REDHAT
mac2unix
dos2unix(1) General Commands Manual dos2unix(1) NAME
dos2unix - DOS/MAC to UNIX text file format converter SYNOPSYS
dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...] Options: [-hkqV] [--help] [--keepdate] [--quiet] [--version] DESCRIPTION
This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format. OPTIONS
The following options are available: -h --help Print online help. -k --keepdate Keep the date stamp of output file same as input file. -q --quiet Quiet mode. Suppress all warning and messages. -V --version Prints version information. -c --convmode convmode Sets conversion mode. Simulates dos2unix under SunOS. -o --oldfile file ... Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used. -n --newfile infile outfile ... New file mode. Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be used or you WILL lost your files. EXAMPLES
Get input from stdin and write output to stdout. dos2unix Convert and replace a.txt. Convert and replace b.txt. dos2unix a.txt b.txt dos2unix -o a.txt b.txt Convert and replace a.txt in ASCII conversion mode. Convert and replace b.txt in ISO conversion mode. Convert c.txt from Mac to Unix ascii format. dos2unix a.txt -c iso b.txt dos2unix -c ascii a.txt -c iso b.txt dos2unix -c mac a.txt b.txt Convert and replace a.txt while keeping original date stamp. dos2unix -k a.txt dos2unix -k -o a.txt Convert a.txt and write to e.txt. dos2unix -n a.txt e.txt Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt. dos2unix -k -n a.txt e.txt Convert and replace a.txt. Convert b.txt and write to e.txt. dos2unix a.txt -n b.txt e.txt dos2unix -o a.txt -n b.txt e.txt Convert c.txt and write to e.txt. Convert and replace a.txt. Convert and replace b.txt. Convert d.txt and write to f.txt. dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt DIAGNOSTICS
BUGS
The program does not work properly under MSDOS in stdio processing mode. If you know why is that so, please tell me. AUTHORS
Benjamin Lin - <blin@socs.uts.edu.au> Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org> MISCELLANY
Tested environment: Linux 1.2.0 with GNU C 2.5.8 SunOS 4.1.3 with GNU C 2.6.3 MS-DOS 6.20 with Borland C++ 4.02 Suggestions and bug reports are welcome. SEE ALSO
unix2dos(1) mac2unix(1) 1995.03.31 dos2unix v3.0 dos2unix(1)