05-18-2006
Fast way to split a tab delimited file
I have searched the forum and tried different options. One of the options work but is very slow. The file has millions and millions of records.
It is a TAB delimited file which contains two types of records. Metadata and Detail records.
M PARTNER 8 LAST_BOOKED_DATE D YYYYMMDD
M PARTNER 8 TRIPS_YTD A 11 TRIPS_TOTAL
D NAME FIRST LAST 209 N SANBORN AVE
D NAME FIRST LAST 6997 COUNTY ROAD D
I need to split the file into two files by looking at the first character. All records that start with 'M' go into one file and all records that start with 'D' go into another file.
The following code works but it is too slow...Is there any fast way of accomplishing it?
#!/usr/bin/ksh
while read line
do
char=`echo "$line" | cut -c1`
if [ "$char" = "M" ]; then
echo "$line" >> M.txt
else
echo "$line" >> D.txt
fi
done < head10000.out
exit 0
Any help would be appreciated.
Thank You,
Madhu
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi there
Just wondered if someone could help me out
I have a file that has been delimited by tabs, ie
field1<tab>fiield2<tab>field3
Does anybody know a command that will convert tabs to commas throughout the entire file?
Note: there are a number of unpopulated fields in the file so... (6 Replies)
Discussion started by: hcclnoodles
6 Replies
2. Shell Programming and Scripting
Hi,
Can anyone let me know on how to convert a Tab delimited file to Comma delimited file in Unix
Thanks!! (22 Replies)
Discussion started by: charan81
22 Replies
3. UNIX for Dummies Questions & Answers
Hi all,
I have a file with single white space delimited values, I want to convert them to a tab delimited file.
I tried sed, tr ... but nothing is working.
Thanks,
Rajeevan D (16 Replies)
Discussion started by: jeevs81
16 Replies
4. UNIX for Dummies Questions & Answers
Hi
I have 100 files each with only one column of 10 numbers that I wish to add to a new file so that I get 100 columns collected in one tab delimited file.
I tried something like:
foreach num (1 2 3)
foreach? gawk -F '\t' '{$num=$1}1' OFS='\t' Eu9_10.2patienter/pospep_10.2patient$num >>... (5 Replies)
Discussion started by: Banni
5 Replies
5. UNIX for Dummies Questions & Answers
Hi Forum
I have a tab delimited file that opens well in Openoffice calc (excel). But when I perform any operation in command line, it reads the file incorrectly. When I 'save As' the same file in office as tab delimited then it works fine.
The file that I think is tab delimited is actually... (8 Replies)
Discussion started by: imlearning
8 Replies
6. Shell Programming and Scripting
I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use
cat file | sed 's/|//t/g'
The above command substituted "/t" not tab in the place of pipe.
Sample file:
abc|123|2012-01-30|2012-04-28|xyz
have to convert to:
abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies
7. Shell Programming and Scripting
Hi How to make tab delimited file to space delimited?
in put file:
ABC kgy
jkh ghj
ash kjl
o/p file:
ABC kgy
jkh ghj
ash kjl
Use code tags, thanks. (1 Reply)
Discussion started by: jagdishrout
1 Replies
8. UNIX for Dummies Questions & Answers
Hi, I have a rquirement in unix as below .
I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column.
ex:
Input Text file:
1|A|apple
2|B|bottle
excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies
9. Shell Programming and Scripting
Hi,
I have requirement to create tab delimited file with values coming from variables.
File will contain only two columns separated by tab.
Header will be added once.
Values will be keep adding upon the script run.
If values already exists then values will be replaced.
I have done so... (1 Reply)
Discussion started by: sukhdip
1 Replies
10. UNIX for Beginners Questions & Answers
Hello Everyone..
I want to replace the retail col from FileI with cstp1 col from FileP if the strpno matches in both files
FileP.txt
... (2 Replies)
Discussion started by: YogeshG
2 Replies
LEARN ABOUT DEBIAN
binstats
BINSTATS(1) Debian Manual BINSTATS(1)
NAME
binstats - display statistics about programs and libraries
SYNOPSIS
binstats [-b[inaries]=bindirs] [-d[ebug]] [-s[kip-debian] [-f[ile]=logfile] [-l[ibraries]=libdirs] [-m[anuals]=mandirs] [-p[ath]=path]
[-t[emp]=tmpdir]
binstats -h[elp]
binstats -v[ersion]
OPTIONS
-binaries bindirs
bindirs is a quoted list of whitespace delimited directories where programs should be searched for.
-debug Store temporary files in the current directory and do not remove them when finished.
-skip-debian
Skip Debian tests that analyse installed packages even if this is a Debian system.
-file logfile
Output report to logfile instead of standard output.
-help Display a usage summary and exit.
-libraries libdirs
libdirs is a quoted list of whitespace delimited directories where libraries should be searched for.
-manuals mandirs
mandirs is a quoted list of whitespace delimited directories where manpages should be searched for.
-path path
Use the specified path instead of the value of the environment variable PATH.
-temp tmpdir
Store temporary files in tmpdir instead of the value of the environment variable TMPDIR, if defined, or /tmp.
-version
Output version information and exit.
AUTHOR
Peter Chang <Peter.Chang@nottingham.ac.uk>. This manual page was originally written by Matej Vela <vela@debian.org>.
Debian Project October 31, 1999 BINSTATS(1)