10-17-2013
Limits are most definitely implementation specific, GNU awk, for example, seems to have no limit on open files, apart from those enforced by the OS limits.
GNU awk 3.1.7 on RHEL 6.4 will allow 24768 and with GNU awk 4.1.0 on cygwin 6.1 I could get 1834.
---------- Post updated at 12:11 PM ---------- Previous update was at 11:54 AM ----------
I'm surprised this solution is running slow for you, awk is fairly efficient and even if it's re-coded in C I wouldn't expect much improvement.
Are you writing the output department files to the same disk as the huge_file.txt is stored on, contention between read and write may be slowing it down. If you have more than 1 drive in the system try writing the output elsewhere eg make current directory on /disk1/tmp and read input file from /disk2/data.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi all,
I have searched and found various threads about removing spaces from a field within a text file. Unfortunately, I have not found exactly what I'm looking for, nor am I adept enough to modify what I've found into what I need.
I use the following command to remove the first line... (3 Replies)
Discussion started by: carriehoff
3 Replies
2. Shell Programming and Scripting
Hi all
I have two files X.txt and Y.txt. Both file contains same number of sentences. The content of X.txt is
The filter described above may be combined.
and the content of Y.txt is
The filter describ+ed above may be combin+ed.
Some of the words are separated with "+"... (2 Replies)
Discussion started by: my_Perl
2 Replies
3. Shell Programming and Scripting
Hello,
Is there anyway that I can align a pipe delimited text file by the maxium field length where the field is separated out by pipes for large text files with more than
100,000 rows?
So, far I have searched other forums and google about aligning text files in unix and I have noticed that... (7 Replies)
Discussion started by: physalis2099
7 Replies
4. Shell Programming and Scripting
I have list of files in a directory 'dir'. Each file is of type HTML. I need to read each file and get the string which starts with 'http' and write them in a new text file. How can i do this shell scripting?
file1.html
<head>
<url>http://www.google.com</url>
</head>
file2.html
<head>... (6 Replies)
Discussion started by: vel4ever
6 Replies
5. Shell Programming and Scripting
Is there a way to read in a two-columned CSV file, and based on the fields in 1st column, output many different files? The input/output looks something like:
input.csv:
call Call Mom.
call Call T-Mobile.
go Go home.
go Go to school.
go Go to gas station.
play Play music.
play Play... (4 Replies)
Discussion started by: pxalpine
4 Replies
6. Shell Programming and Scripting
I dont even have a sample script cause I dont know where to start from. My data lookes like this
> sat#16 #data: 15 site:UNZA baseline: 205.9151
0.008 -165.2465 35.8109 40.6685 21.9148 121.1446 26.4629 -18.4976 33.8722
0.017 -165.2243 48.2201 40.6908 ... (8 Replies)
Discussion started by: malandisa
8 Replies
7. Shell Programming and Scripting
I am trying to use awk skip each line with a ## or # and check each line after for STB= and if that value in greater than or = to 0.8, then at the end of line the text "STRAND BIAS" is written in else "GOOD".
So in the file of 4 entries attached.
awk tried:
awk NR > "##"' "#" -F"STB="... (6 Replies)
Discussion started by: cmccabe
6 Replies
8. UNIX for Beginners Questions & Answers
My program run without error. The problem I am having.
The program isn't outputting field values with the column headers to file.txt.
Each of the column headers in file.txt has no data.
MEMSIZE SECOND SASFoundation Filename
The output results in file.txt should show:
... (1 Reply)
Discussion started by: dellanicholson
1 Replies
9. Shell Programming and Scripting
In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies
10. Shell Programming and Scripting
The below awk executes as is and produces the current output. It isvery close but what Ican not seem to do is add the -exon..., the ... portion comes from $1 and the _exon is static and will never change. If there is + sign in $4 then the ... is in acending order or sequential. If there is a - in... (2 Replies)
Discussion started by: cmccabe
2 Replies
LEARN ABOUT DEBIAN
shtool-path
SHTOOL-PATH.TMP(1) GNU Portable Shell Tool SHTOOL-PATH.TMP(1)
NAME
shtool-path - GNU shtool command dealing with shell path variables
SYNOPSIS
shtool path [-s|--suppress] [-r|--reverse] [-d|--dirname] [-b|--basename] [-m|--magic] [-p|--path path] str [str ...]
DESCRIPTION
This command deals with shell $PATH variables. It can find a program through one or more filenames given by one or more str arguments. It
prints the absolute filesystem path to the program displayed on "stdout" plus an exit code of 0 if it was really found.
OPTIONS
The following command line options are available.
-s, --suppress
Supress output. Useful to only test whether a program exists with the help of the return code.
-r, --reverse
Transform a forward path to a subdirectory into a reverse path.
-d, --dirname
Output the directory name of str.
-b, --basename
Output the base name of str.
-m, --magic
Enable advanced magic search for ""perl"" and ""cpp"".
-p, --path path
Search in path. Default is to search in $PATH.
EXAMPLE
# shell script
awk=`shtool path -p "${PATH}:." gawk nawk awk`
perl=`shtool path -m perl`
cpp=`shtool path -m cpp`
revpath=`shtool path -r path/to/subdir`
HISTORY
The GNU shtool path command was originally written by Ralf S. Engelschall <rse@engelschall.com> in 1998 for Apache. It was later taken
over into GNU shtool.
SEE ALSO
shtool(1), which(1).
18-Jul-2008 shtool 2.0.8 SHTOOL-PATH.TMP(1)