Yes, you can do it in awk and it's not difficult. But I very seriously doubt that awk and shell is appropriate tools for processing 400GB files. Just try to time some very simple awk script, ie:
I believe It may take days or weeks. I suggest to use any compiled language, use available parallel tools and convert your file to noSQL database (with numbers, not strings) before processing. Or maybe use some specialized tools/languages like Matlab (or Octave).
Hi guys I want to print the values by using this script but its giving the no of rows and columns as input instead of values
Would you plz help me on this
FILE- chr1.txt
1981 1
1971 1
1961 1
1941 1
perl script
#!/usr/bin/perl -w
$infile1 = 'chr1.txt';
$outfile3 = 'out3.txt';
... (3 Replies)
Hello guys,
Please help me to solve this problem. I have tried some awk commands but couldn't succeed.
I have a tab delimited file where each record is separated by ------ and 4th column of each record is same.
<INPUT FILE>
------
peon 53931587 53931821 ... (12 Replies)
Hi
My pipe delimited .txt file contains rows with 10 columns.
Can anyone advise how I output to file only those rows with the letters ‘ci'
as the first 2 characters in the 3rd column ?
Many thanks (4 Replies)
Hello,
I want to extract several columns and rows from a huge tab delimited file
for example: I want to print from from column 3 to 68 till row number 30.
I have tried using cut command but it was extracting whole 3rd and 68th column.
Please suggest a solution.
Ryan (8 Replies)
Hi all,
I have a large matrix of 720 x 25. I want to get sum of range of rows and columns. Like, I need sum of all columns and row number 2 to 21, then leaving 22nd row, again sum of all columns and row number 23 to 42 again leaving 43rd row and then sum of 44th to 63. Means I want to add all... (4 Replies)
Hi I have a matrix with n rows and m columns like below example. i want to extract all the pairs with values <200.
Input
A B C D
A 100 206 51 300
B 206 100 72 48
C 351 22 100 198
D 13 989 150 100
Output format
A,A:200
A,C:51
B,B:100... (2 Replies)
Hi all, I'm pretty much a newbie to UNIX. I would appreciate any help with UNIX coding on comparing two large csv files (greater than 10 GB in size), and output a file with matching columns.
I want to compare file1 and file2 by 'id' and 'chain' columns, then extract exact matching rows'... (5 Replies)
Hallo,
I would need to extract only rows which has different value in the second and third column.
Thank you very much for any advices
Input:
A 0 0
B 0 1
C 1 1
D 1 3
Output
B 0 1
D 1 3 (4 Replies)
Hello
I have a file like this:
> cat examplefile
ghi|NN603762|eee
mno|NN607265|ttt
pqr|NN613879|yyy
stu|NN615002|uuu
jkl|NN607265|rrr
vwx|NN615002|iii
yzA|NN618555|ooo
def|NN190486|www
BCD|NN628717|ppp
abc|NN190486|qqq
EFG|NN628717|aaa
HIJ|NN628717|sss
>
I can sort the file by... (5 Replies)
Discussion started by: CHoggarth
5 Replies
LEARN ABOUT MOJAVE
sleep
SLEEP(1) BSD General Commands Manual SLEEP(1)NAME
sleep -- suspend execution for an interval of time
SYNOPSIS
sleep seconds
DESCRIPTION
The sleep command suspends execution for a minimum of seconds.
If the sleep command receives a signal, it takes the standard action.
IMPLEMENTATION NOTES
The SIGALRM signal is not handled specially by this implementation.
The sleep command will accept and honor a non-integer number of specified seconds (with a '.' character as a decimal point). This is a non-
portable extension, and its use will nearly guarantee that a shell script will not execute properly on another system.
EXIT STATUS
The sleep utility exits 0 on success, and >0 if an error occurs.
EXAMPLES
To schedule the execution of a command for x number seconds later (with csh(1)):
(sleep 1800; sh command_file >& errors)&
This incantation would wait a half hour before running the script command_file. (See the at(1) utility.)
To reiteratively run a command (with the csh(1)):
while (1)
if (! -r zzz.rawdata) then
sleep 300
else
foreach i (`ls *.rawdata`)
sleep 70
awk -f collapse_data $i >> results
end
break
endif
end
The scenario for a script such as this might be: a program currently running is taking longer than expected to process a series of files, and
it would be nice to have another program start processing the files created by the first program as soon as it is finished (when zzz.rawdata
is created). The script checks every five minutes for the file zzz.rawdata, when the file is found, then another portion processing is done
courteously by sleeping for 70 seconds in between each awk job.
SEE ALSO nanosleep(2), sleep(3)STANDARDS
The sleep command is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible.
HISTORY
A sleep command appeared in Version 4 AT&T UNIX.
BSD April 18, 1994 BSD