11-12-2012
Faster command to remove headers for files in a directory
Good evening
Im new at unix shell scripting and im planning to script a shell that removes headers for about 120 files in a directory and each file contains about 200000
lines in average.
i know i will loop files to process each one and ive found in this great forum different solutions using grep, sed, awk, head, etc.
But according to the above scenario and your experince and knowledge which command is the best for performance and does the homework faster ?
Thanks in advance
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi
I am running a script (which compares two directory contents) for which I am getting an output of 70 pages in which few pages are blank so I was able to delete those blank lines.
But I also want to delete the headers present for each page. can any one help me by providing the code... (1 Reply)
Discussion started by: raj_thota
1 Replies
2. Shell Programming and Scripting
I have a file with millions of records...Before I experiment, I would like to know which one is faster.
Both the commands work absolutely fine on a smaller set of records.
Please advice.
sed 's/^M//g' ${INPUT_FILE} > tmp.txt
mv tmp.txt ${INPUT_FILE}
tr -d "\15" < ${INPUT_FILE} > ... (11 Replies)
Discussion started by: madhunk
11 Replies
3. Shell Programming and Scripting
I have a data file with over 500,000 records/lines that has the header throughout the file.
SEQ_ID Name Start_Date Ins_date Add1 Add2
1 Harris 04/02/08 03/02/08 333 Main Suite 101
2 Smith 02/03/08 01/23/08 287 Jenkins
SEQ_ID Name ... (3 Replies)
Discussion started by: psmall
3 Replies
4. UNIX for Dummies Questions & Answers
Hello,
So i want to send mails in any way from a solaris 5.8 system, perhaps using mailx or sendmail. My purpose is to stay clear of systems name in head data. So i want to strip at least the "Message-Id" and the "Recieved" headers of the mail. Yet this seems to be a bit of a problem.
Now i... (2 Replies)
Discussion started by: congo
2 Replies
5. Shell Programming and Scripting
Hi,
I'm trying to strip all lines between two headers in a file:
### BEGIN ###
Text to remove, contains all kinds of characters
...
Antispyware-Downloadserver.com (Germany)=http://www.antispyware-downloadserver.c
om/updates/
Antispyware-Downloadserver.com #2... (3 Replies)
Discussion started by: Trones
3 Replies
6. Shell Programming and Scripting
Hi ,
I have a typical situation. I have 4 files and with different headers (number of headers is varible ).
I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only).
For example -
File 1
H1|H2|H3|H4
11|12|13|14
21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies
7. Shell Programming and Scripting
Hi All,
I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies
8. UNIX for Dummies Questions & Answers
Hi,
I have catenated multiple output files (from a monte carlo run) into one big output file. Each individual file has it's own two line header. So when I catenate, there are multiple two line headers (of the same wording) within the big file. How do I use the sed command to search for the... (1 Reply)
Discussion started by: rebazon
1 Replies
9. Shell Programming and Scripting
Good evening
I need your help please, im new at Unix and i wanted to remove the first 5 headers for 100000 records files and then create a control file .ctl that contains the number of records and all seem to work out but when i tested at production it didnt wotk.
Here is the code:
#!... (6 Replies)
Discussion started by: alexcol
6 Replies
10. Shell Programming and Scripting
I have a file called "dsout" with empty rows and duplicate headers.
DATE TIME TOTAL_GB USED_GB %USED
--------- -------- ---------- ---------- ----------
03/05/013 12:34 PM 3151.24316 2331.56653 73.988785 ... (3 Replies)
Discussion started by: Daniel Gate
3 Replies
LEARN ABOUT FREEBSD
sleep
SLEEP(1) BSD General Commands Manual SLEEP(1)
NAME
sleep -- suspend execution for an interval of time
SYNOPSIS
sleep seconds
DESCRIPTION
The sleep command suspends execution for a minimum of seconds.
If the sleep command receives a signal, it takes the standard action. When the SIGINFO signal is received, the estimate of the amount of
seconds left to sleep is printed on the standard output.
IMPLEMENTATION NOTES
The SIGALRM signal is not handled specially by this implementation.
The sleep command allows and honors a non-integer number of seconds to sleep in any form acceptable by strtod(3). This is a non-portable
extension, and its use will nearly guarantee that a shell script will not execute properly on another system.
EXIT STATUS
The sleep utility exits 0 on success, and >0 if an error occurs.
EXAMPLES
To schedule the execution of a command for x number seconds later (with csh(1)):
(sleep 1800; sh command_file >& errors)&
This incantation would wait a half hour before running the script command_file. (See the at(1) utility.)
To reiteratively run a command (with the csh(1)):
while (1)
if (! -r zzz.rawdata) then
sleep 300
else
foreach i (`ls *.rawdata`)
sleep 70
awk -f collapse_data $i >> results
end
break
endif
end
The scenario for a script such as this might be: a program currently running is taking longer than expected to process a series of files, and
it would be nice to have another program start processing the files created by the first program as soon as it is finished (when zzz.rawdata
is created). The script checks every five minutes for the file zzz.rawdata, when the file is found, then another portion processing is done
courteously by sleeping for 70 seconds in between each awk job.
SEE ALSO
nanosleep(2), sleep(3)
STANDARDS
The sleep command is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible.
HISTORY
A sleep command appeared in Version 4 AT&T UNIX.
BSD
April 18, 1994 BSD