Sponsored Content
Top Forums Shell Programming and Scripting Faster command to remove headers for files in a directory Post 302730415 by Yoda on Monday 12th of November 2012 08:31:39 PM
Old 11-12-2012
Jim, do you mean file caching helped sed because of the sequence of execution I chose? If yes, I tried the other way and still sed took less time to complete this particular task.

Code:
# time sed '1d' infile > out

real    0m3.41s
user    0m1.22s
sys     0m1.62s

Code:
# time awk 'FNR>1' infile > out

real    0m5.60s
user    0m2.20s
sys     0m3.21s


Last edited by Yoda; 11-12-2012 at 09:38 PM.. Reason: Code Added
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

help:how to remove headers in output file

Hi I am running a script (which compares two directory contents) for which I am getting an output of 70 pages in which few pages are blank so I was able to delete those blank lines. But I also want to delete the headers present for each page. can any one help me by providing the code... (1 Reply)
Discussion started by: raj_thota
1 Replies

2. Shell Programming and Scripting

Which one is faster to remove control m characters?

I have a file with millions of records...Before I experiment, I would like to know which one is faster. Both the commands work absolutely fine on a smaller set of records. Please advice. sed 's/^M//g' ${INPUT_FILE} > tmp.txt mv tmp.txt ${INPUT_FILE} tr -d "\15" < ${INPUT_FILE} > ... (11 Replies)
Discussion started by: madhunk
11 Replies

3. Shell Programming and Scripting

Remove Headers throughout a data file

I have a data file with over 500,000 records/lines that has the header throughout the file. SEQ_ID Name Start_Date Ins_date Add1 Add2 1 Harris 04/02/08 03/02/08 333 Main Suite 101 2 Smith 02/03/08 01/23/08 287 Jenkins SEQ_ID Name ... (3 Replies)
Discussion started by: psmall
3 Replies

4. UNIX for Dummies Questions & Answers

Remove certain headers using mailx or sendmail

Hello, So i want to send mails in any way from a solaris 5.8 system, perhaps using mailx or sendmail. My purpose is to stay clear of systems name in head data. So i want to strip at least the "Message-Id" and the "Recieved" headers of the mail. Yet this seems to be a bit of a problem. Now i... (2 Replies)
Discussion started by: congo
2 Replies

5. Shell Programming and Scripting

Remove text between headers while leaving headers intact

Hi, I'm trying to strip all lines between two headers in a file: ### BEGIN ### Text to remove, contains all kinds of characters ... Antispyware-Downloadserver.com (Germany)=http://www.antispyware-downloadserver.c om/updates/ Antispyware-Downloadserver.com #2... (3 Replies)
Discussion started by: Trones
3 Replies

6. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

7. Shell Programming and Scripting

Running rename command on large files and make it faster

Hi All, I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies

8. UNIX for Dummies Questions & Answers

Using sed command to remove multiple instances of repeating headers in one file?

Hi, I have catenated multiple output files (from a monte carlo run) into one big output file. Each individual file has it's own two line header. So when I catenate, there are multiple two line headers (of the same wording) within the big file. How do I use the sed command to search for the... (1 Reply)
Discussion started by: rebazon
1 Replies

9. Shell Programming and Scripting

Remove headers thar dont match

Good evening I need your help please, im new at Unix and i wanted to remove the first 5 headers for 100000 records files and then create a control file .ctl that contains the number of records and all seem to work out but when i tested at production it didnt wotk. Here is the code: #!... (6 Replies)
Discussion started by: alexcol
6 Replies

10. Shell Programming and Scripting

Remove white space and duplicate headers

I have a file called "dsout" with empty rows and duplicate headers. DATE TIME TOTAL_GB USED_GB %USED --------- -------- ---------- ---------- ---------- 03/05/013 12:34 PM 3151.24316 2331.56653 73.988785 ... (3 Replies)
Discussion started by: Daniel Gate
3 Replies
ATF-SH(1)						    BSD General Commands Manual 						 ATF-SH(1)

NAME
atf-sh [-s shell] -- interpreter for shell-based test programs SYNOPSIS
atf-sh script DESCRIPTION
atf-sh is an interpreter that runs the test program given in script after loading the atf-sh(3) library. atf-sh is not a real interpreter though: it is just a wrapper around the system-wide shell defined by ATF_SHELL. atf-sh executes the inter- preter, loads the atf-sh(3) library and then runs the script. You must consider atf-sh to be a POSIX shell by default and thus should not use any non-standard extensions. The following options are available: -s shell Specifies the shell to use instead of the value provided by ATF_SHELL. ENVIRONMENT
ATF_LIBEXECDIR Overrides the builtin directory where atf-sh is located. Should not be overridden other than for testing purposes. ATF_PKGDATADIR Overrides the builtin directory where libatf-sh.subr is located. Should not be overridden other than for testing purposes. ATF_SHELL Path to the system shell to be used in the generated scripts. Scripts must not rely on this variable being set to select a specific interpreter. EXAMPLES
Scripts using atf-sh(3) should start with: #! /usr/bin/env atf-sh Alternatively, if you want to explicitly choose a shell interpreter, you cannot rely on env(1) to find atf-sh. Instead, you have to hardcode the path to atf-sh in the script and then use the -s option afterwards as a single parameter: #! /path/to/bin/atf-sh -s/bin/bash ENVIRONMENT
ATF_SHELL Path to the system shell to be used in the generated scripts. SEE ALSO
atf-sh(3) BSD
September 27, 2014 BSD
All times are GMT -4. The time now is 11:44 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy