Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Faster command to remove headers for files in a directory
Top Forums Shell Programming and Scripting Faster command to remove headers for files in a directory Post 302730417 by jim mcnamara on Monday 12th of November 2012 08:42:37 PM
Old 11-12-2012
Yes, that was what I meant. And yes it is very likely the grep, egrep, and sed are better at massive I/O than awk, which is running interpreted. The point, I think, is that a lot of tests like this are a lot of fun, but they may not be informative. Unless you understand why results can be set askew.

On my large m4000 Solaris boxes sed always outperforms awk on simple stream editing of massive files. On cygwin they come out really close.

However, by the time I've set up a fair test and run several candidates through, I could have coded and already processed 24 files in parallel, using any reasonable method.

Which is a lot less fun, I admit.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

help:how to remove headers in output file

Hi I am running a script (which compares two directory contents) for which I am getting an output of 70 pages in which few pages are blank so I was able to delete those blank lines. But I also want to delete the headers present for each page. can any one help me by providing the code... (1 Reply)
Discussion started by: raj_thota
1 Replies

2. Shell Programming and Scripting

Which one is faster to remove control m characters?

I have a file with millions of records...Before I experiment, I would like to know which one is faster. Both the commands work absolutely fine on a smaller set of records. Please advice. sed 's/^M//g' ${INPUT_FILE} > tmp.txt mv tmp.txt ${INPUT_FILE} tr -d "\15" < ${INPUT_FILE} > ... (11 Replies)
Discussion started by: madhunk
11 Replies

3. Shell Programming and Scripting

Remove Headers throughout a data file

I have a data file with over 500,000 records/lines that has the header throughout the file. SEQ_ID Name Start_Date Ins_date Add1 Add2 1 Harris 04/02/08 03/02/08 333 Main Suite 101 2 Smith 02/03/08 01/23/08 287 Jenkins SEQ_ID Name ... (3 Replies)
Discussion started by: psmall
3 Replies

4. UNIX for Dummies Questions & Answers

Remove certain headers using mailx or sendmail

Hello, So i want to send mails in any way from a solaris 5.8 system, perhaps using mailx or sendmail. My purpose is to stay clear of systems name in head data. So i want to strip at least the "Message-Id" and the "Recieved" headers of the mail. Yet this seems to be a bit of a problem. Now i... (2 Replies)
Discussion started by: congo
2 Replies

5. Shell Programming and Scripting

Remove text between headers while leaving headers intact

Hi, I'm trying to strip all lines between two headers in a file: ### BEGIN ### Text to remove, contains all kinds of characters ... Antispyware-Downloadserver.com (Germany)=http://www.antispyware-downloadserver.c om/updates/ Antispyware-Downloadserver.com #2... (3 Replies)
Discussion started by: Trones
3 Replies

6. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

7. Shell Programming and Scripting

Running rename command on large files and make it faster

Hi All, I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies

8. UNIX for Dummies Questions & Answers

Using sed command to remove multiple instances of repeating headers in one file?

Hi, I have catenated multiple output files (from a monte carlo run) into one big output file. Each individual file has it's own two line header. So when I catenate, there are multiple two line headers (of the same wording) within the big file. How do I use the sed command to search for the... (1 Reply)
Discussion started by: rebazon
1 Replies

9. Shell Programming and Scripting

Remove headers thar dont match

Good evening I need your help please, im new at Unix and i wanted to remove the first 5 headers for 100000 records files and then create a control file .ctl that contains the number of records and all seem to work out but when i tested at production it didnt wotk. Here is the code: #!... (6 Replies)
Discussion started by: alexcol
6 Replies

10. Shell Programming and Scripting

Remove white space and duplicate headers

I have a file called "dsout" with empty rows and duplicate headers. DATE TIME TOTAL_GB USED_GB %USED --------- -------- ---------- ---------- ---------- 03/05/013 12:34 PM 3151.24316 2331.56653 73.988785 ... (3 Replies)
Discussion started by: Daniel Gate
3 Replies

LEARN ABOUT FREEBSD

shar

SHAR(1) 						    BSD General Commands Manual 						   SHAR(1)

NAME

     shar -- create a shell archive of files

SYNOPSIS

     shar file ...

DESCRIPTION

     The shar command writes a sh(1) shell script to the standard output which will recreate the file hierarchy specified by the command line op-
     erands.  Directories will be recreated and must be specified before the files they contain (the find(1) utility does this correctly).

     The shar command is normally used for distributing files by ftp(1) or mail(1).

EXAMPLES

     To create a shell archive of the program ls(1) and mail it to Rick:

	   cd ls
	   shar `find . -print` | mail -s "ls source" rick

     To recreate the program directory:

	   mkdir ls
	   cd ls
	   ...
	   <delete header lines and examine mailed archive>
	   ...
	   sh archive

SEE ALSO

     compress(1), mail(1), tar(1), uuencode(1)

HISTORY

     The shar command appeared in 4.4BSD.

BUGS

     The shar command makes no provisions for special types of files or files containing magic characters.  The shar command cannot handle files
     without a newline ('
') as the last character.

     It is easy to insert trojan horses into shar files.  It is strongly recommended that all shell archive files be examined before running them
     through sh(1).  Archives produced using this implementation of shar may be easily examined with the command:

	   egrep -v '^[X#]' shar.file

BSD
								   June 6, 1993 							       BSD
All times are GMT -4. The time now is 08:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy