Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using awk on multiple files in a directory
Top Forums Shell Programming and Scripting Using awk on multiple files in a directory Post 302837919 by SkySmart on Saturday 27th of July 2013 02:04:10 PM
Old 07-27-2013
Quote:
Originally Posted by RudiC
I think this is as close as you can get:
Code:
awk     'FNR == 1               {if (NR > 1) {print fn, "text1", fnr, "text2", nl}
                                 fn=FILENAME; fnr = 1; nl = 0}
                                {fnr = FNR}
         /customer.*processed/  {nl++}
         END                    {print fn, "text1", fnr, "text2", nl}
        ' file?

As you want the line count per file, you need to read every file entirely; I don't see much chance to improve on speed...
thank you!!!

this worked perfectly. is there anyway i can instruct awk to do exactly what you're doing here, but to treat any file it finds that isn't plain text (i.e. gzip files) in a different way?

like, for instance, the grepping for the string wont work on files that are gzipped. i do know you can use the following for reading gzip files:

Code:
( gunzip -cd /path/to/file.gz ; cat /path/to/file ) | grep

the problem im having is being able to incorporate this command into your awk command so it is kicked off ONLY when the awk comes across a file that isn't plain text.

makes sense?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

ftp multiple files from same directory

Hi there Gurus, I have the following ftp script: $ more ftp_dump_arch4.sh #! /usr/bin/ksh # Constant variables HOST='xx.xx.xx.xx' USER='user' PASSWD='password' dir='/export/file' ftp_log='/tmp' ftp -n $HOST > $ftp_log/ftp.log << END user $USER $PASSWD verbose lcd $dir bin (3 Replies)
Discussion started by: lweegp
3 Replies

2. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting: (0 Replies)
Discussion started by: pinnacle
0 Replies

3. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

4. Shell Programming and Scripting

How to Pull out multiple files from DB table and redirect all those files to a differetn directory?

Hi everyone!! I have a database table, which has file_name as one of its fields. Example: File_ID File_Name Directory Size 0001 UNO_1232 /apps/opt 234 0002 UNO_1234 /apps/opt 788 0003 UNO_1235 /apps/opt 897 0004 UNO_1236 /apps/opt 568 I have to... (3 Replies)
Discussion started by: ss3944
3 Replies

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. Shell Programming and Scripting

apply record separator to multiple files within a directory using awk

Hi, I have a bunch of records within a directory where each one has this form: (example file1) 1 2 50 90 80 90 43512 98 0909 79869 -9 7878 33222 8787 9090 89898 7878 8989 7878 6767 89 89 78676 9898 000 7878 5656 5454 5454 and i want for all of these files to be... (3 Replies)
Discussion started by: amarn
3 Replies

8. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

9. UNIX for Dummies Questions & Answers

Multiple files with the same name in the same directory

Hi, Is it possible to have multiple files with the same name in a same unix directory? Eg., in the path \tmp, can we have 2 files with the same name as SALES_data_20120124.TXT? I heard it is possible if the user id that is created the files are different and with some ids, a new gets... (1 Reply)
Discussion started by: Vijay81
1 Replies

10. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

LEARN ABOUT DEBIAN

gri_merge

GRI_MERGE(1)						      General Commands Manual						      GRI_MERGE(1)

NAME

       gri_merge - merge multiple Gri output files into a single PostScript file.

USAGE (style 1):
	      gri_merge [OPTIONS] CxR a.ps b.ps ... > merged_file.ps

       Merges  the  files  onto one page, in 'C' columns and 'R' rows.	The CxR files are given in the order of words on a page.  The page is pre-
       sumed to be 8.5x11in in size, as are all the input files, and the input files are sized to fit, and kept in natural scale.

USAGE (style 2):
	      gri_merge [OPTIONS] xcm ycm enlarge a.ps [b.ps ...] > merged_file.ps

       Where `enlarge' is a scale factor applied after offsetting `xcm' to the right and `ycm' upward.

   EXAMPLE (style 2):
	      The following

       gri_merge 2 12 .5 a.ps 

	      12 12 .5 b.ps 

	      2 2 .5 c.ps 

	      12 2 .5 d.ps > all.ps

       produces 4 panels from gri plots done using margins and sizes as specified in the following lines in a gri commandfile

	      set x margin 2

	      set x size 15

	      set y margin 2

	      set y size 15

       The OPTIONS, available if your 'perl' has 'getopts' library, are:

       -u graylevel -- set graylevel for underlay beneath panels, by default 0.75.

	      Values range from 0 (black) to 1 (white), although a value of precisely 1 means do NOT draw underlay.

       -b graylevel -- Set value for background under individual panels, again 0

	      for black to 1 for white, with 1 meaning no drawing.

       -h     -- Print this help message and quit.

SEE ALSO

       gri(1), gri_unpage(1)

gri_merge							       2009							      GRI_MERGE(1)
All times are GMT -4. The time now is 08:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy