perform 3 awk commands to multiple files in multiple directories Post: 302568188

Sponsored Content

Top Forums Shell Programming and Scripting perform 3 awk commands to multiple files in multiple directories Post 302568188 by amarn on Wednesday 26th of October 2011 02:40:52 PM

10-26-2011

Registered User

perform 3 awk commands to multiple files in multiple directories

Hi,

I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them.

Each of these text files contains records (where this record is unique and there are no duplicates) and i want to initially separate each of these records into its own file and name it based on the second field (where the $2 field is an identifier and have this form : cust_xxx_yyy of the record...I'm currently doing this (example for file hour_1/hour_1.txt) :

(1)

Code:

awk '{print $0 > $2".txt"}' hour_1.txt

which results to multiple .txt files starting with cust_

then i want to have all these files as a single column file, therefore i do this:

(2)

Code:

awk '{print >  "n_"FILENAME}' RS=" " cust_*

and finally i want to remove the first 3 records of the newly created files thus i do the following:

(3)

Code:

awk 'FNR>3 {print > "fin_"FILENAME}' n_cust*

I know that there might be an easier way of doing this even for a single directory, but is there a way to write a universal script and perform these 3 commands in all the directories?

Thanks in advance!

Last edited by Franklin52; 10-27-2011 at 03:11 AM.. Reason: Please use code tags, thank you

amarn

View Public Profile for amarn

Find all posts by amarn

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

How do I perform a command to multiple files? For example, I want to look at all files in a directory and print the ones that do not contain a certain string. How do I go about doing this?

2. AIX

Script to perform some actions on multiple files

I have this Korn script that I wrote (with some help) that is run by cron. I basically watches a file system for a specific filename to be uploaded (via FTP), checks to make sure that the file is no longer being uploaded (by checking the files size), then runs a series of other scripts. The...

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting:

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and...

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace...

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is...

7. Shell Programming and Scripting

FTP multiple files from multiple directories

I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure.. ./dir1/1/TRADE_LOG*.gz ./dir2/10/TRADE_LOG*.gz ./dir11/12/TRADE_LOG*.gz ./dir12/13/TRADE_LOG*.gz when I do ftp uisng mput from the "." dir I am getting the below given error mput...

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Hi, Very unfamiliar with unix/linux stuff. Our admin is on vacation so, need help very quickly. I have directories (eg 40001, 40002, etc) that each have one subdirectory (01). Each subdir 01 has multiple subdirs (001, 002, 003, etc). They are same in each dir. I need to keep the top and...

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Hello all. Newbie here. In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running...

LEARN ABOUT DEBIAN

rdup-up

RDUP-UP(1)							       rdup								RDUP-UP(1)

NAME

       rdup-up - update a directory tree with a rdup archive

SYNOPSIS

       rdup-up [OPTION]...  DIRECTORY

DESCRIPTION

       With rdup-up you can update an (possibly) existing directory structure with a rdup archive.

       The rdup archive has to be given to rdup-up's standard input.

   Username and uids
       rdup  outputs  both  the  username  and	uid, the receiving system (which may be a totally different system) checks if the username and uid
       match. If the username and uid don't match the (numeric) uid is used on the file. The same holds true for the groupname and gid.

OPTIONS

       -n     Do a dry-run and do not create anything on disk.

       -t     Create DIRECTORY (ala mkdir -p) if it does not exist.

       -s N   Strip N path components from a pathname. If the resulting pathname is empty after this operation it is skipped. Be  careful  however
	      with the following structure:

		  /foo
		  /foo/bar
		  /foo/bar/bla.txt
		  /foo/blork/bla.txt

	      With rdup-up -s2 this will leave:

		  <empty>
		  <empty>
		  /bla.txt
		  /bla.txt

	      And the last 'bla.txt' will overwrite the previous one, this will happen without warnings.

       -r PATH
	      This option is related to the -s option, but works different. The string PATH is removed from (the beginning of) each pathname. With
	      -r /home/backup the pathname /home/backup/bin/mycmd becomes /bin/mycmd. The same could be done with -s 2, but then you need to count
	      the slashes. Note -s is always performed before -r.

       -v     Be more verbose and echo the processed files to standard output.

       -vv    Be even more verbose and echo processed file and the uid and gid information to standard output.

       -T     Show  a  table  of  contents of the rdup stream received (ala tar -tf -).  With -T the directory argument is optional. -T unsets any
	      verbose (-v) options.

       -h     A short help message.

       -V     Show the version.

EXIT CODE

       rdup-up return a zero exit code on success, otherwise 1 is returned.

AUTHOR

       Written by Miek Gieben.

REPORTING BUGS

       Report bugs to <miek@miek.nl>.

SEE ALSO

       http:/www.miek.nl/projects/rdup/ is the main site of rdup. Also see rdup(1), rdup-tr(1) and rdup-backups(7).

COPYRIGHT

       Copyright (C) 2005-2010 Miek Gieben. This is free software. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A	PARTICULAR
       PURPOSE.

       Licensed under the GPL version 3. See the file LICENSE in the source distribution of rdup.

1.1.11								    13 Dec 2008 							RDUP-UP(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

Discussion started by: mcgrawa

2. AIX

Script to perform some actions on multiple files

Discussion started by: heprox

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Discussion started by: pinnacle

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Discussion started by: manishabh

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Discussion started by: rich@ardz

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Discussion started by: Liverpaul09

7. Shell Programming and Scripting

FTP multiple files from multiple directories

Discussion started by: prasperl

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Discussion started by: kkouraus1

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Discussion started by: gabrysfe

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Discussion started by: Zeckendorff

LEARN ABOUT DEBIAN

rdup-up