Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: perform 3 awk commands to multiple files in multiple directories
Top Forums Shell Programming and Scripting perform 3 awk commands to multiple files in multiple directories Post 302568188 by amarn on Wednesday 26th of October 2011 02:40:52 PM
Old 10-26-2011
perform 3 awk commands to multiple files in multiple directories
Hi,

I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them.

Each of these text files contains records (where this record is unique and there are no duplicates) and i want to initially separate each of these records into its own file and name it based on the second field (where the $2 field is an identifier and have this form : cust_xxx_yyy of the record...I'm currently doing this (example for file hour_1/hour_1.txt) :

(1)
Code:
awk '{print $0 > $2".txt"}' hour_1.txt

which results to multiple .txt files starting with cust_

then i want to have all these files as a single column file, therefore i do this:

(2)
Code:
awk '{print >  "n_"FILENAME}' RS=" " cust_*

and finally i want to remove the first 3 records of the newly created files thus i do the following:

(3)
Code:
awk 'FNR>3 {print > "fin_"FILENAME}' n_cust*


I know that there might be an easier way of doing this even for a single directory, but is there a way to write a universal script and perform these 3 commands in all the directories?

Thanks in advance!
Last edited by Franklin52; 10-27-2011 at 03:11 AM.. Reason: Please use code tags, thank you
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

How do I perform a command to multiple files? For example, I want to look at all files in a directory and print the ones that do not contain a certain string. How do I go about doing this? (4 Replies)
Discussion started by: mcgrawa
4 Replies

2. AIX

Script to perform some actions on multiple files

I have this Korn script that I wrote (with some help) that is run by cron. I basically watches a file system for a specific filename to be uploaded (via FTP), checks to make sure that the file is no longer being uploaded (by checking the files size), then runs a series of other scripts. The... (2 Replies)
Discussion started by: heprox
2 Replies

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting: (0 Replies)
Discussion started by: pinnacle
0 Replies

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. Shell Programming and Scripting

FTP multiple files from multiple directories

I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure.. ./dir1/1/TRADE_LOG*.gz ./dir2/10/TRADE_LOG*.gz ./dir11/12/TRADE_LOG*.gz ./dir12/13/TRADE_LOG*.gz when I do ftp uisng mput from the "." dir I am getting the below given error mput... (1 Reply)
Discussion started by: prasperl
1 Replies

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Hi, Very unfamiliar with unix/linux stuff. Our admin is on vacation so, need help very quickly. I have directories (eg 40001, 40002, etc) that each have one subdirectory (01). Each subdir 01 has multiple subdirs (001, 002, 003, etc). They are same in each dir. I need to keep the top and... (7 Replies)
Discussion started by: kkouraus1
7 Replies

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Hello all. Newbie here. In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running... (6 Replies)
Discussion started by: Zeckendorff
6 Replies

LEARN ABOUT DEBIAN

dh_installmime

DH_INSTALLMIME(1)						     Debhelper							 DH_INSTALLMIME(1)

NAME

       dh_installmime - install mime files into package build directories

SYNOPSIS

       dh_installmime [debhelperoptions] [-n]

DESCRIPTION

       dh_installmime is a debhelper program that is responsible for installing mime files into package build directories.

       It also automatically generates the postinst and postrm commands needed to interface with the debian mime-support and shared-mime-info
       packages. These commands are inserted into the maintainer scripts by dh_installdeb(1).

FILES

       debian/package.mime
	   Installed into usr/lib/mime/packages/package in the package build directory.

       debian/package.sharedmimeinfo
	   Installed into /usr/share/mime/packages/package.xml in the package build directory.

OPTIONS

       -n, --noscripts
	   Do not modify postinst/postrm scripts.

NOTES

       Note that this command is not idempotent. dh_prep(1) should be called between invocations of this command. Otherwise, it may cause multiple
       instances of the same text to be added to maintainer scripts.

SEE ALSO

       debhelper(7)

       This program is a part of debhelper.

AUTHOR

       Joey Hess <joeyh@debian.org>

9.20120909							    2012-09-10							 DH_INSTALLMIME(1)
All times are GMT -4. The time now is 09:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy