perform 3 awk commands to multiple files in multiple directories Post: 302568188

Sponsored Content

Top Forums Shell Programming and Scripting perform 3 awk commands to multiple files in multiple directories Post 302568188 by amarn on Wednesday 26th of October 2011 02:40:52 PM

10-26-2011

Registered User

perform 3 awk commands to multiple files in multiple directories

Hi,

I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them.

Each of these text files contains records (where this record is unique and there are no duplicates) and i want to initially separate each of these records into its own file and name it based on the second field (where the $2 field is an identifier and have this form : cust_xxx_yyy of the record...I'm currently doing this (example for file hour_1/hour_1.txt) :

(1)

Code:

awk '{print $0 > $2".txt"}' hour_1.txt

which results to multiple .txt files starting with cust_

then i want to have all these files as a single column file, therefore i do this:

(2)

Code:

awk '{print >  "n_"FILENAME}' RS=" " cust_*

and finally i want to remove the first 3 records of the newly created files thus i do the following:

(3)

Code:

awk 'FNR>3 {print > "fin_"FILENAME}' n_cust*

I know that there might be an easier way of doing this even for a single directory, but is there a way to write a universal script and perform these 3 commands in all the directories?

Thanks in advance!

Last edited by Franklin52; 10-27-2011 at 03:11 AM.. Reason: Please use code tags, thank you

amarn

View Public Profile for amarn

Find all posts by amarn

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

How do I perform a command to multiple files? For example, I want to look at all files in a directory and print the ones that do not contain a certain string. How do I go about doing this?

2. AIX

Script to perform some actions on multiple files

I have this Korn script that I wrote (with some help) that is run by cron. I basically watches a file system for a specific filename to be uploaded (via FTP), checks to make sure that the file is no longer being uploaded (by checking the files size), then runs a series of other scripts. The...

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting:

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and...

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace...

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is...

7. Shell Programming and Scripting

FTP multiple files from multiple directories

I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure.. ./dir1/1/TRADE_LOG*.gz ./dir2/10/TRADE_LOG*.gz ./dir11/12/TRADE_LOG*.gz ./dir12/13/TRADE_LOG*.gz when I do ftp uisng mput from the "." dir I am getting the below given error mput...

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Hi, Very unfamiliar with unix/linux stuff. Our admin is on vacation so, need help very quickly. I have directories (eg 40001, 40002, etc) that each have one subdirectory (01). Each subdir 01 has multiple subdirs (001, 002, 003, etc). They are same in each dir. I need to keep the top and...

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Hello all. Newbie here. In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running...

LEARN ABOUT DEBIAN

h5fromh4

H5FROMH4(1)							      h5utils							       H5FROMH4(1)

NAME

       h5fromh4 - convert HDF4 scientific datasets to an HDF5 file

SYNOPSIS

       h5fromh4 [OPTION]... [HDF4FILE]...

DESCRIPTION

       h5fromh4  takes one or more files in HDF4 format and outputs files in HDF5 format containing the datasets from the HDF4 files.  (Currently,
       only a single dataset per HDF4 file is converted.)

       HDF4 and HDF5 are free, portable binary formats and supporting libraries developed by the National Center for  Supercomputing  Applications
       at the University of Illinois in Urbana-Champaign.

       A  single  h5 file can contain multiple data sets; by default, h5fromh4 creates a dataset called "data", but this can be changed via the -d
       option, or by using the syntax HDF5FILE:DATASET with the -o option.  The -a option can be used to append new datasets to an  existing  HDF5
       file.   If the -o option is used and multiple HDF4 files are specified, all the HDF4 datasets are output into that HDF5 file with the input
       filenames (minus the ".hdf" suffix) used as the dataset names.

       The most basic usage is something like 'h5fromh4 foo.hdf', which will output a file foo.h5 containing the scientific dataset from foo.hdf.

OPTIONS

       -h     Display help on the command-line options and usage.

       -V     Print the version number and copyright info for h5fromh4.

       -v     Verbose output.

       -a     If the HDF5 output file already exists, append the data as a new dataset rather than overwriting the file  (the  default	behavior).
	      An existing dataset of the same name within the file is overwritten, however.

       -o file
	      Send  HDF5 output to file rather than to the input filename with .hdf replaced with .h5 (the default).  If multiple input files were
	      specified, this causes all input datasets to be stored in file (rather than in separate files), with the input filenames (minus  the
	      .hdf suffix) as the dataset names.

       -d name
	      Write  to  dataset  name	in  the  output; otherwise, the output dataset is called "data" by default.  Alternatively, use the syntax
	      HDF5FILE:DATASET with the -o option.

BUGS

       Send bug reports to S. G. Johnson, stevenj@alum.mit.edu.

AUTHORS

       Written by Steven G. Johnson.  Copyright (c) 2005 by the Massachusetts Institute of Technology.

h5utils 							   March 9, 2002						       H5FROMH4(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

Discussion started by: mcgrawa

2. AIX

Script to perform some actions on multiple files

Discussion started by: heprox

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Discussion started by: pinnacle

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Discussion started by: manishabh

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Discussion started by: rich@ardz

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Discussion started by: Liverpaul09

7. Shell Programming and Scripting

FTP multiple files from multiple directories

Discussion started by: prasperl

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Discussion started by: kkouraus1

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Discussion started by: gabrysfe

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Discussion started by: Zeckendorff

LEARN ABOUT DEBIAN

h5fromh4