Sponsored Content
Top Forums Shell Programming and Scripting perform 3 awk commands to multiple files in multiple directories Post 302568435 by amarn on Thursday 27th of October 2011 10:00:00 AM
Old 10-27-2011
Hi,

I've been going through some trial-error procedures...

basically :


Code:
!usr/bin/sh

set -xv

find /home/tester/datasets_26_10_11/hour_1/ -name "cust_*.txt" -type f | \


while read fname
do
	fileBaseName=`basename "${fname}"`
	fileDirName=`dirname "${fname}"` 

	echo "fileBaseName: [${fileDirName}][${fileBaseName}] - fname[${fname}]"

	echo "now working on: [${fname}] with [${fileBaseName}]"
	#this is right! - for the shell script being in the same directory , not outside
	awk -v outputPath="${fileDirName}" 'BEGIN{RS =" ";}{print > outputPath "/" "n_"FILENAME}' ${fileBaseName}
	echo "fileBasename AFTER FIRST AWK : [${fileBaseName}]"

        #the following not tested yet
	awk -v outputPath="${fileDirName}" -v inputPath="${fileDirName}" 'FNR>3 {print > outputPath "/" "fin_"FILENAME}' ${fileBaseName}


done

works only if the file is in the actual hour_1/ directory...if not in there i get an awk error saying that it cannot read the file e.g:

Code:
awk: fatal : cannot open `file cust_1209_3000'  for reading (no such file or directory)

so, it can actually find this particular file but cannot read it...i believe it has something to do with the ` or ' surrounding it.....any suggestions?
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perform a command to multiple files

How do I perform a command to multiple files? For example, I want to look at all files in a directory and print the ones that do not contain a certain string. How do I go about doing this? (4 Replies)
Discussion started by: mcgrawa
4 Replies

2. AIX

Script to perform some actions on multiple files

I have this Korn script that I wrote (with some help) that is run by cron. I basically watches a file system for a specific filename to be uploaded (via FTP), checks to make sure that the file is no longer being uploaded (by checking the files size), then runs a series of other scripts. The... (2 Replies)
Discussion started by: heprox
2 Replies

3. Shell Programming and Scripting

Multiple search string in multiple files using awk

Hi, filenames: contains name of list of files to search in. placelist contains the names of places to be searched in all files in "filenames" for i in $(<filenames) do egrep -f placelist $i if ] then echo $i fi done >> outputfile Output i am getting: (0 Replies)
Discussion started by: pinnacle
0 Replies

4. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

5. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. Shell Programming and Scripting

FTP multiple files from multiple directories

I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure.. ./dir1/1/TRADE_LOG*.gz ./dir2/10/TRADE_LOG*.gz ./dir11/12/TRADE_LOG*.gz ./dir12/13/TRADE_LOG*.gz when I do ftp uisng mput from the "." dir I am getting the below given error mput... (1 Reply)
Discussion started by: prasperl
1 Replies

8. UNIX for Dummies Questions & Answers

Deleting multiple directories inside multiple directories

Hi, Very unfamiliar with unix/linux stuff. Our admin is on vacation so, need help very quickly. I have directories (eg 40001, 40002, etc) that each have one subdirectory (01). Each subdir 01 has multiple subdirs (001, 002, 003, etc). They are same in each dir. I need to keep the top and... (7 Replies)
Discussion started by: kkouraus1
7 Replies

9. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

10. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Hello all. Newbie here. In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running... (6 Replies)
Discussion started by: Zeckendorff
6 Replies
clm close(1)							  USER COMMANDS 						      clm close(1)

  NAME
      clm close - Fetch connected components from graphs or subgraphs

      clmclose	is  not  in  actual  fact  a program. This manual page documents the behaviour and options of the clm program when invoked in mode
      close. The options -h, --apropos, --version, -set, --nop are accessible in all clm modes. They are described in the clm manual page.

  SYNOPSIS
      clm close -imx <fname> [options]

      clm close -imx fname (specify matrix input) -abc fname (specify label input) -dom fname (input domain/cluster file) [-o fname (output file)]
      [--is-undirected	(trust	input  graph  to  be  undirected)] [--write-count (output component count)] [--write-sizes (output component sizes
      (default))] [--write-cc (output components as clustering)] [--write-block (output graph restricted to -dom argument)] [--write-blockc  (out-
      put  graph  complement  of  -dom	argument)]  [-cc-bound	num  (select components with size at least num)] [-tf spec (apply tf-spec to input
      matrix)] [-h (print synopsis, exit)] [--apropos (print synopsis, exit)] [--version (print version, exit)]

  DESCRIPTION
      Use clm close to fetch the connected components from a graph. Different output modes are supported (see below). In matrix mode  (i.e.  using
      the  -imx option) the output returned with --write-cc can be used in conjunction with mcxsubs to retrieve individual subgraphs corresponding
      to connected components.

  OPTIONS
      -abc <fname> (label input)
	The file name for input that is in label format.

      -imx <fname> (input matrix)
	The file name for input that is in mcl native matrix format.

      -o fname (output file)
	Specify the file where output is sent to. The default is STDOUT.

      -dom fname (input domain/cluster file)
	If this option is used, clm close will, as a first step, for each of the domains in file fname retrieve the associated subgraph  from  the
	input graph. These are then further decomposed into connected components, and the program will process these in the normal manner.

      --write-count (output component count)
      --write-sizes (output component sizes (default))
      --write-cc (output components as clustering)
      --write-block (output graph restricted to -dom argument)
      --write-blockc (output graph complement of -dom argument)
	The  default  behaviour  is  currently to output the sizes of the connected components. It is also possible to simply output the number of
	components with --write-count or to write the components as a clustering in mcl format with -write-cc. Even more options exist: it is pos-
	sible to output the restriction of the input graph to a domain, or to output the complement of this restriction.

      --is-undirected (omit graph undirected check)
	With  this option the transformation to make sure that the input is undirected is omitted. This will be slightly faster. Using this option
	while the input is directed may lead to erronenous results.

      -cc-bound num (select components with size at least num)
	Transform the input matrix values according to the syntax described in mcxio(5).

  AUTHOR
      Stijn van Dongen.

  SEE ALSO
      mclfamily(7) for an overview of all the documentation and the utilities in the mcl family.

  clm close 12-068						      8 Mar 2012							clm close(1)
All times are GMT -4. The time now is 12:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy