Apply 'awk' to all files in a directory or individual files from a command line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Apply 'awk' to all files in a directory or individual files from a command line
# 1  
Old 11-04-2010
Apply 'awk' to all files in a directory or individual files from a command line

Hi All,

I am using the awk command to replace ',' by '\t' (tabs) in a csv file. I would like to apply this to all .csv files in a directory and create .txt files with the tabs.

How would I do this in a script?

I have the following script called "csvtabs":
Code:
awk 'BEGIN {
                FS = ","
                OFS = "\t"
        }

        {
                $1 = $1
                for (i = 1; i <= NF; i++) {
                        if ($i == "") {
                                $i = "null"
                        }
                }
                print $0
        }' test.csv > test.txt

which makes test.txt with tabs. However, this only works for one file (test.csv) which I have to change each time to the file I want to convert. Since I have many files I want to make the process faster.

Alternatively, if the solution is too difficult, how can I alter the script to take in files at the command line. In other words run:
Code:
$ csvtabs test.csv test.txt

instead
Code:
$ csvtabs

Any help is appreciated,

Cheers,
ScKaSx

Last edited by Franklin52; 11-04-2010 at 04:18 PM.. Reason: Please use code tags
# 2  
Old 11-04-2010
Try this:
Code:
awk 'BEGIN {FS = ","; OFS = "\t"}
f!=FILENAME{f=FILENAME;split(f,a,".")}
{
  $1 = $1
  for (i = 1; i <= NF; i++) {
    if ($i == "") {
      $i = "null"
    }
  }
  print $0 > a[1]".txt"
}' *.csv

# 3  
Old 11-04-2010
Code:
cd /temp/temp1
for i in *.csv
do
awk 'BEGIN {
FS = ","
OFS = "\t"
}

{
$1 = $1
for (i = 1; i <= NF; i++) {
if ($i == "") {
$i = "null"
}
}
print $0
}' $i > ${i%".csv"}".txt"
done

# 4  
Old 11-04-2010
Thanks guys, these work great!

Maybe you can also shed so light on another thing I am trying to do. In these files I have 3 columns with example data as such:

1 2 3
2 3 4
3 4 5
4 3 4
5 2 3
6 3 4
7 4 5
8 5 6
9 6 7
10 5 6

Looking at column 2 or 3 (maybe specified at command line) I want to find the two maximums and write to file (in case of column 2 it would be 4 and 6). Since this is somewhat complicated I first was trying to find just the maximum from one of the columns. Therefore, I tried to add this to the awk command:

Code:
      3 max=1
      4 awk 'BEGIN {
      5       FS = ","
      6       OFS = "\t"
      7    }
      8 
      9    {
     10       $1 = $1
     11       for (i = 1; i <= NF; i++) {
     12          if ($i == "") {
     13             $i = "null"
     14          }
     15          if ($2 > max) {
     16             max = $2
     17          }
     18       }
     19       print $0
     20    }' test.csv > test.txt
     21 echo $max"

As you can probably tell, I have a c background. Anyways, this isn't giving me the maximum number. Is this even the right approach for what I ultimately want?

Cheers,
ScKaSx
# 5  
Old 11-04-2010
awk does not run inside your shell, hence cannot set shell variables. If you want to get a value out of awk, you have to print it somehow. Since you're already using stdout for data, maybe stderr?

Code:
END { print "max is", $max > "/dev/stderr"; }

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Apply command to all files in folder

Hi all! I have this command grep -E '^\To: |^\Date: |^\Subject: ' fileA.txt > fileA_1.txt && grep -v '^\To: |^\Date: |^\Subject: ' fileA.txt >> fileA_1.txt && rm fileA.txt && sed -i -e 's/\(Date: \|Subject: \|To: \)//g' fileA_1.txtHow do I apply it to all the files in the folder (each file has a... (7 Replies)
Discussion started by: guilliber
7 Replies

2. Shell Programming and Scripting

awk to match and apply condtions to matchijng files in directories

I am trying to merge the below awk, which compares two files looking for a match in $2 and then prints the line if two conditions are meet. awk awk 'FNR==NR{A=$0;next} ($2 in A){if($10>30 && $11>49){print A}}' F113.txt F113_tvc.bed This code was improved and provided by @RavinderSingh13,... (18 Replies)
Discussion started by: cmccabe
18 Replies

3. Shell Programming and Scripting

Applying the same awk over a directory of files with individual file output

I am trying to apply an awk action over multiple files in a directory. It is a simple action, I want to print out the 1st 2 columns (i.e. $1 and $2) in each tab-separated document and output the result in a new file *.pp This is the awk that I have come up with so far, which is not giving me a... (6 Replies)
Discussion started by: owwow14
6 Replies

4. Shell Programming and Scripting

Apply argument to all files in directory

Hi all: i need to run a rather simple command-line argument: head -200 input > output However, I need to do it on several files, all in the same directory. Is this possible? (2 Replies)
Discussion started by: owwow14
2 Replies

5. Shell Programming and Scripting

reading information from a table and apply a command on multiple files

Hey gyuz, I wanna calculate the number of mapped reads of a bam file in a region of interest. I used this code to do so : samtools view input.bam chrname:region1 > region1.txt This will store all the reads from given bam file within the region of interest in region1.txt Now I have... (5 Replies)
Discussion started by: @man
5 Replies

6. Shell Programming and Scripting

awk command to compare a file with set of files in a directory using 'awk'

Hi, I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files. To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Discussion started by: anandek
10 Replies

7. Shell Programming and Scripting

apply record separator to multiple files within a directory using awk

Hi, I have a bunch of records within a directory where each one has this form: (example file1) 1 2 50 90 80 90 43512 98 0909 79869 -9 7878 33222 8787 9090 89898 7878 8989 7878 6767 89 89 78676 9898 000 7878 5656 5454 5454 and i want for all of these files to be... (3 Replies)
Discussion started by: amarn
3 Replies

8. Shell Programming and Scripting

How to apply a regular expression in all the files in a directory

I have say 100 text files (with .txt extension) in a directory. An example of the content in the file is given below "NAME" "cgd1_200" "cgd1_3210" "cgd1_560" "cgd2_2760" "cgd2_290" "cgd3_3210" "cgd3_3310" "cgd3_660" "cgd5_2130" "cgd5_4080" "cgd6_3690" "cgd6_4480" "cgd8_1540"... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

9. Shell Programming and Scripting

batch shell script to zip individual files in directory - help

help trying to figure out a batch shell script to zip each file in a directory into its own zip file using this code but it does not work tryed this also nothing seems to work , just ends without zipping any files i have over 3000 files i need to zip up individualy ... (7 Replies)
Discussion started by: wingchun22
7 Replies

10. UNIX for Dummies Questions & Answers

Let GID apply to new files in directory

Hi, Does anyone know if it is possible to override the GID which files have when they are created in a specific folder? I want the given GID for the folder to apply to the new files created in the folder, no matter what group the owner of the files have... I have tried sticky bits but doesn't... (1 Reply)
Discussion started by: linge
1 Replies
Login or Register to Ask a Question