Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

List of filenames where column title matches string and value is in limits

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting List of filenames where column title matches string and value is in limits
# 1  
Old 09-16-2011
List of filenames where column title matches string and value is in limits
I'm somewhat new to BASH scripting but have managed to work my way through most of a problem. I'm trying to get a list of filenames where a column header occurs and any value in that column is within a range.

So far I can sort through the list of files in a directory specified by the user, find the column, and get the column number. The problem is that I can't get the column number from one awk line and into the next. I print the variable in the for loop scanning the file, but when I leave the first awk, the variable goes back to null or 0 thereby returning all of the files which match the column header.

I'm not sitting in front of the machine with the code, I'll try to post it later.

Any help would be great.
# 2  
Old 09-16-2011
Quote:
The problem is that I can't get the column number from one awk line and into the next.
awk saves variables between lines just fine, unless you're loading, running, and quitting awk once for each individual line -- somewhat analogous to making 5 phone calls to say a sentence of 5 words... You can probably do a lot more in one awk call. You might be able to do everything in one awk call.

Or do it all in BASH, for that matter -- it's quite capable of splitting and selecting columns by itself.

I'll need to see your input and output data to write it either way though.
# 3  
Old 09-17-2011
Form of Input:
Code:
Header1,Header2,Header3,....
Unit1,Unit2,Unit3,....
Data,Data,Data,....
Data,Data,Data,....
Etc

Form of output:
Code:
/directory/sub/filename1.csv
/directory/sub/filename2.csv

Cant remember exactly what I had but something like:
Code:
For f in file_list
   awk-f',' -v col=$user_parameter 'NR==1{for(i=1;i<NF;i++){if($i==$user_parameter){c=i;break;}}}END{print c}' $f

c comes out right but then:
Code:
awk-f',' 'if($c>=$user_parameter){print FILENAME}'

prints every file in file_list.

Again, I'm not positive that's the exact code I used but I'll post it early next week.
Last edited by Franklin52; 09-21-2011 at 03:28 AM.. Reason: Please use code tags, thank you
# 4  
Old 09-20-2011
Code:
#!/bin/bash
read -p "Enter a directory: " user_directory
read -p "Enter a file type: " user_type
read -p "Enter the parameter of interest: " user_parameter
found_files=$(find $user_directory -name $user_type -print | xargs)
file_list=$(grep -H $user_parameter $found_files -R | cut -d: -f1)
for f in $file_list
do
        #Remove the portion of the header with 0-5 columns
        awk -F',' 'NF>5' $f > /datashare/script_output_temp/${f##*/}"_modified.csv"
done
file_list2=$(find /datashare/script_output_temp -name *.csv -print | xargs)
read -p "Threshold value for parameter: " user_threshold
#Search the first line for the parameter and return that column number
for f in $file_list2
do
        #echo "Processing " $f
        #units=$(awk -F',' -v col=$user_parameter 'NR==1{for(i=1;i<=NF;i++){if($i==col){c=i;break}}} NR==2{print $c}' $f)
        #awk -F',' -v col=$user_parameter 'NR==1{for(i=1;i<=NF;i++){if($i==col){c=i;break}}} NR==2{print $c}' $f > /datashare/script_output_temp/units.dat
        sed -i '2d' $f
        awk -F',' -v col=$user_parameter 'NR==1{for(i=1;i<=NF;i++){if($i==col){break}}}END{if($i>=$user_threshold){print FILENAME}}' $f >> /datashare/script_output_temp/list.csv
        #echo $col_num
        #awk '{if($col_num>=$user_threshold){print FILENAME}}' $f >> /datashare/script_output_temp/list.csv
done
sort -u /datashare/script_output_temp/list.csv > /datashare/script_output_temp/list.dat

Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!
Last edited by vgersh99; 09-20-2011 at 01:08 PM.. Reason: code tags, please!
# 5  
Old 09-20-2011
Quote:
Originally Posted by hu_r_u2000
Cant remember exactly what I had but something like:
For f in file_list
awk-f',' -v col=$user_parameter 'NR==1{for(i=1;i<NF;i++){if($i==$user_parameter){c=i;break;}}}END{print c}' $f

c comes out right but then:
awk-f',' 'if($c>=$user_parameter){print FILENAME}'

prints every file in file_list.
You'll have to combine the two awk scripts into one, variables aren't shared across two separate executions of awk.

You can put more than one line into awk. You can even put large awk scripts into a script file, and run them with awk -f file.awk datafile so they don't clutter up your script.
# 6  
Old 09-20-2011
How do I do that? I'm a little confused on piping one awk into another. I assume thus is what you're referring to.

---------- Post updated at 12:46 PM ---------- Previous update was at 12:02 PM ----------

Turns out:
Code:
awk -F',' -v col=$user_parameter 'NR==1{for(i=1;i<=NF;i++){if($i==col){break}}}END{if($i>=$user_threshold){print FILENAME}}' $f >> /datashare/script_output_temp/list.csv

Needs to be (single quotes around $user_threshold):
Code:
awk -F',' -v col=$user_parameter 'NR==1{for(i=1;i<=NF;i++){if($i==col){break}}}END{if($i>='$user_threshold'){print FILENAME}}' $f >> /datashare/script_output_temp/list.csv

Last edited by Franklin52; 09-21-2011 at 03:28 AM.. Reason: Please use code tags, thank you
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Soft and hard limits for nproc value in /etc/security/limits.conf file (Linux )

OS version : RHEL 6.5 Below is an excerpt from /etc/security/limits.conf file for OS User named appusr in our server appusr soft nproc 2047 appusr hard nproc 16384 What will happen if appusr has already spawned 2047 processes and wants to spawn 2048th process ? I just want to know... (3 Replies)
Discussion started by: kraljic
3 Replies

2. Shell Programming and Scripting

Difference of the same column when two other column matches and one column differs less than 1 hour

This is my input file : # cat list 20130430121600, cucm, location,76,2 20130430121600,cucm1,location1,76,4 20130430122000,cucm,location,80,8 20130430122000,cucm1,location1,90,8 20130430140000,cucm1,location1,87,11 20130430140000, cucm,location,67,9 This is the required output ... (1 Reply)
Discussion started by: Lakshmikumari
1 Replies

3. Shell Programming and Scripting

How to get difference of the same column between two files when other column matches?

File 1: 20130416,235800,10.78.25.104,BR2-loc,60.0,1624,50.0,0,50.0,0 20130416,235800,10.78.25.104,BR1-LOC,70.0,10,50.0,0,70.0,0 20130416,235800,10.78.25.104,Hub_None,60.0,15,60.0,0,50.0,0 File 2: 20130417,000200,10.78.25.104,BR2-loc,60.0,1626,50.0,0,50.0,0... (3 Replies)
Discussion started by: Lakshmikumari
3 Replies

4. Shell Programming and Scripting

awk - how to get difference of the same column when other column matches

I have a file like this : # cat list cucm, location,76,2 cucm1,location1,76,4 cucm,location,80,8 cucm1,location1,90,8 cucm1,location1,87,11 cucm,location,67,9 and I want output like this : cucm,location,76,2 cucm1,location1,76,4 cucm,location,80, 6 ===> (8-2 =6) cucm1,location1,90,4... (5 Replies)
Discussion started by: Lakshmikumari
5 Replies

5. Shell Programming and Scripting

Replace column that matches specific pattern, with column data from another file

Can anyone please help with this? I have 2 files as given below. If 2nd column of file1 has pattern foo1@a, find the matching 1st column in file2 & replace 2nd column of file1 with file2's value. file1 abc_1 foo1@a .... abc_1 soo2@a ... def_2 soo2@a .... def_2 foo1@a ........ (7 Replies)
Discussion started by: prashali
7 Replies

6. UNIX for Dummies Questions & Answers

Find the list of filenames that have the string 31 at 4th and 5th position

Hi, Can anyone let me know the command to know the list of filenames that have string 31 in their 4th and 5th positions inside the file: grep -l "31" main*.txt The above grep lists all the files which have 31 at any position but I want filenames having 31 at position 4 and position 5. (8 Replies)
Discussion started by: okkadu
8 Replies

7. UNIX for Dummies Questions & Answers

Creating a column based list from a string list

I have a string containing fields separated by space Example set sr="Fred Ted Joe Peter Paul Jean Chris Tim Tex" and want to display it in a column format, for example to a maximum of a window of 100 characters And hopefully display some thing like Fred Ted Joe ... (3 Replies)
Discussion started by: kristinu
3 Replies

8. Shell Programming and Scripting

How to print the last column of the list contain filenames with spaces.

Hi experts, I have the following data: I want the last filed in the output. How to print the last field , It contains the file names and few filenames with white spaces . -rw-r--r-- 1 root root 0 2010-04-26 16:57 file1 2space_File.txt -rw-r--r-- 1 root root 0 2010-04-26... (2 Replies)
Discussion started by: rveri
2 Replies

9. Shell Programming and Scripting

perl script to list filenames that do not contain given string

Hi, I need to write a perl script that should do a search recursively in all the '*.txt' files for the string "ALL -Tcb" and should print only the file names that do not contain this string. (21 Replies)
Discussion started by: royalibrahim
21 Replies
Login or Register to Ask a Question