Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Getting information from various files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Getting information from various files
# 1  
Old 01-22-2011
Getting information from various files
I have got this piece of csh code that looks into various log files and outputs some parameters

For example, I might have 4 files and want to grep for the lines containing "Best Value"

npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log

Doing
Code:
grep "Best Value" *dc0p004-16x12drw*.log

gives

Code:
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:0. Best Value   = 0.0348089
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:1. Best Value   = 0.0348089
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:2. Best Value   = 0.0347595
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:3. Best Value   = 0.0347595
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:4. Best Value   = 0.0343211
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:5. Best Value   = 0.0343211
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:6. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:7. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:8. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:9. Best Value   = 0.0341526
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:10. Best Value   = 0.0341407
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:11. Best Value   = 0.0341101
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:12. Best Value   = 0.0340129
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:13. Best Value   = 0.0340129
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:14. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:15. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:0. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:1. Best Value   = 0.0338745
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:2. Best Value   = 0.0338223
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:3. Best Value   = 0.0338054
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:4. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:5. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:6. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:7. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:8. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:9. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:10. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:11. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:12. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:13. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:14. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:0. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:1. Best Value   = 0.0336147
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:2. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:3. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:4. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:5. Best Value   = 0.0334184
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:6. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:7. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:8. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:0. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:1. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:2. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:3. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:4. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:5. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:6. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:7. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:8. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:9. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:10. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:11. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:12. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:13. Best Value   = 0.033229


Now I want to have something like this stored in a file

Code:
0. Best Value = 0.0348089 
1. Best Value = 0.0348089
2. Best Value = 0.0347595
3. Best Value = 0.0347595
4. Best Value = 0.0343211
5. Best Value = 0.0343211
6. Best Value = 0.034262
7. Best Value = 0.034262
8. Best Value = 0.034262
9. Best Value = 0.0341526
10. Best Value = 0.0341407
11. Best Value = 0.0341101
12. Best Value = 0.0340129
13. Best Value = 0.0340129
14. Best Value = 0.0339073
15. Best Value = 0.0339073
0. Best Value = 0.0339073
1. Best Value = 0.0338745
2. Best Value = 0.0338223
3. Best Value = 0.0338054
4. Best Value = 0.033714
5. Best Value = 0.033714
6. Best Value = 0.033714
7. Best Value = 0.0336161
8. Best Value = 0.0336161
9. Best Value = 0.0336161
10. Best Value = 0.0336161
11. Best Value = 0.0336161
12. Best Value = 0.0336161
13. Best Value = 0.0336161
14. Best Value = 0.0336161
0. Best Value = 0.0336161
1. Best Value = 0.0336147
2. Best Value = 0.0334466
3. Best Value = 0.0334466
4. Best Value = 0.0334466
5. Best Value = 0.0334184
6. Best Value = 0.0333759
7. Best Value = 0.0333759
8. Best Value = 0.0333759
0. Best Value = 0.0333759
1. Best Value = 0.0333759
2. Best Value = 0.0333759
3. Best Value = 0.0333041
4. Best Value = 0.0333041
5. Best Value = 0.0333041
6. Best Value = 0.0332733
7. Best Value = 0.0332733
8. Best Value = 0.0332733
9. Best Value = 0.033229
10. Best Value = 0.033229
11. Best Value = 0.033229
12. Best Value = 0.033229
13. Best Value = 0.033229

As one may notice the 0. value occurs multiple times. I need to remove the entry 0. except on the first occurance which is in the first line

The code below actually does that.

However if I have only one file, the script does not work. Also I want to change the file naming system to

Code:
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run1.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log

which means that if the log files are a continuation of runs, the first run is tagged with -run1- as well. Of course, if only one run is done the lof file will still be
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log, i.e. without the -run1- tag.

I would also need to run the script on such a file and create a file with the parameters of interest.

The csh code is shown below, but I am getting confused how to change it so that I will pass multiple files all having a -run- tag. The code below ignored the -run- tag as currently the first run never has the -run1- tag.

Code:
    foreach f ($fnames)

      echo "(I) ${f}.log, (O) ${f}.mis"

      set fdrwtag = `echo $f | awk '/drw/'`
      set fsmptag = `echo $f | awk '/smp/'`

      if ($fdrwtag != "") then

        grep "Rms Value" $f*.log              \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' > $f.mis

        grep "Best Value" $f*.log             \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' >> $f.mis

      else if ($fsmptag != "") then

        grep "Best Value" $f*.log             \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' >> $f.mis

    end

# 2  
Old 01-23-2011
Try this,

Code:
awk '{if(p!=FILENAME){p=FILENAME;i=0}if(/Best Value/){print i++".",$0}}' *dc0p004-16x12drw*.log

# 3  
Old 01-23-2011
Usually when using grep on various files, one get the file name on each line and then the matching line.

How can I get the file name with the matching line when using just one file?

for example

Code:
grep "Rms Value" *dc0p006-16x12drw-run*.log

gives

Code:
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Rms Value    = 0.00551131
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Rms Value    = 0.0105689
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Rms Value    = 0.0137097
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Rms Value    = 0.0163874
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:4. Rms Value    = 0.0187422
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:5. Rms Value    = 0.0197708
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:6. Rms Value    = 0.0208463
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:7. Rms Value    = 0.0217194
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:8. Rms Value    = 0.0226114
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:9. Rms Value    = 0.0229611
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:10. Rms Value    = 0.0239331
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:11. Rms Value    = 0.025327
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:12. Rms Value    = 0.0250741
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:13. Rms Value    = 0.0255705
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:14. Rms Value    = 0.0266015
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:15. Rms Value    = 0.0268534
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:16. Rms Value    = 0.0264882
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:17. Rms Value    = 0.0269775
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:0. Rms Value    = 0.0269775
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:1. Rms Value    = 0.0271278
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:2. Rms Value    = 0.0267241
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:3. Rms Value    = 0.0267395
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:4. Rms Value    = 0.0267743
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:5. Rms Value    = 0.0268856
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:6. Rms Value    = 0.0273428
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:7. Rms Value    = 0.0283749
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:8. Rms Value    = 0.0278549

but

Code:
grep "Rms Value" *dc0p006-16x12drw-run1.log

gives

Code:
0. Rms Value    = 0.00551131
1. Rms Value    = 0.0105689
2. Rms Value    = 0.0137097
3. Rms Value    = 0.0163874
4. Rms Value    = 0.0187422
5. Rms Value    = 0.0197708
6. Rms Value    = 0.0208463
7. Rms Value    = 0.0217194
8. Rms Value    = 0.0226114
9. Rms Value    = 0.0229611
10. Rms Value    = 0.0239331
11. Rms Value    = 0.025327
12. Rms Value    = 0.0250741
13. Rms Value    = 0.0255705
14. Rms Value    = 0.0266015
15. Rms Value    = 0.0268534
16. Rms Value    = 0.0264882
17. Rms Value    = 0.0269775

# 4  
Old 01-23-2011
Use -H option with grep.
# 5  
Old 01-26-2011
Code:
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Rms Value    = 0.00551131
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Rms Value    = 0.0105689
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Rms Value    = 0.0137097
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Rms Value    = 0.0163874
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:4. Rms Value    = 0.0187422
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:5. Rms Value    = 0.0197708
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:6. Rms Value    = 0.0208463
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:7. Rms Value    = 0.0217194
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:8. Rms Value    = 0.0226114
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:9. Rms Value    = 0.0229611
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:10. Rms Value    = 0.0239331

# 6  
Old 01-26-2011
Grep will print filename at the beginning, try it with awk :
Code:
awk '/Rms Value/' *dc0p006-16x12drw-run*.log

# 7  
Old 01-26-2011
I have file like below. The lines with :0. must be removed except the first one.

Code:
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Best Value   = 0.0346886
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:0. Best Value   = 0.0338255
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:1. Best Value   = 0.0338255
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:2. Best Value   = 0.0337909
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:3. Best Value   = 0.0337666
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:4. Best Value   = 0.0335744
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:0. Best Value   = 0.0335744
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:1. Best Value   = 0.0335052
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:2. Best Value   = 0.0334884
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:3. Best Value   = 0.0334884
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:0. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:1. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:2. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:0. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:1. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:2. Best Value   = 0.0328672
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:3. Best Value   = 0.0328672

I have coded in csh script like this, but am looking to see if I can improve from this to simplify.
Any suggestions welcome.

Code:
          grep -H "Best Value" $f-run*.log                    \
            | awk '{ if (NR == 1) {                           \
                          chisq = $5 / var_rgdt;              \
                          print i++". "$2" "$3" "$4" "$5      \
                      } else {                                \
                          if ($0 !~ /run[0-9]+\.log:0\. /) {  \
                              chisq = $5 / var_rgdt;          \
                              print i++". "$2" "$3" "$4" "$5  \
                          }                                   \
                      }                                       \
                    }' >> $f.csmis

Last edited by kristinu; 01-27-2011 at 11:40 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Matching information from different files

I have a dataset with coordinates of certain points with their names. file1: AAA 5 7 BBB 7 2 CCC 4 4 DDD 4 4 EEE 8 9 FFF 9 9 GGG 4 9 HHH 32 1 III 7 6 JJJ 12 4 KKK 12 3 MMM 15 3The letters represent names of points. In file1 they are unique. I also have file2 with a dataset with... (2 Replies)
Discussion started by: maya3
2 Replies

2. UNIX for Beginners Questions & Answers

Combine information from 2 files

Hi there, I‘m a newbie in linux (ubuntu) working with several files, some of them containing hundred thousands of lines. I started to extract information out of 2 files, combining them by 1 column: I need a Vlookup-like command that reads sampleID (column 2)(line 2,..line by line) in file 1, looks... (1 Reply)
Discussion started by: Nika
1 Replies

3. Shell Programming and Scripting

search information in multiple files and save in new files

hi everyone, im stuck in here with shell :) can you help me?? i have a directory with alot files (genbank files ... all ended in .gbk ) more than 1000 for sure ... and i want to read each one of them and search for some information and if i found the right one i save in new file with new... (6 Replies)
Discussion started by: andreia
6 Replies

4. Shell Programming and Scripting

Updated files information

I am using the below script to remove the rows which contains null values in the 3rd column.My requirement here is want to get the filenames which row is removed .please help me. #!/usr/bin/sh Scripts=/ushhquest/data001/Scripts cd /ushhquest/data011/TgtFiles/MonthlyData ls CUSTADDR*.txt >... (4 Replies)
Discussion started by: katakamvivek
4 Replies

5. UNIX for Dummies Questions & Answers

How to copy files with only certain information

I need to copy a list of files in directory and only files with content DUMMY_B should be copy over to a new directory. How can i do that ? Thanks a lot. (2 Replies)
Discussion started by: Alyssa
2 Replies

6. UNIX for Dummies Questions & Answers

Combining information from Excel files

Hi, I am looking for an AWK or grep script (join will not work here since the data is not sorted) to combine two Excel files that look lke this: Infile1: Georgia Atlanta 1234 1234 Georgia Marrieta 2134 2134 Georgia Scottdale 3414 3414 Georgia Clarkston 2321 2321 Infile2: ... (6 Replies)
Discussion started by: Xterra
6 Replies

7. Shell Programming and Scripting

Compare two files and add new information

Hi, I want to compare two fields in two different files and add a corresponding field in a third output file. Something similar to vlookup. Please see the attached input files and the example output file. I want to compare each entry in column 1 in file1 with column 5 in file2. If both the... (6 Replies)
Discussion started by: chabook
6 Replies

8. UNIX for Dummies Questions & Answers

Prepending information from filenames into files

I would like to know how to take information from a filename and place it into the text of the same file. Let's say I have a file called height_2_width_1.txt containing data that is related to a height of 2 and a width of 1, and the text originally looks like this: where these two columns... (13 Replies)
Discussion started by: Scatterbrain26
13 Replies

9. Solaris

To get the hidden files information only

Hi all, I want to get only hidden files(which are start with '.' or '..') information in a current directory. I tried the below command, $ find . -name "^." -exec ls -la '{}' \; but it's not working. Can anyone give me your outputs. Thanks in advance, Raghu. (5 Replies)
Discussion started by: raghu.iv85
5 Replies

10. Linux

Need to store information relating to certain files

I need to save information relating to certain files that are projected to deleted. I am using find $defPath/archive/sub/subchild -type f -mtime +365 > LOGFILE cat LOGFILE| while read line do ls -l $line | cut -d" " -f 10-15 done But, since cut is cutting columns on space (single... (3 Replies)
Discussion started by: ramu_indian
3 Replies
Login or Register to Ask a Question