Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract information from file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract information from file
# 8  
Old 05-12-2017
when I replaced *.sh by the line in your original code

I am getting below error:

awk: cmd. line:37: fatal: can not open file 'awk' for reading ( no such file or directory )
# 9  
Old 05-12-2017
Quote:
Originally Posted by vedanta
when I replaced *.sh by the line in your original code
I am getting below error:
awk: cmd. line:37: fatal: can not open file 'awk' for reading ( no such file or directory )
Hello Vedanta,

Are you running this awk program too as a .sh script? If yes then you may need to consider that it will take this script also as this script is also ending with .sh. If this is not the case then you may need to post whatever files are present in the directory and we may need to see what's going on.

Thanks,
R. Singh
# 10  
Old 05-12-2017
I am running the program as a .sh script.

Code:
#!/bin/ksh
awk ' NR==1 {
                        print "File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay" # to be replaced by tab
                   }
              /maxdelay/{
                                delay=$NF;
                                next
                        }
              /node/    {
                                node=$NF;
                                next
                        }
              /tname/   {
                                name=$NF;
                                next
                        }
              /pfile/   {
                                file=file?file","$NF:$NF;
                                next
                        }
              /metafile/{
                                metafile=metafile?metafile","$NF:$NF;
                                next
                        }
              /jobname/ {
                                jobname=$NF;
                                next
                        }
              /priority/{
                                pri=$NF;
                                next
                        }
              /exit/    {
                                print FILENAME OFS jobname OFS node OFS file OFS metafile OFS name OFS pri OFS delay;
                                #jobname=node=file=metafile=name=pri=delay="";
                        }
             ' OFS="\t"   awk '/^submit/{print FILENAME;nextfile}' *.sh  # replaced *.sh with awk

Please see last line. I want to pick only selected files ( 1000 files from many files).

I executed like below
Code:
ksh test.ksh

# 11  
Old 05-12-2017
Quote:
Originally Posted by vedanta
I am running the program as a .sh script.

Code:
#!/bin/ksh
awk ' NR==1 {
                        print "File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay" # to be replaced by tab
                   }
              /maxdelay/{
                                delay=$NF;
                                next
                        }
              /node/    {
                                node=$NF;
                                next
                        }
              /tname/   {
                                name=$NF;
                                next
                        }
              /pfile/   {
                                file=file?file","$NF:$NF;
                                next
                        }
              /metafile/{
                                metafile=metafile?metafile","$NF:$NF;
                                next
                        }
              /jobname/ {
                                jobname=$NF;
                                next
                        }
              /priority/{
                                pri=$NF;
                                next
                        }
              /exit/    {
                                print FILENAME OFS jobname OFS node OFS file OFS metafile OFS name OFS pri OFS delay;
                                #jobname=node=file=metafile=name=pri=delay="";
                        }
             ' OFS="\t"   awk '/^submit/{print FILENAME;nextfile}' *.sh  # replaced *.sh with awk

Please see last line. I want to pick only selected files ( 1000 files from many files).

I executed like below
Code:
ksh test.ksh

Hello Vedanta,

I am shocked, as I have already told you not to mix 2 codes. My first code was to get the output in your expected shape from a Input_file and second code which was posted by you I corrected(which I provided fair warning like don't mix them and open a new thread on same).

Would like to request you if you can segregate your requirements as it is very confusing now.

NOTE: Above your code will NOT work in this style you are using 2 awk by providing 1 time Input_file which is *.sh.

Thanks,
R. Singh
# 12  
Old 05-12-2017
Hi,
I want to pull all the files that have the pattern like the input file given. In a directory there are say 5000 files out of which 1000 files have the text starting with 'submit file'.
and from those files I want to get the output I mentioned earlier.

First filter those files which have the text starting with 'submit file' ( out of 5000, I would have 1000 such input file ) and read those input files which have have the text starting with 'submit file' and get the output in the required format I mentioned earlier. So basically, I want to filter out the files and read those file as input only and get required output. thanks
# 13  
Old 05-12-2017
Quote:
Originally Posted by vedanta
Hi,
I want to pull all the files that have the pattern like the input file given. In a directory there are say 5000 files out of which 1000 files have the text starting with 'submit file'.
and from those files I want to get the output I mentioned earlier.

First filter those files which have the text starting with 'submit file' ( out of 5000, I would have 1000 such input file ) and read those input files which have have the text starting with 'submit file' and get the output in the required format I mentioned earlier. So basically, I want to filter out the files and read those file as input only and get required output. thanks
This is very convoluted logic. Using awk to read all of your 5000 files to get a list of files containing a certain string to use as arguments to another awk script is grossly inefficient since you have to read each of your 5000 and then read the selected 1000 files a second time. There is very seldom a need to invoke awk twice, but if you must you have to actually invoke awk twice instead of just using the command line arguments that would be used to invoke awk as operands to awk as in:
Code:
awk '...
...
             ' OFS="\t"   $(awk '/^submit/{print FILENAME;nextfile}' *.sh)  # replaced *.sh with awk

or:
Code:
awk '...
...
             ' OFS="\t"   $(grep -l '^submit' *.sh)  # filter *.sh with grep

But, as I said before, putting it all in a single awk script would be much more efficient:
Code:
#!/bin/ksh
awk ' NR==1 {
                        print "File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay" # to be replaced by tab
                   }
              /maxdelay/{
                                delay=$NF;
                                next
                        }
              /node/    {
                                node=$NF;
                                next
                        }
              /tname/   {
                                name=$NF;
                                next
                        }
              /pfile/   {
                                file=file?file","$NF:$NF;
                                next
                        }
              /metafile/{
                                metafile=metafile?metafile","$NF:$NF;
                                next
                        }
              /jobname/ {
                                jobname=$NF;
                                next
                        }
              /priority/{
                                pri=$NF;
                                next
                        }
              /^submit/{
                                printit=1;
                                next
                        }
              /exit/ && printit{
                                print FILENAME OFS jobname OFS node OFS file OFS metafile OFS name OFS pri OFS delay;
                                #jobname=node=file=metafile=name=pri=delay="";
                                printit=0
                                nextfile
                        }
             ' OFS="\t" *.sh

This User Gave Thanks to Don Cragun For This Post:
vedanta
# 14  
Old 05-15-2017
Many thanks!
One problem I am seeing here is that awk also is reading the comment lines.
Is there a way to ignore all comment lines at the very beginning? Should I need to create a separate post for this? Thanks.


I have used below as a solution

added !/^#/ eg,
/pfile/ && !/^#/ { ..


Last edited by vedanta; 05-15-2017 at 04:11 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk script to extract transcript information from gff3 file

I need help to extract transcript information from gff3 file. Here is the input Chr01 JGI gene 82773 86941 . - . ID=Potri.001G000900;Name=Potri.001G000900 Chr01 JGI mRNA 82793 86530 . - . ID=PAC:27047814;Name=Potri.001G000900.1;pacid=27047814;longest=1;Parent=Potri.001G000900... (6 Replies)
Discussion started by: Maduranga
6 Replies

2. Shell Programming and Scripting

Extract information from file

Gents, If is possible please help. I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Discussion started by: jiam912
11 Replies

3. Shell Programming and Scripting

Extract information from txt file

Hello! I need help :) I have a file like this: AA BC FG RF TT GH DD FF HH (a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain: for the 1° loop:... (11 Replies)
Discussion started by: edekP
11 Replies

4. Shell Programming and Scripting

How to extract information from a file?

Hi, i have a file like this: <Iteration> <Iteration_iter-num>3</Iteration_iter-num> <Iteration_query-ID>lcl|3_0</Iteration_query-ID> <Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def> <Iteration_query-len>197</Iteration_query-len> ... (9 Replies)
Discussion started by: the_simpsons
9 Replies

5. Shell Programming and Scripting

Extract various information from a log file

Hye ShamRock If you can help me with this difficult task for me then it will save my day Logs : ================================================================================================================== ... (4 Replies)
Discussion started by: SilvesterJ
4 Replies

6. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

7. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

8. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

9. Shell Programming and Scripting

extract and format information from a file

Hi, Following is sample portion of the file; <JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver" MaxCapacity="10" Name="MyApp_DevPool" PasswordEncrypted="{3DES}7tXFH69Xg1c=" Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false" ... (12 Replies)
Discussion started by: sujoy101
12 Replies

10. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies
Login or Register to Ask a Question