Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Extract information from file

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 2 Weeks Ago
vedanta vedanta is offline
Registered User
 
Join Date: Mar 2013
Last Activity: 15 May 2017, 3:11 AM EDT
Posts: 26
Thanks: 11
Thanked 0 Times in 0 Posts
Extract information from file

In a particular directory, there can be 1000 files like below.

filename is job901.ksh

Code:
#!/bin/ksh
cront -x << EOJ
submit file=$PRODPATH/scripts/genReport.sh maxdelay=30
      &node=xnode01
      tname=job901
      &pfile1=/prod/mldata/data/test1.dat
      &pfile2=/prod/mldata/data/test2.dat
      &metafile1=test1.met
      &metafile2=test2.met
      &jobname=job901
      &priority=10;
      EOJ
exit


Want to read similar all files and extract the info. That means the output would have info for each
of the files with such format.The output for one file would be expected like below.

Is it possible?


HTML Code:
File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay
job901.ksh| job901  | xnode01| test1.dat,test2.dat | test1.met,test2.met | job901| 10 | 30
Thanks.
Sponsored Links
    #2  
Old Unix and Linux 2 Weeks Ago
RavinderSingh13 RavinderSingh13 is online now Forum Advisor  
Registered User
 
Join Date: May 2013
Last Activity: 29 May 2017, 7:35 AM EDT
Location: Chennai
Posts: 2,477
Thanks: 553
Thanked 1,172 Times in 1,055 Posts
Hello Vedanta,

Could you please try following and let me know if this helps you.

Code:
awk -F'[=/]' 'BEGIN{print "File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay"} /maxdelay/{delay=$NF;next} /node/{node=$NF;next} /tname/{name=$NF;next} /pfile/{file=file?file","$NF:$NF;next} /metafile/{metafile=metafile?metafile","$NF:$NF;next} /jobname/{jobname=$NF;next} /priority/{pri=$NF;next} /exit/{print FILENAME OFS jobname OFS node OFS file OFS metafile OFS name OFS pri OFS delay;jobname=node=file=metafile=name=pri=delay="";}' OFS="|  " *.ksh

I haven't tested it with 1000 or more files, let us know if this helps you.
EDIT: Adding a non-one liner form of solution too here.

Code:
awk -F'[=/]' 'BEGIN{
                        print "File      | Jobname |  node  | pfile               | metafile            | tname | priority | delay"
                   }
              /maxdelay/{
                                delay=$NF;
                                next
                        }
              /node/    {
                                node=$NF;
                                next
                        }
              /tname/   {
                                name=$NF;
                                next
                        }
              /pfile/   {
                                file=file?file","$NF:$NF;
                                next
                        }
              /metafile/{
                                metafile=metafile?metafile","$NF:$NF;
                                next
                        }
              /jobname/ {
                                jobname=$NF;
                                next
                        }
              /priority/{
                                pri=$NF;
                                next
                        }
              /exit/    {
                                print FILENAME OFS jobname OFS node OFS file OFS metafile OFS name OFS pri OFS delay;
                                jobname=node=file=metafile=name=pri=delay="";
                        }
             ' OFS="| "   *.ksh

Thanks,
R. Singh
The Following User Says Thank You to RavinderSingh13 For This Useful Post:
vedanta (2 Weeks Ago)
Sponsored Links
    #3  
Old Unix and Linux 2 Weeks Ago
MadeInGermany MadeInGermany is offline Forum Advisor  
Registered User
 
Join Date: May 2012
Last Activity: 29 May 2017, 2:52 AM EDT
Location: Simplicity
Posts: 3,548
Thanks: 277
Thanked 1,182 Times in 1,070 Posts
Here is a bash solution with associative arrays, requires bash 4.

Code:
#!/bin/bash
cols="node tname pfile metafile"
space=20
declare -A C

# print the header
headersep=""
for col in $cols
do
  printf "${headersep}%${space}s" "$col"
  headersep=" | "
done
printf "\n"

# loop over the files
for jfile in job[0-9]*.ksh
do
  # loop over the lines, collect values in hash C[]
  while IFS="=" read key val
  do
    [ -z "${#val}" ] && continue
    case $key in
    *"&node") C[node]=$val
    ;;
    *"&tname") C[tname]=$val
    ;;
    *"&pfile"*) C[pfile]=${C[pfile]}${C[pfile]:+,}${val##*/}
    ;;
    *"&metafile"*) C[metafile]=${C[metafile]}${C[metafile]:+,}${val##*/}
    ;;
    esac
  done < "$jfile"

  # print and clear C[]
  sep=""
  for col in $cols
  do
    printf "${sep}%${space}s" "${C[$col]}"
    unset C[$col]
    sep=$headersep
  done
  printf "\n"
done

It is not yet complete.
But once understood how it works it is easy to expand.
    #4  
Old Unix and Linux 2 Weeks Ago
vedanta vedanta is offline
Registered User
 
Join Date: Mar 2013
Last Activity: 15 May 2017, 3:11 AM EDT
Posts: 26
Thanks: 11
Thanked 0 Times in 0 Posts
Thanks Guys for the help.
Is there a way to use
Code:
awk '/^submit/{print FILENAME;nextfile}' *.sh

instead of
Code:
*.sh

with awk in the script above.
There can be different files with different pattern and I want to pick only those files (1000 files in this case ) which would have content starting with submit.
can the last line be modified to select only those files that have content starting with line 'submit'

Code:
' OFS="| "   *.ksh


Code:
' OFS="| "   awk '/^submit/{print FILENAME;nextfile}' *.sh # pick only those selected files

@Ravinder, It worked. Many thanks!

Last edited by Scrutinizer; 2 Weeks Ago at 03:24 PM.. Reason: code tags
Sponsored Links
    #5  
Old Unix and Linux 2 Weeks Ago
RavinderSingh13 RavinderSingh13 is online now Forum Advisor  
Registered User
 
Join Date: May 2013
Last Activity: 29 May 2017, 7:35 AM EDT
Location: Chennai
Posts: 2,477
Thanks: 553
Thanked 1,172 Times in 1,055 Posts
Quote:
Originally Posted by vedanta View Post
Thanks Guys for the help.
Is there a way to use awk '/^submit/{print FILENAME;nextfile}' *.sh instead of *.sh with awk in the script above.
There can be different files with different pattern and I want to pick only those files (1000 files in this case ) which would have content starting with submit.
can the last line be modified to select only those files that have content starting with line 'submit'

' OFS="| " *.ksh


' OFS="| " awk '/^submit/{print FILENAME;nextfile}' *.sh # pick only those selected files


@Ravinder, It worked. Many thanks!
Hello Vedanta,

Please always open a new thread for a new question, now coming to your question, if you want to print only the file names out of many files which have string submit in them starting of the line then you could make a slight change into your code.

Code:
 awk '/^submit/{print FILENAME;nextfile}' *.sh

Since you haven't told us about the Input_files and their look so I am removing the OFS part here which is anyways not require since you are printing only the file names, in case your Input_files are | delimited then you could add -F"|" into this above code after awk and your string submit is on a specific field you could look only for that field then.

Thanks,
R. Singh
Sponsored Links
    #6  
Old Unix and Linux 2 Weeks Ago
vedanta vedanta is offline
Registered User
 
Join Date: Mar 2013
Last Activity: 15 May 2017, 3:11 AM EDT
Posts: 26
Thanks: 11
Thanked 0 Times in 0 Posts
Hi,

It does not work when I replaced the *.sh by

Code:
awk '/^submit/{print FILENAME;nextfile}' *.sh

in the orginal solution you provided. It gives error can not open file awk
Sponsored Links
    #7  
Old Unix and Linux 2 Weeks Ago
RavinderSingh13 RavinderSingh13 is online now Forum Advisor  
Registered User
 
Join Date: May 2013
Last Activity: 29 May 2017, 7:35 AM EDT
Location: Chennai
Posts: 2,477
Thanks: 553
Thanked 1,172 Times in 1,055 Posts
Quote:
Originally Posted by vedanta View Post
Hi,
It does not work when I replaced the *.sh by

Code:
awk '/^submit/{print FILENAME;nextfile}' *.sh

in the orginal solution you provided. It gives error can not open file awk
Hello Vedanta,

I just tested with 3 files and it worked for me, could you please paste the exact error here? Also make sure you have at least read permissions to files in which you are trying to search the keyword.

Thanks,
R. Singh
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Extract information from file jiam912 Shell Programming and Scripting 11 11-12-2016 01:06 AM
Extract information from txt file edekP Shell Programming and Scripting 11 11-09-2016 12:32 PM
How to extract information from a file? the_simpsons Shell Programming and Scripting 9 05-20-2014 05:47 PM
Extract various information from a log file SilvesterJ Shell Programming and Scripting 4 09-27-2011 12:00 PM
extract and format information from a file sujoy101 Shell Programming and Scripting 12 04-16-2008 06:58 AM



All times are GMT -4. The time now is 07:41 AM.