Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Find a pattern and traverse left and pick something from another pattern

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find a pattern and traverse left and pick something from another pattern
# 1  
Old 03-08-2018
Find a pattern and traverse left and pick something from another pattern
I have a Text like below ,

Code:
Detailed Table Information      Table(tableName:a1, dbName:default, owner:eedc_hdp_s_d-itm-e, createTime:1520514151, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:columna, type:string, comment:this is first column which has comment), FieldSchema(name:columnb, type:int, comment:null)], location:hdfs://DBDP-Dev/apps/hive/warehouse/a1, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, rawDataSize=0, numRows=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1520514151}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)


from the text i need to

1) Check whether we have comment:null
2) if yes identify the parent table of the column identified by string "Table(tableName:" and a1 is tablename . Then identify the Columnname for which comemnt is null that is columnb from this string "FieldSchema(name:columnb, type:int, comment:null)"
3) Once i have both column name and tablename print

Code:
There is no comment for Column Columnname for table Tablename

as output where both Colmnname and Tablenames are values we found in step2

Can you please help to achieve this.

Note:I have a linux distribution with awk version GNU Awk 3.1.8 and using Bash shell
Last edited by Don Cragun; 03-10-2018 at 04:46 AM.. Reason: Change QUOTE tags to NOPARSE and CODE tags.
# 2  
Old 03-08-2018
How would you apply / adapt what you have learned from your other threads to this new but not that different problem?
# 3  
Old 03-08-2018
Hint: if you use ), ( or , as your record separator and : as your field separator you are close to your solution:

Code:
$ awk -F: 'NF>1{print $1 "->" $2}' RS="[)(,]" infile | grep -E "[Nn]ame|comment"
tableName->a1
 dbName->default
name->columna
 comment->this is first column which has comment
name->columnb
 comment->null
name->null
skewedColNames->[]

# 4  
Old 03-09-2018
Working on it
thanks RudiC and Chubler_XL for the directions.

I am working on the problem. I have picked up Arnold Robbins book on AWK and trying to see how can i solve this .

Hopefully i can have something to show in another couple of days .
# 5  
Old 03-09-2018
The problem might be a bit too complex to solve with a text book in hand. There will be several different approaches to the solution; this may give you a starting point:
Code:
awk -F, '
match ($0, /tableName:.*comment:null[^]]*[]]/)          {$0 = substr ($0, RSTART, RLENGTH)
                                                         printf "There is no comment for Column "
                                                         gsub (/FieldSchema\(name:/, "\n")
                                                         n = split ($0, T, "\n")
                                                         for (i=2; i<=n; i++) if (T[i] ~ /comment:null/)        {printf "%s%s", DL, substr (T[i], 1, index (T[i], ",") - 1)
                                                                                                                 DL = ","
                                                                                                                }
                                                         printf " for table %s\n", substr ($0, 11, index($0, ",") - 11)
                                                        }
' file
There is no comment for Column columnb for table a1

It takes care of several uncommented columns in a table already.
# 6  
Old 03-09-2018
Code:
cat nv186000.file

Code:
Detailed Table Information Table(tableName:a1, dbName:default, owner:eedc_hdp_s_d-itm-e, createTime:1520514151, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:columna, type:string, comment:this is first column which has comment), FieldSchema(name:columnb, type:int, comment:null)], location:hdfs://DBDP-Dev/apps/hive/warehouse/a1, inputFormatLinuxrg.apache.hadoop.mapred.TextInputFormat, outputFormatLinuxrg.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLibLinuxrg.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, rawDataSize=0, numRows=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1520514151}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Detailed Table Information Table(tableName:a2, dbName:default,
Testing filler FieldSchema(name:columnA, type:string, comment:null)
Detailed Table Information Table(tableName:a2, dbName:default, owner:eedc_hdp_s_d-itm-e, createTime:1520514151, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:columnA, type:string, comment:null), FieldSchema(name:columnB, type:int, comment:null)], location:hdfs://DBDP-Dev/apps/hive/warehouse/a1, inputFormatLinuxrg.apache.hadoop.mapred.TextInputFormat, outputFormatLinuxrg.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLibLinuxrg.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, rawDataSize=0, numRows=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1520514151}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

Code:
 perl -nle '($t) = /tableName:(\w+)/ or next; while(/name:(\w+),\s?type:\w+,\s?comment:null/g){print "There is no comment for Column $1 for table $t"}' nv186000.file

Output:
Code:
There is no comment for Column columnb for table a1
There is no comment for Column columnA for table a2
There is no comment for Column columnB for table a2

Last edited by Don Cragun; 03-10-2018 at 04:49 AM.. Reason: Change QUOTE tags to CODE tags.
# 7  
Old 03-12-2018
thank you
Thank you Aia , RudiC.

RudiC,

You are right , this solution could not have been done by me in a short span with a book.

But i do see a pattern here in the solutions and can focus on those .

thanks again Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed -- Find pattern -- print remainder -- plus lines up to pattern -- Minus pattern

The intended result should be : PDF converters 'empty line' gpdftext and pdftotext?xml version="1.0"?> xml:space="preserve"><note-content version="0.1" xmlns:/tomboy/link" xmlns:size="http://beatniksoftware.com/tomboy/size">PDF converters gpdftext and pdftotext</note-content>... (9 Replies)
Discussion started by: Klasform
9 Replies

2. UNIX for Advanced & Expert Users

Traverse backwards based on forward pattern match

Hi, I have a file containing records of this format. I need to show output of all header of the sets containing recn:rvn records. The sets appear in my file like below. #set 1 header1:hv1 rec1:rv1 rec2:rv2 rec3:rv3 ....... ...... recn:rn #set 2 header1:hv1 rec1:rv1 rec2:rv2... (3 Replies)
Discussion started by: adurga
3 Replies

3. UNIX for Dummies Questions & Answers

Find pattern suffix matching pattern

Hi, I am trying to get a result out of this but fails please help. Have two files /tmp/1 & /tmp/hosts. /tmp/1 IP=123.456.789.01 WAS_HOSTNAME=abcdefgh.was.tb.dsdc /tmp/hosts 123.456.789.01 I want this result in /tmp/hosts if hostname is already there dont want duplicate entry. ... (5 Replies)
Discussion started by: rajeshwebspere
5 Replies

4. Shell Programming and Scripting

Pick the last one hour lines from log matching this pattern.

Hello please help me on this, pick the last one hour lines from the log, which have the prefix time format like this. log message log message i tried to do grep, but that failed. my code grep '(date +)' log_file_path This checking only the current time stamp. How to get the log... (16 Replies)
Discussion started by: santosh2626
16 Replies

5. UNIX for Dummies Questions & Answers

Find next line based on pattern, if it is similar pattern skip it

Hi, I am able to get next line if it is matching a particular pattern. But i need a way to skip if next line also matches same pattern.. For example: No Records No Records Records found got it Records found Now i want to find 'Records found' after 'No Records' pattern matches.. ... (5 Replies)
Discussion started by: nagpa531
5 Replies

6. Shell Programming and Scripting

Find required files by pattern in xml files and the change the pattern on Linux

Hello, I need to find all *.xml files that matched by pattern on Linux. I need to have written the file name on the screen and then change the pattern in the file just was found. For instance. I can start the script with arguments for keyword and for value, i.e script.sh keyword... (1 Reply)
Discussion started by: yart
1 Replies

7. Shell Programming and Scripting

sed: Find start of pattern and extract text to end of line, including the pattern

This is my first post, please be nice. I have tried to google and read different tutorials. The task at hand is: Input file input.txt (example) abc123defhij-E-1234jslo 456ujs-W-abXjklp From this file the task is to grep the -E- and -W- strings that are unique and write a new file... (5 Replies)
Discussion started by: TestTomas
5 Replies

8. Shell Programming and Scripting

search a pattern and if pattern found insert new pattern at the begining

I am trying to do some thing like this .. In a file , if pattern found insert new pattern at the begining of the line containing the pattern. example: in a file I have this. gtrow0unit1/gctunit_crrownorth_stage5_outnet_feedthru_pin if i find feedthru_pin want to insert !! at the... (7 Replies)
Discussion started by: pitagi
7 Replies

9. UNIX for Advanced & Expert Users

I am trying to find pattern between two words but unable to get that pattern..

HI.... It's fallow up file .. #./show.sh click enter button.. i am gettng the fallowup file. its keep on running every time why because there are lots of users working on it. In that file i want to search pattern between two words for ex: SELECT DISTINCT... (7 Replies)
Discussion started by: ksr.test
7 Replies

10. Shell Programming and Scripting

what left of the pattern

I have a script which loop through a directory then report any file matches the given pattern, say, the pattern is "a2006", this file would be returned a20061101.txt I would like to know how can I get the remaining of the filename, so a20061101txt - a2006 = 1101.txt Can anybody help? Thank... (2 Replies)
Discussion started by: mpang_
2 Replies
Login or Register to Ask a Question