Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to match strings that sometimes are splitted in 2 lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to match strings that sometimes are splitted in 2 lines
# 1  
Old 08-24-2013
Script to match strings that sometimes are splitted in 2 lines
Hello to all,

I have an hexdump -C format as below:
Code:
31 54 47 55 48 4c 52 31 5f 52 31 32 31 31 32 ff
44 00 00 0E 01 32 14 56 42 17 47 48 0f ff ff ff
44 00 00 01 32 14 56 00 23 83 95 2f 42 17 47 48
00 0f ff ff 00 15 00 0a 48 00 01 5a 00 02 17 00
00 2f 00 00 30 00 00 31 00 00 ff 34 ff 44 00 00
02 32 14 56 01 75 70 69 8f 42 17 47 48 00 1f ff
ff 00 15 00 0c 48 00 01 5a 00 0a 17 00 01 18 00
01 42 00 01 60 00 01 36 00 01 37 00 01 7e ff 44
00 00 2B 32 14 56 79 00 00 94 00 01 93 00 01 22
00 00 21 00 00 09 00 01 0a 00 01 10 00 01 08 00

I want to do a kind of grep to get the strings that follow the pattern:
Code:
ff 44 + (3 or 4 bytes) + 32 14 56 + (5 bytes)

I'm able to match that pattern in a text editor with the next regex:
Code:
FF( |\n)44(( |\n)[0-9a-f]{2}){3,4}( |\n)32( |\n)14( |\n)56(( |\n)[0-9a-f]{2}){5}

But I don't know how to insert this regex in a bash script (using awk, sed, etc) in order to
get the output below, because with awk the issue I see is that the pattern not always is in a unique line, could begin in one and ends in the next one.

Output desired:
Code:
ff 44 00 00 0E 01 32 14 56 42 17 47 48 0f
ff 44 00 00 01 32 14 56 00 23 83 95 2f
ff 44 00 00 02 32 14 56 01 75 70 69 8f
ff 44 00 00 2B 32 14 56 79 00 00 94 00

May some body help with this.

Thanks in advance.
# 2  
Old 08-24-2013
Hi, try:
Code:
awk -v pat="ff 44( [^ ]{2}){3,4} 32 14 56( [^ ]{2}){5}" '{b=p $0} match(b,pat) {print substr(b,RSTART,RLENGTH); sub(pat,x)} {p=$0 FS}' file

Code:
ff 44 00 00 0E 01 32 14 56 42 17 47 48 0f
ff 44 00 00 01 32 14 56 00 23 83 95 2f
ff 44 00 00 02 32 14 56 01 75 70 69 8f
ff 44 00 00 2B 32 14 56 79 00 00 94 00

Only space (no newline) was used in the matching regex, because space was used to glue two lines together in the variable b The expression will never be on more that two lines and there cannot be two matches on a single line... The substitution (sub() ) is necessary for cases where a match occurs on a single line...

It is a bit much for a single line, so this may be more readable:
Code:
awk -v pat="ff 44( [^ ]{2}){3,4} 32 14 56( [^ ]{2}){5}" '
  {
    b=p $0
  }
  match(b,pat) {
    print substr(b,RSTART,RLENGTH)
    sub(pat,x)
  }
  {
    p=$0 FS
  }
' file

This should work with regular awk. With gawk <= version 3 try gawk --re-interval . With mawk it will not function because it cannot do {m,n} interval expressions, so in that case the regex would need to be expanded..
Last edited by Scrutinizer; 08-24-2013 at 08:25 AM..
# 3  
Old 08-24-2013
Hello Scrutinizer,

Thank you, it works just fine.

I'm trying to modify your code in order to group the bytes as follow:
Code:
ff 44 00000E01 321456421747480f
ff 44 000001   321456002383952f
ff 44 000002   321456017570698f
ff 44 00002B   3214567900009400

But trying to separate with substr(b,1,2) substr(3,4) ... is printing in different order, that I'm don't understand yet how is stored in b.

The objetive to group it is to print as decimal the 3rd column as below:
Code:
ff 44 3585 	321456421747480f
ff 44 1 	321456002383952f
ff 44 2 	321456017570698f
ff 44 43	3214567900009400

How can I do this?

Thanks again.
# 4  
Old 08-24-2013
You would need a bit of additional substring wrestling, try something like:
Code:
awk -v pat="ff 44( [^ ]{2}){3,4} 32 14 56( [^ ]{2}){5}" '
  {
    b=p $0
  } 
  match(b,pat) {
    s=substr(b,RSTART,RLENGTH)
    gsub(FS,x,s)
    l1=length(s)-20
    f1=substr(s,5,l1)
    f2=substr(s,5+l1)
    printf "ff 44 %8-d%s\n","0x" f1,f2
    sub(pat,x)
  }
  {
    p=$0 FS
  }
' file

or remove those spaces beforehand:
Code:
awk -v pat="ff44.{6}(..)?321456.{10}" '
  {
    gsub(FS,x)
    b=p $0
  } 
  match(b,pat) {
    s=substr(b,RSTART,RLENGTH)
    l1=length(s)-20
    f1=substr(s,5,l1)
    f2=substr(s,5+l1)
    printf "ff 44 %8-d%s\n","0x" f1,f2
    sub(pat,x)
  }
  {
    p=$0
  }
' file

Last edited by Scrutinizer; 08-24-2013 at 03:44 PM..
# 5  
Old 08-24-2013
Hello Scrutinizer,

Thanks for your time!.

I've trying but is printing "0's" only.
Code:
ff 44 0       321456421747480f
ff 44 0       321456002383952f
ff 44 0       321456017570698f
ff 44 0       3214567900009400

# 6  
Old 08-24-2013
  • Hi did you prepend the first value with "0x" ?
  • Otherwise, what does awk 'BEGIN{printf "%8-d\n", "0x" "00000E01"}' produce?
  • What is your OS and version?
# 7  
Old 08-24-2013
Hello Scrutinizer,

Produces "0". But strtonum produces correct output.
Code:
$ awk 'BEGIN{printf "%8-d\n", "%0x" "00000E01"}'
0
$ awk 'BEGIN{print strtonum(0xE01)}'
3585

I'm using Cygwin, latest version on Windows.

Code:
CYGWIN_NT-6.1-WOW64 2013-08-15 11:55 i686 Cygwin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Use strings from nth field from one file to match strings in entire line in another file, awk

I cannot seem to get what should be a simple awk one-liner to work correctly and cannot figure out why. I would like to use patterns from a specific field in one file as regex to search for matching strings in the entire line ($0) of another file. I would like to output the lines of File2 which... (1 Reply)
Discussion started by: jvoot
1 Replies

2. Shell Programming and Scripting

Delimited records splitted into different lines

Hi I am using delimited sequence file. Delimter we are using is pipe .But for some of the records for one of the column the values are getting split into different lines as shown below "113"|"0155"|"2016-04-27 07:59:04"|"1930"|"TEST@TEST"|"2016-04-27 11:04:04.357000000"|"BO"|"Hard... (13 Replies)
Discussion started by: ginrkf
13 Replies

3. Shell Programming and Scripting

Script to match lines in screen

I'd like to ask people who knows bash scripting to write me a script which would open a specific screen and match lines. Here is algorithm I'm thinking about. Find SCREENS named name1, name2.... and nameX. Open them one by one and type 'STATS' Match last lines of the screen before command... (3 Replies)
Discussion started by: GhostMan
3 Replies

4. Shell Programming and Scripting

Returning two lines if they both match strings

Hi I have a problem where I have a large amount of files that I need to scan and return a line and its following line, but only when the following line begins with a string. String one - line one must begin with 'Bill' String two - line two must begin with 'Jones'. If these two... (7 Replies)
Discussion started by: majormajormajor
7 Replies

5. Shell Programming and Scripting

Print only lines where fields concatenated match strings

Hello everyone, Maybe somebody could help me with an awk script. I have this input (field separator is comma ","): 547894982,M|N|J,U|Q|P,98,101,0,1,1 234900027,M|N|J,U|Q|P,98,101,0,1,1 234900023,M|N|J,U|Q|P,98,54,3,1,1 234900028,M|H|J,S|Q|P,98,101,0,1,1 234900030,M|N|J,U|F|P,98,101,0,1,1... (2 Replies)
Discussion started by: Ophiuchus
2 Replies

6. Shell Programming and Scripting

Script to multi-transfer splitted files via scp

Hey :3 I am moving some stuff between different servers. I do it like this: scp -r -P 22 -i ~/new.ppk /var/www/bigfile.tar.gz user@123.123.123.123:/var/www/bigfile.tar.gz Lets say, this file is 50 GiB. I would like to know, if its possible to split the file in different parts,... (2 Replies)
Discussion started by: Keenora
2 Replies

7. Shell Programming and Scripting

Delete lines in file containing duplicate strings, keeping longer strings

The question is not as simple as the title... I have a file, it looks like this <string name="string1">RZ-LED</string> <string name="string2">2.0</string> <string name="string2">Version 2.0</string> <string name="string3">BP</string> I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies

8. Shell Programming and Scripting

Strings from one file which exactly match to the 1st column of other file and then print lines.

Hi, I have two files. 1st file has 1 column (huge file containing ~19200000 lines) and 2nd file has 2 columns (small file containing ~6000 lines). ################################# huge_file.txt a a ab b ################################## small_file.txt a 1.5 b 2.5 ab ... (4 Replies)
Discussion started by: AshwaniSharma09
4 Replies

9. Shell Programming and Scripting

shell script: grep multiple lines after pattern match

I have sql file containing lot of queries on different database table. I have to filter specific table queries. Let say i need all queries of test1,test2,test3 along with four lines above it and sql queries can be multi lines or in single line. Input file contains. set INSERT_ID=1; set... (1 Reply)
Discussion started by: mirfan
1 Replies

10. Shell Programming and Scripting

Perl script to match a pattern and print lines

Hi I have a file (say 'file1')and I want to search for a first occurence of pattern (say 'ERROR') and print ten lines in the file below pattern. I have to code it in PERL and I am using Solaris 5.9. I appreciate any help with code Thanks Ammu (6 Replies)
Discussion started by: ammu
6 Replies
Login or Register to Ask a Question