Extracting 22-character strings from text using sed/awk?

Login or Register to Ask a Question and Join Our Community

Extracting 22-character strings from text using sed/awk?

Tags

awk, beginners, sed

Top Forums UNIX for Dummies Questions & Answers Extracting 22-character strings from text using sed/awk?

09-15-2013

Registered User

33, 0

Join Date: May 2011

Last Activity: 21 September 2017, 4:01 PM EDT

Location: Maryland, USA

Posts: 33

Thanks Given: 15

Thanked 0 Times in 0 Posts

Extracting 22-character strings from text using sed/awk?

Here is my task, I feel sure this can be accomplished with see/awk but can't seem to figure out how.

I have large flat file from which I need to extract every case of a pairing of characters (GG) in this case PLUS the previous 20 characters. The output should be a list (which I plan to make non-redunatant using uniq) of every 22-character string that ends in these specific 2 characters.

The input is a just flat file of just a long string of characters without line breaks (usually around 2000-10000 characters each):

Code:

OIQWEUQJDSBDSJDFIEHFOIWEHSDCNSKDHOIFUOIQWEUFOQWHFSHDFK.......

The desired output would look like this:

Code:

PONDJSYHEKSNUQFNHGSPWGG
MJAHENSUAHBEKSLRJSNHEGG
WJANAHRWOAMADUWGINDFHGG
WWJDUIANNJWEUAHSIAJFHGG
....

Any help is appreciated!

Last edited by Twinklefingers; 09-15-2013 at 02:57 PM..

Twinklefingers

View Public Profile for Twinklefingers

Find all posts by Twinklefingers

09-15-2013

Registered User

3,733, 1,154

Join Date: Apr 2009

Last Activity: 3 August 2016, 11:03 AM EDT

Posts: 3,733

Thanks Given: 7

Thanked 1,154 Times in 1,124 Posts

Can you post sample input and desired output?

bartus11

View Public Profile for bartus11

Find all posts by bartus11

09-15-2013

Registered User

15,129, 5,008

Join Date: Jul 2012

Last Activity: 4 May 2020, 4:31 PM EDT

Location: Aachen, Germany

Posts: 15,129

Thanks Given: 735

Thanked 5,008 Times in 4,483 Posts

This will work when
a) there's only one single occurrence of GG in a line
b) there's at least 20 char in front of the GG

Code:

sed -rn 's/^.*(.{20}GG).*$/\1/p' file

In case your sed doen't allow for EREs, try

Code:

sed -n 's/^.*\(.\{20\}GG\).*$/\1/p' file

RudiC

View Public Profile for RudiC

Find all posts by RudiC

09-15-2013

Registered User

33, 0

Join Date: May 2011

Last Activity: 21 September 2017, 4:01 PM EDT

Location: Maryland, USA

Posts: 33

Thanks Given: 15

Thanked 0 Times in 0 Posts

@bartus11 - Added in the post, thanks!

@RudiC - There are going to be more than one instances of GG in each line, however I can prepare the files for analysis by inserting a line break after each instances of GG.

Twinklefingers

View Public Profile for Twinklefingers

Find all posts by Twinklefingers

09-15-2013

Registered User

3,733, 1,154

Join Date: Apr 2009

Last Activity: 3 August 2016, 11:03 AM EDT

Posts: 3,733

Thanks Given: 7

Thanked 1,154 Times in 1,124 Posts

Sample input for that output would be nice to have as well.

bartus11

View Public Profile for bartus11

Find all posts by bartus11

09-15-2013

Registered User

1,709, 666

Join Date: Jan 2013

Last Activity: 20 May 2020, 1:43 PM EDT

Location: Loughborough

Posts: 1,709

Thanks Given: 838

Thanked 666 Times in 467 Posts

Some assumptions have to made here:-
1) The flatfile is pure alpha-numeric.
2) The "GG" ALWAYS has at least 20 characters before it that are NOT those two.
3) The file is not enormous.

In this example I have deliberately inserted GGs inside the 22 character bounderies to
show the possible error that could result.

Code:

#!/bin/bash --posix
echo "1234567890qwertyuiopasdfghjklzxcvbnmGGMNBVCXZLKJHGFDSAPOIUYTREWQ0987654321GG1234567890GGABCDEFGGHIJ" > /tmp/flatfile
read textfile < /tmp/flatfile
echo ""
echo "$textfile"
echo ""
for n in $( seq 0 1 ${#textfile} )
do
	if [ "${textfile:$n:2}" == "GG" ]
		then echo "${textfile:$[ ( $n - 20 ) ]:22}"
	fi
done
echo ""

And the display on OSX 10.7.5 default bash terminal:-

Code:

Last login: Sun Sep 15 18:17:17 on ttys000
AMIGA:barrywalker~> chmod 755 flatfile.sh
AMIGA:barrywalker~> ./flatfile.sh

1234567890qwertyuiopasdfghjklzxcvbnmGGMNBVCXZLKJHGFDSAPOIUYTREWQ0987654321GG1234567890GGABCDEFGGHIJ

uiopasdfghjklzxcvbnmGG
POIUYTREWQ0987654321GG
87654321GG1234567890GG
GG1234567890GGABCDEFGG

AMIGA:barrywalker~> _

NOTE:- The "GG"s inside 22 characters also include earlier "GG"s!

wisecracker

View Public Profile for wisecracker

Find all posts by wisecracker

09-15-2013

Registered User

33, 0

Join Date: May 2011

Last Activity: 21 September 2017, 4:01 PM EDT

Location: Maryland, USA

Posts: 33

Thanks Given: 15

Thanked 0 Times in 0 Posts

@bartus11 - Input added!

Last edited by Twinklefingers; 09-15-2013 at 03:49 PM..

Twinklefingers

View Public Profile for Twinklefingers

Find all posts by Twinklefingers

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Extracting strings at various positions of text file

Hi Team - I hope everyone has been well! I export a file from one of our source systems that gives me more information than I need. The way the file outputs, I need to extract certain strings at different positions on the file and echo them to another file. I can do this in batch easily,...

2. Shell Programming and Scripting

Extracting text between two strings, multiple instances

Hi experts, Ive got a text file which has the following text which will occur in this format at least one time: +=========================>> Some stuff that evreryone should knnow other stufsjdokajkajokajda aijhjajcdjajcisajcqsqdqwdqad <<=========================+ It is likely that...

3. Shell Programming and Scripting

Extracting text between two constant strings

Hi All, I have a file whose common patter is like this: .I 1 .U 87049087 .S Some text here too .M This is a text .T Some another text here .P Name of the book .W Some lines of more text. This text needs to be extracted. .A more text goes here too .I 2

4. Shell Programming and Scripting

any savant ? using AWK/SED to remove newline character between two strings : conditional removal

I'd like to remove (do a pattern or precise replacement - this I can handle in SED using Regex ) ---AFTER THE 1ST Occurrence ( i.e. on the 2nd occurrence - from the 2nd to fourth occurance ) of a specific string : type 1 -- After the 1st occurrence of 1 string1 till the 1st occurrence of...

5. Shell Programming and Scripting

Extracting text between two strings

Hi, I've looked at a few existing posts on this, but they don't seem to work for my inputs. I have a text file where I want to extract all the text between two strings, every time that occurs. Eg my input file is Anna said that she would fetch the bucket. Anna and Ben moved the bucket....

6. Shell Programming and Scripting

replace two character strings by two variables with sed command

Hello, I want to writte a script that replace two character strings by two variables with the command sed butmy solution doesn't work. I'm written this: sed "s/TTFactivevent/$TTFav/g && s/switchSLL/$SLL/g" templatefile. I want to replace TTFactivevent by the variable $TTFav, that is a...

7. UNIX for Advanced & Expert Users

bash/grep/awk/sed: How to extract every appearance of text between two specific strings

I have a text wich looks like this: clid=2 cid=6 client_database_id=35 client_nickname=Peter client_type=0|clid=3 cid=22 client_database_id=57 client_nickname=Paul client_type=0|clid=5 cid=22 client_database_id=7 client_nickname=Mary client_type=0|clid=6 cid=22 client_database_id=6...

8. UNIX for Dummies Questions & Answers

Using awk/sed to extract text between Strings

Dear Unix Gurus, I've got a data file with a few hundred lines (see truncated sample)... BEGIN_SCAN1 TASK_NAME=LA48 PDD Profiles PROGRAM=ArrayScan 1.00 21.220E+00 2.00 21.280E+00 END_DATA END_SCAN1 BEGIN_SCAN2 TASK_NAME=LA48 PDD Profiles 194.00 2.1870E+00 ...

9. Shell Programming and Scripting

Extracting text between two strings, first instance only

There are a lot of ways to extract text from between two strings, but what if those strings occur multiple times and you only want the text from the first two strings? I can't seem to find anything to work here. I'm using sed to process the text after it's extracted, so I prefer a sed answer, but...

10. Shell Programming and Scripting

extracting a set of strings from a text file

i have textfiles that contain a series of lines that look like this: string0 .................................................... column3a column4a string1**384y0439 ..................................... column3b column4b...

Login or Register to Ask a Question