Remove first 2 characters and last two characters of each line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove first 2 characters and last two characters of each line
# 1  
Old 12-17-2016
Remove first 2 characters and last two characters of each line

here's what im trying to do.

i have a file containing lines similar to this:

data.txt:
Code:
1hsRmRsbHRiSFZNTTA1dlEyMWFkbU5wUW5CSlIyeDFTVU5SYjJOSFRuWmpia0ZuWXpKV2FHTnRU
1lKUnpWMldrZFZaMG95V25oYQpSelEyWTBka2QyRklhSHBrUjA1b1kwUkJkd3BOVXpWM1lVaG5k
195Um5kaFZ6VnFURzFPZG1KVFFsQlZhVUp0WTFkU2RVOXVRbTVqClIyZzBZek5TYWxsWVFYZE5S
1wxWTBkb05FeHRaR2hqUjJ4MVdYazFhbUl5TUdkVU1VbG5DbHB1Um10aWFuQjNXak5DYjJWSVRq
22oKTWtaM1RVUkJla3h1UW05bFF6VnVXVmhDY0dKdFRYVlpNamwwU1VVNVUwbEhXbmhhUnpRMlkw
ZGtkMkZJYUhwa1IwNW9ZMFJCZDA1RApOWGNLWVVobmRWb3lSbmRoVnpWcVRHMU9kbUpUUWxCVmFV
66qRZMWRTZFU5dVFtNWpSMmcwWXpOU2FsbFlRWGROUkZWMVkwZG9ORXh0ClpHaGpSMngxV1hrMWFt

I need to remove the first two characters and the last two characters of each line. but i also need to be able to put those removed characters back to where they were before they were removed.

so, here's what i'm doing so far:

Code:
sed -e 's ..  ' -e  's ..$  ' data.txt > data.txt.pruned

This only removes the first two characters of each line and the last two.

where im getting stuck is, how do i get back the removed characters?

a thought that crossed my mind is to first get the first two characters of each line line. save it to a file and do the same for the last two characters.

Code:
some_magical_command data.txt > firstwo.characters
some_magical_command data.txt > lastwo.characters

then find a way to merge firstwo.characters and lastwo.characters with data.txt.pruned so they look like the original text in data.txt?

i suppose the commands join may be used here. but im looking for something that can work on all Unix systems (portable), especially those systems that are considered "embedded". if awk can be used here, that'll be great.

Last edited by SkySmart; 12-18-2016 at 12:05 AM..
# 2  
Old 12-18-2016
Hello SkySmart,

Could you please try following and let me know how it goes then, though I am not sure about your complete requirement.
Code:
cat script.ksh 
FIRST_TWO=$(awk '{Q=Q?Q "@@" substr($0,1,2):substr($0,1,2)} END{print Q}' file1)
LAST_TWO_CHAR=$(awk '{Q=Q?Q "@@" substr($0,length($0)-1,2):substr($0,length($0)-1,2)} END{print Q}' file1)
REST_CHAR=$(awk '{Q=Q?Q "@@" substr($0,3,length($0)-4):substr($0,3,length($0)-4)} END{print Q}' file1)

echo "Separate ones:"
echo "First 2 chars of each line: " $FIRST_TWO
echo "Last 2 chars of each line: " $LAST_TWO_CHAR
echo "Lines without first 2 chars and last 2 chars:" "$REST_CHAR"


echo "combining the lines again here..."
awk -vfirst_two="$FIRST_TWO" -vlast_two_char="$LAST_TWO_CHAR" -vrest_char="$REST_CHAR" 'BEGIN{num=split(first_two, A,"@@");split(last_two_char, B,"@@");split(rest_char, C,"@@");for(i=1;i<=num;i++){print A[i] C[i] B[i]}}'

So when we are running the script then following will be the output.
Code:
./script.ksh 
Separate ones:
First 2 chars of each line:  1h@@1l@@19@@1w@@22@@ZG@@66
Last 2 chars of each line:  RU@@5k@@5S@@Rq@@kw@@FV@@Ft
Lines without first 2 chars and last 2 chars: sRmRsbHRiSFZNTTA1dlEyMWFkbU5wUW5CSlIyeDFTVU5SYjJOSFRuWmpia0ZuWXpKV2FHTn@@KUnpWMldrZFZaMG95V25oYQpSelEyWTBka2QyRklhSHBrUjA1b1kwUkJkd3BOVXpWM1lVaG@@5Um5kaFZ6VnFURzFPZG1KVFFsQlZhVUp0WTFkU2RVOXVRbTVqClIyZzBZek5TYWxsWVFYZE@@xWTBkb05FeHRaR2hqUjJ4MVdYazFhbUl5TUdkVU1VbG5DbHB1Um10aWFuQjNXak5DYjJWSV@@oKTWtaM1RVUkJla3h1UW05bFF6VnVXVmhDY0dKdFRYVlpNamwwU1VVNVUwbEhXbmhhUnpRMl@@tkMkZJYUhwa1IwNW9ZMFJCZDA1RApOWGNLWVVobmRWb3lSbmRoVnpWcVRHMU9kbUpUUWxCVm@@qRZMWRTZFU5dVFtNWpSMmcwWXpOU2FsbFlRWGROUkZWMVkwZG9ORXh0ClpHaGpSMngxV1hrMW
combining the lines again here...
1hsRmRsbHRiSFZNTTA1dlEyMWFkbU5wUW5CSlIyeDFTVU5SYjJOSFRuWmpia0ZuWXpKV2FHTnRU
1lKUnpWMldrZFZaMG95V25oYQpSelEyWTBka2QyRklhSHBrUjA1b1kwUkJkd3BOVXpWM1lVaG5k
195Um5kaFZ6VnFURzFPZG1KVFFsQlZhVUp0WTFkU2RVOXVRbTVqClIyZzBZek5TYWxsWVFYZE5S
1wxWTBkb05FeHRaR2hqUjJ4MVdYazFhbUl5TUdkVU1VbG5DbHB1Um10aWFuQjNXak5DYjJWSVRq
22oKTWtaM1RVUkJla3h1UW05bFF6VnVXVmhDY0dKdFRYVlpNamwwU1VVNVUwbEhXbmhhUnpRMlkw
ZGtkMkZJYUhwa1IwNW9ZMFJCZDA1RApOWGNLWVVobmRWb3lSbmRoVnpWcVRHMU9kbUpUUWxCVmFV
66qRZMWRTZFU5dVFtNWpSMmcwWXpOU2FsbFlRWGROUkZWMVkwZG9ORXh0ClpHaGpSMngxV1hrMWFt

This is just an example where you could use the first 2 characters, last 2 characters and rest of the line without them in a script itself rather than saving them into files. If you have more requirements then kindly do let us know on same.

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
# 3  
Old 12-18-2016
What do you mean putting those characters back? You could put the characters back by simply using the original file.


That said, to split the file into there different files, try:
Code:
awk -v n=2 '
{
  len=length($0)
  print substr($0,1,n)>left
  print substr($0,n+1,(len-2*n))>middle
  print substr($0,len-n+1)>right
}
' left=file1 middle=file2 right=file3 file

If you want to combine the files you can use the paste command:
Code:
paste -d'\0' file1 file2 file3

These 2 Users Gave Thanks to Scrutinizer For This Post:
# 4  
Old 12-18-2016
Just for the fun of it, using standard text tools:
Code:
tee >(cut -c-2 > left) <file | cut -c3- | rev | tee >(cut -c-2 | rev > right) | cut -c3- | rev > middle

cf left middle right 
left:
1h
1l
19
1w
22
ZG
66
middle:
sRmRsbHRiSFZNTTA1dlEyMWFkbU5wUW5CSlIyeDFTVU5SYjJOSFRuWmpia0ZuWXpKV2FHTn
KUnpWMldrZFZaMG95V25oYQpSelEyWTBka2QyRklhSHBrUjA1b1kwUkJkd3BOVXpWM1lVaG
5Um5kaFZ6VnFURzFPZG1KVFFsQlZhVUp0WTFkU2RVOXVRbTVqClIyZzBZek5TYWxsWVFYZE
xWTBkb05FeHRaR2hqUjJ4MVdYazFhbUl5TUdkVU1VbG5DbHB1Um10aWFuQjNXak5DYjJWSV
oKTWtaM1RVUkJla3h1UW05bFF6VnVXVmhDY0dKdFRYVlpNamwwU1VVNVUwbEhXbmhhUnpRMl
tkMkZJYUhwa1IwNW9ZMFJCZDA1RApOWGNLWVVobmRWb3lSbmRoVnpWcVRHMU9kbUpUUWxCVm
qRZMWRTZFU5dVFtNWpSMmcwWXpOU2FsbFlRWGROUkZWMVkwZG9ORXh0ClpHaGpSMngxV1hrMW
right:
RU
5k
5S
Rq
kw
FV
Ft

Use paste as Scrutinizer did for reconstruction.

Last edited by RudiC; 12-18-2016 at 11:49 AM..
These 2 Users Gave Thanks to RudiC For This Post:
# 5  
Old 12-19-2016
Code:
perl -nle 'print /^(..).+(..)$/' data.txt > data.txt.pruned

Code:
perl -pe '' data.txt.pruned

1hRU
1l5k
195S
1wRq
22kw
ZGFV
66Ft

# 6  
Old 12-19-2016
My suspicion is that this is a private security key and you want to be able to email it safely. My concern is that you will also transmit the way that the key is reconstituted and thereby expose how to reconstitute your private key to anyone snooping your traffic.

There are various ways that this can be done as shown already, and more besides, but I just wanted to check that we are not encouraging you to follow an insecure design.

Please feel free to tell me I'm wrong - I'd be relieved. Smilie


Kind regards,
Robin
This User Gave Thanks to rbatte1 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Ksh: Read line parse characters into variable and remove the line if the date is older than 50 days

I have a test file with the following format, It contains the username_date when the user was locked from the database. $ cat lockedusers.txt TEST1_21062016 TEST2_02122015 TEST3_01032016 TEST4_01042016 I'm writing a ksh script and faced with this difficult scenario for my... (11 Replies)
Discussion started by: humble_learner
11 Replies

2. Shell Programming and Scripting

I want to remove 1st and last two characters of each line of the file

I want to remove 1st and last two characters of each line of the file Ex: file1 zzfile1ee @xfile2:y qfile3>> @ file4yy and redirect to the file called new Basically file will have any charcter including space, spical character... Please help.... (7 Replies)
Discussion started by: shell1509
7 Replies

3. Shell Programming and Scripting

Remove characters from line

I have a the following line 22.152.25.36 - K##### "GET /DGGKE/GetMail.do;jwebsphere=FADFFFGSFGSFGSDGFSDFGSDFGSDF HTTP/1.1" 200 44948 Need a cut command which should give me the below output 22.152.25.36 - K##### "GET /DGGKE/GetMail.do HTTP/1.1" 200 44948 Note: The value of jwebsphere can... (6 Replies)
Discussion started by: suindar1982
6 Replies

4. Shell Programming and Scripting

remove first few characters from each line

Hi, I have a file with lines like below. I need to remove first few characters from each line until a date format is found. 05/06/12 20:47:02 GUMGUY@98.192.174.74{42B42A72AC955F5926621273E3A15059.tomcat2}TP-Processor15 LogExchUsage: ERROR: 05/06/12 20:47:02... (8 Replies)
Discussion started by: ratheeshjulk
8 Replies

5. UNIX Desktop Questions & Answers

Remove new line characters from a file

I tried using below command tr -cd "" < InputFile.xml > output.txt ============= This removes all the tabs/newline/extra spaces from a file it successfully removed all the extra spaces,tabs and new line characters but then the complete file become one record. I want to retain one new line... (1 Reply)
Discussion started by: saini
1 Replies

6. UNIX for Advanced & Expert Users

Remove new line characters if found between 1 to 10 columns

Hi, I have a file with ';' delimeter which has some new line characters. How can I delete the new line characters if they are found between 1 to 10 fields. Thanks (3 Replies)
Discussion started by: rudoraj
3 Replies

7. Shell Programming and Scripting

How to remove special characters from each line?

Hello, Is there a simpler way to remove special characters (color codes) from each lines in a log file? I use sed like in the example below but I think there should be a more simple way to achieve the same result: $ cat -vet file1 ^, , , , Maybe to convert the file somehow? ... (5 Replies)
Discussion started by: majormark
5 Replies

8. UNIX for Dummies Questions & Answers

remove characters from line

Hello, I have multiple lines in a file, each of which will have data that looks like this: xxxxxyyyyzzzz4abcdXYZXYZXYZ pqrstPQRST2cdPQRSTPQRST lmnopqr6abcdefgRST.3abc I want to be able to remove the number 4 + the following 4 characters (abcd) in the first line. For the second line,... (1 Reply)
Discussion started by: Gussifinknottle
1 Replies

9. Shell Programming and Scripting

sed remove last 10 characters of a line start from 3rd line

hello experts, I need a sed command that remove last 10 characters of a line start from 3rd line. any suggestions? Thanks you (7 Replies)
Discussion started by: minifish
7 Replies

10. UNIX for Advanced & Expert Users

how to remove line greater then 3000 characters.

I am using awk and it stops when it encounter line greater then 3000 character. Is there any command which will help me remove line greater then 3000 characters. (10 Replies)
Discussion started by: naren_14
10 Replies
Login or Register to Ask a Question