line too long using awk or sed or tr


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting line too long using awk or sed or tr
# 1  
Old 06-12-2007
line too long using awk or sed or tr

Goodmorning,

I have MKS Toolkit (K-Shell) running on a windows server. On it I have a c program that in a true unix environment works fine, but here it adds an extra '0000000000000016000A' in various places in the file that the c program produces that I need to remove.

Here is what the file looks like when I pg it:
Code:
00000000000000840022003A00013A40313A98263A013AF63AF2423AF03AF2423AF03A13043A0705163A08233A20513AF03AF53A113AF0
00000000000000840022003A00013A26983A98273A013AF73AF4903AF03AF4903AF03A26983A0705163A12063A20493AF03AF73AF63AF0
00000000000000840022003A00013A26983A98283A013AF53AF1073AF03AF1073AF03A26983A0705163A13283A20483AF03AF13A123AF0

Note that the length of the line exceeds the width of the screen and pg simply does not display the rest of the line; this file shows 17 very long lines.

From the windows environment, here is what it looks like when I use notepad:
Code:
000000000000004C0022163A000000003A00013AF03AF03AF03AF6103A08023A070516220A 
0000000000000016000A 
000000000000004C0022163A000000003A00033AF03AF03AF03AF6083A08033A070516220A 
0000000000000016000A 
000000000000004C0022163A000000003A00043AF03AF03AF03AF6073A08043A070516220A 
0000000000000016000A 
000000000000004C0022163A000000003A00023AF03AF03AF03AF6073A08063A070516220A 
0000000000000016000A 
000000000000004C0022163A000000003A00013AF03AF03AF03AF6093A08123A070516220A

So I have tried sed:
Code:
sed '/0000000000000016000A/d' TL000110.IP.OUT > new.OUT

and I get a zero byte new.OUT. I have tried this identical sed command on a simple text file where I put '00123' strewn throughout and it worked fine.

I tried awk:
Code:
awk '{gsub("0000000000000016000A","",$0)}' TL000110.IP.OUT > new.OUT

and I get this message:
Code:
awk: line 0 (NR=0): line too long: limit 20000

I tried using tr, but I could only ever get it to work on a character by character basis, so when I asked for it to delete '0000000000000016000A' it also removed all instances of the numbers 0,1,6 and the letter A.

Can anyone here think of a way to remove these errant entries from this file that has long lines?

Thanks for the help!
# 2  
Old 06-12-2007
Code:
sed 's/0000000000000016000A$//' TL000110.IP.OUT > new.OUT

# 3  
Old 06-12-2007
Just guessing, give this a try:
tr '\r' '\n' < TL000110.IP.OUT | sed '/0000000000000016000A/d' > new.OUT
# 4  
Old 06-12-2007
The sed produced a new.OUT that was identical to the TL*.OUT

The tr / sed did exactly what was needed. Thanks for the amazing quick reply!

Last edited by philplasma; 06-12-2007 at 09:32 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to tail sed and awk in one line?

Hello, I am trying to create an iptables script with tail ,sed and awk. 1st Request: Search keyword "secret" in access.log file 2nd Request: Get first column matching lines (ip address) 3rd Request: Save it to a file This is what I did so far: grep.sh #!/bin/bash while true; do tail... (23 Replies)
Discussion started by: baris35
23 Replies

2. UNIX for Dummies Questions & Answers

sed and awk for long lines

Hi, I'm trying make a variable length file to a fixed length of 4000. I'm to pad spaces on the right of a record if length is less than 4000 to make the record length 4000. I'm trying to use the below commands awk '{printf "%-4000s\n", $0}' inputfile.dat > outputfile.dat sed -e :a... (12 Replies)
Discussion started by: uxusr
12 Replies

3. Shell Programming and Scripting

Multiple line search, replace second line, using awk or sed

All, I appreciate any help you can offer here as this is well beyond my grasp of awk/sed... I have an input file similar to: &LOG &LOG Part: "@DB/TC10000021855/--F" &LOG &LOG &LOG Part: "@DB/TC10000021852/--F" &LOG Cloning_Action: RETAIN &LOG Part: "@DB/TCCP000010713/--A" &LOG &LOG... (5 Replies)
Discussion started by: KarmaPoliceT2
5 Replies

4. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

5. Shell Programming and Scripting

Command line - awk, sed

My input file gfile values is CTRY=GM&PROJTYPE=SP&PROJECTTYPE=Small+Project If i am giving PROJECTTYPE then it must give Small Project awk -F"&" '{for (i=1; i<=NF; i++) if ($i ~ "^"PAT) {sub ("^"PAT"=", "", $i); sed 's/'+'/""/' $i ; print $i }}' PAT=$1 ... (6 Replies)
Discussion started by: nag_sathi
6 Replies

6. Solaris

Line too long error Replace string with new line line character

I get a file which has all its content in a single row. The file contains xml data containing 3000 records, but all in a single row, making it difficult for Unix to Process the file. I decided to insert a new line character at all occurrences of a particular string in this file (say replacing... (4 Replies)
Discussion started by: ducati
4 Replies

7. Shell Programming and Scripting

sed and Output line too long

hello i try this command in console mode sed -e :a -e '/$/N; s/\(\)\n/\1 /; ta' test.txt > result.txt i have in the output screen "Output line too long" for multiples lines can you please tell me how can i retrieve those long lines during the execution ? Another thing very... (5 Replies)
Discussion started by: ade05fr
5 Replies

8. Shell Programming and Scripting

Replacing or removing a long list of pattern by using awk or sed

Input: >abc|123456|def|EXIT| >abc|203456|def|EXIT2| >abc|234056|def|EXIT3| >abc|340056|def|EXIT4| >abc|456000|def|EXIT5| . . . Output: def|EXIT| def|EXIT2| def|EXIT3| def|EXIT4| def|EXIT5| . . My try code: (9 Replies)
Discussion started by: patrick87
9 Replies

9. Shell Programming and Scripting

awk;sed appending line to previous line....

I know this has been asked before but I just can't parse the syntax as explained. I have a set of files that has user information spread out over two lines that I wish to merge into one: User1NameLast User1NameFirst User1Address E-Mail:User1email User2NameLast User2NameFirst User2Address... (11 Replies)
Discussion started by: walkerwheeler
11 Replies

10. Shell Programming and Scripting

Read logline line by line with awk/sed

Hello, I have a logfile which is in this format: 1211667249500#3265 1211667266687#2875 1211667270781#1828 Is there a way to read the logfile line by line every time I execute the code and put the two numbers in the line in two separate variables? Something like: 1211667249500#3265... (7 Replies)
Discussion started by: dejavu88
7 Replies
Login or Register to Ask a Question