Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Inserting duplicate lines in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Inserting duplicate lines in a file
# 1  
Old 05-26-2012
Inserting duplicate lines in a file
Hi,

I copied the contents of a binary file into a .text file using hd (hexdump) command. The data in binary file is such that I get in many places like following

Code:
00000250  00 00 00 00 3f 2d 91 68  3f 69 fb e7 00 00 00 00  |....?-.h?i......|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000002e0  00 00 00 00 00 00 00 00  c9 0d 19 e0 48 c3 50 00  |............H.P.|
000002f0  49 74 24 00 4a 74 24 00  4b 18 96 80 c3 3a 8e 63  |It$.Jt$.K....:.c|
00000300  c2 bd d6 87 3f 1c e6 5c  00 00 00 00 3c 38 db ff  |....?..\....<8..|

where first column is hexadecimal address, 2nd-17th are hex data and last column contains char values corresponding to hex data. Obviously, * denotes block of arbitrary no of lines which are identical to the very previous line. In all cases, this previous line contains all zeros. Hence, this block is made of all zeros - a fact that can be exploited.

I want to replace * with corresponding block of zeros. Like this

Code:
00000250  00 00 00 00 3f 2d 91 68  3f 69 fb e7 00 00 00 00  |....?-.h?i......|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000270  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  c9 0d 19 e0 48 c3 50 00  |............H.P.|
000002f0  49 74 24 00 4a 74 24 00  4b 18 96 80 c3 3a 8e 63  |It$.Jt$.K....:.c|
00000300  c2 bd d6 87 3f 1c e6 5c  00 00 00 00 3c 38 db ff  |....?..\....<8..|

Actually you need not worry about the last column or 1st column either. All that is required is to insert block of size that depends on addresses of previous and subsequent lines ie, output file can contain only 1st-17th or 2nd-17th columns.

Or, a much simpler and quick alternative is to provide some command or format of output file in which * (or any other symbol) is not used to denote duplicate lines ie, in output file, it prints the entire block of zeros if it's there. Note : This output file must be human readable. I tried copying using hd command in a .txt file and it doesn't open.

Thanks in advance.
Last edited by Scrutinizer; 05-26-2012 at 05:49 AM.. Reason: code tags
# 2  
Old 05-26-2012
Code:
     -v      Cause hexdump to display all input data.  Without the -v option,
             any number of groups of output lines, which would be identical
             to the immediately preceding group of output lines (except for
             the input offsets), are replaced with a line comprised of a sin-
             gle asterisk.

This User Gave Thanks to binlib For This Post:
KidD312
# 3  
Old 05-26-2012
Thanks a lot! It really works. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

2. Shell Programming and Scripting

Inserting lines from one file to a sorted list

Hello friends! I am working a Psychology/Neuro* project where I am sorting inline citations by category. The final step of the process has me a little stuck. I need to take citations from a text list and sort them in another text file. Here is a file X example... (1 Reply)
Discussion started by: danbroz
1 Replies

3. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

6. Shell Programming and Scripting

Inserting lines in between the contents of file.

Hi, I want to insert some lines in between the contents of a file but the file format should not be changed. #!/usr/bin/sh - # Link appropriate OS specific versions of vxicap and vxchk4badblks vxlvmlink() { vxipath=/usr/lib/vxvm/bin relmajor=`uname -v` ... (1 Reply)
Discussion started by: ajilesh
1 Replies

7. Shell Programming and Scripting

inserting a lines in a file

Hi folks, i need to insert the same set of lines between each line input lines 111111 aaaaaa 333333 output should be 111111 1 2 3 aaaaaa 1 2 3 333333 1 (2 Replies)
Discussion started by: Balaji Sukumara
2 Replies

8. Shell Programming and Scripting

removing the duplicate lines in a file

Hi, I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted. eg: file1: ----- data1 value1 data2 value2 data3 value3 file2: ----- data1 value1 data4 value4 data5 value5 file3: ----- data1 value1 data4 value4 (3 Replies)
Discussion started by: Sharmila_P
3 Replies

9. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

10. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies
Login or Register to Ask a Question