Deleting duplicated chunks in a file using awk/sed
Hi all,
I'd always appreciate all helps from this site.
I would like to delete duplicated chunks of strings on the same row(?).
One chunk is comprised of four lines such as:
path name
starting point
ending point
voltage number
I would like to delete duplicated chunks on the same row(?) if "ending point" is duplicated.
For example, ending points of the first and the second chunk are same in the first row and I would like to only keep the first chunk. Therefore, the second chunk is removed on the first row.
In the second row, ending points of the first and the third chunk are same and keep the first chunk.
input.txt:
Expected_output.txt:
The number of columns can be up to 20 in a file.
Actually, I have posted the same question on other website to get a help, and somebody posted replies, but did not work correctly. Any help is appreciated.
Instead of trying to get multiple websites to act as your unpaid programming staff, why don't you show us how you have tried to solved this problem on your own? If you can show us what you have tried, maybe we can help you fix it.
We have helped you with 8 other awk scripts in the last six months. Can't you use the examples provided by those scripts to get a good start on what you need here?
I tried 'uniq' that only show uniq strings first, but don't know how to show uniq chunks. It only works for lines. And then I tried sed.
So now I could find the duplicated one and replaced with "-", but failed to remove the chunk.
If my post is not appropriate, I will remove it soon.
Best,
Jaeyoung
Moderator's Comments:
Please use CODE tags when displaying sample input, sample output, and code segments.
Last edited by Don Cragun; 05-26-2016 at 02:01 AM..
Reason: Add CODE tags.
Maybe something more like:
would work better for you. This uses a single tab character as the output field separator instead of a seemingly random number of spaces (but you can easily change it to a fixed number of spaces if you want to).
From the sed command you're using, I assume that you're not running this on a Solaris system, but if someone else wants to try the above code on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
Dear all,
I always appreciate your help.
I would like to delete lines containing duplicated strings in the second column.
test.txt
658 invert_d2e_q_reg_0_/Qalu_ecl_zlow_e 0.825692
659 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692
660 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692... (1 Reply)
Hi gurus,
I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error.
awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;... (10 Replies)
First time poster, but the forum has saved my bacon more times than... Lots.
Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file
1
2
3
4
5
etc...
I want the out put... (3 Replies)
hi,
Here is excerpt from my xml file
<!-- The custom module to do the authentication for LDAP
-->
</login-module>
<login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient">
<module-option... (1 Reply)
Hello
I have a unique problem of needing to delete large files slowly off of an XSan.
I was wondering if there is a script I could use to delete 100gb chunks of files and folders that get placed in to a watch folder, slowly so as not to disrupt the other users. I would like to use Automator in... (0 Replies)
Input:
:: gstreamer
:: xine-lib
:: xine-lib-extras
Output should be:
gstreamer xine-lib xine-lib-extras
How can it be done with sed or perl? (12 Replies)
cat file.txt
fvnuiehuewf
ruevhxncvkjrh
zxjvurhfuwe
jkhvBEGINvfnvf
ijrgioe
Trying to delete a line that has the pattern "BEGIN"
cat sedtest
filename=file.txt
pattern=BEGIN
sed "/^$pattern/d" "$filename" (9 Replies)
hi there,
i have a text file like that one:
I like to delete the second block with the Start and End Line!
Does anyone have a idea?
Thanks for your help,
Roland (4 Replies)
I can not figure out this one, so I turn to unix.com for help, I have a file, in which there are some lines containing continuously duplicate columns, like the following
adb abc abc asd adfj
123 123 123 345
234 444 444 444 444 444 23
and the output I want is
adb abc asd adfj
123 345... (5 Replies)