Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines which has been repeated 4 times Post 303008625 by Kalia on Monday 4th of December 2017 10:09:34 AM
Old 12-04-2017
1-Getting bug It is printing exact first 4 occurrences but if the lines are repeated more then 4 those line are not printing,need those line also if possible

2-few step am not able to understand if possible could please example with comments
Code:
ps -u gpoper -o pid= -o args=| awk '{p=$1; $1=""; pid[$0,++s[$0]]=p} s[$0]==4 {for (i=1; i<=s[$0]; i++) print pid[$0,i] $0}'

---------- Post updated at 08:39 PM ---------- Previous update was at 08:37 PM ----------

*explain
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

3. Shell Programming and Scripting

AWK Duplicate lines multiple times based on a calculated value

Hi, I'm trying to create an XML sitemap of our dynamic ecommerce sites SEO Friendly URLs and am trying to create the initial page listing. I have a CSV file that looks like the following and need duplicate the lines based on a value which needs calculating. ... (2 Replies)
Discussion started by: jamesfx
2 Replies

4. Shell Programming and Scripting

Need to remove the duplicate lines from a log!!

Hello Folks, Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's. Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies

5. Shell Programming and Scripting

Remove regularly repeated lines

How can i delete some regular repeated lines in a file? example: in_file EDGE 1 2 12 EDGE 2 3 23 EDGE 3 4 34 EDGE 5 6 56 EDGE 6 7 67 EDGE 7 8 78 EDGE 9 10 910 EDGE 10 11 1011 EDGE 11 12 1112 EDGE 13 14 1314 EDGE 14 15 1415 EDGE 15 16 1516 EDGE 17 18 1718 EDGE 18 19 1819 EDGE 19... (8 Replies)
Discussion started by: saeed.soltani
8 Replies

6. Shell Programming and Scripting

How to print the lines which are repeated 3 times in a file?

Hello All, I have a file which has repeated lines. I want to print the lines which are repeated three times. Please help. (3 Replies)
Discussion started by: ailnilanjan
3 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

9. UNIX for Beginners Questions & Answers

Export lines that have first entry repeated 5 times or above

Dears i want to extract lines only that have first entry repeated 3 times or above , ex data : -bash-3.00$ cat INTCONT-IS.CSV M205-00-106_AMDRN:1-0-6-22,12-662-4833,intContact,2016-11-15 02:32:16,50 M205-00-106_AMDRN:1-0-23-17,12-616-0462,intContact,2016-11-15 02:32:23,50... (5 Replies)
Discussion started by: is2_egypt
5 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
IPC::Open2(3pm) 					 Perl Programmers Reference Guide					   IPC::Open2(3pm)

NAME
IPC::Open2 - open a process for both reading and writing using open2() SYNOPSIS
use IPC::Open2; $pid = open2(*CHLD_OUT, *CHLD_IN, 'some cmd and args'); # or without using the shell $pid = open2(*CHLD_OUT, *CHLD_IN, 'some', 'cmd', 'and', 'args'); # or with handle autovivification my($chld_out, $chld_in); $pid = open2($chld_out, $chld_in, 'some cmd and args'); # or without using the shell $pid = open2($chld_out, $chld_in, 'some', 'cmd', 'and', 'args'); waitpid( $pid, 0 ); my $child_exit_status = $? >> 8; DESCRIPTION
The open2() function runs the given $cmd and connects $chld_out for reading and $chld_in for writing. It's what you think should work when you try $pid = open(HANDLE, "|cmd args|"); The write filehandle will have autoflush turned on. If $chld_out is a string (that is, a bareword filehandle rather than a glob or a reference) and it begins with ">&", then the child will send output directly to that file handle. If $chld_in is a string that begins with "<&", then $chld_in will be closed in the parent, and the child will read from it directly. In both cases, there will be a dup(2) instead of a pipe(2) made. If either reader or writer is the null string, this will be replaced by an autogenerated filehandle. If so, you must pass a valid lvalue in the parameter slot so it can be overwritten in the caller, or an exception will be raised. open2() returns the process ID of the child process. It doesn't return on failure: it just raises an exception matching "/^open2:/". However, "exec" failures in the child are not detected. You'll have to trap SIGPIPE yourself. open2() does not wait for and reap the child process after it exits. Except for short programs where it's acceptable to let the operating system take care of this, you need to do this yourself. This is normally as simple as calling "waitpid $pid, 0" when you're done with the process. Failing to do this can result in an accumulation of defunct or "zombie" processes. See "waitpid" in perlfunc for more information. This whole affair is quite dangerous, as you may block forever. It assumes it's going to talk to something like bc, both writing to it and reading from it. This is presumably safe because you "know" that commands like bc will read a line at a time and output a line at a time. Programs like sort that read their entire input stream first, however, are quite apt to cause deadlock. The big problem with this approach is that if you don't have control over source code being run in the child process, you can't control what it does with pipe buffering. Thus you can't just open a pipe to "cat -v" and continually read and write a line from it. The IO::Pty and Expect modules from CPAN can help with this, as they provide a real tty (well, a pseudo-tty, actually), which gets you back to line buffering in the invoked command again. WARNING
The order of arguments differs from that of open3(). SEE ALSO
See IPC::Open3 for an alternative that handles STDERR as well. This function is really just a wrapper around open3(). perl v5.16.2 2012-10-11 IPC::Open2(3pm)
All times are GMT -4. The time now is 07:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy