Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Getting non unique lines from concatenated files
Top Forums UNIX for Dummies Questions & Answers Getting non unique lines from concatenated files Post 302507925 by bartus11 on Friday 25th of March 2011 07:11:45 AM
Old 03-25-2011
Try this:
Code:
#!/bin/sh
perl -l -s0e 'BEGIN{@f=split / /,$f;$N=$#f}for $i (0..$N){for $j (0..$i-1,$i+1..$N){open I,"<$f[$j]";$a.=<I>}open O,">files${i}.tmp";print O $a;$a=""};
for $i (0..$N){print "$f[$i] unique\n";system "bash -c \"comm -23 <(sort $f[$i]) <(sort files$i.tmp);rm -f files$i.tmp\"";print "\n##############\n"}' -- -f="$*"

-s option allows easy passing of variable values to the Perl one-liner (-- -f="$*" part). After passing all the filenames as $f variable, it is split using space as delimiter into @f array - @f=split / /,$f.
This User Gave Thanks to bartus11 For This Post:
pawannoel
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Lines Concatenated with awk

Hello, I have a bash shell script and I use awk to print certain columns of one file and direct the output to another file. If I do a less or cat on the file it looks correct, but if I email the file and open it with Outlook the lines outputted by awk are concatenated. Here is my awk line:... (6 Replies)
Discussion started by: xadamz23
6 Replies

2. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Hi, I have 2 files file1 ******** 01-05-09|java.xls| 02-05-08|c.txt| 08-01-09|perl.txt| 01-01-09|oracle.txt| ******** file2 ******** 01-02-09|windows.xls| 02-05-08|c.txt| 01-05-09|java.xls| 08-02-09|perl.txt| 01-01-09|oracle.txt| ******** (8 Replies)
Discussion started by: shekhar_v4
8 Replies

3. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

4. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

5. UNIX for Dummies Questions & Answers

getting unique lines from 2 files

hi i have used comm -13 <(sort 1.txt) <(sort 2.txt) option to get the unique lines that are present in file 2 but not in file 1. but some how i am getting the entire file 2. i would expect few but not all uncommon lines fro my dat. is there anything wrong with the way i used the command? my... (1 Reply)
Discussion started by: anurupa777
1 Replies

6. Shell Programming and Scripting

compare 2 files and return unique lines in each file (based on condition)

hi my problem is little complicated one. i have 2 files which appear like this file 1 abbsss:aa:22:34:as akl abc 1234 mkilll:as:ss:23:qs asc abc 0987 mlopii:cd:wq:24:as asd abc 7866 file2 lkoaa:as:24:32:sa alk abc 3245 lkmo:as:34:43:qs qsa abc 0987 kloia:ds:45:56:sa acq abc 7805 i... (5 Replies)
Discussion started by: anurupa777
5 Replies

7. Shell Programming and Scripting

Print only lines where fields concatenated match strings

Hello everyone, Maybe somebody could help me with an awk script. I have this input (field separator is comma ","): 547894982,M|N|J,U|Q|P,98,101,0,1,1 234900027,M|N|J,U|Q|P,98,101,0,1,1 234900023,M|N|J,U|Q|P,98,54,3,1,1 234900028,M|H|J,S|Q|P,98,101,0,1,1 234900030,M|N|J,U|F|P,98,101,0,1,1... (2 Replies)
Discussion started by: Ophiuchus
2 Replies

8. Shell Programming and Scripting

Look up 2 files and print the concatenated output

file 1 Sun Mar 17 00:01:33 2013 submit , Name="1234" Sun Mar 17 00:01:33 2013 submit , Name="1344" Sun Mar 17 00:01:33 2013 submit , Name="1124" .. .. .. .. Sun Mar 17 00:01:33 2013 submit , Name="8901" file 2 Sun Mar 17 00:02:47 2013 1234 execute SUCCEEDED Sun Mar 17... (24 Replies)
Discussion started by: aravindj80
24 Replies

9. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Discussion started by: cokedude
7 Replies

10. UNIX for Beginners Questions & Answers

Print number of lines for files in directory, also print number of unique lines

I have a directory of files, I can show the number of lines in each file and order them from lowest to highest with: wc -l *|sort 15263 Image.txt 16401 reference.txt 40459 richtexteditor.txt How can I also print the number of unique lines in each file? 15263 1401 Image.txt 16401... (15 Replies)
Discussion started by: spacegoose
15 Replies

LEARN ABOUT SUNOS

gst-complete

gst-complete-0.8(1)						   User Commands					       gst-complete-0.8(1)

NAME

       gst-complete-0.8, gst-complete - perform bash completion for gst-launch command lines

SYNOPSIS

       complete -Cgst-complete-0.8 gst-launch

       gst-complete [--gst-list-mm] [--gst-mm=string] [--print]

DESCRIPTION

       gst-complete-0.8 enables bash to provide context-sensitive tab completion for gst-launch command lines. See gst-launch(1) for the syntax of
       gst-launch command lines.

       You must have a version of bash that supports programmable completion. Such versions of bash provide the complete built-in command.

       gst-complete is a wrapper script that runs the latest installed version of gst-complete-X.X. For example, if both gst-complete-0.7 and gst-
       complete-0.8 are installed on your system, gst-complete runs gst-complete-0.8.

OPTIONS

       The following options are supported by gst-complete-0.8 and gst-complete:

       --help	       Display detailed usage message. You can also use -? to specify this option.

       The following options are supported by gst-complete only:

       --gst-list-mm   List found major/minor versions. This option displays the versions that are available.

       --gst-mm=string Force  major/minor  version.  This  option  enables you to specify a specific version to run, if you do not want to run the
		       default version.

       --print	       Print wrapped command line. This option displays the command that will be run, and then runs the command.

EXAMPLES

       Example 1: Generating the Registry of Available Plugins by Running gst-compprep

       Run the following command in your .bash_profile or other appropriate place:

       example% complete -C gst-complete-0.8 gst-launch

       Example 2: Running the Wrapper Script and Displaying the Command Name

       example% gst-complete --print
       /usr/bin/gst-complete-0.8

FILES

       The following files are used by this application:

       /usr/bin/gst-complete-0.Executable for bash completion for gst-launch

       /usr/bin/gst-complete   Wrapper script that runs the latest installed version of gst-complete-X.X

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWgnome-media		   |
       +-----------------------------+-----------------------------+
       |Interface stability	     |External			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       gst-compprep(1), gst-feedback(1), gst-inspect(1), gst-launch(1), gst-launch-ext(1), gst-md5sum(1), gst-register(1), gst-thumbnail(1),  gst-
       typefind(1), gst-xmlinspect(1), gst-xmllaunch(1), gstreamer-properties(1), libgstreamer-0.8(3), libgstgetbits(3)

NOTES

       Original man page written by the GStreamer team at http://gstreamer.net/.

       Updated by Brian Cameron, Sun Microsystems Inc., 2004.

SunOS 5.10							    14 Oct 2004 					       gst-complete-0.8(1)
All times are GMT -4. The time now is 10:40 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy