Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

comparing 2 text files to get unique values??

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting comparing 2 text files to get unique values??
# 8  
Old 12-16-2008
Code:
nawk -v OFS='\t' '$1==1 {$1=$1;print}'

" -v OFS='\t' " - set the OutputFieldSeparator to '\t' (tab)
" $1 == 1" - if the value of field 1 ($1) is 1 (the number of occurrences in the combined file), then do {...}

" {$1=$1; print} " - force the reevaluation of the current record/line - forcing the '\t' delimited fields

" | cut -f2- " - given a default field delimiter of '\t', 'cut' everything starting at field 2.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge 4 bim files by keeping only the overlapping variants (unique rs values )

Dear community, I am facing a problem and I kindly ask your help: I have 4 different data sets consisted from 3 different types of array. On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets: x2014: 1 rs3094315... (4 Replies)
Discussion started by: fondan
4 Replies

2. Shell Programming and Scripting

How to identify varying unique fields values from a text file in UNIX?

Hi, I have a huge unsorted text file. We wanted to identify the unique field values in a line and consider those fields as a primary key for a table in upstream system. Basically, the process or script should fetch the values from each line that are unique compared to the rest of the lines in... (13 Replies)
Discussion started by: manikandan23
13 Replies

3. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber... (3 Replies)
Discussion started by: Sharma331
3 Replies

4. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

5. Shell Programming and Scripting

Comparing the values of two files

Hi Am trying to compare the values of two files.. One is a big file that has many values and the other is a small file.. The big file has all values present in small file.. # cat SmallFile 4456602 22347881 7471282 15859891 8257690 21954701 7078068 18219229 2883826 6094959 100000 ... (3 Replies)
Discussion started by: Priya Amaresh
3 Replies

6. UNIX for Dummies Questions & Answers

Comparing two test files and printing out the values that do not match

Hi, I have two text files with matching first columns. Some of the values in the second column do not match. I want to write a script to print out the rows (only the first column) where the values in the second column do not match. Example: Input 1 A 1 B 2 C 3 D 4 Input 2 A 2 B 2... (6 Replies)
Discussion started by: evelibertine
6 Replies

7. UNIX for Dummies Questions & Answers

Comparing two text files by a column and printing values that do not match

I have two text files where the first three columns are exactly the same. I want to compare the fourth column of the text files and if the values are different, print that row into a new output file. How do I go about doing that? File 1: 100 rs3794811 0.01 0.3434 100 rs8066551 0.01... (8 Replies)
Discussion started by: evelibertine
8 Replies

8. UNIX for Dummies Questions & Answers

Comparing two files and dividing the values

Hi all, I am new to unix and I am trying hard to get this requirement, but no luck. I am trying to compare two cloumns in two files and if it matches, the last column in file1 must be divided by file2 and the output must be written in a new file. To elaborate the 2nd column in file1 (EUR) must be... (6 Replies)
Discussion started by: smadderla
6 Replies

9. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Hi, I have 2 files file1 ******** 01-05-09|java.xls| 02-05-08|c.txt| 08-01-09|perl.txt| 01-01-09|oracle.txt| ******** file2 ******** 01-02-09|windows.xls| 02-05-08|c.txt| 01-05-09|java.xls| 08-02-09|perl.txt| 01-01-09|oracle.txt| ******** (8 Replies)
Discussion started by: shekhar_v4
8 Replies

10. UNIX for Dummies Questions & Answers

Need to find only unique values for a given tag across the files

Need to find only unique values for a given tag across the files: For eg: Test1: <Tag1>aaa</Tag1> <Tag2>bbb</Tag2> <Tag3>ccc</Tag3> Test2: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test3: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test4: (8 Replies)
Discussion started by: sudheshnaiyer
8 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

httpindex

httpindex(1)						      General Commands Manual						      httpindex(1)

NAME

       httpindex - HTTP front-end for SWISH++ indexer

SYNOPSIS

       wget [ options ] URL...	2>&1 | httpindex [ options ]

DESCRIPTION

       httpindex is a front-end for index++(1) to index files copied from remote servers using wget(1).  The files (in a copy of the remote direc-
       tory structure) can be kept, deleted, or replaced with their descriptions after indexing.

OPTIONS

   wget Options
       The wget(1) options that are required are: -A, -nv, -r, and -x; the ones that are highly recommended are: -l, -nh, -t, and  -w.	 (See  the
       EXAMPLE.)

   httpindex Options
       httpindex accepts the same short options as index++(1) except for -H, -I, -l, -r, -S, and -V.

       The following options are unique to httpindex:

       -d     Replace the text of local copies of retrieved files with their descriptions after they have been indexed.  This is useful to display
	      file descriptions in search results without having to have complete copies of the remote files thus saving filesystem  space.   (See
	      the extract_description() function in WWW(3) for details about how descriptions are extracted.)

       -D     Delete  the  local copies of retrieved files after they have been indexed.  This prevents your local filesystem from filling up with
	      copies of remote files.

EXAMPLE

       To index all HTML and text files on a remote web server keeping descriptions locally:

	    wget -A html,txt -linf -t2 -rxnv -nh -w2 http://www.foo.com 2>&1 |
	    httpindex -d -e'html:*.html,text:*.txt'

       Note that you need to redirect wget(1)'s output from standard error to standard output in order to pipe it to httpindex.

EXIT STATUS

       Exits with a value of zero only if indexing completed sucessfully; non-zero otherwise.

CAVEATS

       In addition to those for index++(1), httpindex does not correctly handle the use of multiple -e, -E, -m, or -M options  (because  the  Perl
       script uses the standard GetOpt::Std package for processing command-line options that doesn't).	The last of any of those options ``wins.''

       The work-around is to use multiple values for those options seperated by commas to a single one of those options.  For example, if you want
       to do:

	    httpindex -e'html:*.html' -e'text:*.txt'

       do this instead:

	    httpindex -e'html:*.html,text:*.txt'

SEE ALSO

       index++(1), wget(1), WWW(3)

AUTHOR

       Paul J. Lucas <pauljlucas@mac.com>

SWISH++ 							  August 2, 2005						      httpindex(1)