Bash uniq/ diff/ and other I cant figure it out


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash uniq/ diff/ and other I cant figure it out
# 1  
Old 04-05-2008
Bash uniq/ diff/ and other I cant figure it out

First off thank you for any help.
Here is the problem. I have two text files that fit the same format. The first I created using an ls -d command and then with the help of the forums ran awk resulting in the fallowing output.

W00CHZ0103345-I1CZ44
W00E6S1016722-I01JW159
W00E6S1016722-I01JW160
W00E6S1016722-I01JW161
W00EGS10125151-I01JW176
W00EGS10125151-I01JW177
W00EGS10125151-I01JW178
W00EGS10125151-I01JW179
W00EGS10125151-I01JW180
W00EGS1012593-I00EGS1017114

I have a second text file whose format i was trying to successfully managed to match. Here it is.

W00CHZ0103345-I1CZ44
W00EGS1016051-I00EGS1016053
W00EGS1016054-I00EGS1016056
W00EGS1016057-I00EGS1016059
W00EGS1016060-I00EGS1016062
W00EGS1016181-I1PD10388
W00EGS1016199-I00EGS1016201
W00EGS1016202-I1GS65488
W00EGS1016210-I00EGS1016212
W00EGS1016213-I00EGS1016216

Now these lists are nearly 10,000 lines long and what i need to do is compare them and see what only occurs on the second list. I have tried sorting and using uniq and diff and everything else that i can think of but was unable to generate a list of the lines that only appear in txt2. That being said if there were lines that appeared in the first text and not the second i would want to know but i would need to know from which list it came.

Please help if you can.

Thank you
# 2  
Old 04-05-2008
Look at the comm utility.
# 3  
Old 04-05-2008
Quote:
Originally Posted by Movomito
That being said if there were lines that appeared in the first text and not the second i would want to know but i would need to know from which list it came.

Use the comm command.
# 4  
Old 04-06-2008
I looked at comm

i looked at comm and I can't seem to get my desired output.
here is what i am trying maybe someone can help.

first i run sort text1.txt > newtext1.txt

then the same with text2.txt

then i try this:
comm -3 newtext1.txt newtxt2.txt

I have tried a variation of comm -1, comm -2, -1 -2, -3, pretty much everthing that i can think of but i am still getting output that i can find in both files. I want just the lines that occur in only newtext1.txt
# 5  
Old 04-06-2008
Code:
comm -23 newtext1. newtxt2.txt

# 6  
Old 04-06-2008
comm -23 file1 file2 still not working

so i tried using the comm -23 file1 fil2 per the last suggestion and still am not getting the desired result. Using bbedit i can look and do a line count and there is a difference of appx 800 files from file1 to file2. and so that is the output that i am expecting. Now i have tried sorting the files before running comm and outputting that sort to new files, on which i do the comm. there are two strange things happening here one after comparing my new files by eye after running sort i can see that they are not sorted the same way, and secondly after running comm -23 etc. i get a reurn of one entry which i cant find in either file. Any help is appreciated
# 7  
Old 04-06-2008
Can you pinpoint the sort discrepancy, i.e. prune it down to a list of, say, three lines which sort differently? Do you have locales at play? (Still, comm and sort should obey the same locale; but if they don't, what you describe sort of makes sense.) See also the locale manual page. See if adding LC_ALL=C helps.

Code:
LC_ALL=C sort file1 >sorted1.lc_all=c
diff sorted1.lc_all=c newtext1.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Bash diff date doesn't work

Hi everyone, I've an issue trying to soustracte two dates, e.g: d1=$(date -d "Nov 18, 2017" +%s) d2=$(date +%s) # Today we are 2017-11-16 echo "$(( (d1 - d2) / 86400 ))" Output: 1 I don't understand why it doesn't work. for me, it should give "18 - 16 = 2". Much appreciated... (1 Reply)
Discussion started by: Arnaudh78
1 Replies

2. Shell Programming and Scripting

Bash script to diff two arrays

Hi, I am quite scripting illiterate and have been trying to write a bash script to compare to two files which i have populated in two seperate arrays as below and confirmed that all the files are loaded into the array. IFS=$'\n' filea=($(find /var/tmp/dir1 -type f -follow -print))... (12 Replies)
Discussion started by: jlykke
12 Replies

3. Shell Programming and Scripting

Diff 3 files, but diff only their 2nd column

Guys i have 3 files, but i want to compare and diff only the 2nd column path=`/home/whois/doms` for i in `cat domain.tx` do whois $i| sed -n '/Registry Registrant ID:/,/Registrant Email:/p' > $path/$i.registrant whois $i| sed -n '/Registry Admin ID:/,/Admin Email:/p' > $path/$i.admin... (10 Replies)
Discussion started by: kenshinhimura
10 Replies

4. Shell Programming and Scripting

Help with change significant figure to normal figure command

Hi, Below is my input file: Long list of significant figure 1.757E-4 7.51E-3 5.634E-5 . . . Desired output file: 0.0001757 0.00751 0.00005634 . . . (10 Replies)
Discussion started by: perl_beginner
10 Replies

5. Shell Programming and Scripting

Time Stap Diff using Bash

hi all kindly i want to subtract to time stamps like: t1=08:35:20 t2=08:36:58 diff=00:01:38 <<<< or diff=01:38 <<<< using bash thanks alot (9 Replies)
Discussion started by: teefa
9 Replies

6. Shell Programming and Scripting

How do a distinct from a file using sort uniq in bash?

I have an output file .dat. From this file i have to do a distinct of the ID using the sort uniq command in bash script. How can i do it? i found : sort -u ${FILEOUT_DAT} but i don't think is my solution because the id isn't specified.. is there other solution? (7 Replies)
Discussion started by: punticci
7 Replies

7. Shell Programming and Scripting

serach diff filename in diff location using shell scripting

Hi, I am new to shell scripting. please help me to find out the solution. I need a script where we need to read the text file(consists of all file names) and get the file names one by one and append the date suffix for each file name as 'yyyymmdd' . Then search each file if exists... (1 Reply)
Discussion started by: Lucky123
1 Replies

8. Shell Programming and Scripting

Bash script - im missing something and cant's figure out what

I just put together a script for work that will essentially automate the migration of our Windows fileserver to my newly created Debian based SAMBA server. My script will create the necessary directories then copy the data over to my new server, after that it will set the ACL's by using... (3 Replies)
Discussion started by: binary-ninja
3 Replies

9. Shell Programming and Scripting

.procmailrc and uudeview (put attachments from diff senders to diff folders)

Moderator, please, delete this topic (1 Reply)
Discussion started by: optik77
1 Replies

10. Shell Programming and Scripting

diff 2 files; output diff's to 3rd file

Hello, I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3. For example: file 1 123 1234 123456 file 2 123 2345 23456 file 3 should have 2345 23456 I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies
Login or Register to Ask a Question