Visit Our UNIX and Linux User Community


Compare two text files and print the difference against key in bash shell script


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Compare two text files and print the difference against key in bash shell script
# 1  
Old 04-18-2020
Compare two text files and print the difference against key in bash shell script

Shell script, bash, have 2 large files around 1.2 GB data, with key and values, I need to compare both files based on the key and store difference in the value in the third file,

File 2 will always be a subset of File 1, just need to find values(against key) which are not present in file 2 and unique ones in File 1.

File 1:

Code:
test1 marco;polo;angus

test2 mike;zen;liza

test3 tom;harry;alan

test4 bob;june;janet

1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule 343323H;343434311H;454656556H;343343432H

1332240_44557576_CONTI Mazed & Micro kjd $353.50_30062020_lsdf3_some-rule 232324L;2226556H;343223432H


File 2:

Code:
test1 polo;angus

test2 mike

test4 bob;janet

1332240_44557576_CONTI Mazed & Micro kjd $353.50_30062020_lsdf3_some-rule 232324L;343223432H

I would like to compare the first two columns of file1 with file2 (search through the entire contents of file2 in first two columns) if they match print the difference of values. Then search for the second line of file 1 and so on. Also keys unique in file 1 should be printed.

Expected Output:

Code:
test1 marco

test2 zen;liza

test3 tom;harry;alan

test4 june

1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule 343323H;343434311H;454656556H;343343432H

1332240_44557576_CONTI Mazed & Micro kjd $353.50_30062020_lsdf3_some-rule 2226556H

The files I have are huge, containing about 100,000 lines, so I would like to make the execution fast. This is running in shell script, using bash shell scripting. These files file1 and file2 are text file, with this as key (1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule) and these as values : (343323H;343434311H;454656556H;343343432H)
Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!

Last edited by vgersh99; 04-18-2020 at 04:59 PM.. Reason: code tags, please!
# 2  
Old 04-18-2020
what have you done so far and where exactly are stuck?
have you searched for similar threads in these fora?
have you checked the 10 More Discussions You Might Find Interesting at the bottom of this thread for similar threads?

More importantly, what have you tried so far?

Last edited by vgersh99; 04-19-2020 at 12:44 PM..
This User Gave Thanks to vgersh99 For This Post:

Previous Thread | Next Thread
Test Your Knowledge in Computers #565
Difficulty: Medium
The logical and operator is also called a conjunction.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need awk or Shell script to compare Column-1 of two different CSV files and print if column-1 matche

Example: I have files in below format file 1: zxc,133,joe@example.com cst,222,xyz@example1.com File 2 Contains: hxd hcd jws zxc cst File 1 has 50000 lines and file 2 has around 30000 lines : Expected Output has to be : hxd hcd jws (5 Replies)
Discussion started by: TestPractice
5 Replies

2. Shell Programming and Scripting

Simple awk command to compare two files and print first difference

Hello, I have two text files, each with a single column, file 1: 124152970 123899868 123476854 54258288 123117283 file 2: 124152970 123899868 54258288 123117283 122108330 (5 Replies)
Discussion started by: LMHmedchem
5 Replies

3. Shell Programming and Scripting

Compare two text files and output difference

Hi experts, I am trying to compare two text files and output the difference to another file. I'm not strictly looking for differences in text but additional text at the end of one file that isn't in another, so basically comparing the file 2 against file 1 and printing any additional text to... (9 Replies)
Discussion started by: martin0852
9 Replies

4. Shell Programming and Scripting

Compare two text files and print matches

Hi, I am looking for a way to compare two text files and print the matches. For example; File1.txt 89473036 78474384 48948408 95754748 47849030 File2.txt 47849030 46730356 16734947 78474384 36340047 Output: (11 Replies)
Discussion started by: lewk
11 Replies

5. UNIX for Advanced & Expert Users

Shell Script to compare xml files and print output to a file

All, PLease can you help me with a shell script which can compare two xml files and print the difference to a output file. I have attached one such file for you reference. <Group> <Member ID=":Year_Quad:41501" childCount="4" fullPath="PEPSICO Year-Quad-Wk : FOLDER.52 Weeks Ending Dec... (2 Replies)
Discussion started by: kanthrajgowda
2 Replies

6. Shell Programming and Scripting

Compare two columns in two files and print the difference

one file . . importing table employee 119 . . importing table jobs 1 2nd file . . importing table employee 120 . . importing table jobs 1 and would like... (2 Replies)
Discussion started by: jhonnyrip
2 Replies

7. Shell Programming and Scripting

Compare Fields from two text files using key columns

Hi All, I have two files to compare. Each has 10 columns with first 4 columns being key index together. The rest of the columns have monetary values. Using Perl, I want to read one file into hash; check for the key value availability in file 2; then compare the values in the rest of 6... (2 Replies)
Discussion started by: Sangtha
2 Replies

8. Shell Programming and Scripting

Compare two files and print the two lines with difference

I have two files like this: #FILE 1 ABCD 4322 26485 JMTJ 5311 97248 XMPJ 4321 58978 #FILE 2 ABCD 4321 26485 JMTJ 5311 97248 XMPJ 4321 68978 What to do: Compare the two files and find those lines that doesn't match. And have a new file like this: #FILE 3 "from file 1" ABCD 4322 26485... (11 Replies)
Discussion started by: kingpeejay
11 Replies

9. Shell Programming and Scripting

script to compare first column of two files and find difference

Hi, I want to write a script which will compare the 1st column of both the files and will give the difference. e.g:- my 1st file contains: 89 /usr 52 /usr/local 36 /tmp 92 /opt 96 /home 27 /etc/opt/EMCom 1 ... (3 Replies)
Discussion started by: adityam
3 Replies

10. Shell Programming and Scripting

to compare two files and to print the difference

suppose one file P1168S P2150L P85L Q597R R1097C Another file P2150L P85L Q597R R1097C R1379C R1587K Then output shud be R1379C R1587K thanks (5 Replies)
Discussion started by: cdfd123
5 Replies

Featured Tech Videos