UNIX for Dummies Questions & Answers

Hi all,
I hope you are well. I am very happy to see your contribution. I am eager to become part of it.

I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below

For clear picture, please see attached image

File 1 File 2
----------------- -------------------
Time sig_name sig_val Time sig_name sig_val

0ns sig1 0 0ns sig1 0
0ns sig2 0 0ns sig2 1
0ns sig3 0 0ns sig3 1
0ns sig4 1 0ns sig4 1
1ns sig1 0 1ns sig1 0
1ns sig2 0 1ns sig2 0
1ns sig3 0 1ns sig3 0
1ns sig4 0 1ns sig4 0
2ns sig1 0 2ns sig1 1
2ns sig2 0 2ns sig2 0
2ns sig3 0 2ns sig3 0
2ns sig4 0 2ns sig4 0
3ns sig1 1 3ns sig1 0
3ns sig2 0 3ns sig2 1
3ns sig3 0 3ns sig3 0
3ns sig4 0 3ns sig4 0

Given the two files in the above format, how can i print out the following table from "efficient" file comparison. Efficiency is required as file size is over 3GB

signal number_of_mismatches time_of_mismatch
------ -------------------- ----------------
sig1 2 2ns, 3ns
sig2 2 0ns, 3ns
sig3 1 0ns
sig4 0


I shall really appreciate your response.

Unless you have enough RAM to process such volume of data on the flight, it would be advisable to load all those data into an DBMS (database management system) for further processing.

Otherwise : filter out duplicate lines and save lines that appear only once in file3

Code:

cat file1 file2 | sort | uniq -u >file3

(also see the corresponding -T options of sort command (depending on your OS) for using temporary files and/or directory)

Then process file3 (i didn't test it but maybe something like this :

Code:

awk '{N[$2]++;A[$2]=(A[$2]=="")?$1:((A[$2]~$1)?(A[$2]):(A[$2]","$1))}END{print "signal" OFS "num of mismatch" OFS "time of missmatch";for(i in A) {print i OFS N[i]/2 OFS A[i]}}' OFS="\t" file3

---------- Post updated at 07:38 PM ---------- Previous update was at 07:27 PM ----------

Regarding the file3 generation, you could also give a try to something like

Code:

mknod /tmp/pipeline p
sort -u </tmp/pipeline >file3 &
cat file1 file2>/tmp/pipeline

Thanks ctsgnb.


Could you please explain why do you need to sort?

Secondly, could you please explain the details of the awk code?

Thirdly please explain why use mknod?

I shall really appreciate.

Kind Regards,

---------- Post updated at 03:35 PM ---------- Previous update was at 03:01 PM ----------

It produces the WRONG output as follows.Note sig3 has 2.5 as number which makes no sense

signal num of mismatch time of mismatch
sig1 3 0ns,1ns,2ns,3ns
sig2 3 0ns,1ns,2ns,3ns
sig3 2.5 0ns,1ns,2ns,3ns
sig4 2 0ns,1ns,2ns,3ns

instead, the following (correct) output should be produced

signal number_of_mismatches time_of_mismatch
------ -------------------- ----------------
sig1 2 2ns, 3ns
sig2 2 0ns, 3ns
sig3 1 0ns
sig4 0

---------- Post updated at 03:56 PM ---------- Previous update was at 03:35 PM ----------

/********************* IT WORKS AS FOLLOWS *************************/

Here is what works. ctsgnb your awk script is awesome it works perfectly. Here is what you need to modify

cat file1 file2 | sort | uniq -u > file3.txt

Then do awk using your script on file3.txt


Thanks a lot.

It's the simplest and most reliable way to tell whether a line's a duplicate or not. sort is actually pretty sophisticated, capable of handling extremely huge files without bogging down. Once it's done so, any duplicate lines will show up all in a row, letting you use uniq to get rid of them. He's actually making a named pipe. On many systems you can just use mkfifo for that.

A named pipe lets you do this:

Code:

mkfifo fifo
cat file > fifo &
cat fifo # the first cat writes to the fifo, the second cat reads from it.
# This is effectively cat file | cat.
# Named pipes can be a handy way to join together programs which
# read from filenames and can't easily handle stdin/stdout communication.

Hi

Yes you are right i did a little mistake :

sort -u would sort the file and if there are some duplicate lines, it would display them once instead of displaying ONLY lines that DO NOT have duplicate (uniq -u does it).

Note that the uniq command proceed to comparison sequentially so the output need to be sorted at first.

that is why the sort is necessary before extracting the lines that appear only once.

Code:

cat file1 file2 | sort | uniq -u >file3

Then, as i already specified in my previous post, the awk code need to be applied on file3, here are the requested information about the awk code:

Thanks a million ctsgnb and all others.

I am having problems on large files. I am able to get this thing working on 3GB files but when the file size is 11GB, i get this message

sort: write failed: /tmp/sortZgF9MD: No space left on device

despite the fact that i have good hard disk space

Any solution to this.

Please suggest



Really appreciate your help

Kind Regards,
kaaliakahn

I think ctsgnb already mentioned this in his post, but try to see if the sort on your OS, support the "-T" option.
Also post output from . Looks like you are running out of space in /tmp. So basically 2 options, see if your sort can use -T and specify an alternate location for tmp-files or split your file into smaller pieces and do a sort on the smaller split files.