Sponsored Content
Top Forums UNIX for Advanced & Expert Users File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this Post 303025350 by gull04 on Wednesday 31st of October 2018 08:57:09 AM
Old 10-31-2018
Hi kartikirans,

I'd be tempted to look at comm -3 ${file1} ${file2} this will suppress lines common to ${file1} and ${file2} later versions of comm don't require the files to be sorted.

Regards

Gull04
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

search and grab data from a huge file

folks, In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help? For example, if I want to find if these files contain a string like "93849", what command I should use? Also, there is oder_id number... (1 Reply)
Discussion started by: ting123
1 Replies

2. Shell Programming and Scripting

How to extract data from a huge file?

Hi, I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows: 1. Needs to create folders as the strings starts with "item_*" from the input file 2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies

3. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

4. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

5. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

6. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

7. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

8. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

9. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

10. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies
plotit(1)						      General Commands Manual							 plotit(1)

NAME
plotit - experiment data visualization SYNOPSIS
plotit -w FILE plotit [-lp] -q FILE [FILE]... plotit [-lp] Y FILE [FILE...] plotit [-lp] X Y FILE [FILE...] DESCRIPTION
The plotit program is a simple plotting program which can draw line plots and stripcharts using a command language similar to that of gnu- plot. It can also make quick plots of multiple data files from the command-line. Wildcards may be used in the file names. The program takes optional expressions which can be used to manipulate data columns before plotting. Expressions may be as simple as the designation of a column to be plotted or may contain arithmetic expressions and functions. In the latter case the espression must be enclosed in 'single' or "double" quotation marks. Column numbers are designated by the at-sign character (@). A single expression is used for the Y axis of the plot. OPTIONS
-h, --help Print the usage and exit. --version Print the version number and exit. -w Watch the input file for commands to stripchart, or wait to read all of stdin before plotting. Only one file or - for stdin may be specified with this option. No other options may be used in conjuction with this option. -q Plot the first and second columns of each file. -l Plot with lines. -p Plot with points. EXAMPLES
Plot the second column versus the first of all three files with lines only. plotit -lq file1.dat file2.dat file3.dat Plot the quotient of the third column and the second column versus the first column of all files matching the pattern using points only. plotit -p '@3/@2' file??.dat Plot the natural log of the second column divided by the third column versus the fourth column. By default these data are plotted with both lines and points. plotit @4 'log(@2/@3)' file.dat AUTHOR
This manual page was written by Carlo Segre <segre@iit.edu>. February 21, 2009 plotit(1)
All times are GMT -4. The time now is 03:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy