Sponsored Content
Top Forums UNIX for Dummies Questions & Answers File comparison of huge files Post 302588223 by kaaliakahn on Saturday 7th of January 2012 04:41:00 PM
Old 01-07-2012
Thanks a million ctsgnb and all others.

I am having problems on large files. I am able to get this thing working on 3GB files but when the file size is 11GB, i get this message

sort: write failed: /tmp/sortZgF9MD: No space left on device

despite the fact that i have good hard disk space

Any solution to this.

Please suggest



Really appreciate your help

Kind Regards,
kaaliakahn

Last edited by kaaliakahn; 01-07-2012 at 05:50 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

2. UNIX for Advanced & Expert Users

Huge files manipulation

Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text. I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump) In using HP-UX large servers. Any advice will... (8 Replies)
Discussion started by: Klashxx
8 Replies

3. Shell Programming and Scripting

Huge File Comparison

Hi i need to compare two fixed length files and produce the differences if any to a seperate file. I have to capture each and every differneces line by line. Ideally my files should not have any differences but if there are any then it should be captured without any miss. Also my files sizes are... (4 Replies)
Discussion started by: naveenn08
4 Replies

4. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

5. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

6. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

7. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

8. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

9. Shell Programming and Scripting

Aggregation of Huge files

Hi Friends !! I am facing a hash total issue while performing over a set of files of huge volume: Command used: tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f' Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies

10. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies
ln(1B)						     SunOS/BSD Compatibility Package Commands						    ln(1B)

NAME
ln - make hard or symbolic links to files SYNOPSIS
/usr/ucb/ln [-fs] filename [linkname] /usr/ucb/ln [-fs] pathname... directory DESCRIPTION
The /usr/ucb/ln utility creates an additional directory entry, called a link, to a file or directory. Any number of links can be assigned to a file. The number of links does not affect other file attributes such as size, protections, data, etc. filename is the name of the original file or directory. linkname is the new name to associate with the file or filename. If linkname is omitted, the last component of filename is used as the name of the link. If the last argument is the name of a directory, symbolic links are made in that directory for each pathname argument; /usr/ucb/ln uses the last component of each pathname as the name of each link in the named directory. A hard link (the default) is a standard directory entry just like the one made when the file was created. Hard links can only be made to existing files. Hard links cannot be made across file systems (disk partitions, mounted file systems). To remove a file, all hard links to it must be removed, including the name by which it was first created; removing the last hard link releases the inode associated with the file. A symbolic link, made with the -s option, is a special directory entry that points to another named file. Symbolic links can span file sys- tems and point to directories. In fact, you can create a symbolic link that points to a file that is currently absent from the file sys- tem; removing the file that it points to does not affect or alter the symbolic link itself. A symbolic link to a directory behaves differently than you might expect in certain cases. While an ls(1) on such a link displays the files in the pointed-to directory, an `ls -l' displays information about the link itself: example% /usr/ucb/ln -s dir link example% ls link file1 file2 file3 file4 example% ls -l link lrwxrwxrwx 1 user 7 Jan 11 23:27 link -> dir When you use cd(1) to change to a directory through a symbolic link, you wind up in the pointed-to location within the file system. This means that the parent of the new working directory is not the parent of the symbolic link, but rather, the parent of the pointed-to direc- tory. For instance, in the following case the final working directory is /usr and not /home/user/linktest. example% pwd /home/user/linktest example% /usr/ucb/ln -s /var/tmp symlink example% cd symlink example% cd .. example% pwd /usr C shell user's can avoid any resulting navigation problems by using the pushd and popd built-in commands instead of cd. OPTIONS
-f Force a hard link to a directory. This option is only available to the super-user, and should be used with extreme caution. -s Create a symbolic link or links. USAGE
See largefile(5) for the description of the behavior of ln when encountering files greater than or equal to 2 Gbyte ( 2^31 bytes). EXAMPLES
Example 1 The /usr/ucb/ln command The commands below illustrate the effects of the different forms of the /usr/ucb/ln command: example% /usr/ucb/ln file link example% ls -F file link file link example% /usr/ucb/ln -s file symlink example% ls -F file symlink file symlink@ example% ls -li file link symlink 10606 -rw-r--r-- 2 user 0 Jan 12 00:06 file 10606 -rw-r--r-- 2 user 0 Jan 12 00:06 link 10607 lrwxrwxrwx 1 user 4 Jan 12 00:06 symlink -> file example% /usr/ucb/ln -s nonesuch devoid example% ls -F devoid devoid@ example% cat devoid devoid: No such file or directory example% /usr/ucb/ln -s /proto/bin/* /tmp/bin example% ls -F /proto/bin /tmp/bin /proto/bin: x* y* z* /tmp/bin: x@ y@ z@ ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWscpu | +-----------------------------+-----------------------------+ SEE ALSO
cp(1), ls(1), mv(1), rm(1), link(2), readlink(2), stat(2), symlink(2), attributes(5), largefile(5) NOTES
When the last argument is a directory, simple basenames should not be used for pathname arguments. If a basename is used, the resulting symbolic link points to itself: example% /usr/ucb/ln -s file /tmp example% ls -l /tmp/file lrwxrwxrwx 1 user 4 Jan 12 00:16 /tmp/file -> file example% cat /tmp/file /tmp/file: Too many levels of symbolic links To avoid this problem, use full pathnames, or prepend a reference to the PWD variable to files in the working directory: example% rm /tmp/file example% /usr/ucb/ln -s $PWD/file /tmp lrwxrwxrwx 1 user 4 Jan 12 00:16 /tmp/file -> /home/user/subdir/file SunOS 5.11 11 Mar 1994 ln(1B)
All times are GMT -4. The time now is 12:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy