I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it.
Currently, I am using:
sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins.
Is there any other faster way... (15 Replies)
Hello,
I can remove duplicate entries in a file by:
sort File1 | uniq > File2
but how can I remove duplicates without sorting the file?
I tried cat File1 | uniq > File2 but it doesn't work
thanks (4 Replies)
Hello Experts,
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
I need to use bash to remove duplicates without using sort first.
I can not use:
cat file | sort | uniq
But when I use only
cat file | uniq
some duplicates are not removed. (4 Replies)
I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file.
Requirement:
Can both the awk... (4 Replies)
Hi all,
I need some help to remove duplicates from a file before merging.
I have got 2 files:
file1 has data in format
4300 23456
4301 2357
the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file
file 2 has data in same format but is not in... (10 Replies)
I have a file with the following format:
fields seperated by "|"
title1|something class|long...content1|keys
title2|somhing class|log...content1|kes
title1|sothing class|lon...content1|kes
title3|shing cls|log...content1|ks
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
Hi I have a below file structure.
200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,,
200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,,
300,20140223,0.001,0.001,0.001,0.001,0.001
300,20140224,0.001,0.001,0.001,0.001,0.001
300,20140225,0.001,0.001,0.001,0.001,0.001
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
LEARN ABOUT X11R4
libbash
LIBBASH(7) libbash Manual LIBBASH(7)NAME
libbash -- A bash shared libraries package.
DESCRIPTION
libbash is a package that enables bash dynamic-like shared libraries. Actually its a tool for managing bash scripts whose functions you may
want to load and use in scripts of your own.
It contains a 'dynamic loader' for the shared libraries ( ldbash(1)), a configuration tool (ldbashconfig(8)), and some libraries.
Using ldbash(1) you are able to load loadable bash libraries, such as getopts(1) and hashstash(1). A bash shared library that can be loaded
using
ldbash(1) must answer 4 requirments:
1. It must be installed in $LIBBASH_PREFIX/lib/bash (default is /usr/lib/bash).
2. It must contain a line that begins with '#EXPORT='. That line will contain (after the '=') a list of functions that the library
exports. I.e. all the function that will be usable after loading that library will be listed in that line.
3. It must contain a line that begins with '#REQUIRE='. That line will contain (after the '=') a list of bash libraries that are
required for our library. I.e. every bash library that is in use in our bash library must be listed there.
4. The library must be listed (For more information, see ldbashconfig(8)).
Basic guidelines for writing library of your own:
1. Be aware, that your library will be actually sourced. So, basically, it should contain (i.e define) only functions.
2. Try to declare all variables intended for internal use as local.
3. Global variables and functions that are intended for internal use (i.e are not defined in '#EXPORT=') should begin with:
__<library_name>_
For example, internal function myfoosort of hashstash library should be named as
__hashstash_myfoosort
This helps to avoid conflicts in global name space when using libraries that come from different vendors.
4. See html manual for full version of this guide.
AUTHORS
Hai Zaar <haizaar@haizaar.com>
Gil Ran <ril@ran4.net>
SEE ALSO ldbash(1), ldbashconfig(8), getopts(1), hashstash(1)colors(1)messages(1)urlcoding(1)locks(1)Linux Epoch Linux