09-15-2012
Which URL should stay? And is the list a text file?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it.
Currently, I am using:
sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins.
Is there any other faster way... (15 Replies)
Discussion started by: radhika
15 Replies
2. UNIX for Dummies Questions & Answers
Hello,
I can remove duplicate entries in a file by:
sort File1 | uniq > File2
but how can I remove duplicates without sorting the file?
I tried cat File1 | uniq > File2 but it doesn't work
thanks (4 Replies)
Discussion started by: orahi001
4 Replies
3. Shell Programming and Scripting
Hello Experts,
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
4. UNIX for Dummies Questions & Answers
Can u tell me how to remove duplicate records from a file? (11 Replies)
Discussion started by: saga20
11 Replies
5. Shell Programming and Scripting
I need to use bash to remove duplicates without using sort first.
I can not use:
cat file | sort | uniq
But when I use only
cat file | uniq
some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies
6. Shell Programming and Scripting
I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file.
Requirement:
Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies
7. Shell Programming and Scripting
Hi all,
I need some help to remove duplicates from a file before merging.
I have got 2 files:
file1 has data in format
4300 23456
4301 2357
the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file
file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies
8. Shell Programming and Scripting
I have a file with the following format:
fields seperated by "|"
title1|something class|long...content1|keys
title2|somhing class|log...content1|kes
title1|sothing class|lon...content1|kes
title3|shing cls|log...content1|ks
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies
9. Shell Programming and Scripting
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies
10. Shell Programming and Scripting
Hi I have a below file structure.
200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,,
200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,,
300,20140223,0.001,0.001,0.001,0.001,0.001
300,20140224,0.001,0.001,0.001,0.001,0.001
300,20140225,0.001,0.001,0.001,0.001,0.001
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
RBASH(1) General Commands Manual RBASH(1)
NAME
rbash - restricted bash, see bash(1)
RESTRICTED SHELL
If bash is started with the name rbash, or the -r option is supplied at invocation, the shell becomes restricted. A restricted shell is
used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the follow-
ing are disallowed or not performed:
o changing directories with cd
o setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV
o specifying command names containing /
o specifying a file name containing a / as an argument to the . builtin command
o specifying a filename containing a slash as an argument to the -p option to the hash builtin command
o importing function definitions from the shell environment at startup
o parsing the value of SHELLOPTS from the shell environment at startup
o redirecting output using the >, >|, <>, >&, &>, and >> redirection operators
o using the exec builtin command to replace the shell with another command
o adding or deleting builtin commands with the -f and -d options to the enable builtin command
o using the enable builtin command to enable disabled shell builtins
o specifying the -p option to the command builtin command
o turning off restricted mode with set +r or set +o restricted.
These restrictions are enforced after any startup files are read.
When a command that is found to be a shell script is executed, rbash turns off any restrictions in the shell spawned to execute the script.
SEE ALSO
bash(1)
GNU Bash-4.0 2004 Apr 20 RBASH(1)