"Duplicate lines in a file"

Post #302661431 by alister on Monday 25th of June 2012 09:08:17 AM

Full Discussion: Duplicate lines in a file
Quote:
Originally Posted by nsuresh316
My shell is csh
That's the type of information that should always be included in the original post, especially since most people will provide posix (bourne like) sh solutions.

Regards,
Alister
 
Test Your Knowledge in Computers #866
Difficulty: Easy
Lisp and Fortran are the two oldest high-level programming language still in widespread use today.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

2. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

3. UNIX for Dummies Questions & Answers

How to redirect duplicate lines from a file????

Hi, I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file. Suppose I have a file called file_dup.txt which contains some line as file_dup.txt A100-R1 ACCOUNTING-CONTROL ACTONA-ACTASTOR ADMIN-AUTH-STATS ACTONA-ACTASTOR... (3 Replies)
Discussion started by: zing_foru
3 Replies

4. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

5. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. UNIX for Advanced & Expert Users

Inserting duplicate lines in a file

Hi, I copied the contents of a binary file into a .text file using hd (hexdump) command. The data in binary file is such that I get in many places like following 00000250 00 00 00 00 3f 2d 91 68 3f 69 fb e7 00 00 00 00 |....?-.h?i......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00... (2 Replies)
Discussion started by: KidD312
2 Replies

9. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance! (4 Replies)
Discussion started by: vlm
4 Replies

10. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
shell_builtins(1)														 shell_builtins(1)

NAME
shell_builtins, case, for, foreach, function, if, repeat, select, switch, until, while - shell command interpreter built-in commands The shell command interpreters csh(1), ksh(1), and sh(1) have special built-in commands. The commands case, for, foreach, function, if, repeat, select, switch, until, and while are commands in the syntax recognized by the shells. They are described in the Commands section of the manual pages of the respective shells. The remaining commands listed in the table below are built into the shells for reasons such as efficiency or data sharing between command invocations. They are described on their respective manual pages. | Command | Shell alias |csh, ksh bg |csh, ksh, sh break |csh, ksh, sh case |csh, ksh, sh cd |csh, ksh, sh chdir |csh, sh continue |csh, ksh, sh dirs |csh echo |csh, ksh, sh eval |csh, ksh, sh exec |csh, ksh, sh exit |csh, ksh, sh export |ksh, sh false |ksh fc |ksh fg |csh, ksh, sh for |ksh, sh foreach |csh function |ksh getopts |ksh, sh glob |csh goto |csh hash |ksh, sh hashstat |csh history |csh if |csh, ksh, sh jobs |csh, ksh, sh kill |csh, ksh, sh let |ksh limit |csh login |csh, ksh, sh logout |csh, ksh, sh nice |csh newgrp |ksh, sh nohup |csh notify |csh onintr |csh popd |csh print |ksh pushd |csh pwd |ksh, sh read |ksh, sh readonly |ksh, sh rehash |csh repeat |csh return |ksh, sh select |ksh set |csh, ksh, sh setenv |csh shift |csh, ksh, sh source |csh stop |csh, ksh, sh suspend |csh, ksh, sh switch |csh test |ksh, sh time |csh times |ksh, sh trap |ksh, sh true |ksh type |ksh, sh typeset |ksh ulimit |ksh, sh umask |csh, ksh, sh unalias |csh, ksh unhash |csh unlimit |csh unset |csh, ksh, sh unsetenv |csh until |ksh, sh wait |csh, ksh, sh whence |ksh while |csh, ksh, sh Bourne Shell, sh, Special Commands Input/output redirection is now permitted for these commands. File descriptor 1 is the default output location. When Job Control is enabled, additional Special Commands are added to the shell's environment. In addition to these built-in reserved command words, sh also uses: : No effect; the command does nothing. A zero exit code is returned. .filename Read and execute commands from filename and return. The search path specified by PATH is used to find the directory con- taining filename. C shell, csh Built-in commands are executed within the C shell. If a built-in command occurs as any component of a pipeline except the last, it is exe- cuted in a subshell. In addition to these built-in reserved command words, csh also uses: : Null command. This command is interpreted, but performs no action. Korn Shell, ksh, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari- able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not performed. In addition to these built-in reserved command words, ksh also uses: * : [ arg ... ] The command only expands parameters. * .file [ arg ..Read the complete file then execute the commands. The commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing file. If any arguments arg are given, they become the posi- tional parameters. Otherwise, the positional parameters are unchanged. The exit status is the exit status of the last com- mand executed. the loop termination test. intro(1), alias(1), break(1), cd(1), chmod(1), csh(1), echo(1), exec(1), exit(1), find(1), getoptcvt(1), getopts(1), glob(1), hash(1), his- tory(1), jobs(1), kill(1), ksh(1), let(1), limit(1), login(1), logout(1), newgrp(1), nice(1), nohup(1), print(1), pwd(1), read(1), read- only(1), set(1), sh(1), shift(1), suspend(1), test(1B), time(1), times(1), trap(1), typeset(1), umask(1), wait(1), chdir(2), chmod(2), creat(2), umask(2), getopt(3C), profile(4), environ(5) 29 Jun 2005 shell_builtins(1)

Featured Tech Videos

All times are GMT -4. The time now is 04:26 PM.
Unix & Linux Forums Content Copyright 1993-2019. All Rights Reserved.
Privacy Policy