Sponsored Content
Top Forums Shell Programming and Scripting Deleting Repeating lines from a txt file via script Post 302903936 by clx on Saturday 31st of May 2014 01:04:33 AM
Old 05-31-2014
How to identify the lines? I am pretty sure you don't want to just remove the duplicates from the file?

If I am wrong,

Code:
awk '!a[$0]++' file

This will remove the duplicate lines without changing the order of the lines. (unlike sort -u)
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue... (1 Reply)
Discussion started by: ajp7701
1 Replies

2. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

I need some help deleting lines in a file that contain spaces. Im sure awk or sed will work but i dont know much about those commands. Any help is appreciated :D (7 Replies)
Discussion started by: r04dw4rri0r
7 Replies

3. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Hello, i am new to the forum and know nothing about programing, Linux or Unix :( hope somebody can help me out. I have a .txt file that i need to delete certain lines from. After searching the forum i noticed that using "sed" was the way to go, so i installed gnuwin32 (i use windows xp... (4 Replies)
Discussion started by: luis3141
4 Replies

4. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

File 6 dbnawldb010-b office Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/09/11 03:24:04 42 luigi-b IPNRemitDB Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/10/11 00:41:36 6 ebs-sqldev1-b IPNTracking Memphis_Corp_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

5. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an... (3 Replies)
Discussion started by: czar21
3 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

8. UNIX for Dummies Questions & Answers

redirecting arguments in a script to multiple lines in a .txt file

Ok hope my vocab is right here, i'm trying to write multiple sets of arguments to another file for example: I have a script that accepts four arguments and sends them to a another file $write.sh it then out in so the file receiver.txt would contain this: it then out in what... (2 Replies)
Discussion started by: austing5
2 Replies

9. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks (4 Replies)
Discussion started by: leghorn
4 Replies

10. Shell Programming and Scripting

Creating repeating record in between file through script

apprecieate your help to resove this. My source file looke like 1001 000 HEADER 1001 001 RAJESH 1001 002 100 1001 002 200 1001 002 500 1001 006 FOOTER 1002 000 HEADER 1002 001 RAMESH 1002 002 100 1002 002 200 1002 002 500 1002 006 FOOTER my... (8 Replies)
Discussion started by: Ganesh L
8 Replies
SORT(1) 						      General Commands Manual							   SORT(1)

NAME
sort - sort a file of ASCII lines SYNOPSIS
sort [-bcdfimnru] [-tc] [-o name] [+pos1] [-pos2] file ... OPTIONS
-b Skip leading blanks when making comparisons -c Check to see if a file is sorted -d Dictionary order: ignore punctuation -f Fold upper case onto lower case -i Ignore nonASCII characters -m Merge presorted files -n Numeric sort order -o Next argument is output file -r Reverse the sort order -t Following character is field separator -u Unique mode (delete duplicate lines) EXAMPLES
sort -nr file # Sort keys numerically, reversed sort +2 -4 file # Sort using fields 2 and 3 as key sort +2 -t: -o out # Field separator is : sort +.3 -.6 # Characters 3 through 5 form the key DESCRIPTION
Sort sorts one or more files. If no files are specified, stdin is sorted. Output is written on standard output, unless -o is specified. The options +pos1 -pos2 use only fields pos1 up to but not including pos2 as the sort key, where a field is a string of characters delim- ited by spaces and tabs, unless a different field delimiter is specified with -t. Both pos1 and pos2 have the form m.n where m tells the number of fields and n tells the number of characters. Either m or n may be omitted. SEE ALSO
comm(1), grep(1), uniq(1). SORT(1)
All times are GMT -4. The time now is 10:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy