Sponsored Content
Top Forums Shell Programming and Scripting Deleting Repeating lines from a txt file via script Post 302903937 by Gautham on Saturday 31st of May 2014 01:10:20 AM
Old 05-31-2014
Ya that was my concern..

Your code will remove all duplicate lines, but what i need is to remove only duplicate in headers like this:
Quote:
20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,
I thought of using any pattern matching kinda logic but not able to move with it..

Last edited by Gautham; 05-31-2014 at 02:17 AM.. Reason: updated
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue... (1 Reply)
Discussion started by: ajp7701
1 Replies

2. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

I need some help deleting lines in a file that contain spaces. Im sure awk or sed will work but i dont know much about those commands. Any help is appreciated :D (7 Replies)
Discussion started by: r04dw4rri0r
7 Replies

3. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Hello, i am new to the forum and know nothing about programing, Linux or Unix :( hope somebody can help me out. I have a .txt file that i need to delete certain lines from. After searching the forum i noticed that using "sed" was the way to go, so i installed gnuwin32 (i use windows xp... (4 Replies)
Discussion started by: luis3141
4 Replies

4. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

File 6 dbnawldb010-b office Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/09/11 03:24:04 42 luigi-b IPNRemitDB Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/10/11 00:41:36 6 ebs-sqldev1-b IPNTracking Memphis_Corp_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

5. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an... (3 Replies)
Discussion started by: czar21
3 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

8. UNIX for Dummies Questions & Answers

redirecting arguments in a script to multiple lines in a .txt file

Ok hope my vocab is right here, i'm trying to write multiple sets of arguments to another file for example: I have a script that accepts four arguments and sends them to a another file $write.sh it then out in so the file receiver.txt would contain this: it then out in what... (2 Replies)
Discussion started by: austing5
2 Replies

9. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks (4 Replies)
Discussion started by: leghorn
4 Replies

10. Shell Programming and Scripting

Creating repeating record in between file through script

apprecieate your help to resove this. My source file looke like 1001 000 HEADER 1001 001 RAJESH 1001 002 100 1001 002 200 1001 002 500 1001 006 FOOTER 1002 000 HEADER 1002 001 RAMESH 1002 002 100 1002 002 200 1002 002 500 1002 006 FOOTER my... (8 Replies)
Discussion started by: Ganesh L
8 Replies
SHELL-QUOTE(1)						User Contributed Perl Documentation					    SHELL-QUOTE(1)

NAME
shell-quote - quote arguments for safe use, unmodified in a shell command SYNOPSIS
shell-quote [switch]... arg... DESCRIPTION
shell-quote lets you pass arbitrary strings through the shell so that they won't be changed by the shell. This lets you process commands or files with embedded white space or shell globbing characters safely. Here are a few examples. EXAMPLES
ssh preserving args When running a remote command with ssh, ssh doesn't preserve the separate arguments it receives. It just joins them with spaces and passes them to "$SHELL -c". This doesn't work as intended: ssh host touch 'hi there' # fails It creates 2 files, hi and there. Instead, do this: cmd=`shell-quote touch 'hi there'` ssh host "$cmd" This gives you just 1 file, hi there. process find output It's not ordinarily possible to process an arbitrary list of files output by find with a shell script. Anything you put in $IFS to split up the output could legitimately be in a file's name. Here's how you can do it using shell-quote: eval set -- `find -type f -print0 | xargs -0 shell-quote --` debug shell scripts shell-quote is better than echo for debugging shell scripts. debug() { [ -z "$debug" ] || shell-quote "debug:" "$@" } With echo you can't tell the difference between "debug 'foo bar'" and "debug foo bar", but with shell-quote you can. save a command for later shell-quote can be used to build up a shell command to run later. Say you want the user to be able to give you switches for a command you're going to run. If you don't want the switches to be re-evaluated by the shell (which is usually a good idea, else there are things the user can't pass through), you can do something like this: user_switches= while [ $# != 0 ] do case x$1 in x--pass-through) [ $# -gt 1 ] || die "need an argument for $1" user_switches="$user_switches "`shell-quote -- "$2"` shift;; # process other switches esac shift done # later eval "shell-quote some-command $user_switches my args" OPTIONS
--debug Turn debugging on. --help Show the usage message and die. --version Show the version number and exit. AVAILABILITY
The code is licensed under the GNU GPL. Check http://www.argon.org/~roderick/ or CPAN for updated versions. AUTHOR
Roderick Schertler <roderick@argon.org> perl v5.16.3 2010-06-11 SHELL-QUOTE(1)
All times are GMT -4. The time now is 03:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy