Sponsored Content
Top Forums Shell Programming and Scripting grep and delete 2nd duplicated of txt... -part2 Post 302103345 by kapilraj on Wednesday 17th of January 2007 10:26:25 PM
Old 01-17-2007
count=0
cat {1} | while read line
do
count=`echo "${count} + 1" | bc`
cc=`head -${count} ${1} |grep -c ${line}`
if [ ${cc} -eq 2 ]
then
:
else
echo ${line}
fi
done

Would this work ? I have not tested it but it should work. save it as a file , give it execute permission and then use it as "script filename".

All the best. It's a simple logic but i think that would work for you.

Regds,

Kaps
 

10 More Discussions You Might Find Interesting

1. Windows & DOS: Issues & Discussions

sftp not get txt file 2nd time

I use sftp batch script to copy file from a remote server to local machine. Problem is when I rerun this shell after copying txt file it do not allow me to copy txt file again. My question is , is this confrigration problem of remote server. beace same shell work fie with the other remote... (3 Replies)
Discussion started by: Jamil Qadir
3 Replies

2. Shell Programming and Scripting

sed to read x.txt and grep from y.txt

How would I write a command(s) to read from a file (list) that looks like this: 29847374384 and grep from a second file (list) that looks like this: 29847374384, jkdfkjdf,3833,ddd:confused: (1 Reply)
Discussion started by: smellylizzard
1 Replies

3. Shell Programming and Scripting

delete duplicated characters in each line

I'm a biologist trying to analyse some data and I'll appreciate some help with the following problem. I have a column of characters which I'll like to delete the duplicated characters in each line and report only the unique one.No sorting should be done. E.g. The original data: GTG CTC CTC... (5 Replies)
Discussion started by: ivpz
5 Replies

4. Shell Programming and Scripting

Delete first and second lines from .txt

Hi, I have to programming a program in shell script which combine a lot of .txt files. But in all of these files the program had to delete the first both lines. Because I don't know anything about shell script I need your help. Does anyone have a command or a hint, where I can look for? ... (1 Reply)
Discussion started by: mkrol
1 Replies

5. Homework & Coursework Questions

Delete first both lines from .txt

Hi, I have to programming a program in shell script which combine a lot of .txt files. But in all of these files the program had to delete the first both lines. Because I don't know anything about shell script I need your help. Does anyone have a command or a hint, where I can look for? ... (1 Reply)
Discussion started by: mkrol
1 Replies

6. UNIX for Dummies Questions & Answers

If then else for decimal numbers part2

Hi, I have a small problem with my script. I have everything in order but it doesnt seem to compare anything less than 1 correctly. If the input is more than 1, then the results is correct. If the input is 0.xxx (anything) it returns erroneous results. Pls help input=0.12 if ; then ... (7 Replies)
Discussion started by: streddy
7 Replies

7. Shell Programming and Scripting

Delete file2.txt from file1.txt using scripting

Hi, I`m a total newbie, well my requirement is that i have 2 files I want to identify which countries i do not currently have in db.. how can i use the grep or another command to find this file .. i want to match all-countries.txt with countries-in-db.txt so the output is equal to... (11 Replies)
Discussion started by: beanbaby
11 Replies

8. Shell Programming and Scripting

Delete duplicated fields in a line

Hi, I have files with this kind of format (separator is space): A1 B1 C1 D1 E1 F1 D1 C1 G1 H1 A2 B2 C2 D2 E2 F2 D2 C2 G2 H2 A3 B3 C3 D3 E3 F3 G3 D3 C3 H3 A4 B4 C4 D4 E4 F4 G4 D4 C4 H4 I want the output to be: A1 B1 E1 F1 G1 H1 A2 B2 E2 F2 G2 H2 A3 B3 E3 F3 G3 H3 A4 B4 E4 F4 G4... (12 Replies)
Discussion started by: Gr4wk
12 Replies

9. UNIX for Dummies Questions & Answers

Delete files in a txt file

Hi, I have very old files in my server like from 2012 and i want to delete them, Please help. Thanks in advance.. (2 Replies)
Discussion started by: nanz143
2 Replies

10. Shell Programming and Scripting

How to delete 'duplicated' column values and make a delimited file too?

Hi, I have the following output from an Oracle SQL statement and I want to remove duplicated column values. I know it is possible using Oracle analytical/statistical functions but unfortunately I don't know how to use any of those. So now, I've gone to PLAN B using awk/sed maybe or any... (5 Replies)
Discussion started by: newbie_01
5 Replies
rl(1)								   User Commands							     rl(1)

NAME
rl - Randomize Lines. SYNOPSIS
rl [OPTION]... [FILE]... DESCRIPTION
rl reads lines from a input file or stdin, randomizes the lines and outputs a specified number of lines. It does this with only a single pass over the input while trying to use as little memory as possible. -c, --count=N Select the number of lines to be returned in the output. If this argument is omitted all the lines in the file will be returned in random order. If the input contains less lines than specified and the --reselect option below is not specified a warning is printed and all lines are returned in random order. -r, --reselect When using this option a single line may be selected multiple times. The default behaviour is that any input line will only be selected once. This option makes it possible to specify a --count option with more lines than the file actually holds. -o, --output=FILE Send randomized lines to FILE instead of stdout. -d, --delimiter=DELIM Use specified character as a "line" delimiter instead of the newline character. -0, --null Input lines are terminated by a null character. This option is useful to process the output of the GNU find -print0 option. -n, --line-number Output lines are numbered with the line number from the input file. -q, --quiet, --silent Be quiet about any errors or warnings. -h, --help Show short summary of options. -v, --version Show version of program. EXAMPLES
Some simple demonstrations of how rl can help you do everyday tasks. Play a random sound after 4 minutes (perfect for toast): sleep 240 ; play `find /sounds -name '*.au' -print | rl --count=1` Play the 15 most recent .mp3 files in random order. ls -c *.mp3 | head -n 15 | rl | xargs --delimiter=' ' play Roll a dice: seq 6 | rl --count 2 Roll a dice 1000 times and see which number comes up more often: seq 6 | rl --reselect --count 1000 | sort | uniq -c | sort -n Shuffle the words of a sentence: echo -n "The rain in Spain stays mainly in the plain." | rl --delimiter=' ';echo Find all movies and play them in random order. find . -name '*.avi' -print0 | rl -0 | xargs -n 1 -0 mplayer Because -0 is used filenames with spaces (even newlines and other unusual characters) in them work. BUGS
The program currently does not have very smart memory management. If you feed it huge files and expect it to fully randomize all lines it will completely read the file in memory. If you specify the --count option it will only use the memory required for storing the specified number of lines. Improvements on this area are on the TODO list. The program uses the rand() system random function. This function returns a number between 0 and RAND_MAX, which may not be very large on some systems. This will result in non-random results for files containing more lines than RAND_MAX. Note that if you specify multiple input files they are randomized per file. This is a different result from when you cat all the files and pipe the result into rl. COPYRIGHT
Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 Arthur de Jong. This is free software; see the license for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Version 0.2.7 Jul 2008 rl(1)
All times are GMT -4. The time now is 06:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy