Sponsored Content
Full Discussion: remove duplicating lines
Top Forums UNIX for Dummies Questions & Answers remove duplicating lines Post 302355270 by JudithBorg on Tuesday 22nd of September 2009 08:05:55 AM
Old 09-22-2009
it still gives me two lines back, also when i use the command

Code:
$ awk '!arr[$1 $2 $3 $4]++' kk1.txt > uniq.file
or
awk '!arr[$1 $2 $3 $4 $5]++' kk1.txt > uniq.file
or
awk '!arr[$1 $2 $3]++' kk1.txt > uniq.file

the content of my file is quite big, but it pretty much comes down to this:

Code:
38.119.128.203 - - [19/Jul/2006:07:56:25 +0200] "GET /MoodViews/javascript.js HTTP/1.1" 200 343 "http://ilps.science.uva.nl/MoodViews/index.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
38.119.128.203 - - [19/Jul/2006:07:56:25 +0200] "GET /MoodViews/images/menu_bg.png HTTP/1.1" 200 153 "http://ilps.science.uva.nl/MoodViews/mv-fp-styles.css" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
38.119.128.203 - - [19/Jul/2006:07:56:25 +0200] "GET /MoodViews/images/button_selected.png HTTP/1.1" 200 511 "http://ilps.science.uva.nl/MoodViews/mv-fp-styles.css" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
38.119.128.203 - - [19/Jul/2006:07:56:25 +0200] "GET /MoodViews/images/button_normal.png HTTP/1.1" 200 507 "http://ilps.science.uva.nl/MoodViews/mv-fp-styles.css" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
38.119.128.203 - - [19/Jul/2006:07:56:25 +0200] "GET /MoodViews/Moodfeeds/Moodstickers/sticker-linear-bottom-tiny.png HTTP/1.1" 200 2070 "http://ilps.science.uva.nl/MoodViews/index.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"

where i want to remove the last 4 lines, because they start with the exact same IP-adress and date&time as the first line.

Last edited by Franklin52; 09-22-2009 at 09:14 AM.. Reason: Please use code tags!
 

10 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Sorry for duplicating posts

SORRY FOR DUPLICATING POSTS, COULD U JUST REMOVE THE FIRST ONE. Thank u. (1 Reply)
Discussion started by: vrguha
1 Replies

2. Red Hat

Duplicating ethernet speed

Hi guys, Suppose you have a server with two ethernet cards (1GB each) and each cards are connecting to two different switches cisco 3750. My question is: How can I setup my server's network interfaces to increase the throughput up to 2GB? is it possible? If not, do you know another way to up... (3 Replies)
Discussion started by: iga3725
3 Replies

3. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

4. Shell Programming and Scripting

mv duplicating directories

Hi Folks, I've put together a script for sorting my backup files into sub folders to be run from a cron job. Each file is named username.tar.gz and the file /etc/trueuserowners contains all users and their owner in the format "user: owner". The script works fine identifying users and their owners... (10 Replies)
Discussion started by: beddo
10 Replies

5. UNIX for Dummies Questions & Answers

duplicating a line

I have a text file which is the results of running a tests hundreds of times. For simplicity let's say that each test consists of 5 lines of text numbered 1-5 e.g. 1 aaa aaa aaa 2 bbb bbb bbb 3 ccc ccc ccc 4 ddd ddd ddd 5 eee eee eee 1 aaa aaa aaa 2 bbb bbb bbb 3 ccc... (4 Replies)
Discussion started by: millsy5
4 Replies

6. Shell Programming and Scripting

Duplicating and changing sh file

Hello, I have a file named file_1.sh that I want to duplicate into file_2.sh, file_3.sh,..., etc. I also need to change the text within each file so that it would fit the file name. For example, in file_1.sh there is a command to save some output as 'output_1.txt', and also there is an input... (3 Replies)
Discussion started by: haguyw
3 Replies

7. UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

Hi, I have a huge file which has Lacs of lines. File system got full. I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies

8. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

9. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

10. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies
FIREFOX(1)							Linux User's Manual							FIREFOX(1)

NAME
firefox - a Web browser for X11 derived from the Mozilla browser SYNOPSIS
firefox [OPTIONS] [URL] DESCRIPTION
Mozilla Firefox is an open-source web browser, designed for standards compliance, performance and portability. USAGE
firefox is a simple shell script that will set up the environment for the actual executable. OPTIONS
A summary of the options supported by firefox is included below. GTK options --gdk-debug=FLAGS Gdk debugging flags to set --gdk-no-debug=FLAGS Gdk debugging flags to unset --gtk-debug=FLAGS Gtk+ debugging flags to set --gtk-no-debug=FLAGS Gtk+ debugging flags to unset --gtk-module=MODULE Load an additional Gtk module X11 options --display=DISPLAY X display to use --sync Make X calls synchronous --no-xshm Don't use X shared memory extension --xim-preedit=STYLE --xim-status=STYLE --g-fatal-warnings Make all warnings fatal Firefox options -h, -help Show summary of options. -height value Set height of startup window to value. -width value Set width of startup window to value. -v, -version Print /usr/bin/firefox-bin version. -CreateProfile profile Create profile. -P profile Start with profile. -ProfileManager Start with profile manager. -UILocale lang-region Start with lang-region resources. -contentLocale lang-region Start with lang-region resources. -remote command Execute command in an already running Firefox process. For more info, see: http://www.mozilla.org/unix/remote.html -no-remote Don't try to control an already running instance. -jsconsole Start with Javascript Console -chrome url Load the specified chrome. -new-window url Open URL in a new window if Firefox is already running. -new-tab url Open URL in a new tab if Firefox is already running. -preferences Open Firefox preferences menu. -safe-mode Disables extensions and themes for this session. FILES /usr/bin/firefox - shell script wrapping firefox /usr/lib/firefox/firefox - firefox executable VERSION
3.0 BUGS
To report a bug, please visit http://bugzilla.mozilla.org/ SEE ALSO
mozilla(1) AUTHORS
The Mozilla Organization http://www.mozilla.org/about.html firefox January 04, 2006 FIREFOX(1)
All times are GMT -4. The time now is 10:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy