Sponsored Content
Top Forums Shell Programming and Scripting Remove the duplicate content in a file Post 302843251 by ashokvpp on Monday 12th of August 2013 12:51:05 PM
Old 08-12-2013
The occurrences are just random number of times.. all I need is just print the 1st occurrence of the start pattern match..

---------- Post updated at 11:51 AM ---------- Previous update was at 11:42 AM ----------

Code:
awk '!A[$0]++' RS="\n\n\n"  test.txt   # works great .

Have a good day Smilie

Last edited by Scott; 08-12-2013 at 01:52 PM.. Reason: Code tags
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

4. Shell Programming and Scripting

Formatting a file - Remove Duplicate

Hi I have a file in the following format. Basically the file contains tablename and their aliases: TABLE1 TABLE1 A TABLE2 TABLE2 B TABLE3 TABLE4 TABLE4 C TABLE4 Upon formatting an sql statement I am getting such output. Problem: Whenever a tablename appears with alias, it has... (5 Replies)
Discussion started by: freakygs
5 Replies

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

6. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA (2 Replies)
Discussion started by: perl_beginner
2 Replies

7. Shell Programming and Scripting

How to Remove duplicate value from file?

if different branch code is available for same BIC code and one of the branch code is XXX.only one row will be stored and with branch code as XXX .rest of the rows for the BIC code will not be stored. for example if $7 is BIC code and $8 is branch code INPUT file are following... (9 Replies)
Discussion started by: mohan sharma
9 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies
PHOTON(1)																 PHOTON(1)

NAME
photon - generates photo albums SYNOPSIS
photon [OPTIONS] [PATH] DESCRIPTION
Photon generates clean, yet configurable, static HTML photo galleries. All of the work is done from the command line, and the resulting output can easily be configured by changing the template files found in /usr/share/photon/templates. To create a new template (skin), first create a new directory: mkdir -p ${HOME}/.photon/templates/my_theme/ Then copy the contents of /usr/share/photon/templates/photonv1/ to this new directory. Photon only needs three files to build pages: index.html used to display the content of a folder image.html used to display the page for an image movie.html used to display the page for a movie If a file is missing, then photon will throw an error and will not run. HOW TO BUILD A GALLERY
Quick start $ photon -o myoutputdir /photos Use the gimp plugin $ photon --resize-plugin=gimp -v -o myoutputdir /photos Change the size for all images and the thumbnail and keep the original photo $ photon --sizelist=0,1600x1200,800x600 --thumbsize=320x240 -o myoutputdir /photos Create a gallery from your custom template (skin) $ photon -k my_theme -o myoutputdir /photos See photon help $ photon --help OPTIONS
-a | --awstats Insert Javascript for Awstats --awstats-url=URL Change default value for Awstats script url -c NAME | --comment Name of the comment file (default .comments) -d NUM | --display-columns=NUM Number of columns in index (default 3) -E | --no-exif Don't include EXIF information in HTML file -f | --force Overwrite non-image generated files (html, javascript, etc) (default no) -ff | --force --force Overwrite image files (default no) -h | --help Print this help -I | --no-index Do not generate the high level index.html -J | --no-javascript Do not use javascript (no shortcuts, ... ) -k | --skin=THEME Skin selector (default photonv1) --print-skins Print out available skins (templates) -l NUM | --display-lines=NUM Number of lines in index (default 5) -m | --movie Movie mode, include link to movie (*.mov) -o NAME | --output-directory=NAME Name of the output directory -s LIST | --sizelist=LIST Image sizes (default 0,1024x768,800x600,640x480). Zero, which is the default, uses the original resolution. W x H specifies the width and height of the image in pixels. -t SIZE | --thumbsize=SIZE Size of thumbnails (default 160x120) -v | --verbose Verbosely print information -V | --version Print program version number -z | --compress Compress the original image if selected (default copy the image) --exif-bordercolor=COLOR Exif window border color (default #008000) --exif-bgcolor=COLOR Exif window background color (default #f0fff0) --exif-fgcolor=COLOR Exif window text color (default black) --body-bgcolor=COLOR Body background color (default #ccccff) --img-bgcolor=COLOR Image background color (default white) --exclude=PATTERN Exclude files matching PATTERN --resize-plugin=PLUG Where PLUG is the program used to create thumbnails internal: fastest method (default) gimp: use Gimp>1.x (better quality) --resize-quality-low=NUM Quality for small image. 0 (bad) and 100 (good) --resize-quality-high=NUM Quality for big image 0 (bad) and 100 (good) --gimp-program=PROG use PROG for gimp SHORTCUTS WHILE VIEWING AN IMAGE
n/SPACE Go to the next image (with the same resolution) p/BACKSPACE Go to the previous image (with the same resolution) s Start/Stop the slideshow +/- Increase/Decrease by one second the slideshow's period h Show shortcut and help i Show Exif information z Change to the higher resolution [NOT YET IMPLEMENTED] SEE ALSO
gimp(1) AUTHOR
Written by Luc Saillard, <luc@saillard.org> Main web site: http://www.saillard.org/programs_and_patches/photon/ This manual page was written by Kevin Coyner <kcoyner@debian.org> for the Debian project but may be used by others. 12/16/2006 PHOTON(1)
All times are GMT -4. The time now is 06:55 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy