Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Sort and remove duplicates in directory based on first 5 columns: Post 303011783 by rdrtx1 on Tuesday 23rd of January 2018 02:35:27 PM
Old 01-23-2018
Code:
for file in *.marker
do
   base_name="${file//_[0-9][0-9]*_[0-9][0-9]*[.]*/}"
   [[ "$last_base_name" = "$base_name" ]] || echo "$file"
   last_base_name="$base_name"
done


Last edited by rdrtx1; 01-23-2018 at 05:53 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

2. Shell Programming and Scripting

need Shell script for Sort BASED ON FIRST FIELD and PRINT THE WHOLE FILE WITHOUT DUPLICATES

Can some one provide me a shell script. I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique. Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies

3. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

4. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

5. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

6. Shell Programming and Scripting

Sort data by date first and then remove duplicates

Hi , I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points : (1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column) (2)after sorting the file based on latest date... (3 Replies)
Discussion started by: samrat dutta
3 Replies

7. Shell Programming and Scripting

Bash - remove duplicates without sort

I need to use bash to remove duplicates without using sort first. I can not use: cat file | sort | uniq But when I use only cat file | uniq some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies

8. Shell Programming and Scripting

Removing duplicates from delimited file based on 2 columns

Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker... Column #1 is a simple ID, which is used to identify the duplicate. Once dups are identified, I need to only keep the one... (2 Replies)
Discussion started by: kevinprood
2 Replies

9. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

10. Shell Programming and Scripting

Concatenate and sort to remove duplicates

Following is the input. 1st and 3rd block are same(block starts here with '*' and ends before blank line) , 2nd and 4th blocks are also the same: cat <file> * Wed Feb 24 2016 Tariq Saeed <tariq.x.saeed@mail.com> 2.0.7-1.0.7 - add vmcore dump support for ocfs2 * Mon Jun 8 2015 Brian Maly... (4 Replies)
Discussion started by: Paras Pandey
4 Replies
pfsouthdrhtml(1)					      General Commands Manual						  pfsouthdrhtml(1)

NAME
pfsouthdrhtml - Create a web page with an HDR viewer SYNOPSIS
pfsouthdrhtml [<page_name>] [--quality <1-5>] [--image-dir <directory_name>] [--page-template <template_file>] [--image-template <tem- plate_file>] [--object-output <file_name.js>] [--html-output <file_name.html>] DESCRIPTION
The command creates in the current directory an HTML web page containing multi-exposure HDR viewer. The multi-exposure viewer displays a portion of the available dynamic range with minimum contrast distortions and provides a slider control to move the dynamic range window towards brighter or darker tones. The interface is very similar to pfsview, which is a pfstools application for displaying HDR images. The web page employs only JavaScript and CSS opacity property and does not require Java applets or the Flash plugin. Note that because this techniques encodes 20-60 exposures using only few images, the displayed exposures may not be identical to the exposures that are shown in pfsview. For examples and more information, visit http://pfstools.sourceforge.net/hdrhtml/. <page_name> specifies the file name, of the web page to be generated. If <page_name> is missing, the file name of the first image with .html extension will be used. The command can take as input several images and put them all on the same web page. For each image, its file name (from the FILE_NAME tag in the pfsstrem) without extension and a leading path will be used as a name for all JavaScript variables corresponding to that image. If the filename contains illegal characters (such as space, '-', '[', etc), these will be converted to '_'. --quality <1-5>, -q <1-5> Quality of the interpolated exposures, from the worst (1) to the best (5). The default is 2, which is sufficient for most applica- tions. Higher quality will introduce less distortions in the brightest and the darkest tones, but will also generate more images. More images means that there is more data that needs to be transferred to the web-browser, making HDR viewer less responsive. --image-dir <directory_name>, -d <directory_name> Specify where to store the resulting image files. Links to images in HTML will be updated accordingly. This must be a relative path and the directory must exist. Useful to avoid clutter in the current directory. --page-template <template_file>, -p <directory_name>, --image-template <template_file>, -i <template_file> Replaces the template files used to generate an HTML web page. The template files contain all HTML and JaveScript code with special keywords (@keyword@) that are replaced with image specific data, such as width, height, image base name, etc. The default template files can be found in INSTALL_DIR/share/pfstools/hdrhtml_*_templ.html. See TEMPLATE FILE FORMAT below more details. --object-output <file_name.js>, -o <file_name.js> Store JavaScript objects (hdr_<base_name>) associated with each image in a separate file. This is useful if you want to script cre- ating HTML pages. --html-output <file_name.html>, -l <file_name.html> Store HTML code that shows HDRHTML viewer for each image in a separate file. This is useful if you want to script creating HTML pages. TEMPLATE FILE FORMAT
pfsouthdrhtml uses two template files hdrhtml_page_templ.html and hdrhtml_image_templ.html, located in INSTALL_DIR/share/pfstools/, to gen- erate a web page with an HDR HTML viewer. The 'page' file contains the HTML of the entire web page and the 'image' file is used to paste a viewer code for a single image. You can replace one or both these templates with your own using --page-template and --image-template options. Each template contains HTML code with additional keywords surrounded by @ marks (@keyword@), which are replaced with HDR HTML specific code. Most of the keywords are self explanatory, therefore only the most important are described below. @hdr_img_def@ JavaScript objects that must be put in the 'body' section before any images. These define all the parameters needed to control HDR HTML viewer. @cf_array_def@ Pre-computed array of opacity coefficients. The same array is used for all images that use the same quality setting. Currently only one such array could be used per web-page, so images generated with different quality setting cannot be mixed on a single web page. @image_htmlcode@ or @image_htmlcode[base_name]@ Inserts HTML code of all images or a single image with the base_name (name with no file extension) specified as a parameter. This should be put where HDR HTML viewer should be located. EXAMPLES
pfsin memorial.hdr | pfshdrhtml memorial_church Generates a web page memorial_church.html with a set of images memorial_church_*.jpg in the current directory. pfsin ~/hdr_images/*.exr | pfssize --maxx 512 --maxy 512 | pfsouthdrhtml hdr_images Generate a web page with all OpenEXR images from ~/hdr_images/. The images are resized so that they are not larger than 512x512. SEE ALSO
pfsin(1) pfsout(1) BUGS
Please report bugs and comments to the discussion group http://groups.google.com/group/pfstools pfsouthdrhtml(1)
All times are GMT -4. The time now is 06:30 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy