Sponsored Content
Full Discussion: Awk: Remove Duplicates
Top Forums Shell Programming and Scripting Awk: Remove Duplicates Post 302884999 by Don Cragun on Thursday 23rd of January 2014 02:30:33 AM
Old 01-23-2014
Try:
Code:
awk -F, 'dupentries[$1,$2,$3,$4,$5,$6,$7,$8]++ {print > "Duplicates"; next};print' inputfile > inputfile.tmp
mv inputfile.tmp inputfile

This User Gave Thanks to Don Cragun For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

4. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

7. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

awk - Remove duplicates during array build

Greetings Experts, Issue: Within awk script, remove the duplicate occurrences that are space (1 single space character) separated Description: I am processing 2 files using awk and during processing, I am building an array and there are duplicates on this; how can I delete the duplicates... (3 Replies)
Discussion started by: chill3chee
3 Replies
gdcm2vtk(1)							       GDCM							       gdcm2vtk(1)

NAME
gdcm2vtk - Convert a file supported by VTK into DICOM. SYNOPSIS
gdcm2vtk [options] file-in file-out DESCRIPTION
The gdcm2vtk takes as input any file supported by VTK (including DICOM file) and will generate as output a DICOM file. PARAMETERS
file-in input filename (DICOM or VTK supported) file-out DICOM output filename options options --force-rescale force rescale. --force-spacing force spacing. --palette-color when supported generate a PALETTE COLOR file. --argb when supported generate a ARGB file. --compress when supported generate a compressed file. --use-vtkdicom Use vtkDICOMImageReader (instead of GDCM). --modality set Modality. --lower-left set lower left. --shift set shift. --scale set scale. --compress set compression (MetaIO). -T --study-uid Study UID. -S --series-uid Series UID. --root-uid Root UID. compression options -J --jpeg Compress image in jpeg. -K --j2k Compress image in j2k. -L --jpegls Compress image in jpeg-ls. -R --rle Compress image in rle (lossless only). general options -h --help print this help text and exit -v --version print version information and exit -V --verbose verbose mode (warning+error). -W --warning warning mode, print warning information -E --error error mode, print error information -D --debug debug mode, print debug information environment variable GDCM_ROOT_UID Root UID DESCRIPTION
Convert a file supported by VTK into DICOM. Typical usage is: $ gdcm2vtk inputfile output.dcm It uses the internal factory mechanism of VTK to recognize a file (CanRead function). See VTK supported file here: What image file formats can VTK read and write? http://www.vtk.org/Wiki/VTK_FAQ#What_image_file_formats_can_VTK_read_and_write.3F If your input file has 4 components, the 4th comp (alpha) will be removed from the output file as DICOM does not support alpha component anymore (see --argb option). Special care was taken for the following file format: 1. DICOM: Direction Cosines and vtkMedicalImageInformation are passed to the output 2. BMP: The file can be saved with a Lookup Table (see --palette-color) 3. GE Signa: vtkMedicalImageProperties is passed to the output 4. MINC: Direction Cosines is passed to the output 5. TIFF: vtkTIFFReader is currently in bad shape in VTK (different behavior in VTK 5.2 and CVS). Only use it, if you know what you are doing. CONVERT MetaImage (mhd, mha) $ gdcm2vtk inputfile output.mha This command will convert the input DICOM file: inputfile into a MetaImage .mha file. Same goes for .mhd file. CONVERT MHA/MHD $ gdcm2vtk inputfile output.mha or $ gdcm2vtk inputfile output.mhd This command will convert the input DICOM file: inputfile into a MetaImageData .mha/.mhd file. CONVERT VTI $ gdcm2vtk inputfile output.vti This command will convert the input DICOM file: inputfile into a XML VTK ImageData .vti file. CONVERT VTK $ gdcm2vtk inputfile output.vtk This command will convert the input DICOM file: inputfile into an old VTK Structured PointSets .vtk file. CONVERT DICOM
$ gdcm2vtk input.dcm output.dcm vtkGDCMImageReader will be used to read in a DICOM file, not the default vtkDICOMImageReader. See option --use-vtkdicom to use vtkDICOMImageReader. RoundTrip DICOM to MHD to DICOM $ gdcm2vtk input_ybr.dcm output.mhd $ gdcm2vtk --modality US --imageformat 7 output.mhd output.dcm The above section shows how to convert a DICOM using the Photometric Interpretation of YBR_FULL (or even YBR_FULL_422 is lossy) into another file format: MetaImage (mhd). Since this file format does not handle color space, we have to explicitely set it using the --imageformat command line option. The --modality command line option is required in this case since the default Secondary Capture Image Storage Class family does not allow for YBR Photometric Interpretation. gdcm2vtk notes IMPORTANT NOTE: The internal VTK structured will be filled from the input DICOM, and then pass to the output DICOM writer. Some information might be lost during the conversion DICOM to VTK to DICOM. This option is mostly used to test the vtkGDCMImageReader/vtkGDCMImageWriter combination. IMPORTANT NOTE: When converting from a lossy format such as JPEG, the information of lossiness is important. The output DICOM will contains the required Lossy Image Compression attribute that indicates that image was lossy-compressed somewhere along the pipeline. See also gdcmimg (better handling of JPEG in general). IMPORTANT NOTE: When using --use-vtkdicom the output DICOM file will always be written as MR Image Storage as this information is not available from the reader itself. This allow setting the Image Orientation (Patient) properly. SEE ALSO
gdcmdump(1), gdcmviewer(1), gdcmimg(1) COPYRIGHT
Copyright (c) 2006-2011 Mathieu Malaterre Version 2.2.0 Tue Feb 5 2013 gdcm2vtk(1)
All times are GMT -4. The time now is 10:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy