Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to remove duplicated lines?
Top Forums Shell Programming and Scripting How to remove duplicated lines? Post 302825411 by rbatte1 on Monday 24th of June 2013 10:55:20 AM
Old 06-24-2013
I think we've missed the point with the file note being wrapped in CODE tags. If this is one file, then I'm assuming that you want the count for each block. The output requested has the values for b and c in each section.

Not the prettiest solution, but this might work:-
Code:
#!/bin/ksh

mkdir /tmp/$$
while read line
do
   if [ "$line" != "${line#Query=}" ]
   then
      Section="${line#Query=}"
   else
      echo "$line" >> /tmp/$$/$Section
   fi
done < file_name

for file in /tmp/$$/*
do
   echo "Query=$file"
   uniq -c /tmp/$$/$file
done


Does that get you any closer? The output is the wrong way round for the counts, but that could be handled thus:-
Code:
....

for file in /tmp/$$/*
do
   echo "Query=$file"
   uniq -c /tmp/$$/$file | while read col1 col2
   do
      echo "$col2 $col1"
   done
done


I hope that this helps.


Robin
Liverpool/Blackburn
UK

---------- Post updated at 03:55 PM ---------- Previous update was at 03:52 PM ----------

Oh, an update before I posted!

Okay, well addressing the post number 6, try something like:-
Code:
#!/bin/ksh

while read line
do
   grep "^$line" file2 | cut -f2- -d" "
done < file1


Does that do it?



Robin
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicated xml record in a file under unix

Hi, If i have a file with xml format, i would like to remove duplicated records and save to a new file. Is it possible...to write script to do it? (8 Replies)
Discussion started by: happyv
8 Replies

2. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

3. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

4. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

5. Shell Programming and Scripting

Remove rows with first 4 fields duplicated in awk

Hi, I am trying to use awk to remove all rows where the first 4 fields are duplicates. e.g. in the following data lines 6-9 would be removed, leaving one copy of the duplicated row (row 5) Borgarhraun FH9822 ol24 FH9822_ol24_m20 ol Deformed c Borgarhraun FH9822 ol24 ... (3 Replies)
Discussion started by: tomahawk
3 Replies

6. UNIX for Dummies Questions & Answers

Removing duplicated lines??

Hi Guys.. I have a problem for some reason my database has copied everything 4 times. My Database looks like this: >BAC233456 rhjieaheiohjteo tjtjrj6jkk6k6 j54ju54jh54jh >ANI124365 afrhtjykulilil htrjykuk rtkjryky ukrykyrk >BAC233456 rhjieaheiohjteo tjtjrj6jkk6k6 j54ju54jh54jh... (6 Replies)
Discussion started by: Iifa
6 Replies

7. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

8. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

9. AIX

Remove duplicated bootlist entries

Hello. I have a server with 2 boot disk but in the bootlist there are 5 paths of one disk but no path of the other. How can I remove paths from one disk to insert paths from the other disk? Thanks in advance. server074:root:/# bootlist -om normal hdisk0 blv=hd5 pathid=0 hdisk0... (7 Replies)
Discussion started by: Gabriander
7 Replies

10. Shell Programming and Scripting

Remove duplicated records and update last line record counts

Hi Gurus, I need to remove duplicate line in file and update TRAILER (last line) record count. the file is comma delimited, field 2 is key to identify duplicated record. I can use below command to remove duplicated. but don't know how to replace last line 2nd field to new count. awk -F","... (11 Replies)
Discussion started by: green_k
11 Replies

LEARN ABOUT DEBIAN

rdf::query::util

RDF::Query::Util(3pm)					User Contributed Perl Documentation				     RDF::Query::Util(3pm)

NAME

       RDF::Query::Util - Miscellaneous utility functions to support work with RDF::Query.

VERSION

       This document describes RDF::Query::Util version 2.908.

SYNOPSIS

	use RDF::Query::Util;
	my $query = &RDF::Query::Util::cli_make_query;
	my $model = &RDF::Query::Util::cli_make_model;
	$query->execute( $model );
	...

FUNCTIONS

       "cli_make_query_and_model"
	   Returns a query object, model, and args HASHref based on the arguments in @ARGV.  These arguments are parsed using "cli_make_query" and
	   "make_model".

       "cli_make_query"
	   Returns a RDF::Query object based on the arguments in @ARGV. These arguments are parsed using &cli_parse_args. If the -e flag is not
	   present, the query will be loaded from a file named by the argument in @ARGV immediately following the final argument parsed by
	   &cli_parse_args.

       "cli_make_model"
	   Calls "make_model" with arguments from @ARGV, returning the constructed model object.

	   "cli_make_model" will usually be called after cli_make_query, allowing a typical CLI invocation to look like `prog.pl [flags] [query
	   file] [data files]`.

       "make_model ( @files )"
	   Returns a model object suitable for use in a call to "$query->execute", loaded with RDF from files and/or URLs listed in @files. This
	   model may be any of the supported models, but as currently implemented will be a RDF::Trine::Model object.

       "cli_parse_args"
	   Parses CLI arguments from @ARGV and returns a HASH with the recognized key/values.  The allowable arguments are listed below.

       "start_endpoint ( $model, $port )"
	   Starts an SPARQL endpoint HTTP server on port $port.

	   If called in list context, returns the PID and the actual port the server bound to. If called in scalar context, returns only the port.

COMMAND LINE ARGUMENTS

       -e str
	   Specifies the query string str.

       -l lang
	   Specifies the query language lang used. This should be one of: sparql, sparql11, or rdql.

       -O  Turns on optimization.

       -o  Turns off optimization.

       -c class
	   Specifies the perl class used to construct the query object. Defaults to "RDF::Query".

       -f  Implies -c RDF::Query::Federate.

       -F loc
	   Specifies the URL or path to a file loc which contains an RDF service description. The described service is used as an underlying
	   triplestore for query answering. Implies -f.

       -E url
	   Specifies the URL of a remove SPARQL endpoint to be used as a data source. The endpoint is used as an underlying triplestore for query
	   answering. Implies -f.

       -s database-type
	   Specifies the database type to use for the underlying data model.

       -u user
       -p password
       -m model

AUTHOR

	Gregory Todd Williams <gwilliams@cpan.org>

perl v5.14.2							    2012-01-31						     RDF::Query::Util(3pm)
All times are GMT -4. The time now is 05:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy