how to add duplicate lines


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers how to add duplicate lines
# 8  
Old 08-20-2010
and what i meant by horrify was about my horrible newbie script!!!
# 9  
Old 08-20-2010
Quote:
Originally Posted by mikey11415
and what i meant by horrify was about my horrible newbie script!!!
What is important is that you are trying, and learning. We all had to start somewhere and every once in a while I come across some of my old code and wonder why I ever did it that way!!
This User Gave Thanks to agama For This Post:
# 10  
Old 08-20-2010
Thanks!
I just can't believe how much you are helping me solve a major issue!!!
I have another couple of questions.
I needed to rearrange some of the columns in the datafile.
I was able to do this using awk (i am proud of myself!)
But, there are a coiple of places where I am stumped
so for example,
if I have two files

fileA
12 test

and fileB
44 junk

and I want a line in my output that is

12 44

how do I go about that with awk?

I know that i want to do something like this

awk '{print $1}' fileA > outputA
awk '{print $1}' fileB > outputB

but then how do i get the outputA and outputB onto the same line?


you have been amazingly helpful thus far...i hope that you don't mind another couple of questions!

mikey

Last edited by mikey11415; 08-20-2010 at 11:33 PM.. Reason: i figured out the second part
# 11  
Old 08-21-2010
Hello. I hope you don't mind another suggestion. Try to use getline instead:
Code:
awk '
    BEGIN {
        while (getline < ARGV[1]) {
            a = $1

            if (!(getline < ARGV[2])) {
                break
            }

            b = $1

            print a " " b
        }

        exit(0)
    }
' fileA fileB

These 2 Users Gave Thanks to konsolebox For This Post:
# 12  
Old 08-21-2010
well i did this

cat ZZlinecount ZZcharcount > ZZline_char #make one file from two
awk '{printf $1 " "}' ZZline_char > ZZline_char1 #get the first column


and that WORKED

but when I cat this output with anther output, it puts them on the same line (I think because ZZline_char1 does not have an end of line)
I would like ZZline_char1 to have an end of line so that the think that i cat after this is on the next line

i am so close!

thanks for the help!!!

mikey
# 13  
Old 08-21-2010
Quote:
Originally Posted by mikey11415
but then how do i get the outputA and outputB onto the same line?
There are a couple of ways that you could go about this. What makes the difference is what is in the two files. A single line in each, or wanting every line processed, is pretty straight forward. If there are some lines that aren't needed it gets a bit more complicated.

Here's an example that assumes you want the first field of all lines mashed together and written to stdout. You should be able to add some extra pattern matching if you don't need all of the lines.

Code:
( sed 's/^/file1 /' file1; cat file2 ) | awk '
        /^file1/ {
                save[i++] = $2;         # we added a field, so it is $2
                next;
        }

        {
                printf( "%s %s\n", save[j++], $1 );
                if( j >= i )
                        exit( 0 );              # bail if file2 has more lines
        }
'

I think you can figure it out, but I will point out that the parenthesis round the sed and cat commands are very important. Kshell executes the commands placed the parens in a subprocess and all output from that process is piped into the awk.

If the files are largish, then there are better ways of doing this -- stuffing everything from the first file into an array isn't the best form, but is easier to understand and for a few hundred lines it is better to keep it simple.

There are other posts round this forum that do this kind of thing by putting both filenames on the awk command line, and then test FILENAME within the awk programme to determine what to do. Nothing wrong with that, but I prefer this method as it lets you dynamically supply the filenames without having to hard code them in the awk programme (they could be passed into the script and instead of file1/file2 referenced as $1/$2 or somesuch. A small amount of extra overhead in the sed processes, but a big win in terms of flexibility.

Happy to help!

---------- Post updated at 23:26 ---------- Previous update was at 23:17 ----------

Quote:
Originally Posted by mikey11415
well i did this

cat ZZlinecount ZZcharcount > ZZline_char #make one file from two
awk '{printf $1 " "}' ZZline_char > ZZline_char1 #get the first column


and that WORKED

but when I cat this output with anther output, it puts them on the same line (I think because ZZline_char1 does not have an end of line)
I would like ZZline_char1 to have an end of line so that the think that i cat after this is on the next line

i am so close!

thanks for the help!!!

mikey
Puts them on the same line because printf() is different than print. The print command automatically prints a newline while printf() does not.

A small change will solve this:
Code:
awk '{printf( "%s\n", $1 ) }' ZZline_char > ZZline_char1

And a bit of wisdom passed down by one of the original authors of awk was to always use parens with printf().

@konsolebox -- using getline() was what I referred to as the better way, but not as straight forward. You beat me to the punch Smilie
This User Gave Thanks to agama For This Post:
# 14  
Old 08-21-2010
hm
when I use
awk '{printf( "%s\n", $1 ) }' ZZline_char > ZZline_char1

it prints each number onto a separate line. the thing is that i want the first column from two rows from ZZline_char on the same line.
Then, on the next line, I want to cat another file

So ZZline_char looks like this
11 test
22 junk

i want a file that looks like this
11 22

then i have a second file that only has this
100

i want the final output to be
11 22
100

what i was getting was
11 22 100

now i am getting
11
22
100

you are all AMAZINGLY helpful!!!

mikey

---------- Post updated at 11:58 PM ---------- Previous update was at 11:36 PM ----------

OMG I did it!


awk '{printf $1 " "}' ZZline_char > ZZline_char1
echo " " > ZZnewline

then

cat ZZline_char1 ZZnewline ZZthird > dataset

that ZZnewline just put a line in there for me!!!

THANK YOU ALL SO MUCH!!!!

Now I can sleep!!!

All the best

Mikey

---------- Post updated 08-21-10 at 12:16 AM ---------- Previous update was 08-20-10 at 11:58 PM ----------

thanks so much everyone, that solved everything!!!

best

mikey
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

2. Shell Programming and Scripting

Duplicate lines

Dear All, I have a two-column data file and want to duplicate data in second column w.r.t. first column. My file looks like: 2 5.672 1 3.593 3 8.260 ... And the desired format: 5.672 5.672 3.593 8.260 8.260 8.260 ... How may I do so please? I appreciate any help you may... (2 Replies)
Discussion started by: sxiong
2 Replies

3. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

4. Shell Programming and Scripting

Script to duplicate lines

Hello, I'm trying to write an script that in a txt with lines with 2 or more columns separated by commas, like hello, one, two bye, goal first, second, third, fourth hard, difficult.strong, word.line will create another in which if a line has more than 2 columns, it will have another... (4 Replies)
Discussion started by: clinisbud
4 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

Print duplicate lines

I have a file where some of the lines are duplicates. How do I use bash to print all the lines that have duplicates? (2 Replies)
Discussion started by: locoroco
2 Replies

7. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

8. UNIX for Dummies Questions & Answers

Duplicate columns and lines

Hi all, I have a tab-delimited file and want to remove identical lines, i.e. all of line 1,2,4 because the columns are the same as the columns in other lines. Any input is appreciated. abc gi4597 9997 cgcgtgcg $%^&*()()* abc gi4597 9997 cgcgtgcg $%^&*()()* ttt ... (1 Reply)
Discussion started by: dr_sabz
1 Replies

9. Shell Programming and Scripting

Duplicate Lines x 4

Hi Guys and Girls I'm having trouble outputing from a sorted file... i have a looooong list of PVIDs and need to only output only those which occur 4 times!! Any suggestions? ie I need to uniq (but not uniq (i've been through the man pg) this: cat /tmp/disk.out|awk '{print $3}' |grep -v... (6 Replies)
Discussion started by: serm
6 Replies

10. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies
Login or Register to Ask a Question