Sponsored Content
Top Forums Shell Programming and Scripting squeeze duplicates from a table Post 302423933 by Scott on Sunday 23rd of May 2010 01:49:28 PM
Old 05-23-2010
Code:
sort -t\| -nrsk2 file1 | awk -F\| '!A[$1]++'

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

TR squeeze oddity

I discovered that where 'tr -s' works as expected on grepped input, it appears to completely fail on dig results. I am not sure if this is because of some sort of non posix compliancy, or what. Here is what I did: The command below works as expected, squeezing all repeated spaces to a single... (3 Replies)
Discussion started by: bdmeyersc
3 Replies

2. Shell Programming and Scripting

Merge Two Tables with duplicates in first table

Hi.. File 1: 1 aa rep 1 dd rep 1 kk rep 2 bb sad 2 ss sad 3 ee dam File 2 1 apple fruit 2 mango tree 3 lilly flower output: 1 aaple fruit aa,dd,kk rep (7 Replies)
Discussion started by: empyrean
7 Replies

3. UNIX for Advanced & Expert Users

Load average in squeeze too low

sorry, not yelling, its just copied from a bug report and this prob is driving me crazy... ran stress --cpu 4 on a xeon and no problem the load went up to 4, but running mysql server for example the load is almost 0.00, where the same mysql server with same throughput gets 0.8 on another lenny... (15 Replies)
Discussion started by: suffeks
15 Replies

4. UNIX Desktop Questions & Answers

How to squeeze multiple pipe character '|' into single '|' using sed?

Hi, I am trying to convert multiple Unix pipe symbol or bar into single |. I have tried with the following sed statements, but, no success :(. I need it using sed only echo "sed 's/\|\+/\|/g' sed 's/*/\|/' sed 's/\|*/|/' sed -r 's/\|+/\|/' However, the below awk code is working fine.... (4 Replies)
Discussion started by: royalibrahim
4 Replies

5. UNIX for Dummies Questions & Answers

Creating a condensed table from a pre-existing table in putty

Hello, I'm working with putty on Windows 7 professional and I'd like to know if there's a way to gather specific lines from a pre-existing table and make a new table with that information. More specifically, I'd like the program to look at a specific column, say column N, and see if any of the... (5 Replies)
Discussion started by: Deedee393
5 Replies

6. Shell Programming and Scripting

Help me please: UNIX command to extract substring not squeeze spaces

Hi experts, Please help me!... I have a string " test1 test2 test3 ". There are two spaces before "test1"; There are four spaces between "test1" and "test2"; there are two spaces between "test2 and "test3". I want to extract a substring "2 test3" using positions. Below is my test... (5 Replies)
Discussion started by: sophiez16
5 Replies

7. Debian

VPN service fails after update applied in Debian Squeeze

Hello everybody, I used to log in to my office via PPTP VPN, but on last October 5th I updated my installed Debian Squeeze and it caused my VPN service (client-side) to fail. After this upgrade I'm unable to log in to the VPN server. Here follows the log: #tail -f /var/log/messages Plugin... (0 Replies)
Discussion started by: r4ym4r
0 Replies

8. Shell Programming and Scripting

Filtering duplicates based on lookup table and rules

please help solving the following. I have access to redhat linux cluster having 32gigs of ram. I have duplicate ids for variable names, in the file 1,2 are duplicates;3,4 and 5 are duplicates;6 and 7 are duplicates. My objective is to use only the first occurrence of these duplicates. Lookup... (4 Replies)
Discussion started by: ritakadm
4 Replies

9. Web Development

Getting Rid of Annoying Bootstrap Table Borders and Wayward Table Lines

Bootstrap is great; but we have had some issues with Bootstrapped <tables> (and legacy <fieldset> elements) showing annoying, wayward lines. I solved that problem today with this simple jQuery in the footer: <script> $(function(){ $('tr, td, fieldset,... (0 Replies)
Discussion started by: Neo
0 Replies

10. UNIX for Beginners Questions & Answers

Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns

Hello all, I need to filter a dataframe composed of several columns of data to remove the duplicates according to one of the columns. I did it with pandas. In the main time, I need that the last column that contains all different data ( not redundant) is conserved in the output like this: A ... (5 Replies)
Discussion started by: pedro88
5 Replies
funtablerowput(3)						SAORD Documentation						 funtablerowput(3)

NAME
FunTableRowPut - put Funtools rows SYNOPSIS
int FunTableRowPut(Fun fun, void *rows, int nev, int idx, char *plist) DESCRIPTION
The FunTableRowPut() routine writes rows to a FITS binary table, taking its input from an array of user structs that contain column values selected by a previous call to FunColumnSelect(). Selected column values are automatically converted from native data format to FITS data format as necessary. The first argument is the Fun handle associated with this row data. The second rows argument is the array of user structs to output. The third nrow argument specifies the number number of rows to write. The routine will write nrow records, starting from the location speci- fied by rows. The fourth idx argument is the index of the first raw input row to write, in the case where rows from the user buffer are being merged with their raw input row counterparts (see below). Note that this idx value is has nothing to do with the row buffer specified in argument 1. It merely matches the row being written with its corresponding (hidden) raw row. Thus, if you read a number of rows, process them, and then write them out all at once starting from the first user row, the value of idx should be 0: Ev ebuf, ev; /* get rows -- let routine allocate the row array */ while( (ebuf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){ /* process all rows */ for(i=0; i<got; i++){ /* point to the i'th row */ ev = ebuf+i; ... } /* write out this batch of rows, starting with the first */ FunTableRowPut(fun2, (char *)ebuf, got, 0, NULL); /* free row data */ if( ebuf ) free(ebuf); } On the other hand, if you write out the rows one at a time (possibly skipping rows), then, when writing the i'th row from the input array of rows, set idx to the value of i: Ev ebuf, ev; /* get rows -- let routine allocate the row array */ while( (ebuf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){ /* process all rows */ for(i=0; i<got; i++){ /* point to the i'th row */ ev = ebuf+i; ... /* write out the current (i.e., i'th) row */ FunTableRowPut(fun2, (char *)ev, 1, i, NULL); } /* free row data */ if( ebuf ) free(ebuf); } The final argument is a param list string that is not currently used. The routine returns the number of rows output. This should be equal to the value passed in the third nrow</B argument. When FunTableRowPut() is first called for a given binary table, Funtools checks to see of the primary header has already been written (either by writing a previous row table or by writing an image.) If not, a dummy primary header is written to the file specifying that an extension should be expected. After this, a binary table header is automatically written containing information about the columns that will populate this table. In addition, if a Funtools reference handle was specified when this table was opened, the parameters from this Funtools reference handle are merged into the new binary table header. In a typical Funtools row loop, you read rows using FunTableRowGet()() and write rows using FunTableRowPut(). The columns written by FunT- ableRowPut()() are those defined as writable by a previous call to FunColumnSelect(). If that call to FunColumnSelect also specified merge=[update|replace|append], then the entire corresponding raw input row record will be merged with the output row according to the merge specification (see FunColumnSelect() above). A call to write rows can either be done once, after all rows in the input batch have been processed, or it can be done (slightly less effi- ciently) one row at a time (or anything in between). We do recommend that you write all rows associated with a given batch of input rows before reading new rows. This is required if you are merging the output rows with the raw input rows (since the raw rows are destroyed with each successive call to get new rows). For example: Ev buf, ev; ... /* get rows -- let routine allocate the row array */ while( (buf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){ /* point to the i'th row */ ev = buf + i; .... process } /* write out this batch of rows */ FunTableRowPut(fun2, buf, got, 0, NULL); /* free row data */ if( buf ) free(buf); } or Ev buf, ev; ... /* get rows -- let routine allocate the row array */ while( (buf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){ /* process all rows */ for(i=0; i<got; i++){ /* point to the i'th row */ ev = buf + i; ... process /* write out this batch of rows with the new column */ if( dowrite ) FunTableRowPut(fun2, buf, 1, i, NULL); } /* free row data */ if( buf ) free(buf); } Note that the difference between these calls is that the first one outputs got rows all at once and therefore passes idx=0 in argument four, so that merging starts at the first raw input row. In the second case, a check it made on each row to see if it needs to be output. If so, the value of idx is passed as the value of the i variable which points to the current row being processed in the batch of input rows. As shown above, successive calls to FunTableRowPut() will write rows sequentially. When you are finished writing all rows in a table, you should call FunFlush() to write out the FITS binary table padding. However, this is not necessary if you subsequently call FunClose() with- out doing any other I/O to the FITS file. Note that FunTableRowPut() also can be called as FunEventsPut(), for backward compatibility. SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funtablerowput(3)
All times are GMT -4. The time now is 06:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy