Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Extract lines that have dupliucate and count them Post 302986986 by is2_egypt on Saturday 3rd of December 2016 06:24:22 AM
Old 12-03-2016
Extract lines that have dupliucate and count them

Dear friends

i have big file and i want to export the filw with new column for the lines that have same duplicate value in first column : ex : , ex :


Code:
-bash-3.00$ cat INTCONT-IS.CSV
M205-00-106_AMDRN:1-0-6-22,12-662-4833,intContact,2016-11-15 02:32:16,50
M205-00-106_AMDRN:1-0-23-17,12-616-0462,intContact,2016-11-15 02:32:23,50
M205-00-106_AMDRN:1-0-6-22,12-621-0646,intContact,2016-11-15 01:19:01,50
M213-00-312_BJWRM:1-0-8-12,12-621-3479,intContact,2016-11-15 01:19:17,50
M213-00-312_BJWRM:1-0-8-29,12-216-5205,intContact,2016-11-15 01:19:30,50
M213-00-312_BJWRM:1-0-12-28,12-621-7122,intContact,2016-11-15 01:19:44,50
M205-00-106_AMDRN:1-0-6-22,\N,intContact,2016-11-15 01:19:55,50
M205-00-106_AMDRN:1-0-6-22,12-574-4566,intContact,2016-11-15 07:46:00,50
V_TARTEABH_TARU013-A:1-1-1-32,13-823-5712,intContact,2016-11-15 22:46:22,50


ideal output shall export the same original file with new column fo the repetition for the first column in the original file , ex :


Code:
-bash-3.00$ cat INTCONT-IS.CSV
M205-00-106_AMDRN:1-0-6-22,12-662-4833,intContact,2016-11-15 02:32:16,50,4
M205-00-106_AMDRN:1-0-23-17,12-616-0462,intContact,2016-11-15 02:32:23,50,1
M205-00-106_AMDRN:1-0-6-22,12-621-0646,intContact,2016-11-15 01:19:01,50,4
M213-00-312_BJWRM:1-0-8-12,12-621-3479,intContact,2016-11-15 01:19:17,50,1
M213-00-312_BJWRM:1-0-8-29,12-216-5205,intContact,2016-11-15 01:19:30,50,1
M213-00-312_BJWRM:1-0-12-28,12-621-7122,intContact,2016-11-15 01:19:44,50,1
M205-00-106_AMDRN:1-0-6-22,\N,intContact,2016-11-15 01:19:55,50,4
M205-00-106_AMDRN:1-0-6-22,12-574-4566,intContact,2016-11-15 07:46:00,50,4
V_TARTEABH_TARU013-A:1-1-1-32,13-823-5712,intContact,2016-11-15 22:46:22,50,1


another question , what will be the command if i make this based on 3rd column niot first column ?

Thanks alot


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 12-03-2016 at 07:46 AM.. Reason: Added CODE tags.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to count lines - ignoring blank lines and commented lines

What is the command to count lines in a files, but ignore blank lines and commented lines? I have a file with 4 sections in it, and I want each section to be counted, not including the blank lines and comments... and then totalled at the end. Here is an example of what I would like my... (6 Replies)
Discussion started by: kthatch
6 Replies

2. Shell Programming and Scripting

How to extract specific data and count number containing sets from a file?

Hello everybody! I am quit new here and hope you can help me. Using an awk script I am trying to extract data from several files. The structure of the input files is as follows: TimeStep parameter1 parameter2 parameter3 parameter4 e.g. 1 X Y Z L 1 D H Z I 1 H Y E W 2 D H G F 2 R... (2 Replies)
Discussion started by: Daniel8472
2 Replies

3. Shell Programming and Scripting

Extract string from multiple file based on line count number

Hi, I search all forum, but I can not find solutions of my problem :( I have multiple files (5000 files), inside there is this data : FILE 1: 1195.921 -898.995 0.750312E-02-0.497526E-02 0.195382E-05 0.609417E-05 -2021.287 1305.479-0.819754E-02 0.107572E-01 0.313018E-05 0.885066E-05 ... (15 Replies)
Discussion started by: guns
15 Replies

4. UNIX for Dummies Questions & Answers

Extract lines with specific words with addition 2 lines before and after

Dear all, Greetings. I would like to ask for your help to extract lines with specific words in addition 2 lines before and after these lines by using awk or sed. For example, the input file is: 1 ak1 abc1.0 1 ak2 abc1.0 1 ak3 abc1.0 1 ak4 abc1.0 1 ak5 abc1.1 1 ak6 abc1.1 1 ak7... (7 Replies)
Discussion started by: Amanda Low
7 Replies

5. Shell Programming and Scripting

Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s)

I have hundreds of files to process. In each file I need to look for a pattern then extract value(s) from next line and then search for value(s) selected from point (2) in the same file at a specific position. HEADER ELECTRON TRANSPORT 18-MAR-98 1A7V TITLE CYTOCHROME... (7 Replies)
Discussion started by: AshwaniSharma09
7 Replies

6. Shell Programming and Scripting

Extract and count number of Duplicate rows

Hi All, I need to extract duplicate rows from a file and write these bad records into another file. And need to have a count of these bad records. i have a command awk ' {s++} END { for(i in s) { if(s>1) { print i } } }' ${TMP_DUPE_RECS}>>${TMP_BAD_DATA_DUPE_RECS}... (5 Replies)
Discussion started by: Arun Mishra
5 Replies

7. Shell Programming and Scripting

ksh sed - Extract specific lines with mulitple occurance of interesting lines

Data file example I look for primary and * to isolate the interesting slot number. slot=`sed '/^primary$/,/\*/!d' filename | tail -1 | sed s'/*//' | awk '{print $1" "$2}'` Now I want to get the Touch line for only the associate slot number, in this case, because the asterisk... (2 Replies)
Discussion started by: popeye
2 Replies

8. Shell Programming and Scripting

Skip the delimiter with in double quotes and count the number of delimiters during data extract

Hi All, I'm stuck-up in finding a way to skip the delimiter which come within double quotes using awk or any other better option. can someone please help me out. Below are the details: Delimited: | Sample data: 742433154|"SYN|THESIS MED CHEM PTY.... (2 Replies)
Discussion started by: BrahmaNaiduA
2 Replies

9. Shell Programming and Scripting

Extract count of string in all files and display on date wise

Hi All, hope you all are doing well! I kindly ask you for shell scripting help, here is the description: I have huge number of files shown below on date wise, which contains different strings(numbers you can say) including 505001 and 602001. ... (14 Replies)
Discussion started by: VasuKukkapalli
14 Replies

10. Shell Programming and Scripting

Extract paragraphs and count them

Hi, I have a text with a number of paragraphs in them. My problem is I need to locate certain errors/warning and extract/count them. Problem is I do not know how many paras are there with that particular type of error/warning. I had thought that somehow if I could count the number of... (25 Replies)
Discussion started by: dsid
25 Replies
SIGNAL-SAFETY(7)					     Linux Programmer's Manual						  SIGNAL-SAFETY(7)

NAME
signal-safety - async-signal-safe functions DESCRIPTION
An async-signal-safe function is one that can be safely called from within a signal handler. Many functions are not async-signal-safe. In particular, nonreentrant functions are generally unsafe to call from a signal handler. The kinds of issues that render a function unsafe can be quickly understood when one considers the implementation of the stdio library, all of whose functions are not async-signal-safe. When performing buffered I/O on a file, the stdio functions must maintain a statically allocated data buffer along with associated counters and indexes (or pointers) that record the amount of data and the current position in the buffer. Suppose that the main program is in the middle of a call to a stdio function such as printf(3) where the buffer and associated variables have been partially updated. If, at that moment, the program is interrupted by a signal handler that also calls printf(3), then the second call to printf(3) will operate on incon- sistent data, with unpredictable results. To avoid problems with unsafe functions, there are two possible choices: 1. Ensure that (a) the signal handler calls only async-signal-safe functions, and (b) the signal handler itself is reentrant with respect to global variables in the main program. 2. Block signal delivery in the main program when calling functions that are unsafe or operating on global data that is also accessed by the signal handler. Generally, the second choice is difficult in programs of any complexity, so the first choice is taken. POSIX.1 specifies a set of functions that an implementation must make async-signal-safe. (An implementation may provide safe implementa- tions of additional functions, but this is not required by the standard and other implementations may not provide the same guarantees.) In general, a function is async-signal-safe either because it is reentrant or because it is atomic with respect to signals (i.e., its execu- tion can't be interrupted by a signal handler). The set of functions required to be async-signal-safe by POSIX.1 is shown in the following table. The functions not otherwise noted were required to be async-signal-safe in POSIX.1-2001; the table details changes in the subsequent standards. Function Notes abort(3) Added in POSIX.1-2003 accept(2) access(2) aio_error(3) aio_return(3) aio_suspend(3) See notes below alarm(2) bind(2) cfgetispeed(3) cfgetospeed(3) cfsetispeed(3) cfsetospeed(3) chdir(2) chmod(2) chown(2) clock_gettime(2) close(2) connect(2) creat(2) dup(2) dup2(2) execl(3) Added in POSIX.1-2008; see notes below execle(3) See notes below execv(3) Added in POSIX.1-2008 execve(2) _exit(2) _Exit(2) faccessat(2) Added in POSIX.1-2008 fchdir(2) Added in POSIX.1-2013 fchmod(2) fchmodat(2) Added in POSIX.1-2008 fchown(2) fchownat(2) Added in POSIX.1-2008 fcntl(2) fdatasync(2) fexecve(3) Added in POSIX.1-2008 ffs(3) Added in POSIX.1-2016 fork(2) See notes below fstat(2) fstatat(2) Added in POSIX.1-2008 fsync(2) ftruncate(2) futimens(3) Added in POSIX.1-2008 getegid(2) geteuid(2) getgid(2) getgroups(2) getpeername(2) getpgrp(2) getpid(2) getppid(2) getsockname(2) getsockopt(2) getuid(2) htonl(3) Added in POSIX.1-2016 htons(3) Added in POSIX.1-2016 kill(2) link(2) linkat(2) Added in POSIX.1-2008 listen(2) longjmp(3) Added in POSIX.1-2016; see notes below lseek(2) lstat(2) memccpy(3) Added in POSIX.1-2016 memchr(3) Added in POSIX.1-2016 memcmp(3) Added in POSIX.1-2016 memcpy(3) Added in POSIX.1-2016 memmove(3) Added in POSIX.1-2016 memset(3) Added in POSIX.1-2016 mkdir(2) mkdirat(2) Added in POSIX.1-2008 mkfifo(3) mkfifoat(3) Added in POSIX.1-2008 mknod(2) Added in POSIX.1-2008 mknodat(2) Added in POSIX.1-2008 ntohl(3) Added in POSIX.1-2016 ntohs(3) Added in POSIX.1-2016 open(2) openat(2) Added in POSIX.1-2008 pause(2) pipe(2) poll(2) posix_trace_event(3) pselect(2) pthread_kill(3) Added in POSIX.1-2013 pthread_self(3) Added in POSIX.1-2013 pthread_sigmask(3) Added in POSIX.1-2013 raise(3) read(2) readlink(2) readlinkat(2) Added in POSIX.1-2008 recv(2) recvfrom(2) recvmsg(2) rename(2) renameat(2) Added in POSIX.1-2008 rmdir(2) select(2) sem_post(3) send(2) sendmsg(2) sendto(2) setgid(2) setpgid(2) setsid(2) setsockopt(2) setuid(2) shutdown(2) sigaction(2) sigaddset(3) sigdelset(3) sigemptyset(3) sigfillset(3) sigismember(3) siglongjmp(3) Added in POSIX.1-2016; see notes below signal(2) sigpause(3) sigpending(2) sigprocmask(2) sigqueue(2) sigset(3) sigsuspend(2) sleep(3) sockatmark(3) Added in POSIX.1-2004 socket(2) socketpair(2) stat(2) stpcpy(3) Added in POSIX.1-2016 stpncpy(3) Added in POSIX.1-2016 strcat(3) Added in POSIX.1-2016 strchr(3) Added in POSIX.1-2016 strcmp(3) Added in POSIX.1-2016 strcpy(3) Added in POSIX.1-2016 strcspn(3) Added in POSIX.1-2016 strlen(3) Added in POSIX.1-2016 strncat(3) Added in POSIX.1-2016 strncmp(3) Added in POSIX.1-2016 strncpy(3) Added in POSIX.1-2016 strnlen(3) Added in POSIX.1-2016 strpbrk(3) Added in POSIX.1-2016 strrchr(3) Added in POSIX.1-2016 strspn(3) Added in POSIX.1-2016 strstr(3) Added in POSIX.1-2016 strtok_r(3) Added in POSIX.1-2016 symlink(2) symlinkat(2) Added in POSIX.1-2008 tcdrain(3) tcflow(3) tcflush(3) tcgetattr(3) tcgetpgrp(3) tcsendbreak(3) tcsetattr(3) tcsetpgrp(3) time(2) timer_getoverrun(2) timer_gettime(2) timer_settime(2) times(2) umask(2) uname(2) unlink(2) unlinkat(2) Added in POSIX.1-2008 utime(2) utimensat(2) Added in POSIX.1-2008 utimes(2) Added in POSIX.1-2008 wait(2) waitpid(2) wcpcpy(3) Added in POSIX.1-2016 wcpncpy(3) Added in POSIX.1-2016 wcscat(3) Added in POSIX.1-2016 wcschr(3) Added in POSIX.1-2016 wcscmp(3) Added in POSIX.1-2016 wcscpy(3) Added in POSIX.1-2016 wcscspn(3) Added in POSIX.1-2016 wcslen(3) Added in POSIX.1-2016 wcsncat(3) Added in POSIX.1-2016 wcsncmp(3) Added in POSIX.1-2016 wcsncpy(3) Added in POSIX.1-2016 wcsnlen(3) Added in POSIX.1-2016 wcspbrk(3) Added in POSIX.1-2016 wcsrchr(3) Added in POSIX.1-2016 wcsspn(3) Added in POSIX.1-2016 wcsstr(3) Added in POSIX.1-2016 wcstok(3) Added in POSIX.1-2016 wmemchr(3) Added in POSIX.1-2016 wmemcmp(3) Added in POSIX.1-2016 wmemcpy(3) Added in POSIX.1-2016 wmemmove(3) Added in POSIX.1-2016 wmemset(3) Added in POSIX.1-2016 write(2) Notes: * POSIX.1-2001 and POSIX.1-2004 required the functions fpathconf(3), pathconf(3), and sysconf(3) to be async-signal-safe, but this requirement was removed in POSIX.1-2008. * If a signal handler interrupts the execution of an unsafe function, and the handler terminates via a call to longjmp(3) or siglongjmp(3) and the program subsequently calls an unsafe function, then the behavior of the program is undefined. * POSIX.1-2003 clarified that if an application calls fork(2) from a signal handler and any of the fork handlers registered by pthread_atfork(3) calls a function that is not async-signal-safe, the behavior is undefined. A future revision of the standard is likely to remove fork(2) from the list of async-signal-safe functions. Deviations in the GNU C library The following known deviations from the standard occur in the GNU C library: * Before glibc 2.24, execl(3) and execle(3) employed realloc(3) internally and were consequently not async-signal-safe. This was fixed in glibc 2.24. * The glibc implementation of aio_suspend(3) is not async-signal-safe because it uses pthread_mutex_lock(3) internally. SEE ALSO
sigaction(2), signal(7), standards(7) COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the latest version of this page, can be found at https://www.kernel.org/doc/man-pages/. Linux 2017-03-13 SIGNAL-SAFETY(7)
All times are GMT -4. The time now is 10:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy