Sponsored Content
Top Forums Shell Programming and Scripting Bash/shell merge similar lines Post 302966905 by vgersh99 on Wednesday 17th of February 2016 06:55:35 PM
Old 02-17-2016
something along these lines - a bit succinct on explanation, but... :
Code:
awk '
  # idx is an index constructed by concatenating first 3 fields in the line
  {idx=$1 FS $2 FS $3} 

  # a - an array indexed by 'idx'.
  # if idx is in a, add forth field to it; if not, assign forth field as the first entry in array a
  {a[idx]=(idx in a)?a[idx] OFS $4:$4} 

  # after all records/lines have been process...
  # iterate through all the indecies in array a, print out the value of the index and
  # the corresponding value stored in the array with the current index.
  END {for(i in a) print i FS a[i]}' OFS=, myInputFileGoesHere


Last edited by vgersh99; 02-17-2016 at 08:39 PM..
This User Gave Thanks to vgersh99 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Urgent : Merge similar lines

Hi, I have a file like this. please notice that ./usr/orders1/order_new_2627 appears more than once, thus needs to be merged. I would like to merge the lines where the first column match so the output should be like this: Please help (2 Replies)
Discussion started by: rakeshou
2 Replies

2. Shell Programming and Scripting

Help with comparing 2 lines in 2 different file in shell bash

Hi guys i need help with comparing lines in 2 separate files. Both files contain the same amount of lines and i need to output the difference into the 2nd file. The 1st file is always correct. 1st file (Expected.e): Tuesday, 11 August 2009 Wednesday, 12 August 2009 Thursday, 13 August 2009... (2 Replies)
Discussion started by: kcrap
2 Replies

3. Shell Programming and Scripting

merge similar rows

I have a large file (10M lines) that contains two columns: a frequency and a string, ex: 3 aaaaa 4 bbbbb 2 ccccc 5 aaaaa 1 ddddd 4 ccccc I need to merge the lines whose string part is the same, while updating the frequency. The output should look like this: 8 aaaaa 4 bbbbb 5 ccccc... (2 Replies)
Discussion started by: tootles564
2 Replies

4. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

5. UNIX for Dummies Questions & Answers

Merge two files with two columns being similar

Hi everyone. How can I merge two files, where each file has 2 columns and the first columns in both files are similar? I want all in a file of 4 columns; join command removes the duplicate columns. 1 Dave 2 Mark 3 Paul 1 Apple 2 Orange 3 Grapes to get it like this in the 3rd file:... (9 Replies)
Discussion started by: Atrisa
9 Replies

6. UNIX for Dummies Questions & Answers

merge lines within a file that start with a similar pattern

Hello! i have a text file.. which contains the data as follows i want to merge the declarations lines pertaining to one datatype in to a single line as follows i've searched the forum for help.. but couldn't find much help.. how can i do this?? (1 Reply)
Discussion started by: a_ba
1 Replies

7. Shell Programming and Scripting

Help with merge two file based on similar column content

Input file 1: A1BG A1BG A1BG A1CF A1CF BCAS BCAS A2LD1 A2M A2M HAT . . Input file 2: A1BG All A1CF TEMP (5 Replies)
Discussion started by: perl_beginner
5 Replies

8. Shell Programming and Scripting

Merge two files with similar column entries

Hi , I have few files which contains user name and data transfer rate in MBs and this data is collected for year and for each month report is saved in 12 different files I have to merge all the files to prepare the final report Files are as below Similarly I have 10 more files ... (5 Replies)
Discussion started by: pratapsingh
5 Replies

9. Shell Programming and Scripting

Shell script to merge and delete lines

POLY_STATS { EqName 103_tri Id 123 act_polyCount 1 act_polyValue 0 } POLY_STATS { EqName 103_tri Id 123 pass_polyCount 2 pass_polyValue 0 } POLY_STATS { EqName 103 Id 123 mes_polyCount 2 mes_polyValue 0 (5 Replies)
Discussion started by: Jag02
5 Replies

10. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies
SYSPROFILE(8)						      System Manager's Manual						     SYSPROFILE(8)

NAME
sysprofile - modular centralized shell configuration DESCRIPTION
sysprofile is a generic approach to configure shell settings in a modular and centralized way mostly aimed at avoiding work for lazy sysad- mins. It has only been tested to work with the bash shell. It basically consists of the small /etc/sysprofile shell script which invokes other small shell scripts having a .bash suffix which are contained in the /etc/sysprofile.d/ directory. The system administrator can drop in any script he wants without any naming convention other than that the scripts need to have a .bash suffix to enable automagic sourcing by /etc/sysprofile. This mechanism is set up by inserting a small shell routine into /etc/profile for login shells and optionally into /etc/bashrc and/or /etc/bash.bashrc for non-login shells from where the actual /etc/sysprofile script is invoked: if [ -f /etc/sysprofile ]; then . /etc/sysprofile fi For using "sysprofile" under X11, one can source it in a similar way from /etc/X11/Xsession or your X display manager's Xsession file to provide the same shell environment as under the console in X11. See the example files in /usr/share/doc/sysprofile/ for illustration. For usage of terminal emulators with a non-login bash shell under X11, take care to enable sysprofile via /etc/bash.bashrc. If not set this way, your terminal emulators won't come up with the environment defined by the scripts in /etc/sysprofile.d/. Users not wanting /etc/sysprofile to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by simply creating an empty file called $HOME/.nosysprofile in the user's home directory using e.g. the touch(1) command. Any single configuration file in /etc/sysprofile.d/ can be overridden by any user by creating a private $HOME/.sysprofile.d/ directory which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to match exactly the system's default /etc/sysprofile.d/ configuration files. Empty versions of these files contained in the $HOME/.syspro- file.d/ directory automatically disable sourcing of the system wide version. Naturally, users can add and include their own private script inventions to be automagically executed by /etc/sysprofile at login time. OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves. SEE ALSO
The README files and configuration examples contained in /etc/sysprofile.d/ and the manual pages bash(1), xdm(1x), xdm.options(5), and wdm(1x). Recommended further reading is everything related with shell programming. If you need a similar mechanism for executing code at logout time check out the related package syslogout(8) which is a very close compan- ion to sysprofile. BUGS
sysprofile in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we take patches... ;-) AUTHOR
sysprofile was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into something more worthwhile than it currently is. SYSPROFILE(8)
All times are GMT -4. The time now is 09:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy