Sponsored Content
Top Forums Shell Programming and Scripting merging CSV data using a one liner from shell? Post 302131409 by jjinca on Friday 10th of August 2007 01:04:17 PM
Old 08-10-2007
merging CSV data using a one liner from shell?

I'm trying to merge multiple CSV (comma separated value) files into one large master file. All files have a field that is unique to act as the key for entry/merging into the master file & and all files have the same number of fields that are in the master file.

I'll give an example here:

MASTERFILE (BEFORE MERGE):

"userid","logon","name","fname","lname","app1","app2","app3","app4","app5"
"111","","Joe One","","","yes","no","","no",""
"333","","","","","","","","",""
"555","","","Five","","","","","",""
"222","mrtwo","Lou Two","Lou","","no","no","no","no","yes"

MERGEFILE1:

"111","mrone","","Joe","One","","","yes","","no"
"222","","Lou Two","Lou","","no","no","no","no","yes"
"555","mrfive","","","","","","","",""


I imagine it would be something like:
Run the script...
# {somescript} masterfile mergefile1 > masterfile2


So now MASTERFILE2 would become:

"111","mrone","Joe One","Joe","One","yes","no","yes","no","no"
"222","mrtwo","Lou Two","Lou","","no","no","no","no","yes"
"333","","","","","","","","",""
"555","mrfive","","Five","","","","","",""


So the first field would be be scanned from the first line of the mergefile to see if it is in the the first filed of any lines in the masterfile, if it exists in the masterfile, then it would update the fields for that entry accordingly (overwriting any existing fields in masterfile) and continue on to process the next line of the mergefile against the masterfile.


How could I accomplish this? Smilie


Thanks!
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to format a .CSV data

Hi There I needed to write a Unix shell script which will pick up the data from a .CSV file and reformat it as per the requirement and write it to another .CSV file. Currently I am in the proess of Data Import to "Remedy System" (A one kind of incident mangement Application) and this... (8 Replies)
Discussion started by: Uday1982
8 Replies

2. Shell Programming and Scripting

csv to table one-liner

I've googled a lot on this, but could not fine a simple one-liner to do this. I have a .csv file that looks like this: Header one Header two Header three col1,col2,col3 short data, very long data, dataIf I use sed and change the comma to tab, being the colums of variable length I don't get a... (6 Replies)
Discussion started by: ahsog
6 Replies

3. UNIX for Advanced & Expert Users

shell script to format .CSV data

Hi all, I have written a shell script to search a specified directory (e.g. /home/user) for a list of specific words (shown as ${TMPDIR}/wordlist below). The script works well enough, but I was wondering if there was a way to display the line number that the word is found on? Thanks! cat... (1 Reply)
Discussion started by: tmcmurtr
1 Replies

4. Shell Programming and Scripting

Shell snip to import CSV data into BASH array

I have been trying to write a simple snip of bash shell code to import from 1 to 100 records into a BASH array. I have a CSV file that is structured like: record1,item1,item2,item3,item4,etc.,etc. .... (<= 100 items) record2,item1,item2,item3,item4,etc.,etc. .... (<= 100 items)... (5 Replies)
Discussion started by: dstrout
5 Replies

5. Shell Programming and Scripting

creating a csv file from this 1 liner?

I'm trying to create a csv file by running awk and sed on a number of xml files in a directory; I'm using this below: hostname; grep "BuildDate" /dir/ABCD/configuration/*/*.xml | awk -F"/" '{ print $5 }' > /tmp/tempfile.txt; grep "BuildDate" /dir/ABCD/configuration/*/*.xml | awk -F\" '{ print $2... (2 Replies)
Discussion started by: rich@ardz
2 Replies

6. Shell Programming and Scripting

Matching and Merging csv data fields based on a common field

Dear List, I have a file of csv data which has a different line per compliance check per host. I do not want any omissions from this csv data file which looks like this: date,hostname,status,color,check 02-03-2012,COMP1,FAIL,Yellow,auth_pass_change... (3 Replies)
Discussion started by: landossa
3 Replies

7. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 7 columns having values say column 1,column 2.....column 7 as below along with their values. Name, Address,... (7 Replies)
Discussion started by: Vivekit82
7 Replies

8. Shell Programming and Scripting

Shell script to extract data from csv file

Hi everyone, I have a csv file which has data with different heading and column names as below. Static Data Ingested ,,,,,,,,,,,,Known Explained Rejections Column_1,column_2,Column_3,Column_4,,Column_6,Column_7,,% Column_8,,Column_9 ,Column_10 ,... (14 Replies)
Discussion started by: Vivekit82
14 Replies

9. Shell Programming and Scripting

Shell script for .Txt to .csv conversion with data processing

Hi experts, I want to convert a txt file having rows and columns (CNAI_DUMP_raw.txt) by comparing it with another text file (paramaters.txt) and generate a output in CSV which contains only 3rd column from CNAI_DUMP_raw.txt, and the columns mentioned in parameters.txt. FYI: There are two... (16 Replies)
Discussion started by: Gautam Banerjee
16 Replies
DEBCONF-GETTEXTIZE(1)						    po-debconf						     DEBCONF-GETTEXTIZE(1)

NAME
debconf-gettextize - extract translations of debconf templates into PO files SYNOPSIS
debconf-gettextize [-v] [-h] [--podir=DIR] [--choices] [--merge] master [master ...] DESCRIPTION
The /var/lib/dpkg/info/*.templates files read by debconf contain English text and translations in the same file. But in source packages, translations are managed in separate files to help translators. In the initial implementation, a master templates file contained only English text, and templates.xx files contained both original and translated strings for the language xx. The debconf-mergetemplate program was merging master and translated templates files. The new implementation with "po-debconf" is based on "gettext". Maintainers mark translatable fields by prepending them with an underscore, English strings are automatically extracted into a POT file, translators work on normal PO files, and po2debconf generates a merged templates file with the same structure. The debconf-gettextize program was initially designed to help migrating to the new implementation. o It reads a list of master files and their associated translations, and generates po/*.po files for each language containing translated strings. o Each input file receives an ".old" suffix, and a new master file overwrites the old one; it is identical to the previous master file except that an underscore is prepended to translatable fields. Developers can then choose which fields translators have to work on and which ones are skipped because their values are not locale-dependent. o A po/POTFILES.in file is also created, it contains the list of templates files debconf-updatepo has to process. Typically the debconf-gettextize program must be run only once when transforming from the first implementation to the "po-debconf" format, but it can also be used afterwards to transform a "_Choices" field into "__Choices" (or vice-versa) without losing translations, when using the "--merge" flag (alongside with "--choices" or not). The desired template containing the "_Choices" or "__Choices" fields to modify has to be copied into a temporary template file, which is passed to debconf-gettextize as an argument. Then the following steps are performed: 1. po2debconf is run on this template file to generate a translated templates file. 2. This translated templates file is processed as described above and PO files are generated. 3. Newly created PO files are merged with existing ones. When PO files are merged, "_Choices" fields have to be replaced by "__Choices" (or vice-versa) in the original templates file before running debconf-updatepo, otherwise new translations will be fuzzy. OPTIONS
-h, --help Display a usage summary for the program and exit. -v, --verbose Process in verbose mode. --podir=DIR Set directory for PO files. Default is to search for PO files in the po subdirectory below the location of the first master file. --choices By default, debconf-gettextize replaces "Choices" fields by "_Choices". With this flag, "__Choices" fields are written instead. --merge When this flag is set, the strings are merged with existing PO files. The templates files and po/POTFILES.in are not modified. CAVEATS
The "Default" field is special when template type is "Select" or "Multiselect", because a value has to be chosen amongst the English list of choices, even for localized values. Normally this field must not be changed, but in rare circumstances localized values are meaningful (e.g. to choose the default language of an application). In order to let translators know that this localized "Default" field is special, you must by convention call it "_DefaultChoice" instead of "_Default". SEE ALSO
debconf-updatepo(1), po2debconf(1), debconf-devel(7), po-debconf(7). AUTHORS
Denis Barbier <barbier@linuxfr.org> Martin Quinson <martin.quinson@ens-lyon.fr> 2016-10-01 DEBCONF-GETTEXTIZE(1)
All times are GMT -4. The time now is 01:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy