Sponsored Content
Top Forums Shell Programming and Scripting Insert blanks for missing fields and reformat to csv Post 302345680 by edidataguy on Thursday 20th of August 2009 01:19:38 AM
Old 08-20-2009
Quote:
Originally Posted by LoveSquid
jp2542a: Thanks, I will give that a try this morning and see what results I get. I'm not familiar with awk, but plan to learn it so this may be an opportunity.


edidataguy: Thanks for pointing that out! The only constant is that all will begin with UR and all will end with XD and then an empty (after removal of ^M) line. This is obviously generated on a Win machine, buy I am working entirely on Linux desktop and a Linux server.

That said, it may help if I state that I will then use this to insert and delete records in a MySQL database and run as a cron job. Thanks for the help!
Code:
grep '|' reorder.txt | tr '\n' ',' | sed -e 's/,\(UR|\)/\n\1/g; s/,$//'

There are a few pit falls in this. Assumed there will be no ",UR|" anywhere else in the data.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

CSV to SQL insert: Awk for strings with multiple lines in csv

Hi Fellows, I have been struggling to fix an issue in csv records to compose sql statements and have been really losing sleep over it. Here is the problem: I have csv files in the following pipe-delimited format: Column1|Column2|Column3|Column4|NEWLINE Address Type|some descriptive... (4 Replies)
Discussion started by: khayal
4 Replies

2. Shell Programming and Scripting

Insert missing field using perl,sed,awk

sample file (comma as field separators) MessageFlow,1,BusIntBatchMgr,a OOBEvent,1,BusIntBatchMgr,a TaskEvents,1,,a MTTrace,1,,a MTWarning,,1,a MessageFlow,1,Batch,a OOBEvent,1,Batch,a TaskEvents,1,,a EAISAPIdocWizard,1,BusIntMgr,a EAISAPBAPIWizard,1,BusIntMgr,a... (3 Replies)
Discussion started by: vrclm
3 Replies

3. Shell Programming and Scripting

Deleting all the fields(columns) from a .csv file if all rows in that columns are blanks

Hi Friends, I have come across some files where some of the columns don not have data. Key, Data1,Data2,Data3,Data4,Data5 A,5,6,,10,, A,3,4,,3,, B,1,,4,5,, B,2,,3,4,, If we see the above data on Data5 column do not have any row got filled. So remove only that column(Here Data5) and... (4 Replies)
Discussion started by: ks_reddy
4 Replies

4. UNIX for Dummies Questions & Answers

How to combine and insert missing consecutive numbers - awk or script?

Hi all, I have two (2) sets of files that are based on some snapshots of database that I want to merge and insert any missing sequential number. Below are example representation of these files: file1: DATE TIME COL1 COL2 COL3 COL4 ID 01/10/2013 0800 100 ... (3 Replies)
Discussion started by: newbie_01
3 Replies

5. Shell Programming and Scripting

Reformat csv file

Hi, I have a csv file with content like: 1,0,100 1,1,150 2,0,200 2,1,250 3,0,300 3,1,350 I want an output such that all numbers in 3rd col where 2nd col is "0" come in the same col in the output. The same goes for numbers where 2nd col is "1". 1 100 150 2 200 250 3 300 350 Tnx... (2 Replies)
Discussion started by: jamaje
2 Replies

6. Shell Programming and Scripting

Insert missing values

Hi, please help with this, I need to insert missing values into a matrix for a regression analysis. I have made up an example. The first three columns are variables with levels and the next 3 are values, the 4th column missing values should be replaced by 0s, and 5th and 6th column missing... (3 Replies)
Discussion started by: ritakadm
3 Replies

7. Shell Programming and Scripting

Compare 2 files and find missing fields awk

Hello experts! I have 2 files. file1 is a list file containing uniquely names. e.g.: name1 number number name2 number number name5 number number name10 number number ... file2 is a data file arbitrary containing the names of file1 in paragraphs separated by "10" e.g. name4 ... (3 Replies)
Discussion started by: phaethon
3 Replies

8. Shell Programming and Scripting

How to print the missing fields outside the for loop in Korn shell?

I have 2 for loop in my program , first one will list files based on timestamp and second one list the files based on type(RPT / SUB_RPT).Here is my code: #!/bin/ksh STG_DIR=/home/stg for pattern in `find $STG_DIR -type f -name 'IBC*csv' | awk -F'' '{print $(NF-1)}' | sort -u` do echo... (2 Replies)
Discussion started by: ann15
2 Replies

9. Shell Programming and Scripting

awk to insert missing string based on pattern in file

Using the file below, which will always have the first indicated by the digit after the - and last id in it, indicated by the digit after the -, I am trying to use awk to print the missing line or lines in file following the pattern of the previous line. For example, in the file below the next... (4 Replies)
Discussion started by: cmccabe
4 Replies

10. UNIX for Beginners Questions & Answers

How to extract fields from a CSV i.e comma separated where some of the fields having comma as value?

can anyone help me!!!! How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement like i have to extract different field and assign them into different... (4 Replies)
Discussion started by: J.Jena
4 Replies
pilot-addresses(1)						    PILOT-LINK							pilot-addresses(1)

NAME
pilot-addresses - Read and write address book databases to and from a Palm handheld. SECTION
pilot-link: Userland conduits LOCATION
./src SYNOPSIS
pilot-addresses [-p|--port <port>] [--version] [-?|--help] [--usage] [-q|--quiet] [--delete-all] [-t|--delimiter INT] [-d|--delete-cate- gory category ] [-c|--category category] [-a|--augment] [-r|--read file] [-w|--write file ] [-C|--human-readable] DESCRIPTION
pilot-addresses allows the user to read all entries from a Palm handheld such as those made by Palm, Handspring, Handera, TRGPro, Sony or other Palm handheld in the Palm Address Book application, write new entries into the database, and delete a category or delete all entries in the database. COMMANDS
Provide exactly one of --read or --write. -r, --read=<file> Reads records from <file> and install them to the Palm Address Book application. (Use the -w <file> option to get a sample "template file" for input records to see how the fields are laid out). -w, --write=file Reads all address book records from the Palm Address Book application and writes them into file OPTIONS
pilot-addresses options -a, --augment Augments fields in address book records with additional information. The augmented information is placed before and separated from the field with a semi-colon, (;). Augmented information can include any of the following below. category # Placed in front of each record or.. Placed in front of each phone number field: ["Work"|"Home"|"Fax"|"Other"|"E-mail"|"Main"|"Pager"|"Mobile"] Empty fields are not augmented. -c, --category=category Install records to category category by default. Normally pilot-addresses uses the Unfiled category by default. This option is overridden by the category specified in the record with the -a option. -d, --delete-category Delete all records in the specified category category before installing new records. Note that the long name of this option has changed since 0.11.8 -D, --delete-all Delete all address book records in all categories. Obviously, be very careful with this one. Note that the long name of this option has changed since 0.11.8 -t, --delimiter INT Include category in each record, use the delimiter specified to separate all fields of a record. Delimiters are specified as follows: 3=tab, 2=;, 1=,. This overrides the default delimiter of comma between fields and semi-colon between a field's augmented information. (Please note that this may generate confusing results when used with the -a option.) Note that the long name of this option has changed since 0.11.8 -T, --titles Obsolete. This option has been removed since 0.11.8 -C|--human-readable Write generic human-readable output instead of CSV. Conduit Options -p, --port <port> Use device file <port> to communicate with the Palm handheld. If this is not specified, pilot-addresses will look for the $PILOTPORT environment variable. If neither are found or supplied, pilot-addresses will print the usage information. -q, --quiet Suppress 'Hit HotSync button' message -v, --version Display version of pilot-addresses and exit without connecting. Help Options -h, --help Display the help synopsis for pilot-addresses and exit without connecting. --usage Display a brief usage message and exit without connecting. USAGE
The program will connect to the Palm handheld, and perform the requested read or write operation specified by the user. EXAMPLES
To write all address records in a Palm to the file addrbook.csv: pilot-addresses -w addrbook.csv or pilot-addresses -p /dev/pilot -w addrbook.csv To read the address book records in the file addrbook.csv and install them onto the Palm: pilot-addresses -r addrbook.csv To read the address book records in the file addrbook.csv and place them into the Palm address book database category Special after first deleting all current records in the Special category on the palm: pilot-addresses -c Special -d Special -r addrbook.csv AUTHOR
pilot-addresses originally written by Kenneth Albanowski, this manual page was originally written by Robert Wittig <bob.wittig@gt.org> and later updated by David A. Desrosiers <hacker@gnu-designs.com> and Neil Williams <linux@codehelp.co.uk>. REPORTING BUGS
We have an online bug tracker. Using this is the only way to ensure that your bugs are recorded and that we can track them until they are resolved or closed. Reporting bugs via email, while easy, is not very useful in terms of accountability. Please point your browser to http://bugs.pilot-link.org: http://bugs.pilot-link.org and report your bugs and issues there. COPYRIGHT
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER- CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. SEE ALSO
addresses(1) pilot-link(7) 0.12.5 Copyright 1996-2007 FSF pilot-addresses(1)
All times are GMT -4. The time now is 11:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy