Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Modify csv-files with awk
Top Forums Shell Programming and Scripting Modify csv-files with awk Post 303021667 by elRonaldo on Tuesday 14th of August 2018 09:56:01 AM
Old 08-14-2018
Thank you RudiC for your response.

Quote:
Originally Posted by elRonaldo

The filename looks like:
Manufacturer1_FiletypeA_20180502120102.csv
Manufacturer2_FiletypeA_20180502140537.csv
Manufacturer3_FiletypeB_20180501010424.csv

And the data like:
Code:
abcd; XXXX; 1234; XXXX; XXXX; 43563; abcde; XXXX; XXXX; 
cdef; XXXX, 4354; XXXX; XXXX; 43244; cdefg; XXXX;

where XXXX stands for useless data I don't need.
The columns in the new file should look like this
Code:
 abcd; 1234; 43563; abcde; Manufacturer1; FiletypeA; 20180502120102
 cdef; 4354; 43244; cdefg; Manufacturer1; FiletypeA; 20180502120102

Quote:
Originally Posted by RudiC
What does "merge them to one file" mean - ALL input files resulting in one big output file?And, I don't know anythig about that hdfs dfs command - is that really necessary?
I have a few hundred files every day. They aren't huge so it would be good to merge them to one big file. So I need to connect to the DB only once and not a few hundred times.


Yes, the hdfs dfs is necessary because the files are on a Hadoop cluster.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

csv files (with quoted commas) and awk

I have a file as follows: 1,"This is field 2",3,4,5 2,"This is field 2 it can contain one , comma",3,4,5 3,"This is field 2 it also, can, contain, more",3,4,5 4,"This is field 2 without extra commas",3,4,5 and i pass this through to awk: awk -F, ' { if (... (3 Replies)
Discussion started by: Cranie
3 Replies

2. Shell Programming and Scripting

Need to modify csv-file with bash script

Hi Guys, I need to write a script, that exports the "moz_places" table of the "places.sqlite"-file (firefox browser history) into a csv-file. That part works. After the export, my csv looks like this: ... 4429;http://www.sqlite.org/sqlite.html;"Command Line Shell For... (11 Replies)
Discussion started by: Sebi0815
11 Replies

3. Shell Programming and Scripting

Merge 2 csv files with awk

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123$aaa$yyy$zzz 345$xab$yzy$zyz 456$sss$ttt$foo 799$aaa$ggg$dee file 2 123$hhh 345$ddd 456$xxx 888$zzz so if the key is the first field, and the result should be the common key between file 1 and 2 (6 Replies)
Discussion started by: loloAix
6 Replies

4. Shell Programming and Scripting

Read data from .csv file through shell script & modify

I need to read data from a file called "test.csv" through shell script where the file contains values like name,price,descriptor etc. There are rows where descriptor (& in some rows name) are written as string & other characters like "car_+" OR "bike*" etc where it should contains strings like... (3 Replies)
Discussion started by: raj100
3 Replies

5. Shell Programming and Scripting

Calling Pl/sql function in shell script to modify csv

I need to 1.Open a csv 2.Process the csv i.e. Modify 2 column in the csv. To modify the column the value needs to be passed to a pl/sql function and the return value should be updated For eg: If column 2 E,then E will be passed in database function which will return Employee. 3. Write a... (5 Replies)
Discussion started by: Chinky23
5 Replies

6. Shell Programming and Scripting

Perl script to modify csv file

Hi Friends, I want to convert a csv file into a ordinary .txt file. I am able to convert but I want the output to look as shown below in the .txt file table findhost= { {"xyz","abc"}, {"rxz","mmz"}, {"vrr","nnz"}, } default={"NONE"} My current perl script #!/usr/bin/env perl... (12 Replies)
Discussion started by: dbashyam
12 Replies

7. Shell Programming and Scripting

Modify CSV file

Hi, I would like to change my CSV file by adding " and : and moving some of the information around. the CSV file looks as follows: 501254424;500440257;PE PACKS;300467279;PREP;;276476070;655031001867176;Two Block;Olga;25/12/2015 00:00:00;Olga I would like to move the field 7 to the front "... (13 Replies)
Discussion started by: omuhans123
13 Replies

8. Shell Programming and Scripting

Filter using awk in CSV files

Hello Gentlemen, Finding difficulties to play with my Input files:confused: . Your guidance will certainly help as always. After converting to csv file from XLSM file, I am getting some extra ""(double quote) characters which I want to terminate inside shell script and process it further. ... (6 Replies)
Discussion started by: pradyumnajpn10
6 Replies

9. UNIX for Beginners Questions & Answers

awk assistance - Comparing 2 csv files

Hello all, I have searched high and low for a solution to this, many have come really close but not quite what I'm after. I have 2 files. One contains GUID's, for example: 8121E002-96FE-4C9C-BC5A-6AFF20DACECD 84468F30-F3B7-418B-81F0-0908E80792BF A second file, contains a path to the... (8 Replies)
Discussion started by: tirmUK
8 Replies

10. UNIX for Beginners Questions & Answers

awk on csv files

wk on csv files Hi I have two csv files 1 ) keys.csv with data as below XX1,offsite XX2,offsite XX8,none XX3,offsite XX4,onsite XX7,none XX5,onsite XX6,onsite XX15,none 2) source.csv file with data like 1,0123,ppp,XX1 2,0122,sss,XX2 3,1239,yyy,XX8 4,567,kkk,XX5 (2 Replies)
Discussion started by: zozoo
2 Replies

LEARN ABOUT DEBIAN

fntsample

fntsample(1)						      General Commands Manual						      fntsample(1)

NAME

       fntsample - PDF and PostScript font samples generator

SYNOPSIS

       fntsample [ OPTIONS ] -f FONT-FILE -o OUTPUT-FILE
       fntsample -h

DESCRIPTION

       fntsample  program  can	be  used  to generate font samples that show Unicode coverage of the font and are similar in appearance to Unicode
       charts. Samples can be saved into PDF (default) or PostScript file.

OPTIONS

       fntsample supports the following options.

       --font-file, -f FONT-FILE
	      Make samples of FONT-FILE.

       --font-index, -n IDX
	      Font index for FONT-FILE specified using --font-file option.  Useful for files that contain multiple fonts,  like  TrueType  Collec-
	      tions (.ttc).  By default font with index 0 is used.

       --output-file, -o OUTPUT-FILE
	      Write output to OUTPUT-FILE.

       --other-font-file, -d OTHER-FONT
	      Compare FONT-FILE with OTHER-FONT.  Glyphs added to FONT-FILE will be highlighted.

       --other-index, -m IDX
	      Font index for OTHER-FONT specified using --other-font-file option.

       --postscript-output, -s
	      Use PostScript format for output instead of PDF.

       --svg, -g
	      Use SVG format for output.  The generated document contains one page.  Use range selection options to specify which.

       --print-outline, -l
	      Print  document  outlines data to standard output.  This data can be used to add outlines (aka bookmarks) to resulting PDF file with
	      pdfoutline program.

       --include-range, -i RANGE
	      Show characters in RANGE.

       --exclude-range, -x RANGE
	      Do not show characters in RANGE.

       --style, -t "STYLE: VAL"
	      Set STYLE to value VAL.  Run fntsample with option --help to see list of styles and default values.

       --help, -h
	      Display help text and exit.

       Parameter RANGE for -i and -x can be given as one integer or a pair of integers delimited by minus sign (-).  Integers can be specified	in
       decimal,  hexadecimal  (0x...) or octal (0...) format.  One integer of a pair can be missing (-N can be used to specify all characters with
       codes less or equal to N, and N- for all characters with codes greather or equal to N).	Multiple -i and -x options can be used.

EXAMPLES

       Make PDF samples for font.ttf and write them to file samples.pdf:
	      fntsample -f font.ttf -o samples.pdf

       Make PDF samples for font.ttf, compare it with oldfont.ttf and highlight new glyphs. Write output to file samples.pdf:
	      fntsample -f font.ttf -d oldfont.ttf -o samples.pdf

       Make PostScript samples for font.ttf and write output to file samples.ps.  Show only glyphs for characters with	codes  less  or  equal	to
       U+04FF but exclude U+0370-U+03FF:
	      fntsample -f font.ttf -s -o samples.ps -i -0x04FF -x 0x0370-0x03FF

       Make PDF samples for font.ttf and save output to file samples.pdf adding outlines to it:
	      fntsample -f font.ttf -o temp.pdf -l > outlines.txt
	      pdfoutline temp.pdf outlines.txt samples.pdf

AUTHOR

       Copyright (C) 2007 Eugeniy Meshcheryakov <eugen@debian.org>
       Homepage: <http://fntsample.sourceforge.net/>

SEE ALSO

       pdfoutline(1)

								    2010-10-14							      fntsample(1)
All times are GMT -4. The time now is 01:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy