Sponsored Content
Top Forums UNIX for Beginners Questions & Answers awk command to split pipe delimited file Post 303024350 by Peasant on Sunday 7th of October 2018 12:55:57 AM
Old 10-07-2018
Code:
awk -F"|" 'NR>1 { a[$0]=$7 } END { for ( i in a ) print i > "1_"a[i] } ' input

If you have gigabyte files, a different approach would be needed to minimize memory usage, since array a would become huge on those files.

But that's another problem, which would require a bit larger and more efficient program.
If that is the case, get back here, and we shall think of something.

Hope that helps
Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to split pipe delimited file

I have a pipe delimited input file as below. First byte of the each line indicate the record type. Then i need to split the file based on record_type = null,0,1,2,6 and create 5 files. How do i do this in a ksh script? Pls help |sl||SL|SL|SL|1996/04/03|1988/09/15|C|A|sl||||*|... (4 Replies)
Discussion started by: njgirl
4 Replies

2. Shell Programming and Scripting

convert a pipe delimited file to a':" delimited file

i have a file whose data is like this:: osr_pe_assign|-120|wg000d@att.com|4| osr_evt|-21|wg000d@att.com|4| pe_avail|-21|wg000d@att.com|4| osr_svt|-11|wg000d@att.com|4| pe_mop|-13|wg000d@att.com|4| instar_ready|-35|wg000d@att.com|4| nsdnet_ready|-90|wg000d@att.com|4|... (6 Replies)
Discussion started by: priyanka3006
6 Replies

3. UNIX for Dummies Questions & Answers

Delete last value from pipe delimited file

I have a large(ish) pipe delimited file. The last line of the file contains a total row count and a checksum: END|1537451|1328569446 After making other adjustments to the file, I need to strip out the checksum and apply a new value - I have a script to generate the checksum and 'cat' it... (3 Replies)
Discussion started by: relentl3ss
3 Replies

4. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

5. Shell Programming and Scripting

Help with converting Pipe delimited file to Tab Delimited

I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use cat file | sed 's/|//t/g' The above command substituted "/t" not tab in the place of pipe. Sample file: abc|123|2012-01-30|2012-04-28|xyz have to convert to: abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies

6. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

7. Shell Programming and Scripting

Insert a value in a pipe delimited line (unsig sed,awk)

Hi, I want to insert a value (x) in the 3rd position of each line in a file like below a|b|c|d|1 a|b|c|d a|b|c|d|e|1 a|b|cso that output file looks like a|b|x|c|d|1 a|b|x|c|d a|b|x|c|d|e|1 a|b|x|cI can do that using perl as below #!/usr/bin/perl -w use strict; #inserting x at... (5 Replies)
Discussion started by: sam05121988
5 Replies

8. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies

9. Shell Programming and Scripting

Split a content in a file with specific interval base on the delimited values using UNIX command

Hi All, we have a requirement to split a content in a text file every 5 rows and write in a new file . conditions: if 5th line falls between center of the statement . it should look upto after ";" files are below format: 1 UPDATE TABLE TEST1 SET VALUE ='AFDASDFAS' 2 WHERE... (3 Replies)
Discussion started by: KK230689
3 Replies

10. Shell Programming and Scripting

Get the output of w command in pipe delimited format

Since output of w command have variable number of columns I want to get the output in pipe delimited format. I tried export OFS="|"; w but that does not work. Any ideas? (4 Replies)
Discussion started by: Soham
4 Replies
MPB(1)							    MIT Photonic-Bands Package							    MPB(1)

NAME
mpb-split - compute eigenmodes with MPB using multiple processes SYNOPSIS
mpb-split NUM-SPLIT [DEFINITION]... [CTLFILE]... DESCRIPTION
mpb-split is a parallelizing front-end to MIT Photonic Bands (MPB). For a computation with several k points, it splits the list of k points over multiple processes. Of course, this will only benefit you on a system where different processes will run on different proces- sors, such as an SMP or a cluster with automatic process migration (e.g. MOSIX). mpb-split is actually a trivial shell script, though, so you can easily modify it if you need to use a special command to launch processes on other processors/machines. MIT Photonic Bands (MPB) is a free program to compute the band structures (dispersion relations) and electromagnetic modes of periodic dielectric structures, and is applicable both to photonic crystals (photonic band-gap materials) and a wide range of other optical prob- lems. More information on MPB, including a detailed manual, can be found online at the MPB home page: http://ab-initio.mit.edu/mpb/ A typical invocation of mpb-split looks like: mpb-split num-split foo.ctl >& foo.out This causes mpb-split to process the control file foo.ctl, divide the k points into num-split equal chunks, run each list in a separate process with MPB, and redirect the output (in order) to foo.out. (One typically redirects output to a file, as the output is verbose and contains a number of comma-delimited datasets that one can extract by grepping.) Overall, the behavior and arguments are the same as for mpb except that the first argument must be the integer num-split. What mpb-split technically does is to set the MPB variable k-split-num to num-split and k-split-index to the index (starting with 0) of the chunk for each process. If you want, you can use these variables to divide the problem in some other way and then reset them to 1 and 0, respectively. BUGS
Send bug reports to S. G. Johnson, stevenj@alum.mit.edu. AUTHORS
Written by Steven G. Johnson. Copyright (c) 1999, 2000, 2001, 2002 by the Massachusetts Institute of Technology. SEE ALSO
mpb(1), mpb-data(1) MPB
March 13, 2002 MPB(1)
All times are GMT -4. The time now is 08:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy