Visit Our UNIX and Linux User Community

Split file into rows in UNIX

Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Split file into rows in UNIX
# 1  
Old 02-25-2020
Split file into rows in UNIX

Hi Team,
I have delimited file with 6gb of data and it is not hsve newline seperator.

Ask : I would like to split file into rows based on columns.

File.txt contains below data

Expected output:

Last edited by vbe; 02-26-2020 at 10:57 AM..
# 2  
Old 02-25-2020
Please show attempts to complete this task. Otherwise a moderator might have you do it manually with a text editor. Also, search the site for similar or exact solutions to accomplish this task.

Last edited by RavinderSingh13; 02-26-2020 at 12:36 PM..
# 3  
Old 02-26-2020
If you use regexp you can hang the computer.
tr '-' '\n' <file | pr -3tas'-' >new_file

it takes 10 to 20 minutes
You can look at the progress.
In another window, run
progress -c pr -M

pv file | tr '-' '\n' | pr -3tas'-' >new_file

# 4  
Old 02-26-2020
We do encourage users to post their own attempts before helping.
Please in the future, show your effort, as we are not your scripting service.

Give this a shot :
awk -F"-" ' { for ( i=3;i<=NF;i+=3) print $(i-2) FS $(i-1) FS $i }' input

This User Gave Thanks to Peasant For This Post:
# 5  
Old 02-26-2020
<file tr - $'\n' | paste -sd'--\n'

This User Gave Thanks to RudiC For This Post:
# 6  
Old 02-26-2020
It prints the line occurrence of column 1 ($1). a[$1]++ fails for the first occurrence increment since the array value for that key does not exist. The ! means not true so the expression is evaluated to true. The default action for true in awk is to print the line. In other terms: if the previous value of column 1 in array "a" cannot be incremented then print the line. The line could have been written as: '{if (! a[$1]++) print $0}'
# 7  
Old 02-26-2020
Originally Posted by RudiC
<file tr - $'\n' | paste -sd'--\n'

Speed increased 5-10 times!
Originally Posted by Peasant
awk -F"-" ' { for ( i=3;i<=NF;i+=3) print $(i-2) FS $(i-1) FS $i }' input

6GB single line file completely froze the system
This User Gave Thanks to nezabudka For This Post:

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to split one long column into multiple rows with 3 each ?

I have a large csv dataset like this : A value1 A value2 A value3 B value1 B value2 B value3 C value1 C value2 C value3 what I expected output is :A value1 value2 value3 B value1 value2 value3 C value1 value2 value3 I'm thinking of use like awk, columns , but haven't find a proper... (4 Replies)
Discussion started by: nengcheng
4 Replies

2. UNIX for Beginners Questions & Answers

Split column into rows

Hi, I have input dataset as below: Cl.jenn,1051,ABCD JEN.HEA,9740|1517|8119|2145,ZZZZ,REPEAT Rich.Sm, Ann.Car,3972|4051|1064|4323|4122|2394|2574|4507 Sta.for,7777,ABCD,UUUU Sm.Ric, Ch.LRD, Eh.ab,, Expected output: ------------------- Cl.jenn,1051,ABCD... (6 Replies)
Discussion started by: saravananravim
6 Replies

3. Shell Programming and Scripting

Split columns into rows

Any one can help me in converting columns into rows. example I have input file 10000| 10002| 10003| 10004| 10005| I want output in below format PARTY|PART_DT 10000|12080000000 10002|13075200000 10003|13939200000 10004|1347200000 10004|133600000 10004|1152000000 (13 Replies)
Discussion started by: syd
13 Replies

4. Shell Programming and Scripting

Split File based on number of rows

Hi I have a requirement, where i will receive multiple files in a folder (say: /fol1/fol2/). There will be at least 14 to 16 files. The size of the files will different, some may be 80GB or 90GB, some may be less than 5 GB (and the size of the files are very unpredictable). But the names of the... (10 Replies)
Discussion started by: kpk_ds
10 Replies

5. Shell Programming and Scripting

MySql split rows

Dear community, I have to split string in table and list all values. I'll skip the code and jump directly to mysql query. This is the table: category title ======= ======= 7,3 title 1 1,3 title 2 1,2,3 title 3 Now, what I need is split category into single... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

6. Shell Programming and Scripting

split paste them in rows

Hi, I have a file as ABC 123_456_789 234_678_901 XYZ 1100_1250_1580_1680 1175_1440_1620_1890 so on What I want my output file to look is "split by underscore and then place the contents in rows" output ABC 123 234 ABC 456 678 ABC 789 901 XYZ 1100 1175 XYZ 1250 1440... (3 Replies)
Discussion started by: Diya123
3 Replies

7. Shell Programming and Scripting

Split single rows to multiple rows ..

Hi pls help me out to short out this problem rm PAB113_011.out rm: PAB113_011.out: override protection 644 (yes/no)? n If i give y it remove the file. But i added the rm command as a part of ksh file and i tried to remove the file. Its not removing and the the file prompting as... (7 Replies)
Discussion started by: sri_aue
7 Replies

8. Shell Programming and Scripting

Unix Script file to Append Characters before rows in file.

Hi Experts, I am working on HP-UX. I am new to shell scripting. I would like to have a shell script which will prefix: 1. "H|" before first row of my file and, 2. "T" for all other rows of the file. For Example - File before running the script 20100430|4123451810|218.50|TC 20100430 ... (4 Replies)
Discussion started by: phani333
4 Replies

9. Shell Programming and Scripting

Split rows

Hi all, I need a simple bin/sh script FILE1: ab1 gegege swgdeyedg ac2 jxjjxjxjxxjxjx ad3 ae4 xjxjxj zhzhzh ahahs af5 sjsjsjs ssjsjsjsj sjsjsj ag6 shshshshs sjjssj shhshshs has to return: ROW ab1 ROW ac2 ROW ad3 ROW ae4 In other words: "ROW " + the first world... (3 Replies)
Discussion started by: ric79
3 Replies

10. Shell Programming and Scripting

split rows

Hi I wanted to split rows based on the number of 1's present in 21st field(21st field is 40 length field) so I wrote the below awk code. However, the tool that I am using to invoke the command is not recognising the command. So, could you please help me to translate this command to sed? awk... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

Featured Tech Videos