10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hi,
I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues.
If the record delimiter is unix new line, I could use split command either with option l or b.
The problem is that the line terminator is |##|
How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies
2. UNIX for Beginners Questions & Answers
Dear Users,
Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file
input file.txt
scaffold1 928 929 C/T +
scaffold1 942 943 G/C +
scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
3. Shell Programming and Scripting
Hello All,
I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using
sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies
4. Shell Programming and Scripting
Hi Experts,
I am new to shell scripting. Need some help in doing one task given by the customer.
The sample record in a file is as follows:
3538,,,,,,ID,ID1,,,,,,,,,,,
It needs to be the following:
3538,,353800,353800,,,ID,ID1,,,,,COLX,,,,,COLY,
And i want to modify this record in... (3 Replies)
Discussion started by: sugarcane
3 Replies
5. Shell Programming and Scripting
Hi All,
I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix.
ex.
file1.txt is 1000 records
nikki1
nikki2
nikki3
what i want to do is delete the nikki2 record in file1.txt. is it possible?
Please advise,
Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies
6. Shell Programming and Scripting
Hello
i have a big file with a specific format and delimiter is "§" :
§field1§$field2§$field3§$field4§$field5§$field6§$field§
in this file we have a field which are very long (more than 20000 chars !!!!)
so through vi i cant manipulate them.
despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies
7. Shell Programming and Scripting
I have an extremely large csv file that I need to search the second field, and upon matches update the last field...
I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies
8. Shell Programming and Scripting
I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies
9. UNIX for Advanced & Expert Users
while running a script to process a file , it ended in termination . I able to find from the logs that it ended due to the error : fatal: record too large 20480 . Then I tried lots to fix the issue and left in vain . Can any one help me in finding reason for it ? . If you provide me a solution , I... (1 Reply)
Discussion started by: sakthifire
1 Replies
10. UNIX for Dummies Questions & Answers
Hi ,
I have a peculiar case, where my sed command is working on a file which contains lines of small length.
sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt
when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine.
when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies