Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

For loop to get rid of first 2 lines(conditionally)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting For loop to get rid of first 2 lines(conditionally)
# 1  
Old 02-26-2015
For loop to get rid of first 2 lines(conditionally)
hello all,

I get data from different vendors and need to clean it up. Usually it pretty straight forward when i have files that only have headers....but in my case i have files that have a starting line of file name(and some junk info) and 2nd line is headers and the 3rd line is were the actual data starts....

all is good. I do below to get rid of the first 2 lines(junk info and headers)..

Code:
for f in `ls -ltr file* | awk '{print $9}'`; do echo "$(tail -n +3 $f)" > $f; done

the above works just fine except now i am getting some data from vendors that ONLY have headers(file 4.txt below), so if i use the above it will also get rid of my data line as well...so how can i modify my simple for loop that will only get rid of headers(first line) in file4.txt and in other files get rid of first 2 lines...do note most of the data line do start out with a date(not sure if that can help)

hope it make sense...below is a sample file listing


cat file1.txt
Code:
File-name---JUNK
headers
data line 1(starting with date format of something like 8/4/2013)
data line 2
....


cat file2.txt
Code:
File-name---JUNK
headers
data line 1(starting with date format of something like 8/4/2013)
data line 2
....


cat file3.txt
Code:
File-name---JUNK
headers
data line 1(starting with date format of something like 8/4/2013)
data line 2
....


cat file4.txt
Code:
headers
data line 1(starting with date format of something like 8/4/2013)
data line 2
....

Last edited by vgersh99; 02-26-2015 at 11:35 AM.. Reason: code tags, please!
# 2  
Old 02-26-2015
Code:
for f in file* ; do awk '/headers/{p=1;next}p' $f > $f_new; done

# 3  
Old 02-26-2015
i get below error when i use the above...

Code:
[root@server1 user]# for f in file* ; do awk '/headers/{p=1;next}p' $f > $f_new; done
-bash: $f_new: ambiguous redirect
-bash: $f_new: ambiguous redirect
-bash: $f_new: ambiguous redirect
-bash: $f_new: ambiguous redirect
-bash: $f_new: ambiguous redirect
[root@server1 user]#
[root@server1 user]# cat file.txt
File-name---JUNK
headers
8/4/2013, 10/12/2014
data line 2
data line 3
data line 4
data line 5
[root@server1 user]# cat file2.txt
File-name---JUNK
headers
8/4/2013, 10/12/2014
data line 2
data line 3
data line 4
data line 5
[root@server1 user]#

also now that i think about it, i think using SED would be faster by creating a new file...
Code:
sed '2d' file.txt > tmpfile; mv tmpfile file.txt

Last edited by Scrutinizer; 02-26-2015 at 12:03 PM.. Reason: code tags
# 4  
Old 02-26-2015
sorry:
Code:
for f in file* ; do awk '/headers/{p=1;next}p' $f > ${f}_new; done

# 5  
Old 02-26-2015
that does not seem to work either


Code:
[root@server user]# cat file1.txt
File-name---JUNK
headers
8/4/2013, 10/12/2014
data line 2
data line 3
data line 4
data line 5
[root@server user]# for f in file* ; do awk '/headers/{p=1;next}p' $f > ${f}_new; done
[root@server user]#
[root@server user]# cat file1.txt
File-name---JUNK
headers
8/4/2013, 10/12/2014
data line 2
data line 3
data line 4
data line 5
[root@server user]#

Last edited by Scrutinizer; 02-26-2015 at 12:41 PM.. Reason: CODE TAGS
# 6  
Old 02-26-2015
What is the content of file1.txt_new ?

and please use Code tags...
# 7  
Old 02-26-2015
okay that worked, sorry i was looking at wrong file i guess. didnt realized that it created a new file....

is there a way to do the same without creating a new file ?...just use the same file and also could you please explain what exactly is the below doing ?

'/headers/{p=1;next}p'
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Web Development

Getting Rid of Annoying Bootstrap Table Borders and Wayward Table Lines

Bootstrap is great; but we have had some issues with Bootstrapped <tables> (and legacy <fieldset> elements) showing annoying, wayward lines. I solved that problem today with this simple jQuery in the footer: <script> $(function(){ $('tr, td, fieldset,... (0 Replies)
Discussion started by: Neo
0 Replies

2. Shell Programming and Scripting

Picking up files conditionally

Hi I have a scenario: I have a directory say DIR1 (no sub directories) and have few files in that directory as given below: app-cnd-imp-20150820.txt app-cxyzm-imp-20150820.txt app-petco-imp-20150820.txt app-mobility-imp-20150820.txt app-mobility-imp-20150821.txt... (7 Replies)
Discussion started by: Saanvi1
7 Replies

3. Shell Programming and Scripting

get rid of unwanted lines

Hi , My output contains the below contents When i execute df -h | grep / | grep -v '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' i get output as below tpshghghg.fgh.com:/vol/scmvol1/scm-vobstore/WebArch.vbs 1.7T 1.3T 452G 74% /vob/WebArch... (6 Replies)
Discussion started by: ptappeta
6 Replies

4. Shell Programming and Scripting

Reading two lines in a while loop and combining the lines

Dear all, I have a file like this: imput scaffold_0 1 scaffold_0 10000 scaffold_0 20000 scaffold_0 25000 scaffold_1 1 scaffold_1 10000 scaffold_1 20000 scaffold_1 23283 and I want the output like this: scaffold_0 1 scaffold_0 10000 scaffold_0 10000 scaffold_0 20000... (6 Replies)
Discussion started by: valente
6 Replies

5. Shell Programming and Scripting

sed to conditionally delete multiple lines

Hello I'd like to remove any line in an output file that is preceded by one or more warning messages (each warning is on a separate line). Example : WARNING: Estimation of lower confidence limit of \rho failed; setting it to 0. 822 28447 ... (4 Replies)
Discussion started by: jossojjos
4 Replies

6. Shell Programming and Scripting

Loop through file to sum conditionally

Hi, I have a file with header, detail and trailer records. HDR|111 DTL|abc|100|xyz DTL|abc|50|xyz TRL|150 I need to add the values in 3rd field from DTL records. Using awk, I am doing it as follows: awk -F'|' '$1=="DTL"{a += $3} END {print a}' <source_file> However, I want to... (3 Replies)
Discussion started by: delta21
3 Replies

7. Shell Programming and Scripting

Conditionally delete last X lines

delete last X lines,which start with + example file: test1 test2 remove1 remove2 one liner shell is preferred. (8 Replies)
Discussion started by: honglus
8 Replies

8. Shell Programming and Scripting

How to get different lines using a for loop

Hi I am trying to write a small AWK program to extract 324 lines which have a pattern in the line numbers. For examples, I want to extract lines 2, 5, 7, 11 and lines like (2+662), (5+662), (7+662), (11+662) and (2+662+662), (5+662+662), (7+662+662), (11+662+662) and so on for 27 times from... (5 Replies)
Discussion started by: ananyob
5 Replies

9. UNIX for Dummies Questions & Answers

Conditionally joining lines in vi

I've done this before but I can't remember how. Too long away from vi. I want to do a search are replace, but I want the replace to be a join. Example see spot run see spot walk see spot run fast see spot hop %s/run$/<somehow perform a join with the next line>/g so the results... (0 Replies)
Discussion started by: ifermon
0 Replies

10. Shell Programming and Scripting

Get rid of first & last lines from STD_OUT

I have an application that executes, printing a bunch of text to the screen... only problem is that I don't want to see the first line (the internal command being run) or the last line (it says "done") is there something to chop off the first and last line simply through use of piping the... (2 Replies)
Discussion started by: jjinno
2 Replies
Login or Register to Ask a Question