Bash script help - removing certain rows from .csv file
Hello Everyone,
I am trying to find a way to take a .csv file with 7 columns and a ton of rows (over 600,000) and remove the entire row if the cell in forth column is blank.
Just to give you a little background on why I am doing this (just in case there is an easier way), I am pulling information from a PCAP into a .csv file and I only want to view the rows from the .csv file if it lists something in the http.host (forth column) entry (i.e. google.com). If that entry is blank because it is not a http.host website then I would like to remove the row. By doing this it would seriously cut down on the amount of rows I have to review to make sure my users are not visiting sites that they should now be.
So far my script looks like this:
_____
I ran the script on a current PCAP and it wors like a charm getting the information I need from a pcap file to a csv file unfortunately I am running into the aforementioned blank row situation as every entry does not list a value in the http.host cell. In fact of the over 600,000 I am guessing there are only several hundred rows that I need. So adding to the script above (or creating a new script if need be) to remove rows with a blank entry in the forth column of every row would be the perfect solution however I am not sure how to do that. The condition that needs to be met for the loop (assuming a loop is the solution) for the loop to stop would be for each of the 7 columns to be blank a.k.a. the row after the last of the 600,000+ entries.
Can anyone help me edit my current script and or write a new script to loop over (or otherwise remove) blank entries?
Thanks in advance!
Last edited by Franklin52; 04-24-2013 at 04:15 AM..
Reason: Please use code tags
Hi,
I have files with names in the following naming pattern,
Name.of.moviemoretext-moretext.mov
or
Name of moviemoretext-moretext.mov
And I would like to delete the and moretext-moretext leaving just the Name of movie.
The Name of movie will always be different and the year will... (4 Replies)
Hello,
I am supposed to process about 100 csv files. But these files have some extra lines at the bottom of the file. these extra lines start with a header for each column and then some values below. These lines are actually a summary of the actual data and not supposed to be processed. These... (8 Replies)
I have a file like below and want to use awk to solve this problem. The record separator is ">". I want to look at each record section enclosed within ">". Find the row with the 2nd and 3rd columns being 0, such as
10 0 0
I need to take the first number which in this case is 10. Then... (15 Replies)
Hi,
I am not sure if this has already been asked (I tried the search but the search was too broad). Basically I want to remove rows based on another file.
So file1 looks like this (tab seperated):
HHN 3 5 5
HUJ 2 2 1
JJJ 3 1 1
JUN 2 1 3
I have another file (file2)... (2 Replies)
Hi Guys,
I need to write a script, that exports the "moz_places" table of the "places.sqlite"-file (firefox browser history) into a csv-file. That part works. After the export, my csv looks like this:
...
4429;http://www.sqlite.org/sqlite.html;"Command Line Shell For... (11 Replies)
I have CSV file that contains data in the format as shown below:
ABC, 67, 56, 67, 78, 89, 76, 55
PDR, 85, 83, 83, 72, 82, 89, 83
MPG, 86, 53, 54, 65, 23, 54, 75
..
..
..
..
I want to create a script that will pull out the rows from the above sheet and paste it into another CSV file.... (12 Replies)
HI All,
I have a CSV file of 30 columns separated by ,. I want to get a count of all unique rows written to a flat file. The CSV file is around 5000 rows
The first column is a time stamp and I need to exclude while counting unique
Thanks,
Ravi (4 Replies)
Dear community,
maybe I'm asking the moon :rolleyes:, but I'm scratching my head to find a solution for it. :wall:
I have a file called query.out (coming from Oracle query), the file is like this:
ADDR TOTAL
-------------------- ----------
TGROUPAGGR... (16 Replies)
Hi,
I have a requirement where I need to delete given number of top and bottom rows in a flat file which has new line as its delimiter.
For ex: if top_rows=2 & bottom_rows=1
Then in a given file which looks like:
New York
DC
LA
London
Tokyo
Prague
Paris
Bombay
Sydney... (7 Replies)