Sort file data according to a custom list of string


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sort file data according to a custom list of string
# 15  
Old 01-15-2020
Quote:
Originally Posted by RudiC
... after eliminating duplicates in your iplist, and empty lines in your data:
...

I give up. Your new iplist is full of duplicates, your data have IPs not covered in iplist, and your data have at least one empty line responsible for the "bad array subscript" error (on stderr, by the way). I repeat: The script and its output were tested on a recent linux system using a correct, reasonably strucured *nix file as given in your first posts.



Did you use <TAB>s as the delimiter when echoing the results to cut (as can be easily seen necessary in man cut)?
# 16  
Old 01-15-2020
OK, (sigh), dealing with duplicates in iplist, empty lines in data, and IPs not in iplist. Be aware that this is NOT a turnkey solution handling any special case that you / your data sources may come up with but is a proof of concept that was adapted to the ever changing targets from post #1 to post #(n-1). No error handling etc. included (yet). I recommend reading additional documents (e.g. man pages) by yourself should not every pitfall be covered.



Code:
IFS=,
 declare -A IPASS
for i in $iplist;   do  [ ${IPASS[$i]} ] && echo "duplicate $i" || IPASS[$i]=${#IPASS[@]};   done
 MX=${#IPASS[@]}
{ while read LN
    do  [ "$LN" ] || continue
        [ ${#LN} -le 15 ] && {  RC=${IPASS[$LN]:-$MX}
                                unset IPASS[$LN]
                             }
        echo "$RC       $((++CNT))      $LN"
    done < file
  for i in ${!IPASS[@]}
    do  echo "${IPASS[$i]}      $((++CNT))      $i      not in data file"
    done
  echo "$MX     0                       not in iplist:"
} | sort  -k1,1n -k2,2n | cut -f3-
10.9.9.20    not in data file
10.9.9.40    not in data file
10.9.9.41
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgMTT.txt 4006929900
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgBatch.txt 3953673532
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgPostSrv.txt 1014547034
10.9.9.200   not in data file
10.9.9.30    not in data file
              not in iplist:
10.9.9.39
-rwxr-xr-x user1 2019-12-24 19:24 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-24 19:24 /tmp/TomChrgBatch.txt 2552148116
-rwxr-xr-x user1 2019-12-24 19:24 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-24 19:24 /tmp/TomChrgPostSrv.txt 1014547034
10.9.9.202
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgMTT.txt 4006929900
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgBatch.txt 3953673532
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgPostSrv.txt 1014547034
10.9.9.42
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgMTT.txt 4006929900
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgBatch.txt 3953673532
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-28 16:34 /tmp/TomChrgPostSrv.txt 1014547034
10.9.9.43
-rwxr-xr-x user1 2019-12-28 16:35 /tmp/TomChrgMTT.txt 4006929900
-rwxr-xr-x user1 2019-12-28 16:35 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-28 16:35 /tmp/TomChrgBatch.txt 3953673532
-rwxr-xr-x user1 2019-12-28 16:35 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-28 16:35 /tmp/TomChrgPostSrv.txt 1014547034
10.9.9.25
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgMTT.txt 4006929900
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomTDS.txt 1559501379
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgBatch.txt 3953673532
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrg.txt 1681340719
-rwxr-xr-x user1 2019-12-28 16:32 /tmp/TomChrgPostSrv.txt 1014547034


Last edited by RudiC; 01-15-2020 at 05:58 PM..
This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How do I custom sort the files in a directory using the filenames in a text file.?

Hi all, (5 Replies)
Discussion started by: KMusunuru
5 Replies

2. Shell Programming and Scripting

Sort and Split file with header and custom name

Hi, I am using SUN SOLARIS (SunOS sun4v sparc SUNW, T5240). I have a huge data file with header and trailer. This file gets used into an ETL process. ETL skips the header record (which is the first record of the file) and loads the rest of the record. The file can be delimited (comma,... (5 Replies)
Discussion started by: Saanvi1
5 Replies

3. Shell Programming and Scripting

Sort data file by case

Hello, I'm trying to sort a large data file by the 3rd column so that all of the first words in the 3rd column that are in all uppercase appear before (or after) the non uppercase words. For example, Data file: xxx 12345 Rat in the house xxx 12345 CAT in the hat xxx 12345 Dog in the... (4 Replies)
Discussion started by: palex
4 Replies

4. UNIX for Dummies Questions & Answers

Custom sort on a column

Hello all, How do I achieve this? I have A, B and A/B in different variables in a file in col2. I want them to sort in such a way, that the variables appear together, and within a variable, the data is sorted in the order A,B and then A/B. If I sort on the second column, the order becomes A,... (6 Replies)
Discussion started by: senhia83
6 Replies

5. Shell Programming and Scripting

Sort help: How to sort collected 'file list' by date stamp :

Hi Experts, I have a filelist collected from another server , now want to sort the output using date/time stamp filed. - Filed 6, 7,8 are showing the date/time/stamp. Here is the input: #---------------------------------------------------------------------- -rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies

6. Shell Programming and Scripting

Populating File data with custom manipulation on file names

Hi, I am confused how to proceed firther please find the problem below: Input Files: DCIA_GEOG_DATA_OCEAN.TXT DCIA_GEOG_DATA_MCRO.TXT DCIA_GEOG_DATA_CVAS.TXT DCIA_GEOG_DATA_MCR.TXT Output File Name: MMA_RFC_GEOG_NAM_DIM_LOD.txt Sample Record(DCIA_GEOG_DATA_OCEAN.TXT):(Layout same for... (4 Replies)
Discussion started by: Arun Mishra
4 Replies

7. UNIX for Dummies Questions & Answers

List directories with given string, sort by creation date

It is for HP-Unix B.11.31. Requirement: 1. List the directories, which directories name has given particular string. Example: Directories with name "Build" 2. On the output of 1. list the directories by creation date as sort order. I tried with; find . -type d -name "Build*" ... (3 Replies)
Discussion started by: Siva SQL
3 Replies

8. Shell Programming and Scripting

find string(s) in text file and nearby data, export to list help

Hi, So I'm kinda new to shell scripts and the like. I've picked up quite a bit of use from browsing the forums here but ran into a new one that I can't seem to find an answer for. I'm looking to parse/find a string AND the next 15 or so charachters that follow the string within a text file... (1 Reply)
Discussion started by: kar23me
1 Replies

9. Shell Programming and Scripting

Sort a big data file

Hello, I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field. I'm trying to sort with "sort" command and it brings me 6 minutes. I have tried with some transformation methods in perl but it results "Out of memory". I was... (2 Replies)
Discussion started by: rubber08
2 Replies

10. Shell Programming and Scripting

script to sort a string of numerical data in set fields

So, I will be working with someone and basically we are trying to build a form that is submitted most likely via the web and the data is just a string of numbers. like: 19383882872201110929282821818182827349190102837364718191001932873711 Now, each number is part of a numerical value of... (4 Replies)
Discussion started by: tlarkin
4 Replies
Login or Register to Ask a Question