UNIX for Dummies Questions & Answers

Hi there, shameful Linux Newbie here

I was wondering if you could help with my problem...
I have plenty of files I'd like to concatenate. I know how to basically use cat command but that won't be enough from what I need : excluding the last xx bytes from files before assembling since there's some redundancy


the interesting part is the xx bytes to remove is determined by the filename themselves, ie


file 1 is named something like 0-54548
file 2 is named something like 54475-648459
file 3 is named 648345-1269494
etc


so for file 1 I would have to drop bytes between 54475 till the end before joining file 2. etc
it would be easier if it was always the exact same bytes value to remove , alas there are some variations, so it must be calculated from filenames.


I feel like it's possible to script something with not much lines within a loop, but my unix knowledge is way too rudimentary for now ^^


I also read about dd command which could probably be of some help to generate the files without the unwanted part, now I'm a bit clueless about extracting character chains from filenames, turning into numeric values, doing the maths and use this dd or whatever else command so I can assemble the resulting files...

Telling us that filenames are something like x-y where x and y are strings of digits isn't enough for us to figure out how to determine which file is file1, file2, ...

Give us details.

ok, so :
1 - "strings of digits" are bytes positions from the final file I want to recreate. so what I call files "1" "2" etc are exactly in the same order as these string digits in filenames .
if one file is called 0-15000, and another is called 14400-30000, another is called 29800-40000 and a last one 39750-55421
, then it means precisely I want to create a file from byte 0 to byte 55421 minus the redundant bytes (14400-15000 are at the end of 0-15000 and the beginning of 14400-30000, and so on...)

2 yes ("just two strings of digits separated by a single minus sign")

3 let's call it 0-highest digit, or "final" it that's too complicated, I don't mind ^^

4 I'm afraid I'm too newbie to even know what are the different shells. All I know is I use my Xfce4-terminal on my Xubuntu 13.10 to type echo $SHELL , I get "/bin/bash" if that's of any help...

This isn't highly efficient (since it will copy the redundant bytes to the output file multiple time, doesn't try to align block sizes to disk block boundaries, uses small input block sizes if you have a large file that starts at a small [but non-zero] offset), but it seems to do what you want. You can make it considerably more complex to verify that the ranges of bytes specified by the input files don't leave any holes in the output file and to avoid copying duplicated data more than once. The checks for names that contain a "-" but are not just a string of digits followed by a "-" followed by a string of digits could be simplified with ksh and bash specific constructs. But the constructs used here should be portable to any shell that handles basic POSIX shell parameter expansion requirements correctly.

This is too simplistic to work if you want to process a file with a starting offset that is close to your process' maximum available address space. (In other words it probably won't work for terabyte sized files where the starting offset in one or more of your input files is relatively large.) But, it should give you a starting point for a more advanced script:

Code:

#!/bin/ksh
IAm=${0##*/}
of="final.$$"
> $of   # create zero-length temporary output file
maxe=0
for i in *-*
do      printf "Looking at \"%s\"\n" "$i"
        b=${i%%-*}
        e=${i##*-}
        if [ -z "$b" ] || [ -z "$e" ] || [ "$i" != "$b-$e" ] ||
                [ "$b" != "${b#*[!0-9]}" ] || [ "$e" != "${e#*[!0-9]}" ]
        then    continue
        fi
        if [ $e -gt $maxe ]
        then    maxe=$e
        fi
        if [ $b -gt 0 ]
        then    seek="ibs=102400 obs=$b seek=1"
        else    seek='bs=10240'
        fi
        echo starting dd if=$i of=$of $seek conv=notrunc
        dd if=$i of=$of $seek conv=notrunc
done
if [ $maxe -gt 0 ]
then    printf "Creating 0-%d\n" $maxe
        mv $of 0-$maxe
        exit
fi
rm $of
printf "%s: No input files found; no output file created.\n" "$IAm" >&2
exit 1

I use the Korn shell, but this script will work with any POSIX conforming shell without changing anything other than the first line in the script to specify your shell.

fantastic, works perfect at first try, thanks a lot