Remove or truncate trailing nulls from file

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Remove or truncate trailing nulls from file
# 1  
Old 06-10-2016
Remove or truncate trailing nulls from file

I want to remove from a file the trailing null characters (0x00) and stop doing so when a different byte is found (which should not be deleted), and either put the result to the same file or a different one.

Any ideas?
# 2  
Old 06-10-2016
Try:
Code:
tr -d '\0' < file_with_nulls > file_without_nulls

Note that the filename you use for the output file must NOT be a name of your input file.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 06-10-2016
Quote:
Originally Posted by Don Cragun
Try:
Code:
tr -d '\0' < file_with_nulls > file_without_nulls

Note that the filename you use for the output file must NOT be a name of your input file.
That command removes all nulls. I need to remove only the ones that are at the end of the file just after the last non-null character, not in the middle of the file.
# 4  
Old 06-11-2016
I misunderstood your requirements. Even though you said you wanted to remove trailing nulls, the way you said that you wanted to stop removing nulls when a different byte was found sounded like you wanted to stop removing null bytes when a non-null byte was found after a string of one or more null bytes. Just removing trailing NUL bytes could be done in a shell script using a combination of something like od and grep to find the address of the last non-NUL byte in a file and dd to truncate a file to that length, but a relatively simple C program is probably easier. If you save the following in dtn.c:
Code:
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <unistd.h>

char	buf[8192];	// I/O buffer
int	ec;		// exit code

// pe(format_string, format_string_argument, exit_code_modifier);
void
pe(	const char	*fmt,
	const char	*arg,
	int		ecm) {
	int	serrno;	// hold area for errno

	serrno = errno;
	snprintf(buf, sizeof(buf), fmt, arg);
	errno = serrno;
	perror(buf);
	ec |= ecm;
}

// NAME		dtn -- Delete trailing null bytes.
//
// SYNOPSIS	dtn file...
//
// DESCRIPTION	Delete trailing NUL bytes from each file named as an operand.
//		File are updated in place.
//
// OPERANDS
//	file	A pathname of a file to be truncated to have a length that does
//		not include any trailing NUL bytes.
//
// INPUT FILES	The input files must be regular files.
//
// STDERR	The standard error shall be used only for diagnostic messages.
//
// EXIT STATUS
//	0	All input files were successfully processed.
//	>0	An error occurred.
//
// CONSEQUENCES OF ERRORS
//		Default.

int
main(	int	argc,
	char	*argv[]) {

	ssize_t	buflen;	// number of bytes in buf[]
	int	fd,	// file descriptor
		i,	// loop control
		j;	// loop control
	off_t	nsize,	// new file size
		size;	// current file size

	for(i = 1; i < argc; i++) {
		if((fd = open(argv[i], O_RDWR)) == -1) {
			pe("Can't open \"%s\":", argv[i], 1);
			continue;
		}
		nsize = size = 0;
		while((buflen = read(fd, buf, sizeof(buf))) > 0) {
			for(j = 0; j < buflen; j++) {
				size++;
				if(buf[j])
					nsize = size;
			}
		}
		if(buflen) {
			pe("Read error on \"%s\": file will not be truncated:",
			    argv[i], 2);
		} else if(ftruncate(fd, nsize)) {
			pe("Truncation failed on \"%s\":", argv[i], 4);
		}
		close(fd);
	}
	return ec;
}

and then run make dtn to build it, you should have a utility (named dtn) that you can invoke with any number of file operands you want and it will remove trailing NUL bytes from each of those files. (You will need read and write access to each file to do this.) I don't claim that it is highly efficient (it reads from the start of the file noting the offset of the last found non-NUL byte instead of reading from the end and searching for a non-NUL byte), but its performance should be reasonable for most regular files.
This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 06-11-2016
Thank you very much, it works perfectly.
# 6  
Old 06-11-2016
It looks like (even though we are not dealing with *nix text files due to missing <NL> char at the end) GNU sed could do it, not with octal constants, but with hex constants:
Code:
sed ':L;s/\000$//;tL' XX | hd
00000000  73 64 66 65 66 65 65 72  76 30 30 09 00 00 00 00  |sdfefeerv00.....|
00000010  00 00 00 00 00 00 00 00  00 00 00                 |...........|
sed ':L;s/\x00$//;tL' XX | hd
00000000  73 64 66 65 66 65 65 72  76 30 30 09              |sdfefeerv00.|

FreeBSD's sed doesn't work like above.
These 2 Users Gave Thanks to RudiC For This Post:
# 7  
Old 06-11-2016
Not only are non-empty POSIX text files required to end with a <newline> character, they are not allowed to contain any NUL bytes either.

If GNU sed works with:
Code:
sed ':L;s/\x00$//;tL' file

does it also work with:
Code:
sed 's/\x00*$//' file

???
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove leading and trailing spaces from a file

Hi, I am trying to remove leading and trailing spaces from a file using awk but somehow I have not been able to do it. Here is the data that I want to trim. 07/12/2017 15:55:00 |entinfdev |AD ping Time ms | .474| 1.41| .581|green |flat... (9 Replies)
Discussion started by: svajhala
9 Replies

2. Shell Programming and Scripting

Remove the leading and trailing date from a CSV file

I'm a newbie to shell scripting. Can anyone help with the below requirement ? The leading and trailing date of a files to be removed. 2017-07-12_gmr_tag_log_20170711.csv 2017-07-12_gmr_call_log_20170711.csv 2017-07-12_gmr_outgoing_log_20170711.csv I'm looking for output like... (7 Replies)
Discussion started by: shivamayam
7 Replies

3. Shell Programming and Scripting

Remove trailing space from file and folder names

I have a folder that contains many sub folders and files. This tree has to be backed up to an archive system. According to the tech support, one of the archives is failing to back up due to the possibility of trailing spaces on file and folder names. Therefore, I would like to have a script... (16 Replies)
Discussion started by: vipertech
16 Replies

4. Shell Programming and Scripting

Remove trailing number

I have some strings such as ABC1 ABC2 TYFASDD12 They will only have letters and numbers. In each case I want to remove the last digit? The lengths will vary. So a hard coded substr won't work. What do I do? if it doesn't end in a number, I don't want to remove any characters. (6 Replies)
Discussion started by: guessingo
6 Replies

5. Shell Programming and Scripting

Remove trailing 0 from the field

Hi Freinds, I have file1.txt as below file1.txt 1521894~~-0.400~201207 1521794~~-0.486~201207 152494~~-0.490~201207 152154894~~-0.490~201207 1521894354~~-0.489~201207 expected output : 1521894~~-0.4~201207 1521794~~-0.486~201207 152494~~-0.49~201207... (9 Replies)
Discussion started by: i150371485
9 Replies

6. Shell Programming and Scripting

Remove trailing zeros

Hi I have a simple request but can't find the answer. I want to remove trailing zeros, and in some cases the fullstops, from the input data. Example of input file: FR002_15.000_20.000 SD475_5.000_10.500 FG5647_12.250_15.500 BH2463_30.555_32.000 Desired output file would be: ... (10 Replies)
Discussion started by: theflamingmoe
10 Replies

7. Shell Programming and Scripting

Remove trailing spaces from file

I'm currently writing my sql results to a file and they have trailing spaces after each field. I want to get rid of these spaces and I'm using this code: TVXTEMP=$(echo $TVXTEMP|sed -e 's/\ //g') It doesn't work though. I'm not familiar with sedscript, and the other codes I've found online... (6 Replies)
Discussion started by: avillanueva
6 Replies

8. UNIX for Dummies Questions & Answers

I don't want to truncate trailing spaces and ^M at the end of line

I have a script wherein I access each line of the file using a FOR loop and then perform some operations in each line. The problem is each line that gets extracted in FOR loop truncates trailing blank spaces and control characters (^M) that is present at the end of each line. I don't wan this to... (5 Replies)
Discussion started by: Shobana_s
5 Replies

9. Shell Programming and Scripting

Remove trailing G

Hello, I am trying to write a script that will calculate the amount of data remaining in a storage volume. I'm running Tru64 Unix version 5.1B patch kit 6. The script is being run against an AdvFS domain. I am programming in Korn Shell version M-11/16/88f. The basic idea is that I want to run df... (3 Replies)
Discussion started by: Heathe_Kyle
3 Replies

10. UNIX for Dummies Questions & Answers

How to remove trailing spaces

Hi, I have a file like this (ADD_MONTHS((Substr(Trim(BOTH FROM Translate(Maximum(closeDa ------------------------------------------------------------ 2007-06-30 00:00:00 I have a requirement where i need just the date. When i do: tail -1... (2 Replies)
Discussion started by: mahek_bedi
2 Replies
Login or Register to Ask a Question