3

I have encountered a strange issue in concatenating the files using the cat command. I have two files with one string in each of them:

file1:

ABC

file2:

DEF

Either I do cat file1 file2 or I do cat file1 >> file2. I expect an output as shown below:

ABC  
DEF

However, I have a funny output like this:

ABCDEF

I have checked the files and there is no extra space or characters. However, when I do manual deleting from the back of the string, I don't even see a single character. It works fine. I suppose there must be some kind of "hidden" character or line which I can't see.

It has been bugging me a lot because I have tons of files to concatenate. I can't do the same thing manually.

Any help is appreciated.

2
  • 5
    file1 does not have a trailing newline \n - (perhaps file2 is the same). Test file1 via: hexdump file1, and look for 0a at the end. 0a is the hexadecimal value of the newline character
    – Peter.O
    Jul 12, 2015 at 12:40
  • ed "$file" <<< w will write trailing new lines to a file lacking one.
    – kojiro
    Jul 12, 2015 at 23:28

5 Answers 5

5

paste is probably the easiest (which is not to mention extremely efficient) means at your disposal to handle this problem.

printf abc >file1
printf def >file2
paste -sd\\n file[12]

abc
def

When paste is invoked -serially it will read each of its named input files in-turn and paste the output of each line within each file on either a <tab> or a specified -delimiter string. While paste will always end its output of each named infile with a \newline, here the -delimiter is also specified to be a \newline, and so it will just basically cat its input to output with the exception that each file will always end in a \newline.


As Peter points out below, an empty file can cause paste to emit an extra \newline though. If this is an issue, practically the same method might be applied with sed which will not do so:

: > file0
sed '' file[012]

abc
def

Now with this method, though, (with GNU sed at least) there may be a different problem. Any sed will always write out a \newline before pulling in another line, but if it is the very last line of the whole concatenated series of input files, then some seds (such as GNU) may not append a newline at the end. For example, with my input files, the def is not followed on by a newline.

And if that is a problem, well...

sed '' file[012] | paste -sd\\n

...the above pipeline should probably cover all your bases.

1
  • 1
    Great! - with one caveat: an empty file will introduce a spurious blank line into the output. +1
    – Peter.O
    Jul 12, 2015 at 22:09
1

As Peter says, you first file does not have an end-of-line character. You can probably check it with ls -l --- if it's exactly three chars, this is it.

If you want to "cat" the files adding a newline only if the newline wasn't there, you can use the nice trick explained here.

If you have this three files:

[romano:~/tmp] % ls -l f1 f2 f3
-rw-rw-r-- 1 romano romano 3 Jul 12 14:58 f1
-rw-rw-r-- 1 romano romano 4 Jul 12 15:03 f2
-rw-rw-r-- 1 romano romano 4 Jul 12 15:03 f3
[romano:~/tmp] % cat f1 f2 f3
ABCDEF
GHI

where f1 has no end-of-line in the last line, while the others do, you can just do:

[romano:~/tmp] % sed -e '$a\' f1 f2 f3
ABC
DEF
GHI

... sed is a Stream EDitor, and you are instructing it to print all unchanged and the last line adding nothing --- but sed implicitly add a newline when operates, so it solves the problem.

Notice that just using the cat + echo thing will add a newline always. so you'll have two where you had one:

[romano:~/tmp] % for i in f?; do cat $i; echo; done;
ABC
DEF

GHI

[romano:~/tmp] %
1

It seems that your file1 has no trailing newline. If you want to concatenate a list of files. You can first check each one and cat a newline where needed, as follows:

# make some sample files
printf "%s\n" abc > file1  
printf "%s"   def > file2  # no trailing newline
printf "%s\n" ghi > file3
printf "%s"   jkl > file4  # no trailing newline

# find files to concatenate and build a sorted array `f[]`
unset f i; 
while IFS= read -r -d $'\0' path; do f[i++]="$path"
done < <(find . -type f -name 'file[0-9]' -print0 | sort -z)

# build the `cat` command
cmd=cat
tmp="$(mktemp)"; echo >$tmp   # a file which contains only `\n`
for file in "${f[@]}"; do
    lasthex=$(tail -c1 $file | hexdump -ve '1/1 "%02x"')
    [[ -z $lasthex ]] && continue     # skip enpty files
    [[ $lasthex == 0a ]] && nl= || nl=" $tmp"
    cmd="$cmd \"$file\"$nl"
done

# execute the `cat` command
eval "$cmd"

The concatenated result is:

abc
def
ghi
jkl

The generated command is:

cat "./file1" "./file2" /tmp/tmp.z7iKccY0T9 "./file3" "./file4" /tmp/tmp.z7iKccY0T9
0

You have missing newlines. You could do

cat file1; echo; cat file2;

or create a file which has only one linebrake and do

cat file1 NEWLINEFILE file2

or do this with a loop like this

for i in find[1-2]; do cat $i; echo; done;

to run through *, cat each file and add a newline afterwards.

With $(find OPTIONS) instead of find[1-2], you could refine your selection.

0

You can see special characters in a file with od -c filename. Newlines will look like \n. If Windows or source control changed the newlines, you will see \r\n using od -c, but you won't see anything different with cat. You can use dos2unix filename to repair those.

Sometimes you will see a sequence of control characters that cat would print out to look like a "-" or some other valid character.

1
  • 1
    The suggestion to use od to inspect files is indeed a good one. I removed the part about this not being an answer because it doesn't add anything useful (on Stack Exchange sites, answers that aren't answers should be deleted). Also, note that this question deals with files not ending in a newline, which aren't usually related to file conversion across platforms - tools as dos2unix won't actually help in this case.
    – fra-san
    Sep 17, 2020 at 20:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .