0

I have 100's of files in a directory that contain lines of text. I want to concatenate all the lines in a file into a single line. I would like to do this for all files in a single pass. cat does not seem to work here

Example

File_1

atagacat
agataga
ctcatctat
tagcatag

File_1_cat

atagacatagatagactcatctattagcatag

File_2

atagacat
agataga
ctcatctat
tagcatag

File_2_cat

atagacatagatagactcatctattagcatag
  • Your question is not clear. You want to add all the files in single file using cat? – Raghvendra Mar 2 '17 at 7:22
  • No I want to concatenate all lines together into a single line for each file. I want to keep the files separate. The final product will be 100s of files with one line each. @ Raghvendra – Josh Mar 2 '17 at 8:31
  • ok. now i understand your problem. – Raghvendra Mar 2 '17 at 15:54
4

Simple cat + tr should be enough. If not maybe something is wrong with your system:

sh-4.2$ cat file1 file2 
atagacat 
agataga 
ctcatctat 
tagcatag 
atagacat 
agataga 
ctcatctat 
tagcatag
sh-4.2$ cat file1 file2 |tr -d '\n'
atagacatagatagactcatctattagcatagatagacatagatagactcatctattagcatag

Test it online here: http://www.tutorialspoint.com/execute_bash_online.php?PID=0Bw_CjBb95KQMdUM4UER3SzE5Sk0

Update

After your clarification in comments that you need to remove new lines from each file separately and not join all files together (ps: don't put usefull clarifications in comments - edit your main question instead) , you can use something like :

perl -pe 's/\n//g' file >file_cat #perl -pe is directly equivalent to `sed`.  

You can also use perl -pe to make batch replacements in more files at once:

perl -pe 's/\n//g' -i file{1..100) #-i : in-place replacements in each file
OR
perl -pe 's/\n//g' -i.bak file{1..100) #in-place replacement keeping also a backup of original file

ALSO this works for all txt files in a directory
perl -pe 's/\n//g' -i.bak *.txt #in line replacement of all txt files in current directory keeping also a back up file

You can offcourse use a kind of loop like mentioned in other answers or like this:

while IFS= -r -d '' fname;do 
  perl -pe 's/\n//g' "$fname" >"${fname}_cat"
  #any other command you might need like echo,cat,whatever
done < <(find . -maxdepth 1 -type f -name 'file*' -print0)
  • The second bit is what the OP wants, but for each file separately. – Kusalananda Mar 2 '17 at 9:46
  • @Kusalananda Yeap, it was a clarification came in after my answer though. I will update. – George Vasiliou Mar 2 '17 at 9:49
2
for name in ./*; do
  test -f "$name" && (tr -d '\n'; echo) <"$name" >"${name}_cat"
done

This will remove all newlines in all regular files (separately) in the current directory (this is what tr -d '\n' does). The echo will make sure that there is a newline at the end of the line in each generated file.

1

Don't use tr for that. tr -d '\n' removes all the newline characters. A non-empty file that has no newline character is not a text file. You need one newline character at the end for a sequence of characters to be considered as a proper line of text.

The canonical command to join all lines into a single one is the paste command:

 for file in ./*; do paste -sd '\0' "$file" > "${file}_cat"; done

That would do it for all the non-hidden files (regardless of their type -- regular, symlink, directory...) in the current directory. If there's no non-hidden file, in shells other than zsh, that would cause paste to give an error and create an empty *_cat file.

It would also create an empty file_cat files for files that can't be open. To avoid some of those problems, you may prefer:

for file in ./*; do
  [ ! -f "$file" ] || paste -sd '\0' - < "$file" > "${file}_cat"
done

instead (-f is to only consider regular files or symlinks to regular files), or better and shorter with zsh:

file (./*(N-.)) do paste -sd '\0' - < $file > ${file}_cat

(add the D glob qualifier to include hidden files).

-d '\0' is the standard way to specify an empty separator (it does not mean a NUL character separator). Some paste implementations support -d '' for that as well, though that's not portable (other paste implementations would complain about the missing delimiter).

0

You could use tr in a loop, this way:

for file in *; do
  [[ -f "$file" ]] || continue  # skip if not a regular file
  printf "%s\n" $(tr -d '\n' < "$file") > "$file"_cat
done

The \n in printf makes sure that we do add a single newline at the end of the concatenated line.

  • A slight issue with this is if the file is immensely big. – Kusalananda Mar 2 '17 at 10:04
  • Using the split+glob operator on the output of tr like that doesn't make any sense. – Stéphane Chazelas Mar 2 '17 at 13:23
0

With awk:

awk '{printf $0> (FILENAME"_cat")}' *

If you need new line at the end of new file:

awk '{printf $0> (FILENAME"_cat")}' *; sed -i -e '$a\' *_cat
  • 1
    The first argument of printf is the format, you must use printf "%s", $0 instead. – Stéphane Chazelas Mar 2 '17 at 13:22
0

Using Vim:

vim -Nesc 'bufdo! %j! | w %_cat' -c 'qa!' *
  • -Nesc <command> is used to run <command> in non-compatible, silent ex-mode, which is useful for scripting Vim.
  • bufdo! <command> runs the command on each buffer, not stopping for changed files
  • %j! joins all lines without inserting any spaces (just j adds spaces), % being all lines in a range context
  • w %_cat saves the current file to <filename>_cat, % being the current filename in a filename context
  • -c 'qa!' then finally, close without saving all files.
0

You could use awk for this:

for i in file*; do
    awk '{gsub(/[[:space:]]*/, "");printf $0}' $i > ${i}_cat
done

where gsub does the following (cited from the manual):

gsub(regexp, replacement [, target])

Search target for all of the longest, leftmost, nonoverlapping matching substrings it can find and replace them with replacement. The ‘g’ in gsub() stands for “global,” which means replace everywhere. For example:

{ gsub(/Britain/, "United Kingdom"); print }

replaces all occurrences of the string ‘Britain’ with ‘United Kingdom’ for all input records.

The gsub() function returns the number of substitutions made. If the variable to search and alter (target) is omitted, then the entire input record ($0) is used. As in sub(), the characters ‘&’ and ‘\’ are special, and the third argument must be assignable.

0
find . -maxdepth 1 -type f -exec perl -i -l -0777pe 'y/\n//d' {} +

. find picks up files from the current directory without recursing and hands them over to Perl in a bunch, which then goes ahead and strips them of newlines and stores back the changes within each file.

find . -maxdepth 1 -type f -exec sed -i -e '$!{N;s/^/\n/;D;};s/\n//g' {} +

Yet other methods to one-line a file is:

perl -i -lpe '$\ = eof ? "\n" : ""' yourfile
sed  -i -e   'H;$!d;g;s/\n//g'      yourfile

< yourfile sed 's/.*/[&]/' | dc -e '[[]pq]sq [?z0=q n z0=?]s? l?x'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.