18

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently using awk or other linux programs/commands specifically for large files?

input.txt

(s,2,4,5,6)
"s,1,5,5,2"
{z,0,4,5,3}
[y,2,4,5,5]
(y,4,4,5,7)
(r,20,4,5,7)
(e,9,4,5,2)

Expected output.txt

s,2,4,5,6
s,1,5,5,2
z,0,4,5,3
y,2,4,5,5
y,4,4,5,79
r,20,4,5,7
e,9,4,5,2
Improve this question

5 Answers 5

Reset to default
22

Another way just for the heck of it:

rev input | cut -c2- | rev | cut -c2-

(Note: with GNU cut, it only works for characters made of only one byte (as in your example)).

Improve this answer
5
  • Nice! This is significantly faster than the sed and awk solutions proposed so far.
    – Gilles 'SO- stop being evil'
    Jan 10, 2013 at 23:13
  • 1
    I proposed this answer for people afraid of sed/awk/regex syntax, but wouldn't have imagined it's faster, especially for large files, with three pipes and passing the entire contents through each one. Would have thought sed or awk reading a line at a time would be more efficient for large files.
    – Drake Clarris
    Jan 14, 2013 at 13:35
  • 5
    Guess that's what 40+ years of optimization of many of these *nix utilities will get you!
    – Drake Clarris
    Jan 14, 2013 at 13:36
  • @Gilles, it's faster to GNU sed in utf8 locales for some forms of input, and it depends whether you're considering wall clock time, or CPU time. ssed or the Heirloom toolchest sed can achieve better performance.
    – Stéphane Chazelas
    Jan 18, 2013 at 15:09
  • @Gilles Theres no MAN entry for rev in Solaris 5.10. I ended up using sed
    – ayrton_senna
    Jun 12, 2015 at 18:22
14

As per you question delete last and first word from input file as below :

sed 's/.$//; s/^.//' inputfile
Improve this answer
4
  • It would be cool if you could benchmark these against the other solution, s/.\(.*\).$/\1/. It might be faster on account of not using backreferences, and the question did mention "large files".
    – l0b0
    Jan 10, 2013 at 13:52
  • 4
    @l0b0 I tested with time yes | head -n 10000000 | COMMAND >/dev/null. I get rev input | cut -c2- | rev | cut -c2- → 0.14s, sed 's,.\(.*\).$,\1,' → 3.38s; awk '{print substr($0,2,length()-2);}' → 3.50s; sed 's/.$//; s/^.//' → 5.09s.
    – Gilles 'SO- stop being evil'
    Jan 10, 2013 at 23:09
  • @Gilles +1 That should be an answer.
    – l0b0
    Jan 11, 2013 at 7:58
  • 2
    @Gilles, that's very short lines. I find that for 30 character wide lines, @RahulPatil's solution is 3 times as fast with GNU sed than @juampa's. Also. sed 's/.\(.*\)./\1/' appears to be faster than sed 's/^.\(.*\).$/\1/' (GNU sed again). Also, the performance is dependant on the locale (interpretation of what a character is) and the sed implementation (in that regard, sed from the heirloom toolchest is considerably faster than GNU sed).
    – Stéphane Chazelas
    Jan 18, 2013 at 15:00
7

There are many possibilities, as always

sed 's,.\(.*\).$,\1,g' your_file

Explanation

  • , -- the sed delimiter, can be any other character as well, given it is escaped where-ever needed.
  • . Match a single character
  • \(.*\) - Group the remaining part, and this is stored to be retrieve further.
  • . Match a single character again
  • $ - End of line
  • \1 - output the text matched by group above
  • g replace globally on the line.
Improve this answer
2
  • 2
    Why g? there will be only one match per line.
    – njsg
    Jan 10, 2013 at 10:07
  • Note that it won't remove anything from lines that have fewer than 2 characters.
    – Stéphane Chazelas
    Jan 18, 2013 at 14:46
5 
tr -d '()[]{}"' < your_file

This should work as well. It well "translate" each of the characters to nothing (delete).

The downside is that it will delete them if they are not the first/last character as well. It will also miss any ending characters you don't list in the ()[....

Improve this answer
4

You can also do it with awk if you prefer

awk '{print substr($0,2,length()-2);}' input.txt > output.txt
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .