11

Imagine an output of a command like

44444
55555
11111
22222
33333

how can I yank the first N lines (first two in the example above) and append them at the end, but without using temp file (so only using pipes)?

11111
22222
33333
44444
55555

Something along the lines of | sed -n '3,5p;1,2p' (which obviously doesn't work as sed doesn't care about the order of the commands).

  • 2
    Why we can't use a temp file? – Braiam Apr 20 '16 at 17:21
13

Just copy those lines to the hold buffer (then delete them) and when on last line append the content of hold buffer to pattern space:

some command | sed '1,NUMBER{           # in this range
H                                       # append line to hold space and
1h                                      # overwrite if it's the 1st line
d                                       # then delete the line
}
$G'                                     # on last line append hold buffer content

With gnu sed you could write it as

some command | sed '1,NUMBER{H;1h;d;};$G'

Here's another way with ol' ed (it reads the output of some command into the text buffer and then moves lines 1,NUMBER after the la$t one):

ed -s <<IN
r ! some command
1,NUMBERm$
,p
q
IN

Note that - as pointed out - these will both fail if the output has less than NUMBER+1 lines. A more solid approach would be (gnu sed syntax):

some command | sed '1,NUMBER{H;1h;$!d;${g;q;};};$G'

this one only deletes lines in that range as long as they're not the last line ($!d) - else it overwrites pattern space with hold buffer content (g) and then quits (after printing the current pattern space).

  • 2
    sed -e '1,NUMBER{H;1h;d;}' -e '$G' also works portably (note that some sed implementations cannot hold more than a few kilobytes in the hold space, so NUMBER can't be too big there) – Stéphane Chazelas Apr 20 '16 at 10:02
  • @StéphaneChazelas - thanks for the input - I usually go with one command per line as I know for sure that's portable - the multiple expressions syntax has always been a bit confusing to me e.g. the posix docs say "The <right-brace> shall be preceded by a <newline>" so according to them shouldn't that be sed -e '1,NUMBER{H;1h;d' -e'}' -e '$G' ? – don_crissti Apr 20 '16 at 10:19
  • 4
    Typically, a new -e replaces a newline. d;} is not POSIX yet but portable. That will be fixed in the next POSIX spec. See austingroupbugs.net/view.php?id=944#c2720 – Stéphane Chazelas Apr 20 '16 at 10:26
  • 2
    @don_crissti thank you! it would be cool if you could also include a short explanation of why it works. (Of course I'll go look it up, but it would make for a more valuable answer.) – Peter Uhnak Apr 20 '16 at 11:57
  • In my head, the unportable thing about 1,NUMBER{H;1h;d;} is not having a semicolon immediately after the opening brace. That might just have been a bug in SunOS 4.1 sed whose workaround is still wired into my fingers 20 years later, though. – zwol Apr 21 '16 at 0:51
11

An awk approach:

cmd | awk -v n=3 '
  NR <= n {head = head $0 "\n"; next}
  {print}
  END {printf "%s", head}'

One benefit over @don_crissti's sed approach is that it still works (outputs the lines) if the output has n lines or fewer.

  • You may want to replace the hardcoded \n with ORS, so that this would work with other record separators (say you want to use paragraphs, etc). – fedorqui Apr 20 '16 at 11:09
6

I have xclip and with it this can be done in this way:

./a_command | xclip -in && xclip -o | tail -n +3 && xclip -o | head -n 2

Here is its description:

xclip - command line interface to X selections (clipboard)

NAME
       xclip - command line interface to X selections (clipboard)

SYNOPSIS
       xclip [OPTION] [FILE]...

DESCRIPTION
       Reads from standard in, or from one or more files, and makes the data available as an X selection for pasting into X applications. Prints current X selection to standard out.

       -i, -in
              read text into X selection from standard input or files (default)

       -o, -out
              prints the selection to standard out (generally for piping to a file or program)
  • 3
    +1 for the creative (mis-)use of xclip. The answer need an accessible/running X server. – jofel Apr 20 '16 at 13:38
3

A Perl way:

perl -ne '$.<3?($f.=$_):print;}{print $f'

Or, the same thing written less cryptically:

perl -ne 'if($.<3){ $f.=$_ } else{ print } END{print $f}'

For example:

$ cat file
44444
55555
11111
22222
33333

$ cat file | perl -ne '$.<3?($f.=$_):print;}{print $f'
11111
22222
33333
44444
55555

Explanation

  • -ne : read the input file/stream line by line and apply the script given by -e to each line.
  • $.<3 : $. is the current line number, so change 3 to the number of lines you want to shift.
  • $.<3?($f.=$_):print; : this is the conditional operator, the general format is condition ? case1 : case2, it will run case1 if condition is true and case2 if it is false. Here, if the current line number is less than 3, it appends the current line ($_) to the variable $f and, if the line number is greater than 3, it prints.
  • }{ print $f : the }{ is perl shorthand for END{}. It will run after all input lines have been processed. At this point, we will have collected all the lines we want to shift and will have printed all the ones we want to leave alone, so print the lines saved as $f.
  • 1
    regarding your golfed version a few character can be removed perl -ne '$.<3?$f.=$_:print}{print $f – 123 Apr 20 '16 at 15:02
1

Use POSIX ex. Yes, it's intended for file editing, but it will work in a pipeline.

printf %s\\n 111 222 333 444 555 | ex -sc '1,2m$|%p|q!' /dev/stdin

This can have any arbitrary commands added at the beginning or end of the pipeline and will work the same. Better yet, given the presence of /dev/stdin, it's POSIX compliant.

(I don't know if /dev/stdin is specified in POSIX or not, but I see it's present both in Linux and Mac OS X.)

This has a readability advantage over using sed's hold space—you just tell ex "move these lines to the end" and it does so. (The rest of the commands mean "print the buffer" and "exit," which are fairly readable as well.)

NB: The ex command given above will fail if given less than 2 lines as input.

Further reading:

0

A short python snippet:

#!/usr/bin/env python3
import sys
file_ = sys.argv[1]
lines = int(sys.argv[2])
with open(file_) as f:
    f = f.readlines()
    out = f[lines:] + f[:lines]
    print(''.join(out), end='')

Pass the filename as first argument and number of lines to move as second argument.

Example:

$ cat input.txt
44444
55555
11111
22222
33333

$ ./sw_lines.py input.txt 2
11111
22222
33333
44444
55555

$ ./sw_lines.py input.txt 4
33333
44444
55555
11111
22222
0

If you can store the entire output in memory:

data=$(some command)
n=42                  # or whatever
{ tail -n +$((n+1)) <<<"$data"; head -n $n <<<"$data"; } > outputfile
0

Here is a another option which requires GNU sed:

(x=$(gsed -u 3q);cat;printf %s\\n "$x")

-u makes GNU sed unbuffered so that the sed command does not consume more than 3 lines of STDIN. The command substitution removes empty lines, so that the command does not include empty lines at the end of the output if the third, third and second, or third, second, and first lines are empty.

You can also do something like this:

tee >(sponge /dev/stdout|sed -u 3q)|sed 1,3d

Without sponge /dev/stdout| the command would fail with long inputs. sponge /dev/stdout can also be replaced with tac|tac, even though that prints for example a\ncb if the input is a\nb\nc where \n is a linefeed, or with (x=$(cat);printf %s\\n "$x"), even though that removes empty lines from the end of the input. The command above deletes the first line when the number of lines of the input is one or two.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.