3

I have a 25k characters.

I wish to write a script to print (printf is most portable I'm told) an arbitrary number of characters; stepping through them in order.

say:

command number

Where number can be any value of 1-25000, and get that output.

I would prefer to not have the data in a separate file (the easiest solution?), and I'd prefer to use only POSIX shell commands (to make the script as portable as possible: I'm aware awk or perl could whack this out simply).

Should I store this data in a variable? Or run my complete printf through a cut command (cut -c -$1)? Or is there another (better?) solution? Why might I choose one option over another option?

What other problems/caveats am I over-looking?

  • awk -v l="$1" '{print substr($0, 0, l)}' <file – cuonglm Dec 13 '15 at 15:45
  • Awk (without the GNU extensions) is part of the POSIX standard, so Awk scripts should be as portable as POSIX shell scripts. – nwk Dec 13 '15 at 16:16
2

Have you considered the dd command? It lets you skip any number of bytes, then output any number of bytes.

dd if=infilename bs=1 skip=sk count=ct 2>/dev/null

dd, input file name, block size 1, skip first sk bytes of input file, then copy ct bytes to stdout (or specify a file with of=name). Redirect error messages to avoid the status messages it usually prints at the end.

  • dd works with bytes, not characters. – cuonglm Dec 13 '15 at 15:43
  • 1
    And OP did not say anything about coding, so that might be fine. – Tom Zych Dec 13 '15 at 15:44
  • Hmm, so dd would work on the file from which it's called from? I did say that I didn't want the data in a separate file (ie: hard-coded into the script). What's the advantage to doing it this way over another way? But yeah, it's nice to see some different ways to skin cats. – user3082 Dec 13 '15 at 15:47
  • Ah, sorry, missed that detail. So it's in a shellvar? Then printf '%s' "$var" | dd will work. Advantages are that dd is pretty old and standard, and lets you specify how many bytes to skip and then to output. – Tom Zych Dec 13 '15 at 15:50
  • 1
    Well, in that case, I suppose you might put some kind of unique sentinel value at the beginning, have the script examine itself, and have it write data from its own file. You might also read about a shar. – Tom Zych Dec 13 '15 at 16:57
1

Probably not a good idea to store large data as a variable, for portability and reliability reasons. As for a non-awk solution that is also POSIX so as to be more likely portable, make use of sed.

Explanation

For large amounts of data, avoid storing as a variable. Even though Bash itself does not impose a limit, but the OS may do so

Suppose you say, "it works on my OS". But,

  • Different OS will have different limits
  • so if you want to maximize portability, why risk your script working on one OS and but crashing on another just because they have different limits ?
  • so avoid this issue by not storing in a variable in the first place

So then we store it in a file. Specifically, break up your string into one character (or whatever smallest unit you wish), on to separate lines.

Then, use sed:

Additionally think about code maintenance advantages of using a file. Updating lines stored in a file may be easier than navigating code in a script.

Example

Have the data, one character (or whatever smallest unit you want to "step through") per line, for example in a file data.lst:

a
b
c
d
e

Have your script.sh contain:

#!/bin/bash

stop_number="$1"

sed -n "1,${stop_number}p" data.lst

So, you test this on command prompt and see:

$ ./script.sh 3
a
b
c
  • it uses sed to print line 1 through to the number specified by $stop_number. We wrote out $stop_number instead of $1 directly, for clarity
  • $stop_number of course is obtained through the positional parameter $1, which is the arbitrary number input you wanted
  • so it successfully stepped through 3 characters of data.lst, in the sequence it appears in the data.lst
  • at the moment if you enter a number larger than the actual number of lines, it will just show all the lines.
  • currently data.lst is just in the same directory as script.sh but if you don't want that, if you actually have it somewhere else such as ~/some/dir/data.lst then you just have to adjust it to say ~/some/dir/data.lst

So once you have your actual data in data.lst you can test this script yourself.

  • Except, with 2 files I now need to put in a script check for the integrity/existence of the data file, and worry about managing it. I get that a variable is sub-optimal for storage (I'd guessed so, but didn't know until now), but how about a long print statement? The hard-coded data will never change, so updating is not an issue. – user3082 Dec 13 '15 at 16:49
  • Bourne-like shells don't have a limit (other than available memory) on the size of a scalar variable (csh is the shell that has such limits. some kshs have low limits on the number of array elements though). Most systems however have a limit on the size or arguments passed along the execve() system call (cumulative (with env) or of a single argument) which will affect the running of non-builtin commands. – Stéphane Chazelas Dec 13 '15 at 18:26
1

Borrowing from Tom a little bit:

#!/bin/sh
skp(){  dd bs="$1" skip=1 count=0; }    # direct seek to target
rd (){  dd bs="$1" skip=0 count=1; }    # single read at target
tail=$(sed -ne'/^don/{=;q;}' <"$0")     # skip script by line#
while   [ 1 -gt "$#" ] && exit          # exit when args exhausted
        exec <&- <"$0" || exit          # exec <"$0" each iteration
do      head -n "$tail" >&3             # only consider the tail
        case ${2+$1}  in                # test args
        (*[1-9]*|-*[!0]*)               # skp() when ${2++} && $1 != 0
          skp "$1";esac 2>&3            # send stderr to dev/null
          rd  "${2-$1}" 2>&3            # else just rd() from head of offset
        echo; shift ${2+"2"}            # append a newline and shift args away
done    3>/dev/null                     # put your data below this

don't but it in a variable - put it in your file. a 25k variable isn't going to be fun for the shell to handle, and your file can be seeked in a single, practically atomic action. so if you want to print bytes 23843 - 24843, you could do something like the above, and then call it with:

myscript 23843 1000

...and first a head will drop from the shared standard in file-descriptor all the lines of your script so that the offset is set exactly to the head of your 25k string, then the first dd will seek that offset ~ 23k in, and the second dd will read it out. it's the most simple way to do it. the shell is made for reading character by character - a typical shell's read builtin, for example, does a one-byte read() in a loop until it finds a newline - and doesn't stop until it does. dd will do one read per argument pair.

I tested it like this:

# after a copy to my clipboard
ddscr(){ sh /tmp/ddscr.sh "$@"; }
{ xsel; man man; } > /tmp/ddscr.sh
{ echo show the size; ls -l /tmp/ddscr*
  echo read from the top; ddscr 80
  echo from the middle;   ddscr 15k 160
  echo from the tail;     ddscr 64k | tail -n5
}

show the size
-rw-r--r-- 1 mikeserv mikeserv 37564 Dec 13 11:27 /tmp/ddscr.sh
read from the top
MAN(1)                           Manual pager utils                          MAN
from the middle
lso use manconv(1) directly.
              However, this option allows you to convert several manual pages to  a
              single  encoding  without  having
from the tail
       31st  March  2001  -  present day: Colin Watson <cjwatson@debian.org> is now
       developing and maintaining man-db.

2.7.5                                2015-11-06                              MAN(1)

...and...

ddscr 10k 10 20k 10 10250 10

       fi
is  option
le.   If
  • You say #!/bin/sh but you have functions. I'm pretty sure the old Bourne shell doesn't have functions, so I'm guessing the idea is to allow it to run under bash, ksh, pdksh, zsh, etc.? – Tom Zych Dec 13 '15 at 20:07
  • @TomZych - no it will run in any POSIX shell. it's at least Single Unix 2 or 3 or something. But functions were standardized for sh in the early nineties? something like that. all of the command syntax is standard as well - a genuine unix is pretty much required to run the above as expected, and it can be pretty well counted on in any linux as well. by the way, it was the Bourne shell that introduced the function() style functions. – mikeserv Dec 13 '15 at 20:09
  • Hmm, I had misremembered. This has been an educational Q&A for me all around. Thanks. – Tom Zych Dec 13 '15 at 20:16
  • @TomZych - you have a pretty good point, though. while a standard unix is required to have a compliant sh, it isn't required to keep it in /bin. it usually is there, but until Solaris 11 it was the old Bourne shell and the POSIX sh was in the xpg directory or whatever. i dont feel like handling it. but if you take anything from this, let it be bs=num skip=num count=0 - which i only grokked a week or two ago and think is awesome - i constructed a 20 line script with it capable of arbitrary - and super fast - binary searches on input of basically any size. – mikeserv Dec 13 '15 at 20:18
1

Plain Bytes

If the string contains only ASCII bytes, and no new lines, you could use cut. The command cut could work with characters only bytes.

$ echo "ajgjkggéóskmæßðasgbmdéóskmæßðushghsvéóskmæßð" | cut -b 1-5
ajgjk

But that will fail as soon as the cut string contains multi-byte characters:

$ echo "ajgjkggéóskmæßðasgbmdéóskmæßðushghsvéóskmæßð" | cut -b 7-12
géós

There are 6 bytes (7,8,9,10,11, and 12) but those are not 6 characters.
And cut also breaks if there are new lines in the string.

Characters

To work with "multi-byte" characters, we need a tool that understand such multi-byte characters, both sed and awk do.
The command sed could extract characters from an string:

$ s=5;l=3;echo "ajgjkggéóskm" | sed -E 's/^.{'"$s"'}(.{'"$l"'}).*/\1/'
ggé

But the -E option is a GNU extension, so we need to change the line to make it compatible with a POSIX sed (only BRE regex):

$ s=5;l=3;echo "ajgjkggéóskm" | sed 's/^.\{'"$s"'\}\(.\{'"$l"'\}\).*/\1/'
ggé

If the string does not contain new-lines. As sed breaks the input is lines at each new-line character.
That breaks as soon as there are new lines:

$ s=1;l=3;echo $'ajéw\nóskmæß\nðqwee' | sed 's/^.\{'"$s"'\}\(.\{'"$l"'\}\).*/\1/'
jéw
skm
qwe

The output is indeed 3 characters starting with the first (1), but for each line.

Chars and New-Lines.

The only other tool available is awk. Which does have a POSIX specification. Using the available String Functions from AWK:

$ s=6;l=4;echo "ajgjkggéóskm" | awk -v m="$s" -v n="$l" '{print substr($0,m,n)}'
ggéó

But that also breaks on new-lines:

$ s=1;l=3;echo $'ajéw\nóskmæß\nðqwee'  | awk -v m="$s" -v n="$l" '{print substr($0,m,n)}'
ajé
ósk
ðqw

However, we can specify one character that should not be in use in the string, as the RS (record separator): the null byte (\0). That forbids strings that contain NULs (\0), a very very rare problem.

Please note that I am not talking about the empty string: '', as that will make awk use an "empty line" as the record separator.

To do that, I will use a capacity of bash (not all shells could do this) of writing a zero byte as this: $'\0'. For other shells, the solution must be different.
If AWK is set with that RS, it will get the whole input as one record.

$ s=1;l=3;echo $'ajéw\nóskmæß\nðqwee'  |
awk -v RS=$'\0' -v m="$s" -v n="$l" '{print substr($0,m,n)}'
ajé

No more new-line interference. Well, we need to use printf to avoid some issues with print an new lines. With that, we could build an script.
Some notes on the script, as is not so standard:

  • The script starts execution on the last line: main "$@". That ensures that the whole script has been read by bash and that both defined function have been parsed.
  • The line(s) between the two _safe_place_for_string_ should be filled with any text that you need to include with the file.
  • The last _safe_place_for_string_ should be at the start of a line, have no additional text after it (not even spaces), and start at the start of the line (or after a tab character).
  • After that, there must be the close of the function } and the start of execution: main "$@"

The script:

#!/bin/bash
main(){
    while [ $# -gt 0 ]; do
        s=${1//[^0-9]/}; s="$((${s?Missing start of text.}+0))"
        l=${2//[^0-9]/}; s="$((${l?Missing start of text.}+0))"
        echo "from $s read $l characters"
        shift 2
        readchars "$s" "$l"
    done
}

readchars(){
    awk -v RS=$'\0' -v sstr="$1" -v lstr="$2" '
    {printf("%s\n",substr($0,sstr,lstr))}
    ' <<-\_safe_place_for_string_
aéóskmæßðasgbmdéóskmæßðushghsvéóskmæßð
aéóskmæßðasgbmdéóskmæßðushghsvéóskmæßð
aéóskmæßðasgbmdéóskmæßðushghsvéóskmæßð
_safe_place_for_string_
}

main "$@"

Call the script as:

$ ./script.sh 35 12
from 35 read 12 characters
mæßð
aéóskmæ

Note that the first "new-line" comes from inside the string. The last "new-line" was added by the \n in the printf, you could remove it, if needed.

Or even as ./script.sh 35 12 17 12, the internal while will process the repeated calls. Remember to place your text in the script to get the output you expect.

  • hey - you got one of those copernicus hats. thats a good one. your horse should sport that! – mikeserv Dec 16 '15 at 4:40
0

If you really want to wrap program and data in the same file, the best way is to use perl. I'm not sure why you think it's non-portable: It's standard with any Unix distribution you'll come across (including Linux and OS X); you won't find it standard on Windows, but you won't find bash there either.

#!/usr/bin/env perl
print substr(<DATA>, $ARGV[0], $ARGV[1]), "\n";

__DATA__
Just add all your text after 
the __DATA__ line... no fuss, no quoting, 
no tricks

E.g., suppose you name it selective_print and want to print 30 characters starting at 10:

% selective_print 10 30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.