I work with a proof-reader whom I pay certain sum per each N characters (like 500000). He (usually) creates/edits files in alphabetical order. Content/filenames are in (non-latin) utf-8. Simple plain text markdown (.md) or org (.org) files.

I need to find an efficient way to note for myself his "N characters milestones", so that I know when to pay him. For the sake of example, let's say I paid him last time till the character number 3036 in the file "aab.md". He finished that file, went on with "aac.md", "aad.md" and is now on "csdw.md".

  1. How can I "measure" N characters (everything included: punctuation, spaces, digits, newlines, braces, etc) within that stream (assuming all of them reside in the same directory)? I.e. I need a bash command that will get "aab.md", 3036, and N as input and provide something like: "csaw.md", 5023 (meaning that N chars ended in that file on that exact letter).
  2. How can I list the files, that were accounted for in the previous command?
  3. It's less important, but if possible - how will the command from #1 look like, if the files are spread over several directories (also in alphabetical order), like he stopped last time on the character 3036 in the file "a/aab.md" and is now on "np/csdw.md"?

I figured out that cat * | wc -m can provide amount of chars in all the files, but it is still far away from what I need.

  • Do you want to count any character (including punctuation, spaces, digits, markdown formatting) or just alphabetic ones? Is 5023 the offset in byte, in characters or in alphabetic characters within the file? What about graphemes expressed using more than one character (like those using Unicode combining characters)? – Stéphane Chazelas Nov 6 '18 at 10:50
  • @StéphaneChazelas, updated my question: everything included: punctuation, spaces, digits, newlines, braces, etc. Any graphemes. The 5023 offset is in any characters (as above). – user1876484 Nov 6 '18 at 11:09

I'm going to suggest using zsh instead of bash, which is going to make it easier to get a sorted recursive list of the new files since aab.md.

#! /bin/zsh -
last_file=aab.md offset_in_last_file=3036 n=500000

(($#new_files)) && perl -Ci -sne '
   $l = length; $go += $l; $o += $l;
   if ($go >= $n) {
     printf qq(file="%s", line=%d, char-offset=%d\n), $ARGV, $., $o + $n - $go;
   $o = 0 if eof' -- -go=-$offset_in_last_file -n=$n ./$^new_files

For number of grapheme clusters instead of characters, replace $l = length with $l = () = /\X/g. For instance é written as U+0065U+0301 is one grapheme cluster expressed with two characters (and 3 bytes in UTF-8), while it's one grapheme cluster and one character (and 2 bytes) when written as U+00E9.

With bash 4.4+, and GNU awk, you could do something similar for constructing the $new_files array with

shopt -s nullglob extglob globstar
readarray -td '' new_files < <(
    printf '%s\0' **/*.@(md|org) |
      L=$last_file awk -v RS='\0' -v ORS='\0' '$0 == ENVIRON["L"], 0'

With bash, you'd also need to replace ./$^new_files with "${new_files[@]/#/.\/}". (we're adding a ./ prefix to avoid isssues with file names starting with - or |, <, >, whitespace...

|improve this answer|||||
  • Where do I enter "aab.md" as input? This might be not the first file in the directory. This might be the file after I have paid him several times already... – user1876484 Nov 6 '18 at 13:03
  • @user1876484 ./*.@(md|org) is the list of .md and .org files in the current directory, the first ones of which meant to be the files he has already worked on (those 3036 characters that offset our search). – Stéphane Chazelas Nov 6 '18 at 13:07
  • I have to be able to specify any file in the directory as a start point for counting (filename + offset), otherwise I need to rearrange files in the directory each time a try to check whether I have to pay him, which I don't want to do (it's a git repository). – user1876484 Nov 6 '18 at 13:11
  • If 3036 is meant to be the offset in the last file he modified last time, and not the total number of characters he has written in the past, then instead of ./*.@(md|org), give the list of files starting with that last-modified one. – Stéphane Chazelas Nov 6 '18 at 13:14
  • How? Is there a regex meaning [aab.."and upwards"].@(md|org) ? – user1876484 Nov 6 '18 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.