4

I compiled a list of materials I needed in a game, starting from the very top down to its most primitive ingredients. However, now I'm looking for a way to quickly tally up the numbers.

21 reinforced alloy
    21 damascus steel
        21 steel
            21 iron dust
            21 carbon
            21 iron
        21 iron dust
        21 carbon
        21 iron
    21 hardened metal
        21 damascus steel
            21 steel
                21 iron dust
                21 carbon
                21 iron
            21 iron dust
            21 carbon
            21 iron
        21 duralmin
            21 aluminum dust
            21 copper dust
            21 aluminum
                21 aluminum dust
        21 compressed carbon
            84 carbon
        21 aluminum bronze
            21 aluminum dust
            21 bronze
                21 copper dust
                21 tin dust
                21 copper
            21 aluminum
                21 aluminum dust
    21 corinthian bronze
        21 silver dust
        21 gold dust
        21 copper dust
        21 bronze
            21 copper dust
            21 tin dust
            21 copper
    21 solder
        21 lead dust
        21 tin dust
        21 lead
            21 lead dust
    21 billon
        21 silver dust
        21 copper dust
        21 silver
            21 silver dust
    21 gold 24 carat

The top levels don't matter, since I'm looking for the raw materials I need to collect. For example, 21 hardened metal and 21 damascus steel don't matter, because I'm looking for the total of 42 damascus steel, which also doesn't matter, because I'm looking for 42 iron dust, 42 carbon, and 42 iron (this example doesn't count the rest of the list), the total raw materials count.

So far I did this on a regex testing website, but eventually I'd like to be able to use grep so I don't have to open up a website to do the counting. I'd like to get something like "there are 5 occurences of carbon, here are the matching lines" so I can calculate easier, since if I know there are 5 occurences of carbon with 4 of them being 21 carbon and 1 being 84 carbon, I can now easily calculate that I need a total of 21*4 + 84 = 168 carbon.

I'm trying to count lines that don't have another line with a larger amount of tabs following it, since presumably if it does then it's not the raw material.

/(\t+)\d+ aluminum\n(?!\1)/g (replacing "aluminum" with whatever raw material count I'm trying to find)

This is not finding anything though. Is there a way to achieve what I'm trying to achieve with regex at all? If so, how?

Thank you for your time.


I'm not sure whether to put this on SO or this SE, but given that I eventually want to be able to use grep I thought this might be the more appropriate place.

1
  • Please edit your question to show the full expected output given your posted sample input so we have something to test a potential solution against.
    – Ed Morton
    Commented Dec 25, 2022 at 22:04

3 Answers 3

5

If you want to use perl-like regexps, why not using the real thing:

<your-file perl -l -0777 -ne '
  while (m{^(\s*+)(\d+) (.*)$(?!\n\1\s)}mg) {
    $count{$3} += $2
  }
  END {
    printf "%4d %s\n", $count{$_}, $_ for sort keys %count
  }'

Which gives:

  84 aluminum dust
 168 carbon
  42 copper
 105 copper dust
  21 gold 24 carat
  21 gold dust
  84 iron
  84 iron dust
  42 lead dust
  63 silver dust
  63 tin dust

-0777 -n means the whole input is slurped into $_. The multiline flag to the m{...} operator makes so that ^ and $ match at the beginning and end of each line within $_ rather than just at the start and end of $_. Without the s flag, . doesn't match on a newline character, but beware that \s does which could throw things off here if there were blank lines in the input.

\s*+ is the non-backtracking version of \s*. Not strictly necessary here since what follows (\d+) can't match a whitespace.

Standard grep doesn't support perl-like regexps such as those \d and (?!\1) perl RE operators you're using, but you could use pcregrep which happens to also support -o and a multiline mode with -M:

<your-file pcregrep -Mo '^(\s*+)\K.*$(?!\n\1\s)'

You'd still need to pipe to something else like perl or awk to do the sums, so that has little advantage over using perl for everything.

If the indentation may have a mix of tabs and spaces, you may want to have the input go through either expand or unexpand first to consolidate those into just spaces or just tabs. By default, they consider tab stops to be 8 columns apart like most terminals or browsers do (but not stackexchange which annoyingly has them 4 columns apart), but see the -t option to change that.

6
  • Maybe someday I'll learn Perl... thanks for the full-package solution! Commented Dec 22, 2022 at 14:26
  • "...but not stackexchange which annoyingly has them 4 columns apart..." I also consider a tab to be 4 spaces :( Commented Dec 22, 2022 at 14:27
  • @404NameNotFound although standard grep does indeed not support perl regexes as Stéphane mentioned, the default grep on Linux systems, which is not the standard grep, does support them with the -P flag. If you're on Linux, you don't need to install pcregrep, just use your regular grep with -P.
    – terdon
    Commented Dec 22, 2022 at 16:08
  • @terdon, GNU grep (nothing to do with Linux) if built with PCRE support for that -P, does support the same regexp as pcregrep indeed, but GNU grep doesn't have a multiline mode. It does have a -z option for NUL-delimited records, but that's NUL delimited records on output as well. And see the PCRE support is here to stay, but consider this option experimental when combined with the ‘-z’ (‘--null-data’) option, and note that ‘grep -P’ may warn of unimplemented features. in its manual Commented Dec 22, 2022 at 16:14
  • @StéphaneChazelas I only wanted to clarify that when you said that "standard grep does not support PCRE", you did not mean "the grep you will find on your Ubuntu". I think most Linux users will just assume their grep is the "standard" and I wanted to clarify that you were referring to the actual standard, POSIX grep which is a different beast. We don't know what system the OP is using which usually means they're on Linux.
    – terdon
    Commented Dec 22, 2022 at 16:19
4

A line is a "primitive ingredient" (primi) if its' level is <= level of the next element. This is equivalent to:

The previous line is a primi if its' level is <= current Level (or if it is the last)

Using awk with field-separator "\t", levels are NF, the ingredients are the last field $NF:

awk -F '\t' 'prevlev>=NF  {print primi}; 
                          {prevlev = NF; primi=$NF } 
             END          {print $NF}'

In Order to sum them, you could run something along the lines of

... | sed 's/ /\t/' | datamash -g 2 -s sum 1
2

You need to use lookbehind and lookahead. You also need to treat the entire input together, rather than line-by-line. The following command should do what you want:

grep -Pzo '(?<=\n)(\s+)(\S[^\n]*)(?!\n\1\s)' input_file
  • -P enables Perl syntax.

  • -z uses null terminator, rather than newlines.

  • -o outputs only the match.

  • (?<=\n) looks behind for a newline. This is in place of ^, which would normally match the beginning of each line. For negative look behind, use (?<!...). I'm ignoring the first line because presumably there will always be a deeper level. If this is not the case, you can add a new line to the beginning of the input before sending it to grep. There are probably better ways to do this, but here is one:

    ( echo ; cat input_file ) | grep ...
    
  • (\s+) captures the indentation level. This is referred to later as \1. \s matches white space. A potential problem with this is newlines could be considered part of the indentation. For instance, double newlines are often used as paragraph separators. You can replace \s with the specific white space you expect to be used for indentation, [\ \t].

  • (\S[^\n]*) captures the text of interest. \S matches non-white-space. [^\n] matches anything that is not a newline.

  • (?!\n\1\s) negative look ahead to ensure the next line is not indented any deeper than the current line. For positive look ahead, use (?=...).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .