Inserting command options from a file into a command

Question

Context: I'm trying to track down a Mercurial extension bug.

Mercurial has extensions, which extend the base functionality of Mercurial. You can choose to load any number of these extensions, as you wish. Sometimes they interfere with each other.

The command hg fold in the Evolve extension isn't working as expected. It seems that the problem is with one of the other extensions that are loaded, because with just the Evolve extension loaded, the command works correctly. In the case with just Evolve loaded, it looks as follows.

HGRCPATH=/dev/null hg fold --config extensions.hgext3rd.evolve= --exact  -r  2024:8679fc70eae8  -r  2021:8b3257871eac -m "Fold"

I want to do a binary search on all the extensions that I have loaded to isolate the problem one. The list I seem to have (from hg debugconfig) is as follows.

extensions.hgext.churn=
extensions.hgext.color=
extensions.hgext.convert=
extensions.hgext.extdiff=
extensions.hgext.graphlog=
extensions.hgext.hgk=
extensions.hgext.journal=
extensions.hgext.keyword=
extensions.hgext.mq=
extensions.hgext.pager=
extensions.hgext.patchbomb=
extensions.hgext.purge=
extensions.hgext.rebase=
extensions.hgext.record=
extensions.hgext.schemes=
extensions.hgext.progress=
extensions.hgext.histedit=
extensions.hgext.shelve=
extensions.hgext.narrow=
extensions.hgext.show=
extensions.hgext.share=
extensions.hgext3rd.evolve=
extensions.hgext3rd.topic=

What I'd like is a command that lets me specify a list of these extensions to include by line number. So something like

HGRCPATH=/dev/null hg fold something(1, 3, 5) --exact  -r  2024:8679fc70eae8  -r  2021:8b3257871eac -m "Fold"

which would include the extensions on lines 1, 3 and 5 of the file. This should output to

HGRCPATH=/dev/null hg fold --config extensions.hgext.churn= --config extensions.hgext.convert= --config extensions.hgext.graphlog= --exact  -r  2024:8679fc70eae8  -r  2021:8b3257871eac -m "Fold"

Checking the bug using the command line requires me to eyeball the results, by looking at the results of hg log. I could make it an automated check using Mercurial's Python API, but that would be significant effort, unless I got someone to tell me how to do it, and not worth it for such a simple issue. Given that, it makes sense to do the test as few times as possible. Hence the binary search.

Since the --config option is global, it can be put at the end of the command, if that makes things simpler.

What’s the result of something(1, 3, 5) supposed to look like? extensions.hgext.churn=,extensions.hgext.convert=,extensions.hgext.graphlog=? — Stephen Kitt, Jan 8 at 16:14
Do you believe the problem is one extension (at which point you could try the command with each extension in turn), or possibly a combination of extensions? — Jeff Schaller♦, Jan 8 at 16:35
@StephenKitt Question edited. Hopefully that answers your question. — Faheem Mitha, Jan 8 at 16:51
@JeffSchaller It's unlikely to be a combination, I think. Most likely one. But wouldn't narrowing it down still require a binary search? — Faheem Mitha, Jan 8 at 16:52
@JeffSchaller That's a good question. First, spotting the bug requires me to eyeball it. So the test passing cannot be checked automatically. I suppose I could switch to Mercurial's Python API and do it there, but that would be a lot more effort. Because of that, the idea is to keep the number of tests down to a minimum, which means using a bisection method. Which brings me back to a convenient way of scripting the different options. — Faheem Mitha, Jan 8 at 18:00

glenn jackman · Accepted Answer · 2021-01-08 18:38:35Z

How about this function?

hg_fold() {
    local -a config
    mapfile -t config < <(hg debugconfig)

    local -a opts
    for i in "$@"; do
        opts+=( --config "${config[i-1]}" )
    done
    opts+=(
        --exact  
        -r  2024:8679fc70eae8  
        -r  2021:8b3257871eac 
        -m "Fold"
    )

    echo HGRCPATH=/dev/null hg fold "${opts[@]}"
}

hg_fold 1 3 5

Remove the echo if it looks OK.

If you want something more interactive, perhaps

hg_fold() {
    local -a opts
    select_options opts

    opts+=(
        --exact  
        -r  2024:8679fc70eae8  
        -r  2021:8b3257871eac 
        -m "Fold"
    )

    echo HGRCPATH=/dev/null hg fold "${opts[@]}"
}

select_options() {
    local -n options=$1
    local -a config
    mapfile -t config < <(hg debugconfig)

    local selecting=true
    PS3="Select a debugconfig item: "

    while $selecting; do
        select item in "${config[@]}" Done; do
            if [[ $item == "Done" ]]; then
                selecting=false
                break
            elif [[ -n $item ]]; then
                options+=( --config "$item" )
                config[REPLY-1]="*$item"  # mark it as selected
                break
            fi
        done
    done
}

hg_fold

Looks reasonable, except that I would prefer to get the config options from a file as given above, not use hg debugconfig directly, which contains a lot of other stuff besides the config options. I've not reviewed your answer carefully, though. — Faheem Mitha, Jan 8 at 18:34

Jeff Schaller · Accepted Answer · 2021-01-12 20:03:49Z

I want to do a binary search on all the extensions that I have loaded to isolate the problem one.

Here's a solution that approaches this problem from a divide and conquer perspective. The basic idea is to prepare the input file a tiny bit, then recursively divide it in halves; each half is output as a sample command and then recurses on those halves. You would choose which of the sample commands to run based on the visual inspection that's required to determine success or failure of the command. You should only need to execute four of the commands in order to find the culprit.

Here's a bare-bones demonstration of the process, to get a better feeling for what's happening. The script recurses against 23 lines and prints each line in this format:

$key = sed -n $start, $end p file

The output is:

11 = sed -n 1,12p file
12 = sed -n 13,23p file
111 = sed -n 1,6p file
112 = sed -n 7,12p file
1111 = sed -n 1,3p file
1112 = sed -n 4,6p file
11111 = sed -n 1,2p file
11112 = sed -n 3,3p file
11121 = sed -n 4,5p file
11122 = sed -n 6,6p file
1121 = sed -n 7,9p file
1122 = sed -n 10,12p file
11211 = sed -n 7,8p file
11212 = sed -n 9,9p file
11221 = sed -n 10,11p file
11222 = sed -n 12,12p file
121 = sed -n 13,18p file
122 = sed -n 19,23p file
1211 = sed -n 13,15p file
1212 = sed -n 16,18p file
12111 = sed -n 13,14p file
12112 = sed -n 15,15p file
12121 = sed -n 16,17p file
12122 = sed -n 18,18p file
1221 = sed -n 19,21p file
1222 = sed -n 22,23p file
12211 = sed -n 19,20p file
12212 = sed -n 21,21p file

Each recursion is suffixed by "1" or "2" from the previous run. Follow "11" as being the first division of 1..23 into lines 1..12. From there, it's divided into "111" and "112", which are the division into lines 1..6 and lines 7..12 respectively. If the "11" command succeeded, you'd skip all of the subsequent "11.."-prefixed lines. If the "11" command failed, you'd focus on only the subsequent "11.."-prefixed lines. Rinse and repeat on each subdivided group of lines.

The script below uses the same underlying algorithm, but massages the output into the actual commands and sed output.

#!/bin/sh

# prepare the file a bit
file=./extensions-file
sed 's/^/ --config /' "$file" > "$file".prepared

# specifically a () subshell instead of {} grouping, to insulate "N" variable changes
divide() (
  # args = prefix, start, end, suffix
  # output: recursive breakout of:
  #   $prefix $(sed start .. N) $suffix
  #   $prefix $(sed N+1 .. end) $suffix
  # N = (end-start)/2

  n=$(( $2 + ($3-$2)/2 ))
  printf '%s %s %s\n' "${1}" "$(sed -n $2,${n}p "$file".prepared | tr -d '\n')"       "$4"
  printf '%s %s %s\n' "${1}" "$(sed -n $((n+1)),${3}p "$file".prepared | tr -d '\n')" "$4"
  if [ $((n - $2)) -gt 1 ]
  then
    divide "${1}" "$2" "$n" "$4"
  fi
  if [ $(( $3 - (n+1) )) -gt 1 ]
  then
    divide "${1}" "$((n+1))" "$3" "$4"
  fi
)

divide  "HGRCPATH=/dev/null hg fold"    \
        1                               \
        "$(wc -l < "$file".prepared)"   \
        "--exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m \"Fold\""

rm "$file".prepared

The script prepares the extensions output file a bit by prefixing each line with a space and the "--config " text so that we don't have to manipulate it every time we print it out. The temporary file is removed at the end.

The divide() function is defined and then called. The divide() function first computes (the floor of) the midpoint of the incoming range. It then prints the sample commands, based on using lines (start)..(n) and (n+1)..(end). The three arguments to printf are the prefix, the output from sed, and the suffix. We defined the prefix and suffix in the initial call to divide() -- they're just the text that sits on either side of the extensions bits. The sed command prints the requested lines from the file then passes them through tr to remove the newlines in order to make the output all one line. This is where the preparations helped a bit -- having a space prefix each line means that we can join the lines fairly simply.

In order to keep track of things more easily, you may want to filter the above script's output through nl in order to give an identifier to each line. Track those identifiers against the output from the script below, also piped through nl.

The following script is the one that produced the initial "barebones" output above.

#!/bin/sh
# specifically a () subshell instead of {} grouping, to insulate "N" variable changes
divide() (
  # args = prefix, start, end
  # output: recursive breakout of:
  #   "$prefix"1 = start .. N
  #   "$prefix"2 = N+1 .. end
  # N = (end-start)/2

  n=$(( $2 + ($3-$2)/2 ))
  printf '%s = sed -n %d,%dp file\n' "${1}1" "$2" "$n"
  printf '%s = sed -n %d,%dp file\n' "${1}2" "$((n+1))" "$3"
  if [ $((n - $2)) -gt 1 ]
  then
    divide "${1}1" "$2" "$n"
  fi
  if [ $(( $3 - (n+1) )) -gt 1 ]
  then
    divide "${1}2" "$((n+1))" "$3"
  fi
)

divide 1 1 23

For completeness, here's the output of the final script. Note -- some of the lines are very long!

HGRCPATH=/dev/null hg fold  --config extensions.hgext.churn= --config extensions.hgext.color= --config extensions.hgext.convert= --config extensions.hgext.extdiff= --config extensions.hgext.graphlog= --config extensions.hgext.hgk= --config extensions.hgext.journal= --config extensions.hgext.keyword= --config extensions.hgext.mq= --config extensions.hgext.pager= --config extensions.hgext.patchbomb= --config extensions.hgext.purge= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.rebase= --config extensions.hgext.record= --config extensions.hgext.schemes= --config extensions.hgext.progress= --config extensions.hgext.histedit= --config extensions.hgext.shelve= --config extensions.hgext.narrow= --config extensions.hgext.show= --config extensions.hgext.share= --config extensions.hgext3rd.evolve= --config extensions.hgext3rd.topic= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.churn= --config extensions.hgext.color= --config extensions.hgext.convert= --config extensions.hgext.extdiff= --config extensions.hgext.graphlog= --config extensions.hgext.hgk= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.journal= --config extensions.hgext.keyword= --config extensions.hgext.mq= --config extensions.hgext.pager= --config extensions.hgext.patchbomb= --config extensions.hgext.purge= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.churn= --config extensions.hgext.color= --config extensions.hgext.convert= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.extdiff= --config extensions.hgext.graphlog= --config extensions.hgext.hgk= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.churn= --config extensions.hgext.color= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.convert= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.extdiff= --config extensions.hgext.graphlog= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.hgk= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.journal= --config extensions.hgext.keyword= --config extensions.hgext.mq= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.pager= --config extensions.hgext.patchbomb= --config extensions.hgext.purge= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.journal= --config extensions.hgext.keyword= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.mq= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.pager= --config extensions.hgext.patchbomb= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.purge= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.rebase= --config extensions.hgext.record= --config extensions.hgext.schemes= --config extensions.hgext.progress= --config extensions.hgext.histedit= --config extensions.hgext.shelve= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.narrow= --config extensions.hgext.show= --config extensions.hgext.share= --config extensions.hgext3rd.evolve= --config extensions.hgext3rd.topic= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.rebase= --config extensions.hgext.record= --config extensions.hgext.schemes= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.progress= --config extensions.hgext.histedit= --config extensions.hgext.shelve= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.rebase= --config extensions.hgext.record= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.schemes= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.progress= --config extensions.hgext.histedit= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.shelve= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.narrow= --config extensions.hgext.show= --config extensions.hgext.share= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext3rd.evolve= --config extensions.hgext3rd.topic= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.narrow= --config extensions.hgext.show= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"
HGRCPATH=/dev/null hg fold  --config extensions.hgext.share= --exact -r 2024:8679fc70eae8 -r 2021:8b3257871eac -m "Fold"

To use the output to find the problematic extension, number the previous output (perhaps by piping it through nl). We now have a list of 28 commands. Run command #1. Base your next step on these instructions:

If #1 succeeds, run command #17. If #1 fails, run command #3.
If #3 succeeds, run command #11. If #3 fails, run command #5.
If #5 succeeds, run command #9. If #5 fails, run command #7.
If #7 succeeds, the problem is with extension 3. If #7 fails, the problem is with extension 1 or 2.
If #9 succeeds, the problem is with extension 6. If #9 fails, the problem is with extension 4 or 5.
If #11 succeeds, run command #15. If #11 fails, run command #13.
If #13 succeeds, the problem is with extension 9. If #13 fails, the problem is with extension 7 or 8.
If #15 succeeds, the problem is with extension 12. If #15 fails, the problem is with extension 10 or 11.
If #17 succeeds, run command #25. If #17 fails, run command #19.
If #19 succeeds, run command #21. If #19 fails, run command #23.
If #21 succeeds, the problem is with extension 15. If #21 fails, the problem is with extension 13 or 14.
If #23 succeeds, the problem is with extension 18. If #23 fails, the problem is with extension 16 or 17.
If #25 succeeds, the problem is with extension 22 or 23. If #25 fails, run command #27.
If #27 succeeds, the problem is with extension 21. If #27 fails, the problem is with extension 19 or 20.

The problematic extensions are identified by their line number in the hg debugconfig output file.

Stack Exchange Network

current community

your communities

more stack exchange communities

Inserting command options from a file into a command

2 Answers 2

Your Answer

Not the answer you're looking for? Browse other questions tagged shell-script shell or ask your own question.

Hot Network Questions

Inserting command options from a file into a command

2 Answers 2

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged shell-script shell or ask your own question.

Related

Hot Network Questions