0

Goal

Clone repos into subdirectories named .owner.login (argument in Github's REST API) with jq, git clone, and xargs.


Preface

I took a code somewhere that allowed me to clone repos with jq, git clone, and xargs. However, I was not able to set it up so that it would make a new parent directory for each repos. (I am a Windows user but for what I am trying to achieve I was not able to query any other solution but this bash script. I do not have any knowledge of how GNU commands interact together and this is the most I can put together)

Original code

UserName=CHANGEME; \
curl -s "https://api.github.com/users/$UserName/repos?per_page=1000" |\
    jq -r '.[].html_url' |\
    xargs -l git clone

This is my modification:

UserName=CHANGEME; \
curl -s "https://api.github.com/users/$UserName/repos?per_page=1000" |\
    jq -r '.[] | .html_url, .full_name' |\
    xargs -l git clone

I got the same result as the original code. And this error log: fatal: repository 'repoauthor/reponame?' does not exist

I don't know where the \? came from.

So I tried to debug it

by splitting the code into

UserName=CHANGEME; \
curl -s "https://api.github.com/users/$UserName/repos?per_page=1000" |\
    jq -r '.[] | .html_url, .full_name'

which returned this:

https://github.com/repo1name/repo1author
repo1name/repo1author
https://github.com/repo2name/repo2author
repo2name/repo2author
... etc

It returned .html_url, .full_name in 2 different lines instead of a single line.

I thought that it was the issue but then I tried to run xargs separately:

https://github.com/repoauthor/reponame |\
xargs -l git clone

It just make me go into the git help file.

tl;dr

I need to integrate the strings generated by jq into xargs. However, jq generated the important string into two different lines, and MAYBE that's what's causing the bug but I don't know how to resolve it.

1 Answer 1

1

As you don't give us a specific Github user for reproducing your specific issue, I have tested with other Github accounts that I know of myself.

There are two main issues with your attempts.

  1. The individual arguments read by xargs should ideally be quoted.
  2. xargs needs to call git clone with two separate arguments: the repository URL and the destination directory into which to clone it.

You may sort out the quoting of the arguments like this:

curl -s "https://api.github.com/users/$UserName/repos?per_page=1000" |
jq -r '.[] | [ .html_url, .full_name ] | @sh'

This extracts the wanted info from the response to the curl request into an array, and then uses the @sh operator to output a each such array as a line of shell-quoted words.

Strictly speaking, we could have used just .[] | .html_url, .full_name | @sh as the jq expression to get a stream of lines with single shell-quoted strings on them too, it doesn't matter for xargs the way we're going to use it.

Given this stream of words, we then call git clone via xargs:

xargs -n 2 git clone

The -n 2 means xargs will call the utility with two arguments from its input stream at a time.

Putting it together:

curl -s "https://api.github.com/users/$UserName/repos?per_page=1000" |
jq -r '.[] | [ .html_url, .full_name ] | @sh' |
xargs -n 2 git clone
6
  • I got this: fatal: could not create work tree dir 'repoauthor/reponame?': Invalid argument. The issue seems to stem from the fact that either jq or xargs generated an invalid character (a \? in this case).
    – Xyd
    Commented Sep 14, 2021 at 14:02
  • @Xyd It would be so much easier if you cold mention what Github user you are working with. The error message mention a name with a question mark at the end, so I don't know why you're talking about \?. Both ? and \? are valid characters and strings in Unix directory names. In short, I'm unable to reproduce your error, so I'm inclined to close your question as "unclear".
    – Kusalananda
    Commented Sep 14, 2021 at 14:21
  • @Xyd I have no issues cloning into subdirectories with ? and \? in their names, using the code in my answer.
    – Kusalananda
    Commented Sep 14, 2021 at 14:23
  • It seems like jq generated a '\r. I tried using this to debug:$ curl -s "https://api.github.com/users/Xyde/repos?per_page=1000" |\ > jq '.[] | .owner.login'|\ > xargs -t echo 'Xyde'$'\r' Xyde
    – Xyd
    Commented Sep 14, 2021 at 14:37
  • 1
    @Xyd Insert tr -d '\r' in your pipeline, just after curl, or after jq, depending on what's generating the carriage-return character.
    – Kusalananda
    Commented Sep 14, 2021 at 14:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .