1

I have a list of paths stored in a shell variable tmp for example:

/abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_R1.tar
/abc/bcd/def/ZRT834/ZRT834_9/5678/S2_L001_I1.tar
/abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_I2.tar
/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R1.tar
/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R2.tar
/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_I2.tar

I want to create new directories based on the matching patterns from the paths. In the above example, I want to create directories ZRT834_9 and ZRT207_1 and create soft links for the tar files into their corresponding directories.

My output should be something like: ZRT834_9 directory having S1_L001_R1.tar, S2_L001_I1.tar, and S1_L001_I2.tar

How do I achieve this?

Improve this question
9
  • Will all of the tar file names be unique (at least with respect to their destination directories)? If not, what is supposed to happen if two or more files with the same name need to be moved to the same directory?
    – fra-san
    Sep 14, 2021 at 20:25
  • All the file names under a particular directory for example path with pattern ZRT834_9 are unique. But, the same filenames can be repeated in other paths.
    – botloggy
    Sep 14, 2021 at 20:29
  • How should we identify the path component that you want used for your new directories? In the example above the first set matches the fifth item from the left or the second from the right, and the second set matches the seventh item from the left or the second from the right.
    – Chris Davies
    Sep 14, 2021 at 20:39
  • Yes, you are right. This is what is making the folder structure more complicated.
    – botloggy
    Sep 14, 2021 at 20:41
  • Can you please suggest if we assume the folder structure as uniform across all tar files? @roaima
    – botloggy
    Sep 14, 2021 at 20:44

3 Answers 3

Reset to default
2

In zsh:

files=(
  /abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_R1.tar
  /abc/bcd/def/ZRT834/ZRT834_9/5678/S2_L001_I1.tar
  /abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_I2.tar
  /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R1.tar
  /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R2.tar
  /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_I2.tar
)
# or files=(${(f)"$(<list.txt)"}) to get the file list from the non-empty
# lines of list.txt, or files=($=tmp) for word splitting the contents
# of a $tmp scalar variable according to the current value of $IFS.

for file ($files) {
  dir=${file:h5:t}
  mkdir -p -- $dir && ln -s -- $file $dir/
}

Where ${file:h5} gets the 5-component¹ head of $file, and :t the tail of the result. Or you could do ${file:t3:h1} to count from the end.

¹ here component being path component. For instance, in ../a//b///c/./d/e, the components will be .., a, b, c, ., d and e. :h5 would give ../a//b///c/., and :t would then yield .. See also ${file:A:h5:t} to get a canonical absolute path before applying the :h5 and :t.

Improve this answer
1

The following assumes, based on your description, that:

  • variable tmp contains a newline-separated list of filenames[1]
  • you want to extract the fifth path element from the filename (e.g. ZRT834_9 and ZRT207_1)
  • you want to create a sub-directory with that path element if it doesn't already exist
  • you want to symlink the filename into that newly created directory.
#!/bin/bash

tmp="/abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_R1.tar
     /abc/bcd/def/ZRT834/ZRT834_9/5678/S2_L001_I1.tar
     /abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_I2.tar
     /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R1.tar
     /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R2.tar
     /abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_I2.tar"

while read -r f ; do
  d="$(echo "$f" | sed -E 's:^(/+[^/]+){4}/+([^/]*)/.*:\2:')"
  [ -z "$d" ]   && echo "Error: no fifth element in path: '$f'" && exit 1
  mkdir -p "$d" || exit 1
  ln -s "$f" "$d/"
done <<< "$tmp"

The sed script uses Extended Regular Expressions (-E option) and (roughly translated into English) captures the first 4 groups (the {4}) of /+[^/]+ (one-or-more slashes followed by one-or-more non-slash characters) into capture group 1, then the next [^/]+ after one-or-more slashes into capture group two and replaces the entire input line with just capture group two (\2).

The "one-or-more slashes" is because it's perfectly valid to have a pathname like /foo/////////////////bar////baz - the excess /s will be ignored. BTW, some programs (e.g. smbclient) will interpret the first element of a pathname beginning with 2 slashes as a server name prefix, but that's not the case with most programs.

[1] you really should use an array for this. e.g.

#!/bin/bash

# double-quote each array element even though your sample
# data doesn't need to be quoted - because other filenames
# might contain white-space or shell metacharacters.
tmp=("/abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_R1.tar"
     "/abc/bcd/def/ZRT834/ZRT834_9/5678/S2_L001_I1.tar"
     "/abc/bcd/def/ZRT834/ZRT834_9/5678/S1_L001_I2.tar"
     "/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R1.tar"
     "/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_R2.tar"
     "/abc/bcd/def/ZRT207/ZRT207_1/5678/S1_L001_I2.tar")

for f in "${tmp[@]}" ; do
  d="$(echo "$f" | sed -E 's:^(/+[^/]+){4}/+([^/]*)/.*:\2:')"
  [ -z "$d" ]   && echo "Error: no fifth element in path: '$f'" && exit 1
  mkdir -p "$d" || exit 1
  ln -s "$f" "$d/"
done
Improve this answer
5
  • Thanks for explaining your answer in a very detailed manner. My initial question was a bit more complicated. Is there any way we can generalize the answer where the subdirectory can be subsetted without the assumption of position?. Forex, Without the assumption of the 5th element. I had to edited the question earlier to reduce the complexity.
    – botloggy
    Sep 15, 2021 at 2:01
  • sure, you can use a regex to match and extract anything. the difficulty is in crafting the regex so that it matches exactly what you want and only what you want. You haven't provided any details on exactly what pattern you want to match. Without such details, your question is too broad to answer but I"ll guess ZRT[^/]+_[^/]+ to match anything beginning with ZRT and containing an underscore followed by some more characters. To extract that with sed: sed -E 's:.*/(ZRT[^/]+_[^/]+).*:\1:'
    – cas
    Sep 15, 2021 at 2:39
  • The broader question is off-topic here (because it's effectively asking for learning materials) but can be answered with "teach yourself regular expressions". Regular-Expressions.info is a good site to help with that.
    – cas
    Sep 15, 2021 at 2:42
  • This is a great resource.
    – botloggy
    Sep 15, 2021 at 3:47
  • I just realized that this is a better solution compared to any other as it works well in my case. Accepting this as my answer.
    – botloggy
    Sep 16, 2021 at 17:38
1

GNU sed has the ability to execute shell commands with parameters from backlinks in a substitution construct:

sed 's%.*/\([^/]*/\)[^/]*/[^/]*%mkdir -p "\1";ln -s "&" "\1"%e;d' <<<"$tmp"
Improve this answer
3
  • 2
    I would stay well clear of that e flag. First, it runs a shell in addition to those mkdir and ln command for each line, making it less efficient, you might as well omit the flag and pipe to sh. But more importantly, it makes it hard to avoid command injection vulnerabilities. Here you forgot to quote the paths to make it a string in the shell syntax, so for instance, if the dir name is $(reboot), that will reboot.
    – Stéphane Chazelas
    Sep 15, 2021 at 10:06
  • I completely agree with your remarks, but the main thing I have shown is that such an opportunity exists and is interesting in itself, thank you. Please do not delete your comment, it will be a good addition.
    – nezabudka
    Sep 15, 2021 at 10:26
  • Your solution actually works for the question asked. But, if the file structure is different. Would it work?. My initial question was a bit more complicated so I had to ask it as a different question based on suggestions from the community.
    – botloggy
    Sep 15, 2021 at 10:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .