Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to find out which files are in machineB and which files are in machineC.

And I need to run my shell script on machineA to figure out which files are in machineB and which files are in machineC

I have around 300 files distributed in machineB and machineC.

Now I would like to find out which file is in which machine (machineB or machineC).

Below is my shell script which I am running on machineA and PRIMARY_PARTITION has some file numbers and SECONDARY_PARTITION has some file numbers. And the files are present in this directory -

/data/snapshot/20140317

in machineB or machineC .. And the file path is like this -

t1_1980_0_200003_5.data
t1_1980_3_200003_5.data
t1_1980_5_200003_5.data
t1_1980_7_200003_5.data
t1_1980_9_200003_5.data

Here the file number is 0, 3, 5, 7, 9. Other names in the filename are always going to be same apart from the file numbers.

#!/bin/bash

readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/snapshot
PRIMARY_PARTITION=(0 3 5 7 9)
SECONDARY_PARTITION=(1 2 4 6 8)

dir1=20140317
dir2=20140317

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    for el in "${PRIMARY_PARTITION[@]}"
    do
        # do something here may be?
    done
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        # do something here may be?
    done
fi

Now my question is how would I find out all the files in machineB and machineC. Meaning out of (0 3 5 7 9) and (1 2 4 6 8) which files are in machineB and which are in machineC.

As an example I have shown very small set of files above. In general, I have lot of files which I am thinking its pretty difficult for me to login in machineB and machineC to find out the files.

share|improve this question
1  
Why not rsync?. –  Braiam Mar 20 at 1:30
    
@Braiam: I haven't worked with rsync before so there will be some learning curve for me. I would like to learn it but given the time frame I have, I thought its best to do using ssh. –  SSH Mar 20 at 1:34
1  
rsync can work over ssh just fine and seems easy too. Now, depending of what your final goal is rsync can in a single command do it (which may not be helpful if you are doing an assignment) –  Braiam Mar 20 at 1:39
    
@Braiam: It's not an assignment. I need to do this on our production system. I can try using rsync for this. Can you provide an example if possible? And do I need to install something on our production servers on machineA? bcoz from machineA I will be running my shell script. –  SSH Mar 20 at 1:40
1  
In that case, please edit your question, the opening paragraph is very misleading, you state that your problem is that you need to copy files to machineA but don't know if they are on C or B. Are you trying to figure out which file came from which after having successfully copied them (which begs the question of why not before)? How do you connect to the machines? Can you simply do ssh user@machineB ls /path and collect the files that way? –  terdon Mar 20 at 2:37
show 4 more comments

2 Answers

up vote 2 down vote accepted

There are 2 approaches that I can think of off the top of my head. The first would involve using rsync the second would be to use a combination of ssh & diff.

NOTE: Both of these approaches would compare a directory on machineA that has all the files with a subset of these files on machine's B and C showing you which machine (B or C) was the originator of said files.

Example data

Say I had the following setup of files on machineA.

$ ls
t1_1980_10_200003_5.data  t1_1980_4_200003_5.data  t1_1980_8_200003_5.data
t1_1980_1_200003_5.data   t1_1980_5_200003_5.data  t1_1980_9_200003_5.data
t1_1980_2_200003_5.data   t1_1980_6_200003_5.data
t1_1980_3_200003_5.data   t1_1980_7_200003_5.data

Method #1 - rsync

Now let's find out what files in this directory came from machineB.

$ rsync --dry-run -avz machineB:~/20140317/ .
receiving incremental file list
./
t1_1980_1_200003_5.data
t1_1980_2_200003_5.data
t1_1980_3_200003_5.data
t1_1980_4_200003_5.data
t1_1980_5_200003_5.data

sent 29 bytes  received 165 bytes  129.33 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

While machineC:

$ rsync --dry-run -avz machineC:~/20140317/ .
receiving incremental file list
./
t1_1980_10_200003_5.data
t1_1980_6_200003_5.data
t1_1980_7_200003_5.data
t1_1980_8_200003_5.data
t1_1980_9_200003_5.data

sent 29 bytes  received 166 bytes  390.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Method #2 - ssh + diff

As an alternative you can use some semi-basic ssh and diff to do something similar.

$ cd /dir/with/machBC
$ diff -B <(ssh user@machineB "ls -R ~/20140317/") <(ls -R .)
> .:
> t1_1980_10_200003_5.data
> t1_1980_6_200003_5.data
> t1_1980_7_200003_5.data
> t1_1980_8_200003_5.data
> t1_1980_9_200003_5.data
share|improve this answer
    
What is /dir1/ here in your example for rsync? –  SSH Mar 20 at 2:31
    
I am confuse what is /dir1/ here? This folder 20140317 is in machineB and machineC not machineA. And the files are inside this folder 20140317. –  SSH Mar 20 at 4:46
    
@SSH - it was just as directory that I was using for my simulation of your problem, I've replaced it with your 20140317. –  slm Mar 20 at 5:23
    
Sure.. In your ssh + diff example. I would be running that diff command on machineA where the files are located right? If yes, then what is this path cd /dir/with/machBC? –  SSH Mar 20 at 6:34
    
@SSH = correct, on machineA. That's a path to denote that this is some location on machineA with the files from machine's B & C. –  slm Mar 20 at 12:09
add comment

If I understand your question correctly (big if), I would do it with something like this running on machineA:

#!/usr/bin/env bash

file_pattern='t1_1980_*_200003_5.data';
target_dir='/data/snapshot/20140317/'
## Save the files from each machine into bash array
files_on_B=$(ssh user@machineB find "$target_dir" -name "$file_pattern")
files_on_C=$(ssh user@machineC find "$target_dir" -name "$file_pattern")

## Do whatever you like with them, here I am simply printing
echo "Files from B: ${files_on_B[@]}"
echo "Files from C: ${files_on_C[@]}"

Note that the script assumes sane file names, like in your example, with no spaces, new lines or other strange characters.

share|improve this answer
    
Thanks terdon. I am suppose to run this script from machineA where the files are located.. Right? –  SSH Mar 20 at 21:22
    
@SSH yes, from machineA. –  terdon Mar 20 at 22:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.