2

Under the folder /usr/hdp/2.6.4.0-91/tez_hive2/lib/ We have .jar files as the following

$ ls  /usr/hdp/2.6.4.0-91/tez_hive2/lib/ | grep ".jar"
async-http-client-1.8.16.jar
azure-data-lake-store-sdk-2.1.4.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.2.jar
commons-collections4-4.1.jar
commons-io-2.4.jar
commons-lang-2.6.jar
commons-math3-3.1.1.jar
guava-11.0.2.jar
hadoop-aws-2.7.3.2.6.4.0-91.jar
hadoop-azure-2.7.3.2.6.4.0-91.jar
hadoop-azure-datalake-2.7.3.2.6.4.0-91.jar
hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar
hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar
hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.4.0-91.jar
jersey-client-1.9.jar
jersey-json-1.9.jar
jettison-1.3.4.jar
jetty-6.1.26.hwx.jar
jetty-util-6.1.26.hwx.jar
jsr305-3.0.0.jar
metrics-core-3.1.0.jar
protobuf-java-2.5.0.jar
RoaringBitmap-0.4.9.jar
servlet-api-2.5.jar
slf4j-api-1.7.10.jar

I want to count all characters from the .jar files by wc , in order to understand if .jar files renamed

So I do the following command in order to count all characters from all .jar files

ls  /usr/hdp/2.6.4.0-91/tez_hive2/lib/ | grep ".jar" | sed s'/\// /g' | awk '{print $NF}' | wc | awk '{print $NF}'
758

So in this case we get 758 characters from all .jars

But the command isn’t elegant

How we can improve the command to be better?

|improve this question
  • 1
    You want to get total string length of all the filename? That sounds like a strange request. Smells like an XY problem. – glenn jackman Feb 28 at 19:08
  • what I want it to count all characters from the output by wc or any other suggestion – yael Feb 28 at 19:17
  • 1) Wouldn't it make sense to get the length of "each" filename? 2) That wouldn't really tell you a whole lot as it can be renamed to something with the same amount of letters. 3) There really isn't a way to tell if any of the files have been renamed unless a script runs something like ls -l | awk '{print $NF} > jarlog.txt so that you can compare the filenames. – Nasir Riley Feb 28 at 19:22
  • look only I want is to improve the cli - ls /usr/hdp/2.6.4.0-91/tez_hive2/lib/ | grep ".jar" | sed s'/\// /g' | awk '{print $NF}' | wc | awk '{print $NF}' – yael Feb 28 at 19:24
  • 1
    printf '%s\n' *.jar | md5sum wouldn't be blind to renaming to something with the same amount of letters. – Kamil Maciorowski Feb 28 at 19:32
4

Most probably you are looking for

basename -a /usr/hdp/2.6.4.0-91/tez_hive2/lib/*.jar | wc -c

The path with wildcard list all jar files, the basename command strips directories (-a is needed to accept many arguments), and wc -c just counts bytes (if some filenames consist of 2 byte characters then perhaps wc -m (characters count) would be a better choice).

However, to if the goal is to check if files have been modified then perhaps stat (for modification time) or md5sum/shasum for checksum would be a better tools.

|improve this answer
  • 1
    wc -c counts the number of bytes, not characters. wc -m counts the number of characters. Also note that you're also counting one extra newline character per file. – Stéphane Chazelas Feb 28 at 19:47
  • what is the diff between wc -c to wc -m ? ( because on both I get the same results ) – yael Feb 28 at 19:51
  • 2
    @yael, wc -m counts the number of characters, wc -c the number of bytes. That makes a difference in the case of characters made of more than one byte (in UTF-8, that's all the non-ASCII ones (over a million of them)). Compare printf € | wc -c with printf € | wc -m. – Stéphane Chazelas Feb 28 at 19:53
5

Counting the number of letters in the filenames would not be a safe way of detecting a renamed filename.

Instead, create a simple file listing of the names, and compare it to an existing list. By using diff, you would be shown exactly which line(s) in the list had changed.

#!/bin/sh

LC_ALL=C

newlist=$HOME/filelist.new
oldlist=$HOME/filelist.old

echo /usr/hdp/2.6.4.0-91/tez_hive2/lib/*.jar >"$newlist"

if [ -f "$oldlist" ]; then
    diff -u "$oldlist" "$newlist"
fi

mv "$newlist" "$oldlist"

Obviously, the first time you do this, filelist.old would not exist, so the diff would not run.

Note that I save the full path to each file in the output file. This does not matter since the directory path is static.

Change echo to ls -l if you want to also compare timestamps etc. Change it to stat if you want to compare even more meta data (this would generate diff output when even the last-access timestamp on a file changed). Install wdiff and change diff to wdiff to get a word-based diff rather than a line-based one.

The LC_ALL=C is to guarantee a consistent sorting of the expansion of the shell glob.

|improve this answer
3

To just get the number of characters in the (non-hidden) jar filenames, I would do

cd /usr/hdp/2.6.4.0-91/tez_hive2/lib/ && printf %s *.jar | wc -m

This will not count any newlines, just the filename characters (replace wc -m with wc -c for the number of bytes instead of characters).

Purposefully, I'm not parsing ls output.

|improve this answer
  • Note that in shells like bash, if there's no .jar file in the current directory, that will output 5 (the number of characters in *.jar). In bash, you can do shopt -s nullglob to get 0 in that case. – Stéphane Chazelas Feb 28 at 19:51
1

I don't understand the point of the sed command, you are replacing a slash with a space? Why?

Aside from that, it seems that you want to count the total number of characters in all the file names of the .jar files. IF so, try this:
ls /usr/hdp/2.6.4.0-91/tez_hive2/lib/*.jar | sed s'/// /g' | wc -c

Two other possibilities for your sed command:
sed "s|/| |g" -- or -- tr '/' ' '
Since your ls command won't show directory names, I am not sure you need it.
This will also count the LF at the end of each file name. Is that OK?

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.