Tagged Questions
1
vote
2answers
79 views
shell script to do some text manipulation of text file data structure and slight content changes
Apologies in advance for the wall of text, not sure how else to represent the existing data structure.
I have been handed about a years worth of logs collected every hour from a server.
Sadly, ...
0
votes
2answers
70 views
read file record by record and do transformation to the subsequent record based on above record and write into another file
Data file is fixed length file, and I want to read the file record by record and do transformations to the subsequent records based on the prior records (and write the results into another file).
...
11
votes
4answers
269 views
How to do a continous 'wc -l' with gnu texttools?
I know of course that
cat logfile.txt | wc -l
120
will tell me the number of lines in a file.
Whereas
tail -f logfile.txt
will show me the new lines that another program writes to logfile.txt.
...
3
votes
3answers
46 views
How to separate numerical values from identifiers
I'm currently writing a shell script that seperate values from their identifiers (retrieved from grep).
For example, if I grep a certain file I will retrieve the following information:
value1 = 1
...
5
votes
2answers
210 views
How to cut (select) a field from text line counting from the end?
I know how to select a field from a line using the cut command. For instance, given the following data:
a,b,c,d,e
f,g,h,i,j
k,l,m,n,o
This command:
cut -d, -f2 # returns the second field of the ...
1
vote
1answer
127 views
How to match 2 large file and print the difference in shell script
I have 2 files.
File1 has 400k numerical records.
e.g:
1
2
3
4
5
6
.. and so on
File 2 also has 420k numerical records.
e.g:
1
2
3
4
6
.. and so on
Both these file are in unsorted manner. I want ...
3
votes
3answers
467 views
split long line on a delimiter
What command can I use to split input like this:
foo:bar:baz:quux
into this?
foo
bar
baz
quux
I'm trying to figure out the cut command but it seems to only work with fixed amounts of input, like ...
3
votes
2answers
118 views
How can I use column to delimit on tabs and not spaces?
I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
7
votes
10answers
506 views
What's a good way to filter a text file to remove empty lines?
I have a .csv file (on a mac) that has a bunch of empty lines, e.g.:
"1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
2
votes
1answer
189 views
intersection of two files according to the first column
I have two files
in file A, there are sequence_numbers
in the other file B, there are many columns, and the first column is sequnce numbers,
I want to get a files with all the lines in the B with the ...
11
votes
2answers
654 views
echo a file without the first and last lines
Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
3
votes
2answers
154 views
Limit stdout from a continuously running process
I haven't had much luck finding an answer to my problem, but maybe I'm not asking for it correctly.
I have a process I startup like the following:
nohup ping 127.0.0.1 > log.txt >2>&1 &
...
4
votes
3answers
261 views
Find files that have words in common
What would be the best way to create a list of files that have common words with a given file. For example, if I had:
$ ls
mainFile file1 file2 file file4
$ cat mainFile
exquisite malicious ...
2
votes
4answers
189 views
How to do an IF statement from the result of a executed command
I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this?
if [ "`netstat -lnp | ...
3
votes
2answers
273 views
Skip first 3 byte of a file
I am using AIX 6.1 ksh shell
I want to use one liner to do something like this
cat A_FILE | skip-first-3-bytes-of-the-file
I want to skip the first 3 bytes of the first line, is there a way to ...
8
votes
2answers
951 views
How to print only the duplicate values from a text file?
Suppose there is a column of numeric values like following:
File1:
1
2
3
3
3
4
4
4
5
6
I want the output:
3
4
That is, only the repeated lines. Is there any command line tools to find this ...
1
vote
1answer
153 views
Join problem: throwing error, join extra operand
I want to join 3 files on a column which has sorted unique numeric values (those files have only one column of values though) and starts with same prefix for an example "usi".
Now, while I am doing ...
1
vote
2answers
160 views
Substitute pattern within a file with the content of other file
I have a text file (devel.xml).
I added the word REPLACETHIS to it in order to replace this string with the content within a different file (temp.txt).
The closest thing I have is this:
sed -i -e ...
1
vote
1answer
225 views
What's the most appropriate way of parsing values from this output?
I have some netstat output I'm scripting and want to parse. What's the most efficient way of parsing values from the output below? It's important to point out that I'm using the old /bin/sh shell ...
4
votes
2answers
588 views
delete first line of file only if blank using sed
I have a solution for this in awk:
awk '{if (NR==1 && NF==0) next};1' somefile
but was unable to find one that worked in sed. E.g.,
sed -i.bak '/^$/{1,1d;}' somefile
ended up deleting ...
3
votes
3answers
312 views
Using CSV line as command parameters
I have a CSV file like:
Name,Age,Address
Daniel Dvorkin,28,Some Address St. 1234
(... N ...)
Foo Bar,90,Other Address Av. 3210
And I have a command that take this parameters:
./mycommand ...
4
votes
1answer
231 views
inexact text search
Is there any utility like grep or even uniq but for inexact search or I should write it myself?
I mean it will look at 90% (number may vary) matching, or smth like that.
For example I have file with ...
3
votes
3answers
124 views
Regarding separate a single file into multiple files according to line separation
Currently, I have a plain text file, A, such as
lowest priority
very high significance.
outstanding
very novel
In this file, every line contains a sentence. I want to separate this file into ...
2
votes
1answer
173 views
Delete repeated words between brackets inline
Our input looks something like
2012-04-17 [GBPGBP]
2012-04-13 [GBP GBP]
2012-04-13 [GBP]
2012-04-11 [GBPGBP]
2012-04-11 [GBP GBP]
2012-04-10 [GBPGBP]
2012-04-06 [GBP GBP GBP]
2012-04-17 ...
1
vote
2answers
224 views
Is there any simple way to change one line in a lot of files?
I'm trying to use PHP CodeSniffer, and here's the result:
-----------------------------------------------------------------
A TOTAL OF 3008 ERROR(S) AND 380 WARNING(S) WERE FOUND IN 46 FILE(S)
...
3
votes
2answers
286 views
How to skip file in sed if it contains regex?
I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed:
find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+
As ...
2
votes
3answers
264 views
loop to paste specific files in different directories
I some directories that contain a similarly named file eg (*Sample_name*.base.coverage.txt). And I would like to paste all of the *base.coverage.txt files together. I have something written, but its ...
4
votes
1answer
443 views
Linux sort second last column
I would like know how to sort the second last column ? It has the word "days=" in front of the number. I'm able to get the column with awk '{print $(NF-1)}', but sorting is wrong.
457000 ...
3
votes
1answer
128 views
Character classes: construct my own
I want to construct my own character class in a script, then modify (and use) it, for example:
[:myclass:] contains a, *, \n (as linefeed) and [WHITESPACE].
I want to add all characters to ...
4
votes
3answers
501 views
Why do I keep destroying my text files?
I have destroyed a bunch of non-essential files and I don't know why. I have been executing commands like:
tr -sc 'A-Za-z' '\n' > somefile.txt | less
there is no output (blank page with flashing ...
6
votes
1answer
98 views
counting the number of occurences in a file of data list
I have a data file, like
7
2
10
9
10
3
2
4
2
4
6
Each line has a single value. I want to count the occurence of each value. For instance, 10 occurs two times in this file.
Is there a simple way ...
6
votes
4answers
219 views
How can I print a conditional header BEFORE stdout, if there is any output on stdout
I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
7
votes
2answers
960 views
A shell script for joining two files
I want to write a shell script that get two files A and B, and get a result like this:
File A:
user_a tel_a addr_a
user_b tel_b addr_b
File B:
process_1 user_a
process_2 user_a
process_3 user_b
...
13
votes
4answers
14k views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid off those messages?
4
votes
2answers
211 views
How to expand tabs based on content?
I've got some tab-delimited data coming out of a Unix pipe. I'd like to format this data into a compact human-readable table.
How can I expand these tabs into spaces, and automatically set the tab ...
4
votes
2answers
207 views
Is it possible to use split to make character chunks out of Chinese unicode bytes?
For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me.
But there's one ...
4
votes
2answers
126 views
Extract part of the first line of a file
I am trying to configure some external commands in Gedit3, for compiling LaTeX files. All works well, except for the following. I have a LaTeX document, consisting of many parts. Each file begins with
...
8
votes
5answers
1k views
Randomly draw a certain number of lines from a data file
I have a data list, like
12345
23456
67891
-20000
200
600
20
...
Assume the size of this data set (i.e. lines of file) is N. I want to randomly draw m lines from this data file. Therefore, the ...
0
votes
2answers
1k views
list the difference and overlap between two plain data set [duplicate]
Possible Duplicate:
Linux tools to treat files as sets and perform set operations on them
I have two data sets, A and B. The format for each data set is one number per line. For instance,
...
2
votes
2answers
680 views
How to execute this particular shell command from Python?
OK, so, I have this non-functional shell script, which I am rewriting piece by piece in python, except I am getting an "unexpected "|" error" from the shell (see below):
#/bin/sh
LINES=`cat $@ | wc ...
2
votes
4answers
763 views
Trouble getting awk output in for loop
I'm trying to create a script that will check a website for a word. I have a few to check so I'm trying to input them via another file.
The file is called "testurls". In the file I list the keyword ...
2
votes
3answers
251 views
Returning only the portion of a line after a matching pattern
I have a file (file_name) which contains exactly one occurance of the string Result:, at the start of a line. I want to print all the characters after the string Result: in that line until I encounter ...
3
votes
4answers
267 views
Find a string (like grep -q) within only a section of a file
I want to write some Bash that can verify that a string exists in a configuration file. I can't change the file format, it belongs to a different application.
The file is subdivided into groups named ...
2
votes
1answer
315 views
How to insert a first column to many files + how to convert unix time to normal time
There are many-many files in a directory that has e.x.: the following content:
INPUT:
root 1324711901
sshd 1272725792
And I'm searching for a solution how to convert the content of the ...
2
votes
2answers
2k views
extract middle section of lines of a text file?
I am writing a php script to parse a large text file do to database inserts from it. However on my host, the file is too large, and I hit the memory limit for php.
The file has about 16,000 lines; I ...
2
votes
2answers
206 views
Comparing files and their properties
I get information of a certain set of files in my mail every day, which looks like this:
58623208 Sep 14 20:08 blbn_blfbe_drv
57904920 Sep 14 19:54 blbn_cycmn
55814208 Sep 14 06:02 clsa_Upd
38912000 ...
2
votes
1answer
743 views
Enable line breaks when assigning multi line outputs to var in bash
I'm using the following very basic shell script I copied off the net to list the contents of my database backup folder and mail them regularily:
MYSQLLIST=$(ls -lhG /var/backups/mysql/daily/)
...
3
votes
2answers
367 views
In `while IFS= read..`, why does IFS have no effect?
I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect.
The outer IFS (outside the while ...
17
votes
4answers
5k views
Why is `while IFS= read` used so often, instead of `IFS=; while read..`?
It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
6
votes
5answers
981 views
Why does shell Command Substitution gobble up a trailing newline char?
As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens
x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p
# ...
