18
votes
3answers
982 views

Unix command to check if any two lines in a file are same?

Is there a unix command that can check if any two lines in a file are the same? For e.g. Consider a file sentences.txt This is sentence X This is sentence Y This is sentence Z This is sentence X ...
1
vote
2answers
55 views

Mailing out a customised content based on specific search in a logfile

Using below test.sh, I am getting result in dump.txt as below: test.sh: #!/bin/bash #"Maxl-script" zone starts..... essmsh -l admin password -s localhost -i << EOF spool on to 'dump.txt'; ...
1
vote
1answer
110 views

Binary search for a failing line

git has bisect run to figure out which revision an error was introduced. I have a big file (100GB) and at least one of the lines is bad, but the program I have to check it with will not tell me which ...
4
votes
2answers
983 views

How to remove lines included in one file from another file in bash?

I have two lists, a complete list and a partial list. I want a bash script that will search the complete list for any lines that are in the partial, if there are, they will be removed from the ...
2
votes
3answers
279 views

Help searching in files for regex patterns, recursively, with specialized output

My problem is that I need to: Find all lines matching regex_pattern in all files (deep search) in a given root directory For each line that matches, output: File name Line number containing the ...
0
votes
1answer
582 views

comparing lines with awk vs while read line

I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line ...
5
votes
2answers
387 views

Find all files with a Python Shebang

I'm trying to run a PEP8 check against a large source tree. The tree is comprised of a mix of files in various languages. The idea is to check all the Python scripts without having to explicitly list ...
1
vote
1answer
42 views

Replaced matched text in Vim with pattern

Here's a minimal example of what I'm trying to do. Start with: "this") "that") [hundreds of lines like this] End with: "this",&this) "that",&that) [hundreds of lines like this] I have gotten this ...
5
votes
4answers
477 views

Look for English word in a file via terminal

How to find and print out English words contained in a file via linux command line?
4
votes
3answers
160 views

Smart way to perform a similar replacement (not simple enough for diff) in multiple files?

I have a bunch of files (45, so too many to edit them manually without going crazy) which all require the same changes (as seen in the three diffs at the end of this question). Most tools used for ...
3
votes
1answer
2k views

How to do multiline search and replace text (delete) recursively

I want to go recursively through a directory and search for specific content in all files. The content should then be replaced if found. With a single line I would make something like this: find . ...
2
votes
2answers
400 views

remove only specific text occurrences from string using sed

I have a text file that contains many rows of this sort of thing: /*[17:51:27][1 ms]*/ UPDATE `country` SET `region_id` = '4' WHERE `country_id` = '36'; Is there a way that I can use sed to remove ...
3
votes
3answers
617 views

Longest common substring using grep

I have a huge text file called dictionary.txt with entries like ABC_SEQ_NUM This represents.... ABC_RANK This represents.... ABC_BSC_ID This represents... PQR_TA_DATE_AF This ...
1
vote
1answer
242 views

Tools to dump entries from arbitrary sets of log files between 2 dates

I had a outage incident a few days ago and the support team has asked for any available logs for the period, while thinking about how to do this, it occurred to me that it would be generally handy to ...
4
votes
2answers
2k views

How to search and replace multiple needles by one word via one expression?

Assume you have a text file: foo fnord bar bizz foo poit And now I would want to replace both "foo" and "bar" into "narf". I know I could use: sed -e 's/foo/narf/g' -e 's/bar/narf/g' fileName ...
6
votes
4answers
370 views

Fastest way to determine if two sorted lists contain unique elements

I have two sorted files A and B where the size of A is much greater than B, e.g. A is 100GB while B is 50MB. I want to quickly determine if there are any lines in B that are contained in A, stopping ...
3
votes
5answers
1k views

How to replace a list of placeholders in a text file?

I have a text (configuration) file, but the program that reads the file unfortunately doesn't allow using any kind of variables. So I'd like to use a preprocessor that replaces a set of placeholders ...
4
votes
1answer
613 views

Is there a search utility which can binary split search a large sorted (sequential) text file?

Yes, I know it sounds odd. Sequential and Binary-splits don't mix.. That is unless the sequence is the byte offset within the file itself... I've scrambled together a binary split search in bash ...