Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

19
votes
9answers
29k views

Command to display first few and last few lines of a file

I have a file with many rows, and each row has a timestamp at the starting, like [Thread-3] (21/09/12 06:17:38:672) logged message from code..... So, I frequently check 2 things from this log file. ...
4
votes
1answer
3k views

How to find out common elements between two files?

For an example, I have 2 files having following info: File #1: 12 13 14 15 File #2: 12 1 13 2 14 2 15 6 16 7 17 8 Output File: 1 2 2 6 In the output file, I want only the second column ...
1
vote
1answer
360 views

Sort -u Function Command Not Working

As you can see in the picture attached screen shot After I used the sort command to sort a VERY large wordlist file I then split them up using the Split command. From there I confirmed that the ...
12
votes
5answers
15k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
3
votes
5answers
14k views

Bash iterate over a list of strings

I have this bash script: for opt in string1 string2 string3 ... string99 do somestuff It works, but I would like to replace the explicit listing of my strings with a file which actually contains ...
2
votes
2answers
3k views

Replace the shortest match of a string pattern

I have this string: update mytable set mycol=myvalue where mycol=yourvalue; I need to convert it to: insert into mytemp select * from mytable where mycol=youvalue; I can accomplish it like this ...
0
votes
1answer
286 views

what is the best way to convert CRLF to LF and the viceversa for C++ files in my project [duplicate]

Possible Duplicate: How to bulk convert all the file in a file system branch between Unix and Windows line break format? which is the best tool to convert the CRLF to LF for my project in ...
9
votes
5answers
334 views

How to count rows ordered by the first field in bash

Here is a snippet from the INPUT: ... #################### Bala Bela;XXXXXX12345;XXXXXX12345678;A SERVER345Z3.DOMAIN.com0 SERVER346Z3.DOMAIN.com0 SERVER347Z3.DOMAIN.com0 SERVER348Z3.DOMAIN.com0 ...
4
votes
3answers
192 views

Smart way to perform a similar replacement (not simple enough for diff) in multiple files?

I have a bunch of files (45, so too many to edit them manually without going crazy) which all require the same changes (as seen in the three diffs at the end of this question). Most tools used for ...
3
votes
4answers
381 views

Grep one liner to remove blocks of random text

Using a script to backup some Cisco routers and compare startup and running configs. My output is tarnished by some ssl certificates it returns - I wish to remove them. Given: ********* ERROR ...
2
votes
2answers
863 views

How do you truncate all columns in a file to a specific length?

Given columned text in a file that is delimited by a certain character, how do you truncate the width of either a specific column(s) or all columns? The use case is that I am reading some log files ...
2
votes
1answer
475 views

Using sed, how can I replace a line containing a pattern with contents from another file?

input_file.txt: one fffds two xxxx three ffff four ffde data_file.txt: six seven I want to replace the line containing two xxxx in input_file.txt with the content of data_file.txt. The output ...
3
votes
4answers
5k views

How to get all lines between first and last occurrences of patterns?

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar? For instance consider the following ...
4
votes
6answers
452 views

What is the easiest way to check if column A and B values goes both ways with AWK? [closed]

What is the easiest way to check if column A and B values goes both ways? Output to check: Mike John John Mike Pamela Barbara Barbara Pamela Mike Paul Roger Paul Desired output Mike <-> ...
8
votes
4answers
5k views

Concatenate lines by first column by awk or sed

How can I use awk in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
3
votes
1answer
194 views

grep for n or more periods on a given line

I want to be able to arbitrarily match lines that contain a lot of periods, like this: $ echo "there's. someone on the wing. some. thing." | grep -stuFF [[:idunno:]]{4,} there's. someone on the wing. ...
3
votes
2answers
247 views

Add info under specific line

I want to know how could I add files contain information under specific line in file. For example, I have file1.txt with the following lines: line 1 line 2 line 3 and file2.txt has the ...
2
votes
1answer
809 views

Regarding generating intersection and union of two csv files

I have two csv files, there are some overlap columns between these two files. Assume one file is called as A.csv and another is called as B.csv. The intersection of A and B is called as C. I would ...
1
vote
4answers
125 views

Number lines according to their content

I would like to number lines according to their content: the first line gets number 1, the second line gets number 2 if it's identical to the first and number 1 if it's different, and so on. For ...
8
votes
3answers
3k views

How do I make python programs behave like proper unix tools?

I have a few Python scripts laying around, and I'm working on rewriting them. I have the same problem with all of them. It's not obvious to me how to write the programs so that they behave like ...
2
votes
2answers
3k views

Reorder fields/characters with cut command [duplicate]

Possible Duplicate: Combining 2 different cut outputs in a single command? I have One file - temp.txt - which contains ABCDF PQRST LMNOP I tried this cut command $ cut -c 2-5 temp.txt ...
9
votes
7answers
4k views

Combining 2 different cut outputs in a single command?

I have a file such as the following: 1234 ABCD EFGH I'd like to convert it to the following: 2341 BCDA FGHE The actual file has 4,000 words, so I would like to do this in an efficient manner. I ...
3
votes
1answer
310 views

What is the difference between > and >> (especially as it relates to use with the cat program)?

Suppose that I have a file called temp.txt. Using the cat program, I would like to add the contents of this file to the end of myfile.txt -- creating myfile.txt if it does not exist and appending to ...
3
votes
4answers
481 views

Display lines inbetween text with grep

I know that I can use the -A and -B command for grep to get a lot of what I am looking for however that is not quite what I want. I am looking to parse the httpd.conf file to search for a domain. ...
4
votes
2answers
670 views

AWK help for filling up rest of the columns with &nbsp in file

I'm trying to get AWK filling up following text file for "empty" columns". Basic idea of finding longest line with AWK NF and checking which line has no value on that column. Then adding x times N/A ...
1
vote
1answer
154 views

Using join with two files fails on larger file sizes

I am having some issues with a script that is using join to join two files. Eaxmple input files contains lines like this: Here are the input file and the output of the join command: ...
12
votes
11answers
14k views

How can I split a text file into multiple text files?

I have a text file called entry.txt that contains the following: [ entry1 ] 1239 1240 1242 1391 1392 1394 1486 1487 1489 1600 1601 1603 1657 1658 1660 2075 2076 2078 2322 2323 2325 2740 2741 2743 ...
8
votes
7answers
495 views

Swapping an unlimited number of columns

I have a file with columns. See below for an example: a b c ... z 1 2 3 ... 26 I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc.. z y x ...
3
votes
1answer
352 views

grep list of names and information from bigger file

I have two files: one with list of names (500 entries) and other having some more information for each entry in A.txt and extra entries too. File A.txt (each line is starting with > (fasta format) ...
3
votes
3answers
300 views

How to make and apply (patch) one side diff?

I want to make a shell script that updates a config file. The file present on the server has information, like various IP addresses. The new file has more code, from new configs added to the system, ...
4
votes
2answers
173 views

Column mismatch and substituting

input.txt (tab-delimted) TTTTOTTT00000000008 RTTTT899 5.00E-28 TTTTOTTT00000000046 RTTTWRR 3.00E-31 TTTTOTTT00000000051 2.00E-11 TTTTOTTT00000000051 7.00E-12 TTTTOTTT00000000054 ...
5
votes
3answers
1k views

Extracting lines by key from very large file

I have a 42M line text file. Th first nine characters of each line are a numeric key. What is the most efficient way to extract only the lines whose key exists in another list of about 1.5M keys? Both ...
5
votes
4answers
5k views

Compare two files: lines present in one, not in the other, by one column comparison

I need to compare 2 files. Column 1 is the same in both files. Column 2 is what I want to compare: I want all lines in file 1 that are not in file 2 when comparing column 2. Column 3 is different in ...
2
votes
2answers
113 views

reformat a text file to have all text on one line using CentOS 5.6

I have a 210-line block of HTML that I have stored in a HTML for debugging purposes. The HTML, as you would expect, is formatted accordingly so is stored in the file with spaces and tabs. Is it ...
4
votes
1answer
6k views

Delete last character in a word but only if the character is there - in bash

How to remove last character only if it's there? input: OpenOffice.org/m openOffice.org/ozm Pers. Pfg. phil. Prof. resp. Roonstr./m roonstr./ozm desired output: OpenOffice.org openOffice.org ...
2
votes
2answers
472 views

How to Wrap a file with Tags in Bash

I have an xml file named test.xml, without a root tag like this: <cat></cat> <name></name> <id></id> <cat></cat> <name></name> ...
4
votes
4answers
1k views

Sort input file by the results of a regex

I'd like to sort a file based on the results of a regex expression. For example, if I have the following property declarations in Obj-C @property (nonatomic, strong) id <AlbumArtDelegate, ...
4
votes
3answers
216 views

Merge files using zipper method / late merge

I am searching for a nice method of merging two or more files line by line using the zipper method (also called late merge). Assuming we have three files, the result should look like this: line1 ...
3
votes
1answer
3k views

How to do multiline search and replace text (delete) recursively

I want to go recursively through a directory and search for specific content in all files. The content should then be replaced if found. With a single line I would make something like this: find . ...
1
vote
1answer
476 views

What's the most appropriate way of parsing values from this output?

I have some netstat output I'm scripting and want to parse. What's the most efficient way of parsing values from the output below? It's important to point out that I'm using the old /bin/sh shell ...
2
votes
5answers
13k views

awk-Printing column value without new line and adding comma

input.txt EN1 EN2 EN3 EN4 EN5 output EN1,EN2,EN3,EN4,EN5 I have tried awk.But it is not printing with comma awk 'BEGIN { OFS = ","} { printf $1}' input.txt I have GNU Awk 4.0.0 version
3
votes
2answers
4k views

AWK-Find maximum value in a row & print with header

I need to read a file, find the maximum value in each row, and print the ID of the row, the column the maximum value came from (sno#), and the value from the associated lc# column. If the maximum ...
5
votes
1answer
296 views

How to adjust numeric fields in a text file

Suppose I have the following <Tab> separated text file: file name size owner file1.txt 12.345 root file2.txt 0.172222 user1 file3.txt 2.46e2 user2 file4.txt 12345 ...
26
votes
11answers
8k views

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
8
votes
1answer
9k views

Find any lines exceeding a certain length

Is it possible to find any lines in a file that exceed 79 characters?
3
votes
1answer
1k views

How to do df only on root partition?

How can get df results only for / partition. The partition name/identification (/dev/sda2, /dev/cciss/c0d0p1) could vary on different computers.
1
vote
4answers
137 views

“Multipass” scripted modification of large file in-place (file-system level)?

I have just came to the problem of having to cut some lines from a large (gigabyte) sized file, and being aware of potential CPU hog trying to read it in memory, I wanted to edit it in-place ...
2
votes
1answer
245 views

text processing-perl

I am a biologist. I would like help of computer experts to help in text file editing. Here is what the input file looks like: ##dsfsd2 ##sdf-sdf sasg 5.6.3 gi34_ex Gen CDS 161 317 . ...
3
votes
2answers
739 views

remove only specific text occurrences from string using sed

I have a text file that contains many rows of this sort of thing: /*[17:51:27][1 ms]*/ UPDATE `country` SET `region_id` = '4' WHERE `country_id` = '36'; Is there a way that I can use sed to remove ...
2
votes
1answer
676 views

Duplicate selected words with sed and replace it

I have to modify a document containing xml data; one modification I couldn't set up with sed. I have to modify the following expressions from: <Bild ...