Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

4
votes
6answers
296 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
1
vote
3answers
123 views

Joining files on disk

Say I have several file with the following naming: 000001 000002 000003 ... Each of these files is a csv file (may include escape characters). In total the folder has ~20GB of data. How can I ...
3
votes
2answers
145 views

How to count how many different word do i have in a file, in a column

I have a CSV file. "AGNOLI Valerio","ITA","AST" In this example, the 2nd column says "ITA". I expect there are about 100 or so different nationalities listed in this file. I want to know exactly ...
3
votes
1answer
51 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
-2
votes
2answers
81 views

Text processing using unix tools

I have a file in the following format. abc|xyz|mno x up x up up x x x up abc|xyz|mno x x x up x x .... .... Basically the first line of the file are device names (|ed) and ...
2
votes
1answer
35 views

Why might these values not be uniq'ed together?

I am trying to figure out which IP addresses hit the server most often. I received this surprising output from uniq: $ cat access_log | awk '{print $1}' | uniq -c | sort -bnr | head 26 73.23.5.25 ...
4
votes
6answers
187 views

grep with a lot of files which contains a lot of lines?

Imagine we have a lot of huge(30000 line)files: a.2014-05-06, a.2014-05-07, a.2014-05-08 and so on. I know that we can grep a line with word using this command: grep "word" a.* First I guess this ...
2
votes
4answers
151 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
3
votes
3answers
155 views

How can I delete text before a delimiter in Linux

I need to delete text from a line before delimiter. For example: (123434): hello::{apple,orange,mango}. I need to delete text before first :. i.e. (123434). Is there any command in linux to ...
1
vote
4answers
54 views

Increment a date in a file

I have a file with single line of text inside.How can I modify its content without using the vi editor.Below is the example: PROCESS DATE =2014-08-01 I want to increment the "2014-08-01" by the ...
2
votes
3answers
144 views

Text File Renaming Using Script

I have thousands of text files inside a folder and they are named in a certain way. For example: Image_234_Data_7778_n0.txt Image_234_Data_7778_n1.txt Image_234_Data_7778_n2.txt ...
2
votes
6answers
562 views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
1
vote
3answers
107 views

merging all files into one file?

I was trying to merge all the text files in to one file. All the text files has one column and equal number of rows. My data file names are data_1.txt, data_2.txt, data_3.txt, . . . data_n.txt. ...
5
votes
2answers
118 views

Extract tomcat parameters from ps output

I'm trying to write a little diagnostic tool and one of the functions I want from it is that I want to run is: ps axu | grep tomcat | grep -v grep | awk '{ print $<list of desired fields> }' ...
0
votes
2answers
100 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
2
votes
2answers
144 views

Collate tab-delimited files

I want to compare the information from mapping files to retrieve crosslinked information in a tab-delimited file format. For example, one file contains COG0001 882.DVU3168 COG0002 883.DvMF_2502 ...
4
votes
5answers
156 views

Insert Lines after Multi-Line Regex Match

I've got an XML file I need to append to after a specific set of tags. Basically the file structure looks like this: <key>servers</key> <dict> ... Server Details ... </dict> ...
0
votes
3answers
46 views

Delete everything not matching this pattern 'something.swf' in txt file

Can sed or awk do this, what I want do to is delete everything not matching this pattern 'something.swf' It doesn't need to be these commands. And im kind off new the text processing via the terminal ...
2
votes
3answers
85 views

How to convert html entities to readable text?

I want html number entities like &#x119; and want to convert it to real character. I have emails mostly from linkedin that look like this: chcia&#x142;abym zapyta&#x107;, czy ...
4
votes
2answers
182 views

Convert file with integers written in ascii to binary file of integers

I have a file that contains a long list of integers written in ascii separated with newlines, like such: -175 2 -19345 345592 -45 2355 etc... I want to convert this file into a "binary" file ...
1
vote
4answers
139 views

Bash: Convert Multi-line output to Single line

I am getting a multi-line output as follows: Actual Output: GenuineIntel,Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz,CPU0,64,EM64T Family 6 Model 45 Stepping 7,(null),3093,0 GenuineIntel,Intel(R) ...
-3
votes
2answers
133 views

Extract certain column from CSV files [closed]

I have a directory with a group of CSV files. All files have the same column headers. I need to extract values in a certain column from all files. The common part in files names is ...
0
votes
1answer
36 views

Delete all lines containing “..” and “@”

I have a long text file from a tcpdump. I need some of the information but most lines are just like this: .....@isa..g..3..2..2.@...3.3.2.1.12.. I want to use sed to search and replace all lines ...
2
votes
3answers
120 views

Replace a word after a significant line and white spaces (inline) using sed?

I have the following text: A Hello world B Hello world C Hello world I know that I can replace Hello by Hi using sed: sed 's/Hello/Hi/g' -i test but this replace each Hello ...
2
votes
3answers
34 views

Delete until given character starting from given string

root@server:~/foo$ cat INPUT.txt limit. Current u'^'ZO limit. Current usage'^'FR limit. Current us'^'AS limit. Current usage 95 %.'^'FR limit. Cur'^'AEW root@server:~/foo$ sed ...
4
votes
1answer
164 views

Using sed command convert only first character in specific line with uppercase

How can I convert the first character of each word in the line after gr-description: to uppercase? For example : $ cat /tmp/test1 groMail: test@xyz.com gr-description: abc-location-999 ...
1
vote
2answers
57 views

Delete all words surrounded by parentheses with sed

How can I do this in sed, I have a long long list of things and some are surrounded by a parentheses. lol xml lol lol (347) idiot (67) How can delete (347) and (67) but for a huge list with numbers ...
1
vote
1answer
73 views

Replace a string in a file and overwrite the file

I wrote my script using the the directory ~/deleted (it's a long script). after using ~/deleted for the whole script. I want to let the user choose the directory by putting the path in ~/.rm.cfg. My ...
0
votes
1answer
67 views

Print lines if pattern is not matched

Print lines of File 1 only if they don't contain patterns specified in File 2 Exemple: $ cat File_1 L1 pattern1 L2 pattern pattern2 L3 pattern3 pattern L4 pattern4 $ cat File_2 ...
1
vote
3answers
47 views

Print all lines between a searched pattern of a text file to different file

I have a file called test.txt that contains: ....Request.... asaksa sda dsad dsad ....Request... 21mklk nnm212 mkmr543 849238 ....Request... 4392840 kndska 94i0-jkfjdk smdla .....Request.. 839281 ...
2
votes
2answers
92 views

Remove rows from tab delimited files based on a common column with another file

I am having trouble processing a tab delimited file based on the common column Column_4 with another file. One file is likely to be very small (less than 100 rows), the second one, however, will be ...
2
votes
5answers
234 views

extracting/copy ranges of columns from large data file

I was trying to prepare data files from one large file by extracting columns and pasting into a new file. The number of columns of the given data is very large. I want to prepare a number of data ...
4
votes
2answers
186 views

Joining two files

I have a 2 files with the following: File1.txt A 1 B 2 C 5 Z 3 File2.txt A 4 B 7 C 10 D 11 What I would like to do is create something like A 1 4 B 2 7 C 5 10 D - 11 Z 3 - Is there a ...
5
votes
4answers
223 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
9
votes
4answers
259 views

Store output of a command into a ring-buffer

I've got a long-running command that generates a lot of output on stdout. I'd like to be able to retain, for instance, only the last three days or the last gibibyte (avoiding cutting lines in the ...
1
vote
2answers
200 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
1
vote
2answers
83 views

How to add words to an existing column

I have a tab delimited file with 10 columns and in one of the columns (with about 40 million rows), I would like to add a word before the existing entry in each row (same word in each row!) and a ; ...
2
votes
3answers
65 views

Shifting command output to the right

Say I'm writing a .bashrc file to give me some useful information in my login terminals and I'm telling it to run the cal command (a nice one). How would I go about shifting the calendar produced to ...
0
votes
3answers
54 views

String index processing

My input file looks like: #key string pos(string) key1 AA000AA000000AAA0A 2, 3, 18, 12 key2 00A00AAA000AAAA00A 3, 18 And I'd like to add a new column at ...
4
votes
4answers
288 views

Convert key=value blocks to CSV

I am trying to transpose a content of a file into another. I am new to shell scripting any help would be really appreciated Input file Test.txt: HLRSN = 3 IMSI = 404212109727229 KIVALUE = ...
0
votes
3answers
69 views

Remove all capital letters from a particular column in a CSV file

I am having a CSV file containing 3000+ lines in the following format: R1C1, ABC123abc, R1C3 R2C1, DEFG456defg, R2C3 R3C1, HIJKL789hijkl, R3C3 I need to remove all the capital letters from the ...
5
votes
3answers
63 views

How to put the output of the trailing lines of grep in on-line?

Assuming that I have this file: Thu May 8 15:32:07 2014 User-Name = "Mark" Framed-IP-Address = 0.0.0.0 Acct-Status-Type = Interim-Update Acct-Input-Octets = 95684 ...
1
vote
2answers
51 views

Calculate relative positions in strings

Sample input file: #name complete(cs) len(cs) simple(ss) len(ss) position(ss) NAME1 A0AAA000AAA00A 14 AAAAAAAA 8 4,6 NAME2 AAAA0AA00000A 13 AAAAAAA 7 7 ...
0
votes
3answers
54 views

replace comma in specific column value

I have a file which contains 6 columns, each field separated by "I" the second last column contains amount field. 140121059|01/01/201400:00:45|[1390]|[387]|17.64|10 ...
1
vote
0answers
181 views

How to create CSV file without unnecessary spaces

I use the xls2csv binary in order to translate XLS doc to CSV in my linux red-hat machine, example: ( from man page ) xls2csv -x "1252spreadsheet.xls" -b WINDOWS-1252 -c "ut8csvfile.csv" -a UTF-8 ...
2
votes
2answers
78 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
6
votes
5answers
197 views

Remove backslash + newline sequences

We are having a file in Linux which contains one line per record, but problem comes when the line contains some new line characters. In this case, the backslash is appended at the end of line and the ...
5
votes
4answers
87 views

Search a pattern and print preceding lines starting with another pattern

I need to grep for a pattern and once I find it, I need to look for another pattern preceding that and print all those lines in between those. Better example: 1 a 2 a 3 a 4 a 5 a 6 b 7 c 8 d 9 xyz ...
1
vote
1answer
47 views

How to align columns of multiple lines at a fixed distance with vim

Here is another one of my vim question (They seem to become my trademark here.). I have a really long file basically consisting of lines of the following form: ;07 :ANZ textv S9_1 9_1_K3_2 ...
4
votes
2answers
39 views

How to sort a collection of lines from different files?

I want to sort second line in each file by second column, and also print name of the corresponding file. I am doing it like this - rm tmp; for filename in file*; do num=`head -2 $filename | tail ...