Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

3
votes
3answers
28 views

Removal of lines with more than 'N' fields?

I am working on mac with sed, perl, awk, bash.. I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I ...
3
votes
1answer
26 views

Tokenize string from $REPLY in bash script

This is my first post, and I have no idea how I managed anything before StackExchage, Google, Wiki, GNU, Internet, the list goes on :) I am trying to find a way to construct a SQL database generator ...
3
votes
2answers
25 views

Extract text between two specific lines

How can I get the part of the output of a command between two specific lines? A dummy example: Command: git for-each-ref --sort='*authordate' --format='%(tag)' refs/tags | grep -v '^$' Output: ...
0
votes
1answer
36 views

Add text to end of every line

I have tried a lot of commands, like this: sed -i '' "s/$/<@string>/" txt.txt > txt2.txt This makes txt2.txt empty, why?!? sed 's/$/<@string>/' txt.txt > txt2.txt This does ...
6
votes
6answers
171 views

Obtain values of a section of a file

I have a configuration file in the following format. <Title> [part1] A.File = a A.Val = val1 B.File = a B.Val = val1 [part2] A.File = a1 A.Val = val2 B.File = a B.Val = val1 ...
9
votes
3answers
462 views

Why does “wc -c” print a value of one more with echo?

When running echo abcd | wc -c it returns 5. But the word abcd is only 4 characters long. Is echo printing some special character after the word abcd? And can I prevent echo from printing that? ...
1
vote
4answers
160 views

Iteratively edit a single file

I have a single file with a statement that looks like this : myName_tx_1 VARCHAR(255) myName_in_1 VARCHAR(255) myName_tx_2 VARCHAR(255) myName_in_2 VARCHAR(255) myName_tx_3 VARCHAR(255) myName_in_3 ...
1
vote
5answers
59 views

How to overwrite stdout using echo? [on hold]

We know to overwrite bar.txt, we can use this: echo "foo" > bar.txt but how can we overwrite to the standard output? I tried this command: echo "foo" > but it failed. I want this because ...
3
votes
2answers
114 views

Remove duplicate fields in a given column (after ID check)

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated), but only if the ID column ($1 in the example) is the same. Input: A 1,2,3,4 A 8,9,10,11 A ...
3
votes
3answers
106 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
2
votes
2answers
26 views

removing each . after ] appear in a line in complete text file

I have a file in which I need to eliminate . after the ] on every line e.g xyz[1.2]35.23 xyz[1.2.3]35.234 output should be xyz[1.2]3523 xyz[1.2.3]35234
1
vote
2answers
26 views

Transform custom tags with arguments

I want to write my diploma theses with markdown. Sadly, there is no perfect editor and so I have to improvise. For bibtex citations I want to transform my own definition with some sort of bash-filter ...
-3
votes
0answers
23 views

Parsing the log file test into tabular format

How to display below test file in tabular format as below: Output should be like: ma1-test1.cor.dns.com cmd - ...
2
votes
2answers
195 views

Get text between a word and the last line?

According to the answers in Stack Overflow question How to use sed/grep to extract text between two words?, we can get text between two words: sed -n "/first/,/second/p" file But what if I want to ...
2
votes
1answer
27 views

Calculate Apache's response time for last 10 seconds

I have the next apache's log format: LogFormat "%h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" where's %D - The time taken to serve the request, in microseconds. I want to ...
0
votes
4answers
61 views

How print only the current logged in “root” user IP address?

Currently I can see my (root) IP address on top of the output for this command: last But what I want is just that the IP address is printed out to me in a clean way. root pts/1 1.1.1.1 ...
1
vote
3answers
33 views

Trouble in adding a column with awk

I'm trying to add a column to a csv file by using: awk -F "," '{print $0"new_col"}' file.csv but I got : new_col8,0.405,0.101 new_col4,0.374,0.100 new_col23,0.342,0.100 new_col130,0.298,0.100 ...
5
votes
4answers
423 views

How can I “lazily” read output from xrandr?

I have a bash script that I use to adjust my monitor brightness that uses xrandr --verbose to get the current brightness. It works fine, but using xrandr is kind of slow on my machine, as you can see ...
5
votes
4answers
206 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
-1
votes
2answers
14 views

Design the Code to combine the splitted files based on serial,pin,denomination [on hold]

source is a directory contain two files s1 and s2. These files contain some data in columns: serial,pin,denomination datas. This should be moved to another directory called destination with ...
0
votes
1answer
31 views

How do I validate a settings(ini) file for the values expected using bash?

After I check if the ini file exists where it is expected to be, which I've learned how to do, I would like to check the file for proper formatting. I'd like to validate any number of "site" entries ...
0
votes
1answer
22 views

Compare strings

I have 2 txt files, in one i have alias lists and the other is the servers name. So what im trying to do is to loop from the alias txt file line by line and where it will find the name that is on the ...
2
votes
1answer
22 views

Get each column which contains multiple spaces?

My file: subnet 172.25.10.0 netmask 255.255.255.0 { option domain-name-servers 172.25.10.9, 8.8.8.8; default-lease-time 300; max-lease-time 7500; range dynamic-bootp 172.25.10.10 172.25.10.30 ; ...
-3
votes
0answers
48 views

How to make webpages concise? [closed]

concise:I've programmed >20 years taught ~1000 hours ~6 courses. Mixed list of wisdom I tell my students ≥1 time: normal:I've been programming for over 20 years, and I've run about half a dozen ...
-2
votes
3answers
55 views

How to sed two same value without affecting the other

I have an xml file and I want to change one value on that file: param name=swam_license_ports_overdraft_thresh 99 param name="strike_license_ports_overdraft_thresh" 99 The outcome I want is to ...
3
votes
2answers
94 views

Identify lines that are out of order

I have a process that generates output mostly in lexicographically sorted order according to a (timestamp) field, but occasionally the lines will be output in the wrong order: 2014-08-14 ...
4
votes
6answers
280 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
1
vote
3answers
120 views

Joining files on disk

Say I have several file with the following naming: 000001 000002 000003 ... Each of these files is a csv file (may include escape characters). In total the folder has ~20GB of data. How can I ...
3
votes
2answers
135 views

How to count how many different word do i have in a file, in a column

I have a CSV file. "AGNOLI Valerio","ITA","AST" In this example, the 2nd column says "ITA". I expect there are about 100 or so different nationalities listed in this file. I want to know exactly ...
3
votes
1answer
38 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
-2
votes
2answers
71 views

Text processing using unix tools

I have a file in the following format. abc|xyz|mno x up x up up x x x up abc|xyz|mno x x x up x x .... .... Basically the first line of the file are device names (|ed) and ...
1
vote
1answer
33 views

Why might these values not be uniq'ed together?

I am trying to figure out which IP addresses hit the server most often. I received this surprising output from uniq: $ cat access_log | awk '{print $1}' | uniq -c | sort -bnr | head 26 73.23.5.25 ...
4
votes
6answers
148 views

grep with a lot of files which contains a lot of lines?

Imagine we have a lot of huge(30000 line)files: a.2014-05-06, a.2014-05-07, a.2014-05-08 and so on. I know that we can grep a line with word using this command: grep "word" a.* First I guess this ...
2
votes
4answers
69 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
3
votes
3answers
81 views

How can I delete text before a delimiter in Linux

I need to delete text from a line before delimiter. For example: (123434): hello::{apple,orange,mango}. I need to delete text before first :. i.e. (123434). Is there any command in linux to ...
1
vote
3answers
30 views

Increment a date in a file

I have a file with single line of text inside.How can I modify its content without using the vi editor.Below is the example: PROCESS DATE =2014-08-01 I want to increment the "2014-08-01" by the ...
0
votes
2answers
40 views

Text File Renaming Using Script

I have thousands of text files inside a folder and they are named in a certain way. For example: Image_234_Data_7778_n0.txt Image_234_Data_7778_n1.txt Image_234_Data_7778_n2.txt ...
2
votes
6answers
417 views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
1
vote
3answers
63 views

merging all files into one file?

I was trying to merge all the text files in to one file. All the text files has one column and equal number of rows. My data file names are data_1.txt, data_2.txt, data_3.txt, . . . data_n.txt. ...
5
votes
2answers
73 views

Extract tomcat parameters from ps output

I'm trying to write a little diagnostic tool and one of the functions I want from it is that I want to run is: ps axu | grep tomcat | grep -v grep | awk '{ print $<list of desired fields> }' ...
0
votes
2answers
100 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
2
votes
2answers
105 views

Collate tab-delimited files

I want to compare the information from mapping files to retrieve crosslinked information in a tab-delimited file format. For example, one file contains COG0001 882.DVU3168 COG0002 883.DvMF_2502 ...
4
votes
5answers
69 views

Insert Lines after Multi-Line Regex Match

I've got an XML file I need to append to after a specific set of tags. Basically the file structure looks like this: <key>servers</key> <dict> ... Server Details ... </dict> ...
0
votes
3answers
34 views

Delete everything not matching this pattern 'something.swf' in txt file

Can sed or awk do this, what I want do to is delete everything not matching this pattern 'something.swf' It doesn't need to be these commands. And im kind off new the text processing via the terminal ...
1
vote
3answers
38 views

How to convert &#x119; to letter?

I want html number entities like &#x119; and want to convert it to real character. I have emails mostly from linkedin that look like this: chcia&#x142;abym zapyta&#x107;, czy ...
4
votes
2answers
83 views

Convert file with integers written in ascii to binary file of integers

I have a file that contains a long list of integers written in ascii separated with newlines, like such: -175 2 -19345 345592 -45 2355 etc... I want to convert this file into a "binary" file ...
1
vote
4answers
85 views

Bash: Convert Multi-line output to Single line

I am getting a multi-line output as follows: Actual Output: GenuineIntel,Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz,CPU0,64,EM64T Family 6 Model 45 Stepping 7,(null),3093,0 GenuineIntel,Intel(R) ...
-3
votes
2answers
49 views

Extract certain column from CSV files [closed]

I have a directory with a group of CSV files. All files have the same column headers. I need to extract values in a certain column from all files. The common part in files names is ...
0
votes
1answer
34 views

Delete all lines containing “..” and “@”

I have a long text file from a tcpdump. I need some of the information but most lines are just like this: .....@isa..g..3..2..2.@...3.3.2.1.12.. I want to use sed to search and replace all lines ...
2
votes
3answers
83 views

Replace a word after a significant line and white spaces (inline) using sed?

I have the following text: A Hello world B Hello world C Hello world I know that I can replace Hello by Hi using sed: sed 's/Hello/Hi/g' -i test but this replace each Hello ...