Manipulation of text by programs, scripts, etc.
5
votes
2answers
68 views
awk + paste for cleaning up PATH?
I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it.
setenv PATH `echo $PATH | ...
1
vote
3answers
160 views
How to get data count between two strings with multiple occurences in the same file
I have the data in my file as
START-OF-FIELDS
ID
NAME
DEPT
END-OF-FIELDS
START-OF-DATA
1|joy|cs
2|sam|ec
END-OF-DATA
START-OF-FIELDS
ID
NAME
DOB
DEPT
ADDRESS
END-OF-FIELDS
START-OF-DATA
...
1
vote
2answers
120 views
Unix command for pattern matching
I want to count the number of words and print the matched pattern lines which matches exactly with following pattern:
abc-ERROR:
The input File contains:
# abc-ERROR: xyxxkkfgfr
# def-Error: ...
1
vote
4answers
114 views
Number lines according to their content
I would like to number lines according to their content: the first line gets number 1, the second line gets number 2 if it's identical to the first and number 1 if it's different, and so on. For ...
13
votes
3answers
10k views
How to merge all (text) files in a directory into one?
I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?
2
votes
3answers
37 views
Replace certain strings with value from former line
I have the following text:
Source IP,URL
1.1.1.1,example1.com
Blank,example2.com
2.2.2.2,example3.com
Blank,example4.com
Blank,example5.com
Blank,example6.com
Blank,example7.com
7.7.7.7,example8.com
...
16
votes
7answers
20k views
Looping through files with spaces in the names?
I wrote the following script to diff the outputs of two directores with all the same files in them as such:
#!/bin/bash
for file in `find . -name "*.csv"`
do
echo "file = $file";
diff ...
1
vote
4answers
99 views
“Multipass” scripted modification of large file in-place (file-system level)?
I have just came to the problem of having to cut some lines from a large (gigabyte) sized file, and being aware of potential CPU hog trying to read it in memory, I wanted to edit it in-place ...
4
votes
3answers
75 views
Why is sed giving me an error about an unterminated `s'?
I have a set of sed replacements in a bash script and am getting an error about an unterminated `s' command. Here's what the sed line looks like:
sed -n -e "s/TMPFOO1/$FOO1/" -e "s/TMPFOO2/$FOO2/" ...
3
votes
3answers
73 views
sed + how to remove character/s that start or ended on each number
How do I remove the . character(s) that start in the beginning of each number or end on each number?
Remark – perl one liner also good alternative for sed.
Example input:
.23.12.44.5.
.233.3.3.3
...
3
votes
3answers
64 views
checking data in columns when a data or some may be missing or present?
I am not sure if this possible.
say i have columns like :
Team Colour Game Rainfall PlayerName
XYZ Blue Cricket Yes Kapil
suppose i need to search ...
4
votes
2answers
114 views
sort part of a file
How can I sort the file by the second column from the terminal. The content of the file is:
Nome Note
------------
Mehdi 0
Sunday 20
Others 10
Shnou 5
Using sort -t' ' -nk2 is not ...
6
votes
1answer
53 views
How to do `head` and `tail` on null-delimited input in bash?
find command can output names of files as a null-delimited strings (if -print0 is provided), and xargs can consume them with -0 option turned on. But in between, it's hard to manipulate that ...
1
vote
2answers
83 views
shell script to do some text manipulation of text file data structure and slight content changes
Apologies in advance for the wall of text, not sure how else to represent the existing data structure.
I have been handed about a years worth of logs collected every hour from a server.
Sadly, ...
1
vote
2answers
35 views
Better string escaping for hg
So periodically I remove files that are listed as missing by the version control system hg. What I've noticed is that this (admittedly ugly) script I have does not properly escape strings:
hg status ...
1
vote
1answer
40 views
Using awk to parse AIX's lsuser output in stanza format
I'm creating a simple awk script to parse AIX's lsuser -f ALL output and return information in one line. A sample of the output is something like this.
root:
id=0
pgrp=something
...
2
votes
6answers
87 views
How can I separate these two outputs using awk command
Here's my output:
MessageID=3990592283244651750-30192b51.13df831d93a.7eb4;EsbTiming=2013-04-12T01:07:46.099
I want only the time.
How can I separate them?
9
votes
3answers
778 views
Parsing log files for frequent IP's
So, I hacked this together while undergoing a DDOS attack to pull naughty ips out of my logs. Anyone have any improvements or other suggestions to make it better?
Here's the general idea:
pull ...
0
votes
1answer
64 views
How does -k flag work in sort command? [duplicate]
I have got a dataset like this:
manufacturer,model,year,mileage,price
plym fury 1970 73 2500
chevy malibu 1999 60 3000
ford mustang 1965 45 10000
volvo s80 1998 102 9850
ford thundbd 2003 15 10500
...
78
votes
5answers
20k views
Why is printf better than echo?
I have heard that printf is better than echo and I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL ...
4
votes
6answers
90 views
Getting two fields from same line together
I've got a situation where I've got several lines, and I need two different fields out of each of them. To be specific, I have a list of references in a bibliography and I want to get the last name ...
2
votes
3answers
61 views
Scripting to domain allocation to server
I have two text files:
file #1: domain.txt
cocacola.com
airtel.com
pepsi.com
file #2: servers.txt
192.0.53.42 , 4 # 4 domains already allocated on server 192.0.53.42
192.53.1.2 , 1 # 1 ...
3
votes
3answers
92 views
Split file into multiple small files, separate by the newline symbol
Is there a utility that split file by newline symbol? e.g if a file contains the following lines,
aa
bbb
cccc
If I want to split it to 3 files, the desired output would be:
aa, bbb And cccc (in 3 ...
5
votes
4answers
582 views
A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1
Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ?
I have my ...
13
votes
8answers
14k views
How to count the number of a specific character in each line?
I was wondering how to count the number of a specific character in each line by some text processing utilities?
For example, to count " in each line of the following text
"hello!"
Thank you!
The ...
2
votes
3answers
119 views
How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?
CentOS 6.3
I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting).
The template file contains the following:
...
4
votes
3answers
63 views
Replace all the occurences of a term in the files contained in a folder
In a folder on my system, I have a C++ project that I'm trying to compile.
This process uses some C++ idioms which are not supported by my compiler version. In particular, often inside the code it is ...
4
votes
4answers
102 views
How to remove character and space from a string
Hello I have one file with output
Name : ABC
Name : CDE
Name : ZYS
I want to get the result
ABC
CDE
ZYS
How can i get it through SED command.
I have tried. sed s/'Name' but ...
6
votes
4answers
3k views
How do I recursively grep through compressed archives?
I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files ...
3
votes
1answer
43 views
tac command's option creates strange output [comprehension question]
Say I have this file, containing nothing than
a
b
c
b
a
Using tac --separator=a file in BASH [on a Debian based Linux], I get this:
# empty line
# empty line
b
...
0
votes
1answer
93 views
comparing lines with awk vs while read line
I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line ...
3
votes
2answers
74 views
How to adjust column width in Linux command output
When I used this command on Red Hat Linux
/usr/sbin/ss -i
I get the following output below:
State Recv-Q Send-Q Local Address:Port ...
0
votes
2answers
114 views
How to delete a line containing pattern which is first found after a specific pattern
I am writting a shell script to delete an entry from a specific group.
eg: file name is dest.xml
<domain id="1" group_name="group1">
<node id="ABC">
<node id="PQR">
...
3
votes
2answers
53 views
Can I make grep treat escaped multiline as a single line?
How can I make tools like grep treat multilines escaped with a backslash as a long single line?
Let's say I have some files that looks like this:
testvar=test1 \
test2 \
test3
...
7
votes
3answers
909 views
How can I wrap text at a certain column size?
I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
7
votes
6answers
1k views
Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?
grep -A 2 -B 3
prints 2 lines after the grep string and prints 3 lines before.
grep -C 3
prints 3 Lines before and 3 lines after
Unfortunately, the grep I'm using does not support these ...
6
votes
5answers
987 views
Why does shell Command Substitution gobble up a trailing newline char?
As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens
x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p
# ...
0
votes
0answers
24 views
sybase and iconv
I inherited an old Ubuntu 6.10 server running php 5.1.6 which is using sybase to connect to MSSQL. A user submitted some text containing an en-dash and crashed one of my script. It turns out sybase ...
1
vote
3answers
63 views
awk + print line only if the first field start with string as Linux1
how to print the line in case the first field start with Linux1
for example:
echo Linux1_ver2 12542 kernel-update | awk '{if ($1 ~ Linux1 ) print $0;}'
the target is to print the line , ...
9
votes
3answers
294 views
Deleting all C comments with sed
I am trying to write a script that will delete all comments and everything in between inside C files in my current directory. I've been using sed, and this is what I have so far:
sed -i '/ * [^()] ...
0
votes
2answers
70 views
read file record by record and do transformation to the subsequent record based on above record and write into another file
Data file is fixed length file, and I want to read the file record by record and do transformations to the subsequent records based on the prior records (and write the results into another file).
...
0
votes
2answers
32 views
How to delete lines where the given part of the line is more than 100 chars?
I know I can delete lines that are longer than 100 chars with this:
sed '/^.\{100\}..*/d'
But I have the following example line:
<a ...
3
votes
2answers
286 views
How to skip file in sed if it contains regex?
I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed:
find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+
As ...
13
votes
5answers
3k views
sort but keep header line in the at the top?
I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
2
votes
2answers
83 views
add “#” in begining to selected lines in File
Requirement : ADD someting (i.e #) in begining of every line in file A which is grep from file B
File A
abcd
abdc
sdfg
asdfa
jon
ram
File B
jon
abcd
grep file B from file A and add "#" ...
0
votes
1answer
54 views
read line by line and take a string in line and assign that string to the subsequent line using perl
Data file is fixed length file, and I want to read file line by line and take a string from the current line, and then append that string to the subsequent line and write into another file.
For ...
2
votes
2answers
931 views
Why does xargs strip quotes from input?
Why does xargs strip quotes from input text?
Here is a simplified example:
echo "/Place/='http://www.google.com'" | xargs echo
outputs
/Place/=http://www.google.com
Is there any way to ...
3
votes
2answers
155 views
Limit stdout from a continuously running process
I haven't had much luck finding an answer to my problem, but maybe I'm not asking for it correctly.
I have a process I startup like the following:
nohup ping 127.0.0.1 > log.txt >2>&1 &
...
-4
votes
1answer
47 views
How to transform two delimited ASCII files
These are 2 common delimited, text file transformation problems I'd like to find scripts to do (I dont know if awk or perl is best for these tasks).
csh is the default shell on my Solaris10 unix ...
3
votes
5answers
180 views
evaluate multiple patterns from program output and write into pattern specific files
I have a script outputting some value/numbers and I want to split those into two files. I am looking at something like:
./runme.sh | grep 'ook' >> ook.out | grep 'eek' >> eek.out
Where ...
