Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
1answer
194 views

Merge columns with the same value with awk

It is possible merge columns with the same value in awk? Something like if awk '$1 = $2' then merge I got this AZ14962 AZ14962 ctg_ 104 AZ14962 AZ14962 ctg_ 4 AZ14962 AZ14962 ctg_ 61 ...
0
votes
2answers
282 views

Using AWK to Print Output of Combination of Unspecified Number of Files

My objective is to move into a subdirectory, call it SUB, which has many text files with 5 columns each and always the same number of rows. I want one large 'sum' file that adds all columns from 2-5, ...
21
votes
2answers
2k views

Pipes, how do data flow in a pipeline?

I don't understand how the data flows in the pipeline and hope someone could clarify what is going on there. I thought a pipeline of commands processes files (text, arrays of strings) in line by line ...
1
vote
2answers
132 views

add a new line to a delimited file

So I have a file name test.txt inside that file I have about 20 lines of text that are delimited by pipe | Example: John|freshman|seatle|math|4|fulltime Bob|senior|Tacoma|biology|4|part-time I ...
3
votes
2answers
4k views

Find the line number which contains the pattern using custom regex delimiter

As in example I'm trying to get line numbers which contains the pattern. My pattern contains slashes so I wanted to add custom delimiter. This simple one works: sed -n '/file/=' temp.txt Using ...
0
votes
1answer
63 views

Compare two files [closed]

I want to compare two files in which the data is delimited by a pipe |. Both the files have the same results but only few fields might vary. File 1: A|B|C|D File 2: A|B|F|D I want to compare ...
2
votes
3answers
227 views

Writing a program for editing .txt data - Python or Unix?

I only have little experience in programming, and I'm currently working on improving my skills. Basically, I need to write a program, that can do some specific processes to some data in a .txt file. ...
1
vote
2answers
480 views

Copy the first 2 lines from a file and then remove the lines

I need to process a file using a shell script. The file arrives with the data as well as some embedded information releavant to how it is processed. The file is in the following format; ...
4
votes
5answers
1k views

How to replace only the Nth occurrence of a pattern in a file?

How to replace the third occurrence of the string in the file using sed command. Example: Change only the third occurrence of is to us in the file. My input file contains: hai this is linux. hai ...
1
vote
3answers
94 views

Display distinct values of a list and number of occurences

I have a program that writes on the standard output a list of string values, one per line, and I would like to display in real time the list of distinct values along with the number of occurences for ...
3
votes
3answers
86 views

Unix: Add unique names after signs in text

I have a file looking like the following: abcdefghijklmnopqrst // abcdefghijklmnopqrst // abcdefghijklmnopqrst I use the following code to insert a line with // to the beginning of the document (as ...
6
votes
8answers
7k views

can we print the last word of each line in linux using sed command?

suppose , if there is a file consisting of following lines , if they are 12345 567 7878 66 er3 t45t y6y46y 4y6 y656y y5y 46y6 65y7 y66uyuy yy46y6y The output has to look like: 66 ...
1
vote
3answers
2k views

Replace date with current date

I have a file abc.fil whose content is in the format yyyymmdd.xyz.doc. I want to cat that file abc.fil such that I get the output as 20150127.xyz.doc. Can any one please help me to resolve this ...
4
votes
3answers
1k views

Replace multiline string in files

I have a number of files I want to update by replacing one multi-line string with another multi-line string. Something along the lines of: * Some text, * something else * another thing And I want ...
2
votes
4answers
127 views

Removing an empty line and the line above it

I have a large .fna file containing millions of DNA sequences. Due to some write errors after truncating my primers, I'm left with some sequence headers with no sequence following it. An example is ...
0
votes
1answer
193 views

Extracting info from blocks inside text file

I have a text file which has blocks like dir1/dir2/dir3/name_run_number1: line1_run_number1_part1 line2_run_number1_part2 line3_run_number1_part3... Each block is separated with a blank line and ...
0
votes
3answers
88 views

how to replace a character within in a multiple double quoted words within a line using sed?

I have text file including text format like this. 012345,"[ThinkPadT2/3Gband,Mac]",Lenovo,"iPhone3G,A1241" How do i replace comma inside words only within double quote by pipe(|) operator using ...
0
votes
2answers
71 views

Multiply a column using awk

How do I multiply a column by an integer (say 3) and replace the older value by new one, using awk? Input: Data 9390.900391 10573.089844 80.000000 200.000000 2.700000 Data 17762....
-1
votes
1answer
115 views

grep and print all pattern in one lane

I have a file with a list of pattern like this K00001 K00003 K00005 I want to grep and print my pattern in a tab delimited table (the original table is without blank line) like this: K00001 ...
0
votes
2answers
98 views

How do I download a PDF, then output a chunk of text?

My city water bill is emailed to me every month as a link to the current PDF hosted on their server. So I can easily download this PDF any time I want. How do I take that PDF and extract my account ...
1
vote
2answers
319 views

Formatting ifconfig using sed/awk

I'm using ifconfig on OpenSUSE. When I run ifconfig eth0 I get eth0 Link encap:Ethernet HWaddr CE:FD:75:DF:A5:6D inet addr:172.16.4.177 Bcast:172.16.5.255 Mask:255.255.254.0 inet6 ...
3
votes
5answers
403 views

Edit Nth occurence interactively in a search in vim

Given a text with multiple occurrences of the word foo, I would like to do an interactive search through the text, and upon the Nth occurrence of foo, edit it. I can only find the search and replace ...
1
vote
4answers
104 views

Two operations with sed on the same pattern

I use this following sed command for display names of files with a specific form: ls -1 *|sed 's/^\(.*\).png/\1\/\1,/g' If I have two files named BOH_Contour.png and BOV_Web.png, I obtain ...
1
vote
1answer
4k views

How to remove junk characters from the file generated by script command in linux [duplicate]

I am using script command to record everything from terminal. But when I am opening the generated file its having lots of junk character. Can anyone help me to remove these junk characters from file ...
3
votes
1answer
86 views

Show exact number of CRLF occurrences in file

grep -c $'\r\n' file returns the exact same number as grep -c $'\n' file, it shows the number of LF in the file instead of the number of CRLF. (The file definitely does not contain that number of CRLF....
-5
votes
1answer
485 views

Merge two csvfiles by column header

I have two csv files: success.csv "ID","Legacy ID","Field1", "Field2" 1,1111,google,news 2,2222,yahoo,news error.csv "Legacy ID","Field1","Field2","Message" 3333,aol,news,"Failed to upload data"...
2
votes
2answers
967 views

How to remove all CRLF in file (not replace with LF)

I'd like to remove all carriage returns followed by line feeds (CRLF), such as \r\n in a file. How can I do that? I can't use dos2unix because that replaces CRLF with LF. And I can't use tr because ...
1
vote
2answers
53 views

Help with Script to Process txt file to CSV

I am trying/attempting to write a script to turn this: Mikrotik01#show unique routing entry | 192.168.255.232 | inc ID Thu June 10 15:21:32.808 CST Unique Entry: 192.168.255.21 Unique Entry: ...
2
votes
1answer
325 views

How do I print hyphens if a variable is empty?

I am collecting windows system information using wmic on Linux system. For that purpose, I make various wmic calls and their output is stored in some variables. After required data is collected, I ...
1
vote
5answers
651 views

Using sed to cut a string starting with n spaces and ending with n spaces

How to use sed to cut text starting with x spaces and ending with y spaces? For instance This is my string: kkk 111 fff aaabbb 5d98 ccc mmmppp 9369d and I want to get this output: ...
3
votes
5answers
4k views

Awk - output the second line of a number of .dat files to one file

I have searched through previously asked questions and can't find an answer to what I am trying to do. note, I am new to linux. I have multiple files something like: (in reality i have 80) file1.dat ...
3
votes
3answers
2k views

Strip off word pattern at the end of string using sed

I have following string: ksh -x SCRIPT2.ksh $BUSSINESSDATE & PID=$!; ksh -x SCRIPT1.ksh $BUSSINESSDATE & PID=$!; I want to strip off the last word & PID=$!;. Desired result is: ksh -x ...
1
vote
1answer
53 views

Computing differences of column entries

I have a textfile with: 1 100 2 80 3 70 4 64 Now I need another file that contains the first column equal to above and the second column should be the difference of to consecutive values of the ...
2
votes
2answers
95 views

Search and Replace question

I need to find all occurrences of AAsomeArbitraryStringBB and replace it with CCsomeArbitraryStringDD. So AAHelloBB Text AAByeByeBB becomes CCHelloDD Text CCByeByeDD. It's important to note ...
2
votes
2answers
207 views

Extracting values from a file keyed by multiple keys

Consider a file with key=value pairs, and each key is optionally a concatenation of multiple keys. In other words, many keys can map to one value. The reason behind this is that each key is a ...
3
votes
1answer
906 views

Awk compare 2 files, print match together with non match lines filled with 0

I have two files. I want to match the 1st column of first file to 1st column of second file. If there is a match I'd like to print the 2st column of second file, if not 0. I have tried awk (awk 'FNR=...
3
votes
4answers
469 views

Fastest way to read last line within pattern of multiple files with the same extension

I have in a directory /home/files/ some files that all have the same extension .txt. For example 1.txt 2.txt 3.txt 4.txt 5.txt test.txt These files are all the same in data format but the data are ...
1
vote
2answers
223 views

splitting a file by column number

I have a file with 10,671 columns. I am trying to split it into 10 smaller files, each with 1000 columns until all columns have been printed, named file_transposed01, file_transposed02, ... ...
2
votes
2answers
326 views

How to print out every other line to the monitor

Have this tiny script down below and want to print out to the screen every other result. What are the ways of doing it? #!/bin/bash for ((n=5, j=1; j<=9;j++)) do echo "rez: $n*$j"=$((n*j)) done
1
vote
2answers
79 views

find and replace a value in a line of text

I have a text file which contain only 1 line of text 1314:Battery:1.90:50:30 I'm trying to figure out if is there anyway to do a grep and sed to change the value 50 and 30 to the following 1314:...
0
votes
1answer
49 views

Prepending text read from a file on the top of another file [duplicate]

I want to add "void mainfunction()" on the top of a file. I am reading the actual name of the file from another file and the name of the main function from another file. I was using sed -i '1i void $...
3
votes
2answers
467 views

Select lines starting with specific numbers

I have to show the PID, PPID and STATUS of the procceses on the system, but only the ones which PID starts with numbers in the range 5-8. So i tried something like this... ps xao pid,ppid,s | grep ' ...
2
votes
2answers
288 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
1
vote
2answers
89 views

Modifying part of a string in bash script

I have a file named variables.f90 , having many lines defining different variables, as follows :: integer::n_monomer=6800 real*8::rx=5.0d0 #... randomly integer and real numbers defined real*8::...
1
vote
2answers
82 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
158 views

output neighbouring lines with line numbers while grepping

Is there a way to output the neighbouring lines along with the line numbers while grepping the contents of a file? I'm trying to get outputs of the form: $ ~/myscript.sh download file # 'download' ...
3
votes
3answers
3k views

Need to insert single quotes in text file for use as SQL query using sed

I have created a line of text that looks like this: INSERT INTO radcheck(id, username, attribute, op, value) VALUES (,,00:23:32:c2:a9:e8,Auth-Type,:=,Accept); I want to use sed to make it look like ...
3
votes
2answers
817 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
2
votes
3answers
4k views

Compare two files and print unmatched lines [duplicate]

I have two files with the below data; I need the difference between two files. I tried with diff but it also shows line which are common in the two files: (22372 Dec 4 15:36 /opt/apache-tomcat-6.0....
1
vote
1answer
209 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() "/var/www/html/images/images/Outlet-Celine-Boston-Square-...