Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

-4
votes
1answer
173 views

Merge two csvfiles by column header

I have two csv files: success.csv "ID","Legacy ID","Field1", "Field2" 1,1111,google,news 2,2222,yahoo,news error.csv "Legacy ID","Field1","Field2","Message" 3333,aol,news,"Failed to upload ...
2
votes
2answers
277 views

How to remove all CRLF in file (not replace with LF)

I'd like to remove all carriage returns followed by line feeds (CRLF), such as \r\n in a file. How can I do that? I can't use dos2unix because that replaces CRLF with LF. And I can't use tr because ...
1
vote
2answers
38 views

Help with Script to Process txt file to CSV

I am trying/attempting to write a script to turn this: Mikrotik01#show unique routing entry | 192.168.255.232 | inc ID Thu June 10 15:21:32.808 CST Unique Entry: 192.168.255.21 Unique Entry: ...
2
votes
1answer
233 views

How do I print hyphens if a variable is empty?

I am collecting windows system information using wmic on Linux system. For that purpose, I make various wmic calls and their output is stored in some variables. After required data is collected, I ...
1
vote
5answers
79 views

Using sed to cut a string starting with n spaces and ending with n spaces

How to use sed to cut text starting with x spaces and ending with y spaces? For instance This is my string: kkk 111 fff aaabbb 5d98 ccc mmmppp 9369d and I want to get this output: ...
3
votes
5answers
319 views

Awk - output the second line of a number of .dat files to one file

I have searched through previously asked questions and can't find an answer to what I am trying to do. note, I am new to linux. I have multiple files something like: (in reality i have 80) file1.dat ...
3
votes
3answers
244 views

Strip off word pattern at the end of string using sed

I have following string: ksh -x SCRIPT2.ksh $BUSSINESSDATE & PID=$!; ksh -x SCRIPT1.ksh $BUSSINESSDATE & PID=$!; I want to strip off the last word & PID=$!;. Desired result is: ksh -x ...
1
vote
1answer
43 views

Computing differences of column entries

I have a textfile with: 1 100 2 80 3 70 4 64 Now I need another file that contains the first column equal to above and the second column should be the difference of to consecutive values of the ...
2
votes
2answers
72 views

Search and Replace question

I need to find all occurrences of AAsomeArbitraryStringBB and replace it with CCsomeArbitraryStringDD. So AAHelloBB Text AAByeByeBB becomes CCHelloDD Text CCByeByeDD. It's important to note ...
2
votes
2answers
104 views

Extracting values from a file keyed by multiple keys

Consider a file with key=value pairs, and each key is optionally a concatenation of multiple keys. In other words, many keys can map to one value. The reason behind this is that each key is a ...
2
votes
4answers
108 views

Fastest way to read last line within pattern of multiple files with the same extension

I have in a directory /home/files/ some files that all have the same extension .txt. For example 1.txt 2.txt 3.txt 4.txt 5.txt test.txt These files are all the same in data format but the data are ...
1
vote
2answers
48 views

splitting a file by column number

I have a file with 10,671 columns. I am trying to split it into 10 smaller files, each with 1000 columns until all columns have been printed, named file_transposed01, file_transposed02, ... ...
2
votes
2answers
54 views

How to print out every other line to the monitor

Have this tiny script down below and want to print out to the screen every other result. What are the ways of doing it? #!/bin/bash for ((n=5, j=1; j<=9;j++)) do echo "rez: $n*$j"=$((n*j)) done ...
1
vote
2answers
54 views

find and replace a value in a line of text

I have a text file which contain only 1 line of text 1314:Battery:1.90:50:30 I'm trying to figure out if is there anyway to do a grep and sed to change the value 50 and 30 to the following ...
0
votes
1answer
32 views

Prepending text read from a file on the top of another file [duplicate]

I want to add "void mainfunction()" on the top of a file. I am reading the actual name of the file from another file and the name of the main function from another file. I was using sed -i '1i void ...
3
votes
2answers
129 views

Select lines starting with specific numbers

I have to show the PID, PPID and STATUS of the procceses on the system, but only the ones which PID starts with numbers in the range 5-8. So i tried something like this... ps xao pid,ppid,s | grep ' ...
2
votes
2answers
95 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
1
vote
2answers
46 views

Modifying part of a string in bash script

I have a file named variables.f90 , having many lines defining different variables, as follows :: integer::n_monomer=6800 real*8::rx=5.0d0 #... randomly integer and real numbers defined ...
1
vote
2answers
66 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
23 views

output neighbouring lines with line numbers while grepping

Is there a way to output the neighbouring lines along with the line numbers while grepping the contents of a file? I'm trying to get outputs of the form: $ ~/myscript.sh download file # 'download' ...
3
votes
3answers
533 views

Need to insert single quotes in text file for use as SQL query using sed

I have created a line of text that looks like this: INSERT INTO radcheck(id, username, attribute, op, value) VALUES (,,00:23:32:c2:a9:e8,Auth-Type,:=,Accept); I want to use sed to make it look like ...
2
votes
1answer
168 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
1
vote
3answers
146 views

Compare two files and print unmatched lines [duplicate]

I have two files with the below data; I need the difference between two files. I tried with diff but it also shows line which are common in the two files: (22372 Dec 4 15:36 ...
1
vote
1answer
42 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
-1
votes
1answer
47 views

Count values in first column where second column occurrences are the same in UNIX

I have to count values in first column of a file for which the second column occurrences are the same , using Unix. 12,abn 23,abn 45,hh 52,hh 89,mm 89,mm My output should be something like ...
1
vote
3answers
67 views

Grep lines that cotain “1111-11-11” or “2382-21-20”. (Any Numbers)

I have tried to do this and I came up with this grep -E '\<[0-9]{4}"-"[0-9]{2}"-"[0-9]{2}\>' It doesn't work and the reason for that is the "-" and multiple grep things, so I tried dividing ...
1
vote
1answer
60 views

iconv terminated by signal 7

I was running a little bash script on some text files: find . -name "*.dat" -exec iconv --from-code='UTF-8' --to-code='ASCII//TRANSLIT' {} --output={} \; My machine is a Ubuntu 14.04 LTS. After a ...
4
votes
5answers
468 views

What command(s) will feed a tab-delimited text file and cut each line to 80 characters?

I've got multiple-line text files of (sometimes) tab-delimited data. I'd like to output the file so I can glance over it - so I'd like to only see the first 80 characters of each line (I designed the ...
4
votes
1answer
103 views

How is ksh93 so fast?

So, in general, I tend to look to sed for text processing - especially for large files - and usually avoid doing those sorts of things in the shell itself. I think, though, that may change. I was ...
4
votes
2answers
304 views

Replace or append a line with string within a file

I am looking for advice on how to search a part of a string within a file and replace the complete line OR append that string to that file if not found. I "played" with sed for a while now, but ...
4
votes
4answers
114 views

Remove a character between 2 known strings

I have a data set like below: \"XXX \ START sapiodj \\" aj \d 2387 END hddo\" START bbcc \\" END ss My Requirement: I want to remove all occurrences of backslash\ and double quotes " between ...
3
votes
4answers
150 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
5
votes
3answers
190 views

Replace uppercase to lowercase except first letter of each word in specific line

I want to replace names that I received in uppercase letters only. The names are mixed with other information which should stay as it is. AUTH: at the beginning of the line identifies it as a line ...
1
vote
1answer
67 views

using bash script to sort text in a file

I would like to sort the output of multipath -ll into a csv type of file using bash script. For example # multipath -ll would give the following output 360060e80056fc30000006fc30000513c dm-41 ...
6
votes
3answers
223 views

sed with multiple expression for in-place arguement

I am trying to replace multiple words in file by using sed -i #expression1 #expression2 file Something 123 item1 Something 456 item2 Something 768 item3 Something 353 item4 Output (Desired) ...
5
votes
2answers
63 views

Merging/Combining 2 text files according to numeric field

I have 2 text files file_1: 14595|Age 35|Salary xx|Position ax|2013|Info 1|Info 2|Info 3|Info 4|Info 5 14649|Age 30|Salary xx|Position az|2015|Info 1|Info 2|Info 3|Info 4|Info 5 file_2: ...
0
votes
2answers
104 views

Extract multiple instances of text between two strings [closed]

I'd like to extract text between two strings in a text file that contains Spotify chart results Text file extract: ...
3
votes
1answer
76 views

sed repeating the last line in the stream

I have code in a script similar to this... smbconffile="/etc/samba/smb.conf" sed -i 's/.*[\[CMI\]]/\[CMI\$\]/' $smbconffile && echo "Success" || "Failed" sed -i 's/.*[\[LOCAL\]]/\[LOCAL\$\]/' ...
-2
votes
1answer
49 views

Output a colon delimited file to another file [closed]

I have the following example file: root:server1:2014-12-12:x:Open:adm bin daemon lp mail other root sys users sectmp:server1:2014-12-12:x:Open:users smbnull:server1:2014-12-12:x:Locked:ids ...
0
votes
2answers
42 views

Take the value of another column if the query column is between two column values

So i have a file with column that has information on salary (eg. 2674 dollars) - indi salary sam 2674 john 6375 max 9547 and another file with the scale of the salary ...
2
votes
1answer
267 views

Cut and Paste Commands

So I have: $ cat fruits 2 bananas 3 cherries 4 figs 5 dates 6 elderberries 7 apples 8 grapes and 1 $ cat prices 2 2.18 3 4.11 4 1.69 5 4.52 6 1.73 7 1.01 8 1.09 Every line from 'fruits' ...
6
votes
4answers
207 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
2
votes
3answers
245 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
2answers
67 views

Command to see if a word is in a list of files and output how many files it is in

There is a similar question to mine already but mine has a slight difference. I want to see how many files contain each word. When a file has a word then it increments the counter and then continues ...
1
vote
3answers
208 views

Replace a string and replace it with a part of itself

I am cleaning up my data that is stored in text files. Each line starts with a category label followed by the actual data that I want to clean up. There are many text files in different subfolders, so ...
0
votes
2answers
71 views

output a variable value using echo command [duplicate]

I want to output number of lines in text file.So I am using echo command but I am getting filename along with number of lines when I run the following code: echo "$page" > t total="$(wc -l t)" ...
0
votes
1answer
57 views

Removing contents from a file

I have a file like this >gene1*ENSG24 CTTGGGGGGCTGGGGGCCAGGTGAAAGGGAAATGGAGGGCAGCACCCGCG AGCCCTCATTGCCTATAGTGGTTTCCATGGCGATCATGTAAGAGTCAATG TCGTCATTGGCAAAGTCGTCCGGGTGGGGTGTGCTGTAGGCAGAATCGGA ...
4
votes
3answers
375 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
3
votes
2answers
70 views

How do I efficiently select specific line numbers from a list of records?

I want to take a list of numbers representing line numbers of a source file I want to filter and filter those lines from the source file. How I can build a unix pipeline to extract these lines from ...
1
vote
5answers
131 views

How to get the unique count of a particular part of a string

I have a set of data in a file. psf7433-nlhrms unit7433-nobody unit7333-opera bpx7333-operations app7333-osm unit7330-partners psf7331-pdesmond unit7333-projm mnp7330-redirect unit7333-retailbanking ...