Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
2answers
23 views

Append mth and nth columns of a file with the columns of another file

I have 2 files having different number of columns and with huge number of rows: bash-3.00$ cat fileA a1,a2,a3,a4,a5,a6,a7,a8,a9 q,w,,,,y,u,, a,z,,,,q,n,, ......................... z,p,,,,w,e,, ...
13
votes
8answers
8k views

How can I print lines from file backwards (without using “tac”)?

I want to print lines from a file backwards without using tac command. Is there any other solution to do such thing with bash?
2
votes
5answers
67 views

bash expanding to same value instead of line by line value in sed command

I have a file containing numbers in it say numbers.txt as follows. 9374541632553 51243747879841 32030098896914 84654557358238 11000656847765 I want to convert it to the following format. ...
2
votes
3answers
105 views

Remove the ^L (aka \f/FF/form-feed/page-break) character

How do I remove the ^L character and the blank lines that come after it on a unix file? I have tried the below and have been able to remove the VT and spaces but am failing to remove the ^L character ...
3
votes
3answers
114 views

If statement string comparison

I'm reading a csv file with 14 values per line. I want to convert the HEX value from 10th [f10] value to a DEC value. In order to do this if the 7th [f7] value must hold EnHr or EnSt or SpJb or Chem ...
1
vote
3answers
36 views

Adding a pattern to each field of the following line

I need some help here please. I have the following input: MX04A; DMX04A; DMX04A; LMX04A; LMX04A; -17.2; -15.3; -14.3; -13.6; -16.8; -15.4; -16.0; -15.3; LH36A; DLH36A; DLH36A; -11; -117.2; -11; ...
93
votes
3answers
192k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
2
votes
3answers
131 views

How to print the before lines of search keyword with out using grep in a gz file?

How to print the lines before a search keyword without using grep in a .gz file? ID:342N000390AAAAAAAA 07/14/15 10:26 (MV90 ) * Register Data Imported * Warning - No ...
2
votes
2answers
47 views

Awk - keep word “example” and “EXAMPLE” with one input

I am attempting to to only keep customers that contain certain words. Here is an example < newname.txt awk 'BEGIN {FS=","} $7~/USA|United States/' > usa.txt The problem I am having is if they ...
4
votes
6answers
170 views

Extracting lines based on conditions

Each line in a comma-separated file has 5 fields. a,b,c,d,e f,g,c,i, j,k,c,m,n o,p,c,r,s t,u,c,w, x,y,z,aa,bb How can I extract the lines which have c in the 3rd field and their 5th field is NOT ...
0
votes
0answers
32 views

Drawing a histogram from a bash command output

I have the following output from a miRNA Predictor: Read Sequence:hsa-miR-576-3p MIMAT0004796 (22 nt) Read Sequence:gi|182396|gb|M74088.1|HUMFAPAPC Human APC gene mRNA, complete cds (8972 nt) ...
3
votes
3answers
202 views

awk comparison using arrays

I have the following file: 6180,6180,0,1,,1,0,1,1,0,0,0,0,0,0,0,0,4326,4326,,0.440000, 6553,6553,0,1,,1,0,1,1,0,0,0,0,1,0,1,0,4326,4326,,9.000000, ...
2
votes
3answers
233 views

How to 'drop'/delete characters from in front of a string?

I have a string that I would like to manipulate. The string is H08W2345678 how would I be able to manipulate it so the output is just W2345678? Similarly if the I wanted to drop the last 4 ...
3
votes
2answers
771 views

Remove slashes/parent paths from filenames inside CSS and Javascript content

I need to go through all my css and js files and if there is a filename referenced that has any slashes (/) at all then the slash should be removed. What I want is: if any files referenced are ...
3
votes
2answers
637 views

Parsing through log file and printing out regex backreferences

I want to parse my /var/log/maillog to print out the email addresses in from=<> and to=<> (mainly to do a quick check for false positives in the DNS RBLs that I've configured) A maillog ...
2
votes
1answer
2k views

Enable line breaks when assigning multi line outputs to var in bash

I'm using the following very basic shell script I copied off the net to list the contents of my database backup folder and mail them regularly: MYSQLLIST=$(ls -lhG /var/backups/mysql/daily/) ...
3
votes
2answers
831 views

How can I count lines with title-case words of certain length?

How can I create a shell script in unix/linux to find the number of lines in a text file that have words which are 5 to 10 character long and begin with a capital letter?
8
votes
5answers
618 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
2
votes
1answer
125 views

The cursor to a match, edit, the cursor to the next match, edit — like in a directory?

Algorithmic blackout: problem: find the files that match problem: stack/buffer the matches move the cursor to the next file of the stack and thereby to the location of the match repeat step 2 ...
3
votes
6answers
357 views

Grep the text which matches with a pattern

I was looking for a command which will print the selected portion of a line which matches the conditions. Eg, my text line is like this: p1=X||p2=Y||p3=X||p4=X||p5=X||p6=Y||p7=X I want to print ...
2
votes
2answers
18 views

How to replace value for a given condition in specific column of file

I have file separetd by pipe | and I want that when column 6 contain letter I print 0758000 in column 7 and when the column 6 contain letter A print 0800000 in column 7, I can not find how to do it!!! ...
1
vote
2answers
33 views

Escaping double quotes in string

I have a string message which contains data in double quotes. I need to put \ in front of every double quote how can u do it in shell ? I used Perl and it works fine, but I'm looking for sh command: ...
3
votes
4answers
96 views

Efficient data extraction from multiple files to a single CSV file

I have a large collection of XML files with the same exact structure: $ cat file_<ID>.xml ... ... ... <double>1.2342</double> <double>2.3456</double> ...
1
vote
0answers
44 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or ...
5
votes
4answers
134 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
2
votes
2answers
29 views

Shell script - text processing help

I want to create a ldif modify file to update the users on LDAP. I have an input file(alluserdns.ldif) with list of all Dn entries in ldap: dn: cn=testuser2@abc.com, ou=persons,ou=pp,dc=abc,dc=com ...
3
votes
2answers
342 views

What is the difference between > and >> (especially as it relates to use with the cat program)?

Suppose that I have a file called temp.txt. Using the cat program, I would like to add the contents of this file to the end of myfile.txt -- creating myfile.txt if it does not exist and appending to ...
1
vote
1answer
25 views

Loop over lines of file, partition file and classify each partition

In the following I've put up my code for iterating over lines of a file in order to collect data within a segment from the file in following fashion: Illustration for understanding procedure: segL ...
0
votes
1answer
27 views

How to replace the content of specific column with awk? Tab Delimited File

Here's a similar question but I ran into a problem that was not addressed in the question. How to replace the content of specific column with awk? I want to do something similar, however my file is ...
1
vote
1answer
19 views

AWK: where is my error?

I have a file with :-delimited fields. If the 3rd field of a line is greater than 22, that line should be printed to file44. The file looks like this: 1:2:3:4 1:2:3:411 1:2:33:42 1:2:3:44444 ...
2
votes
3answers
60 views

Read a column from a file and append a specific column of another file

I have two files as such: fileA: Name,Site Name,Product UPEI,Product Name,NMS Flag,Product Model, SW FileB: NE_Name SW ABC 4.4 ASD 4.3 ... ... Note that ...
1
vote
4answers
134 views

UNIX to print a content from a .gz file

I have a .gz file with contents like the following: ID:123455 23-07-2015 mvni warning: RTM post warning : no profile data Using the string no profile data, I want to print the ID. I tried with ...
19
votes
15answers
1k views

How to find line with least characters

I am writing a shell script, using any general UNIX commands. I have to retrieve the line that has the least characters (whitespace included). There can be up to around 20 lines. I know I can use ...
2
votes
2answers
331 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
8
votes
4answers
3k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
2
votes
5answers
163 views

Perl script to compare two consecutive lines for the first character and ignore the second line if both first characters are same

Can anyone help how to compare two consecutive lines from a text file for the first character and if both the first characters are same ignore/delete the second line and print into new file only the ...
5
votes
4answers
1k views

How can I delete a trailing newline in bash?

I'm looking for something that behaves like Perl's chomp. I'm looking for a command that simply prints its input, minus the last character if it's a newline: $ printf "one\ntwo\n" | ...
0
votes
0answers
16 views

Adapt text lines to fixed length [duplicate]

With the shell command fmt -w 30 myfile it is possible to truncate the myfile lines so that their length is at most 30 characters. The result is however not a justified file: some lines will be ...
9
votes
7answers
6k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
7
votes
4answers
316 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
0
votes
3answers
70 views

Search for string(s) in files and report four lines of context (match and three following)

I have a string (or a list of strings) that is/are to be searched in all the files of a directory.  For example, if I am searching for ABC, and file1.txt contains ABC on line 6, then I want lines 6, ...
7
votes
6answers
588 views

How do I display the nth result of an ls command?

I used this command to display the first result of files in my directory. ls | head -n 1 My simple question is, how can I modify this command to display say the nth result? Thanks!
10
votes
5answers
662 views

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
2
votes
0answers
100 views

Bash - Convert escape character colored text to BBCode

I want to convert a terminal escape-character colored text file to BBCode colored text. For this i've created an Android logfile with logcat -Cd > /sdcard/logcat.txt. The -C switch adds the color ...
1
vote
2answers
50 views

Reinterpret terminal output

I have a program that executes some commands in a terminal (for example apt-get update) and stores the output. How can I reinterpret the output, so I can remove all information that is not needed ? ...
1
vote
3answers
69 views

Bash convert string to array of strings?

I've got a string like this: 8080 "ac ac df asd" 9019 "f v adfs" 1 "123 da 123x" Is there a clever way to convert this into arrays like this using Bash? 8080 "ac ac df asd" 9019 "f v adfs" 1 "123 ...
-1
votes
2answers
51 views

Substitute a string containing && with sed [closed]

I've wrote some script for automated technical acceptance tests. Since we're not allowed to write password in clear text, the script I used is associated with many variables, one of which is the ...
2
votes
4answers
191 views

Need to print the matching lines in different file

Eg: I have data like this 1111111 test1 test2 test3 1111111 block 1 block 2 block 3 1111111 page1 page2 page3 I would like to print the content as different files as bellow. Test1 - File 1111111 ...
1
vote
2answers
103 views

Remove lines from CSV file if the second column is in a word list

I have found a long list of free email providers that I want to remove from my email lists - https://gist.github.com/tbrianjones/5992856 Below are two commands I currently use that do the same job ...
1
vote
2answers
72 views

How to remove characters of text in one line before specific character

How to remove a characters of text in one line before specific character I will remove characters to the left of the first letter "F" Example: null###F01|54646466|00K4234001|IFD|1990101 ...