Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

2
votes
1answer
37 views

How to find out the biggest number in many documents that contains different numbers

For example, there are some temperature data at those folders in different time. temps.txt contains the temperature number. So how can I use bash script to find out the maximum temperature? (the ...
2
votes
1answer
18 views

grep fixed string at end of line

I need to grep for string containing special characters and make sure it is at the end of the line For example, in list: abc defg[h]ij I want to grep for defg[h]ij$ xyz 123*456 I want to grep for ...
27
votes
2answers
2k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loops are conceptual, reliability, legibility, ...
4
votes
5answers
1k views

How to remove character and space from a string

Hello I have one file with output Name : ABC Name : CDE Name : ZYS I want to get the result ABC CDE ZYS How can i get it through SED command. I have tried. sed s/'Name' but ...
1
vote
1answer
39 views

Add 10 to a value in large xml file

I have in a large xml file some hundred svg graphs where I want to change the <svg:g transform="matrix(1 0 0 1 59.06 88.74)"> 88.74 to 98.74 i.e to add 10 to every line that starts with ...
0
votes
1answer
23 views

Is there a standalone tool which will write reports from CSV data files

I download CSV files (e.g. my bank account transactions) and would like to build reports (to print) from the data. I have searched and found several report writer libraries, but all would force me to ...
0
votes
1answer
29 views

script - recursively iterate over newline delimited list using a given depth range [on hold]

Years ago I encrypted a disk back-up image with an extremely long passphrase which I can no longer recall. However the passphrase is comprised of a number of smaller phrases which I do recall. ...
8
votes
4answers
1k views

Any (text based) plotting tools in unix toolbox?

I was wondering if there are any generic plotting tools available in the unix toolbox? By plotting I mean simple bars or things you see in many curses based utilities. And by unix toolbox I mean ...
0
votes
0answers
30 views

Using s command to subsitute variables [migrated]

I need to replace a incorrect Illumina fasta header for formatting in QIIME, and this script doesn't quite work like I want. I need to able to put in variables and use the unix "s" command. Any ideas? ...
3
votes
7answers
169 views

Rows to column conversion of file

Suppose I have a file: File1: PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc File2 I want: PAPER TEAM MANISH NISHA GARIMA JYOUTI Rows to column conversion of File1.
0
votes
0answers
47 views

Cut specific characters into columns [on hold]

I have a file as below: 2043630001007316220473533459926000000000003960CANDOVER INVEST S LONGHURST EXP 143292043630001007304400000000000000000000000000000 ...
4
votes
7answers
444 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
4
votes
1answer
83 views

extracting a subset of lines from a large text file

I have a large file with 9 columns, tab delimited. This file is about 39MB, with about 250,000 lines. The last column, column 9, has information like this (note that this is all one column - thie ...
4
votes
7answers
538 views

Remove everything that is not a 4 character number

I need to remove everything that is not a 4 character number such as 9838, 6738, 1337 or 1889. I though that this command would work: sed 's/....[^0-9]//g' . Means any character in regex, and ...
5
votes
2answers
3k views

Single command taking two strings to extract string between them, like 'tr' (without expressions)

Saw here a way to use sed to get text between two other strings in a line, like: sed 's/.*starting_text\(.*\)ending_text.*/\1/' but I'd like a simple command (like tr, but for string extraction) ...
1
vote
3answers
139 views

sort CSV by number of column in rows?

I have multiple CSV files with many records. The total number of rows should be 134. I have many files, and each rows has its own number of columns (from 15 to 200). I need to sort them according to ...
0
votes
3answers
66 views

How to print only certain characters?

I have to translate the permissions of a file this way: r ► c w ► s x ► e Then, I must extract the group of characters which are related to owner group. Last step is to print that group of ...
0
votes
1answer
37 views

sed command to print all line starting from and end to specific words present in a file

Suppose I have one file File1 consist two Words File1 : PAPER TEAM Now i need to print all the lines of a file M starting from word PAPER end to word Team sed command to print between two words ...
13
votes
14answers
3k views

How to read an IP address backwards?

If we have this string (IP address): 192.168.1.1 How can I derive the (DNS reverse record form) from this string, so it will be shown like 1.1.168.192.in-addr.arpa using a shell script?
3
votes
4answers
124 views

Get pairs of lines and multiply [closed]

I'm trying to parse an html file for some values using bash. So I've got a big text file with numbers like this 3 2 4 2 What I want to do is get two lines, multiply both numbers and then store the ...
3
votes
4answers
283 views

How to use sed to selectively remove leading zeros

How can I substitute/remove 0's in this, using sed: 2001:0db8:03cd:0000:0000:ef45:0006:0123 to make it looks like this: 2001:db8:3cd:::ef45:6:123 There are 8 pieces of 4's divided by : in the ...
10
votes
2answers
10k views

How to print only the duplicate values from a text file?

Suppose there is a column of numeric values like following: File1: 1 2 3 3 3 4 4 4 5 6 I want the output: 3 4 That is, only the repeated lines. Is there any command line tools to find this ...
11
votes
5answers
33k views

How to strip multipe spaces to one using sed?

sed on AIX is not doing what I think it should. I'm trying to replace multiple spaces with a single space in the output of IOSTAT: # iostat System configuration: lcpu=4 drives=8 paths=2 vdisks=0 ...
25
votes
12answers
51k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
47
votes
11answers
21k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M ...
3
votes
1answer
92 views

Sorting lists when there are no leading zeros

I would like to sort this numerically and the sort command keys off the first character. CPL_V11.01.00_1 CPL_V11.01.00_10 CPL_V11.01.00_2 CPL_V11.01.00_3 CPL_V11.01.00_35 CPL_V11.01.00_36 ...
0
votes
1answer
47 views

No output from grep -P 'com\K\w+'

I have a source file saved off the internet which has various links I want to pull data from. For example https://blahblah.com/uniquetext I'm using grep -oP 'com\K\w+' source.txt > list.txt ...
1
vote
2answers
39 views

Oneliner which prints groups and their numbers which start with a '1'?

I have to write an oneliner which print system groups and their identifiers, for all groups whose identifiers start with '1'. For example, the result of cat command on /etc/group is this: I must ...
3
votes
1answer
78 views

How do I use awk to create an index of words in file?

I have a problem with my homework I need to create a shell script using awk. The script will output only the words in the file in an alphabetical order, and after the word is : after that a space then ...
0
votes
3answers
55 views

Group by and sum in shell script without awk

I have a file like: $ cat input.csv 201,100 201,300 300,100 300,500 100,400 I want to add the values in column 2 which has same value in column 1. Expected output is as follows: $ cat output.csv ...
1
vote
3answers
51 views

Merging multiple rows into a single one using awk

I'm working with a very large data set in the format: chr1 45162 . . C T chr1 45162 . . C T chr1 45162 A . . T chr1 45162 . . C T chr1 45257 A . . T chr1 45257 A . . T chr1 45413 A . . T chr1 46844 ...
0
votes
1answer
33 views

How to Trim below highlighted lines from Shell output

How to Trim below highlighted lines from Shell output. Source shell program is below the output. $ ./checkhome.sh The Oracle base remains unchanged with value /opt/oracle Connecting ...
3
votes
2answers
54 views

How to remove specific keys from a wordlist?

I have generated a wordlist of keys starting from 03000000000 and ends at 03999999999 through crunch. So what I want is to delete a specific number of keys from it, for example, I want to delete keys ...
4
votes
3answers
754 views

paste command: setting (multiple) delimiters

In Linux, I have the following problem with paste from (GNU coreutils) 8.13: Trying to set another delimiter than the default (TAB) results in either just printing the first character of the defined ...
14
votes
6answers
24k views

How to cut part from log file?

I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?
13
votes
8answers
2k views

Replace string containing newline in huge file

Anyone know of a non-line-based tool to "binary" search/replace strings in a somewhat memory-efficient way? See this question too. I have a +2GB text file that I would like to process similar to what ...
1
vote
1answer
80 views

How to export string about France to Microsoft excel file?

One string file Str.rc about French language is here: ID_STR_BRIGHTNESS;,"Luminosité" ID_STR_CHILE_EASTER_ISLAND;,"Île de Pâques" ID_STR_CURRENT_CH;,"Saisie chaîne" ID_STR_DETAILS;,"Détails" ...... ...
3
votes
3answers
24 views

Parsing Log Files with sed -e. Need to count unique class names

I have a file, let's call it filename.log, in it I have something like this (2014-11-18 14:09:21,766), , xxxxxx.local, EventSystem, DEBUG FtpsFile delay secs is 5 [pool-3-thread-7] (2014-11-18 ...
17
votes
5answers
5k views

echo a file without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
0
votes
1answer
441 views

How to print gmail inbox messages

I want to print inbox messages one by one (like cat someFile) so that I can parse the message contents. something like printGmail InboxMesg1 | grep some | sed some Using curl I can access the ...
1
vote
1answer
40 views

Removing duplicate lines after comparing first pattern and special pattern

I'm looking for code to removing duplicate lines after comparing the first tab-delimited pattern: Example : (↦ = tab) car ↦ ab ↦ 38/43 car ↦ ca ↦ 2/34 house ↦ ca ↦ 34/34 house ↦ d ↦ 2/3 house ↦ ht ↦ ...
3
votes
5answers
164 views

removing extensions in a column

I have a file like this ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402.5 ILMN_1343295 CTTCAACAGCGACACCCACTCCTCCACCTTTGACGCTGGGGCTGGCATTG NM_002046.3 ILMN_1651209 ...
1
vote
2answers
77 views

Search for pattern and append line to another file

I have a file like this (five tab-separated columns) head allKO.txt Metabolism Carbohydrate metabolism Glycolisis K07448 Metabolism Protein metabolism protesome K02217 and I want to search for the ...
8
votes
4answers
16k views

How to use bash script to read binary file content?

I want to read a character and then a fixed length of string (the string is not null terminated in the file, and its length is given by the preceding character). How can I do this in a bash script? ...
0
votes
2answers
27 views

Grep matching cell of the csv file and index of that row

I have csv file and I want to see numbers that start with 0.99 and their row's index which is the first cell of that row. This is what I have so far: cat fil.csv | grep '0\\.99'| tee > (cut -d, ...
3
votes
4answers
266 views

Command to remove a portion of JSON data from each line?

I have test.json file with different lengths of rows. Some fictitious example: { a: 123, b: sd, c: x45, d: 1, e: '' } { a: 5, b: bfgg, c: x4c, d: 31, e: '' } I want to cut the whole substring after ...
2
votes
1answer
31 views

How can I add four spaces before each lines of programs code in terminal

I usually go to stackexchange when I face programming terminal and I spend most of time trying to append four spaces before my pasted programming code. Is there any faster way to do it from a ...
3
votes
3answers
128 views

Grep for searching pattern in a file

I want to use grep to search pattern from one file in a second one. My pattern file is something like: K02217 K07448 KO8980 The file to search in is: >aai:AARI_24510 proP; proline/betaine ...
8
votes
6answers
406 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
3
votes
2answers
144 views

Bash - echo line by line, ignoring the space between the lines

I had read the post on: bash - replace space with new line, but it does not helped to solve my issue. [My Directory] [root@testlabs Config]# ls Archive Backup_Files config_file.tgz ...