Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
0answers
7 views

extracting a subset of lines from a large text file

I have a large file with 9 columns, tab delimited. This file is about 39MB, with about 250,000 lines. The last column, column 9, has information like this (note that this is all one column - thie ...
4
votes
6answers
318 views

Remove everything that is not a 4 character number

I need to remove everything that is not a 4 character number such as 9838, 6738, 1337 or 1889. I though that this command would work: sed 's/....[^0-9]//g' . Means any character in regex, and ...
1
vote
3answers
127 views

sort CSV by number of column in rows?

I have multiple CSV files with many records. The total number of rows should be 134. I have many files, and each rows has its own number of columns (from 15 to 200). I need to sort them according to ...
0
votes
3answers
60 views

How to print only certain characters?

I have to translate the permissions of a file this way: r ► c w ► s x ► e Then, I must extract the group of characters which are related to owner group. Last step is to print that group of ...
22
votes
2answers
1k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loops are conceptual, reliability, legibility, ...
0
votes
1answer
32 views

sed command to print all line starting from and end to specific words present in a file

Suppose I have one file File1 consist two Words File1 : PAPER TEAM Now i need to print all the lines of a file M starting from word PAPER end to word Team sed command to print between two words ...
4
votes
6answers
389 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
3
votes
4answers
278 views

How to use sed to selectively remove leading zeros

How can I substitute/remove 0's in this, using sed: 2001:0db8:03cd:0000:0000:ef45:0006:0123 to make it looks like this: 2001:db8:3cd:::ef45:6:123 There are 8 pieces of 4's divided by : in the ...
3
votes
1answer
92 views

Sorting lists when there are no leading zeros

I would like to sort this numerically and the sort command keys off the first character. CPL_V11.01.00_1 CPL_V11.01.00_10 CPL_V11.01.00_2 CPL_V11.01.00_3 CPL_V11.01.00_35 CPL_V11.01.00_36 ...
0
votes
1answer
47 views

No output from grep -P 'com\K\w+'

I have a source file saved off the internet which has various links I want to pull data from. For example https://blahblah.com/uniquetext I'm using grep -oP 'com\K\w+' source.txt > list.txt ...
1
vote
2answers
39 views

Oneliner which prints groups and their numbers which start with a '1'?

I have to write an oneliner which print system groups and their identifiers, for all groups whose identifiers start with '1'. For example, the result of cat command on /etc/group is this: I must ...
0
votes
3answers
52 views

Group by and sum in shell script without awk

I have a file like: $ cat input.csv 201,100 201,300 300,100 300,500 100,400 I want to add the values in column 2 which has same value in column 1. Expected output is as follows: $ cat output.csv ...
3
votes
1answer
77 views

How do I use awk to create an index of words in file?

I have a problem with my homework I need to create a shell script using awk. The script will output only the words in the file in an alphabetical order, and after the word is : after that a space then ...
1
vote
3answers
51 views

Merging multiple rows into a single one using awk

I'm working with a very large data set in the format: chr1 45162 . . C T chr1 45162 . . C T chr1 45162 A . . T chr1 45162 . . C T chr1 45257 A . . T chr1 45257 A . . T chr1 45413 A . . T chr1 46844 ...
0
votes
1answer
30 views

How to Trim below highlighted lines from Shell output

How to Trim below highlighted lines from Shell output. Source shell program is below the output. $ ./checkhome.sh The Oracle base remains unchanged with value /opt/oracle Connecting ...
3
votes
2answers
53 views

How to remove specific keys from a wordlist?

I have generated a wordlist of keys starting from 03000000000 and ends at 03999999999 through crunch. So what I want is to delete a specific number of keys from it, for example, I want to delete keys ...
3
votes
3answers
24 views

Parsing Log Files with sed -e. Need to count unique class names

I have a file, let's call it filename.log, in it I have something like this (2014-11-18 14:09:21,766), , xxxxxx.local, EventSystem, DEBUG FtpsFile delay secs is 5 [pool-3-thread-7] (2014-11-18 ...
3
votes
5answers
164 views

removing extensions in a column

I have a file like this ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402.5 ILMN_1343295 CTTCAACAGCGACACCCACTCCTCCACCTTTGACGCTGGGGCTGGCATTG NM_002046.3 ILMN_1651209 ...
1
vote
2answers
72 views

Search for pattern and append line to another file

I have a file like this (five tab-separated columns) head allKO.txt Metabolism Carbohydrate metabolism Glycolisis K07448 Metabolism Protein metabolism protesome K02217 and I want to search for the ...
2
votes
1answer
30 views

How can I add four spaces before each lines of programs code in terminal

I usually go to stackexchange when I face programming terminal and I spend most of time trying to append four spaces before my pasted programming code. Is there any faster way to do it from a ...
0
votes
2answers
26 views

Grep matching cell of the csv file and index of that row

I have csv file and I want to see numbers that start with 0.99 and their row's index which is the first cell of that row. This is what I have so far: cat fil.csv | grep '0\\.99'| tee > (cut -d, ...
3
votes
4answers
262 views

Command to remove a portion of JSON data from each line?

I have test.json file with different lengths of rows. Some fictitious example: { a: 123, b: sd, c: x45, d: 1, e: '' } { a: 5, b: bfgg, c: x4c, d: 31, e: '' } I want to cut the whole substring after ...
3
votes
3answers
126 views

Grep for searching pattern in a file

I want to use grep to search pattern from one file in a second one. My pattern file is something like: K02217 K07448 KO8980 The file to search in is: >aai:AARI_24510 proP; proline/betaine ...
1
vote
1answer
40 views

Removing duplicate lines after comparing first pattern and special pattern

I'm looking for code to removing duplicate lines after comparing the first tab-delimited pattern: Example : (↦ = tab) car ↦ ab ↦ 38/43 car ↦ ca ↦ 2/34 house ↦ ca ↦ 34/34 house ↦ d ↦ 2/3 house ↦ ht ↦ ...
0
votes
3answers
36 views

How can I convert timestamps in a column to a date?

I have a file containing this: 1415602803,LOGIN SUCCESS,AUTH,user2,192.168.203.63,10.146.124.73,59996,22 1415602807,LOGIN SUCCESS,AUTH,user1,172.24.31.10,172.32.1.1,48191,22 1415602811,LOGIN ...
3
votes
4answers
61 views

Replace a marker in a text file with arbitrary new text

I have a file which contains some text and - on a single line - a marker which indicates where new content should be added foo bar %SUBSTITUTE% foo The line line substitute should be replaced by a ...
3
votes
4answers
123 views

Get pairs of lines and multiply [closed]

I'm trying to parse an html file for some values using bash. So I've got a big text file with numbers like this 3 2 4 2 What I want to do is get two lines, multiply both numbers and then store the ...
3
votes
2answers
198 views

Delete every Nth line in shell

I'm trying to delete a line after N lines using awk and I can't seem to get it right. The file format is like this YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX The real example would be office3 3 1 ...
3
votes
2answers
141 views

Bash - echo line by line, ignoring the space between the lines

I had read the post on: bash - replace space with new line, but it does not helped to solve my issue. [My Directory] [root@testlabs Config]# ls Archive Backup_Files config_file.tgz ...
1
vote
1answer
28 views

Using “ps -C java -o pid,user,cmd” Can I reduce the output of cmd?

I am trying to get this command down to just the essential information that I need. In the command cmd comes back with this output... /usr/local/jre1.7.0_51/bin/java -Dwd.tag=boxect ...
8
votes
6answers
404 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
1
vote
0answers
31 views

How can i compare 2 files and add lines that aren't on the first one?

I have run into a problem while doing some coding, and I need to compare two files, here is an example input file inputfile1.txt: car php module so js node array and another input file: ...
0
votes
3answers
80 views

Simple way to extract value from HTML

I have a very simple html file with a value inside. Value is 57 in this case. <eta version="1.0"><value uri="/user/var/48/10391/0/0/12528" strValue="57" unit="%" decPlaces="0" ...
7
votes
4answers
298 views

Output part of each line to a separate file

I have a file like this: a AGTACTTCCAGGAACGGTGCACTCTCC b ATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCAT c ATATTAAATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCATCCACTCCACAC d ...
2
votes
2answers
33 views

Match missing values in text files

Referring to this question : Finding missing value in text file i have 2 files containing below data example Name Feature Marry Lecturer Marry Student Marry ...
1
vote
4answers
34 views

Sort datetime string in 12-hour format

I have a log file with contents like this: 11-12-2014 - 03:03:59AM lat = 41.990516; lon = -93.430704<br> 11-12-2014 - 03:05:15AM lat = 41.001546; lon = -93.443352<br> 11-12-2014 - ...
1
vote
2answers
21 views

Comparing value in same column and directing output to new column

I have my input file as DPortal=ECCN RemoteFile=4004_130122 0256 A02 141111 0940 29343414 11 110005 DPortal=ECCN RemoteFile=4004_130122 0256 A02 141111 2336 29843714 11 110006 DPortal=ECCN ...
1
vote
1answer
31 views

How to replace a symbol in sed only if it occurs before a letter?

In my documents, I have lots of occurences like these: [["A [["b [["8 [["c [["F [["1... What I need to do is to replace double brackets with single brackets, but only if there is a letter after a ...
8
votes
8answers
497 views

Grep for pattern at start or middle of a line

I'll start by saying that I think this problem is a little less innocent than it sounds. What I need to do: check for a folder within the PATH environment variable. It could be at the start or ...
0
votes
3answers
41 views

Reduce length of specific column in delimited text file

I need for the 5th column in a delimited text file to be reduced to just the first 5 characters. All other columns must remain unedited. Input: file1.txt column1 column2 column3 ...
3
votes
2answers
74 views

Finding missing value in text file

I have a text file with following data. Name Feature Marry Lecturer Marry Student Marry Leader Bob Lecturer Bob Student ...
3
votes
3answers
162 views

Extracting part of lines with specific pattern using awk,sed

I have a question regarding the awk/sed operators. I have a big file which has following set of lines repeated Expression loweWallrhoPhi : sum=-6.97168e-09 Expression leftWallrhoPhi : ...
1
vote
3answers
35 views

How to add a column from a file to the last column of another file using awk?

I want to add the second column of file2 to file1, that contains a number an unknown number of columns, so becoming the last column of file1. In this aim, I tried to create a file3 composed by pasting ...
0
votes
1answer
39 views

insert comma after the last letter

I use a txt file, and I need to convert it in csv file Saint Petersburg 0 10 0.1 - N Moscow - 9 0 - N Novgorod 0 7 1 30 Y In ...
3
votes
3answers
284 views

Take the first column from 2 files and write it to a 3rd file

Suppose I have 2 Files, ABC.txt & PQR.txt with the data shown below as an example: ABC.txt: ABC DEF PQR.txt: PQR XYZ I want to grep column 1 from both files and write into a third text ...
3
votes
2answers
228 views

random permutation of lines of text

If I have a file with following content: 0001 0002 0003 0004 0132 0137 0138 0141 How can I get a random permutation of them in bash?
7
votes
2answers
233 views

How can I delete all text between nested curly brackets in a multiline text file?

This question comes from How can I delete all text between curly brackets in a multiline text file? (just the same, but without the requirements for nesting). Example: This is { {the multiline ...
3
votes
3answers
200 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
0
votes
1answer
36 views

Macbook System Profiler formatted

I want to format the output generated from macbook terminal command system_profiler and transpose the row values to columns under the same Application. Here's a snippet of the file: Firefox: ...
2
votes
5answers
142 views

Re-order text on each line in a file

I'm processing a log file and I need reorder each line (not sort). The log file looks like this: 11-06-2014 - 10:49:06PM lat = 41.858657; lon = -91.345142 11-06-2014 - 10:49:49PM lat = ...