Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

2
votes
2answers
64 views

How to sort data stream by 2 column names (column number may vary)?

I get a stream of data from an API with which look like this: redID blueID whiteID 1 22 2 44 15 41 2 15 15 31 2 14 What I need to do, is to sort ...
1
vote
2answers
60 views

Extract Values from Formatted Text

Is there an easy way to extract variables from a text file? E.g. given the output from ab: This is ApacheBench, Version 2.3 <$Revision: 1638069 $> Copyright 1996 Adam Twiss, Zeus Technology ...
2
votes
2answers
31 views

How to alter a column in a row in a tab delimited text file if it's duplicated

Lets say I have these many rows chr1 66999638 66999638 SGIP1 1 + chr1 66999251 66999251 SGIP1 1 + chr1 33545778 33549778 AZIN2 1 + chr1 8376144 8380144 ...
5
votes
4answers
260 views

convert table to ini file using bash arrays

I have a challenge to take an output file ("inventory.list") in this format: hostname1.env1.domain | abc | environment1 hostname2.env1.domain | abc | environment1 hostname3.env2.domain | abc ...
0
votes
2answers
45 views

How to Pivot the data by awk in unix shell script?

I'm trying to pivot a file using awk. This is an example of my input File: P2P,P2P,MP,DL,AIR,UP,HR,UPE,GSM,S,PP,1,NA,0.2,0.2 P2P,P2P,MP,DL,AI,UP,HR,UPE,GSM,F,PP,4,NA,0.2,0.8 ...
2
votes
2answers
61 views

Selective extraction of data

I have the following block of data in a txt file : Informatica(r) PMCMD, version [9.5.1 HotFix4], build [313.0217], SOLARIS 64-bit Copyright (c) Informatica Corporation 1994 - 2014 All Rights ...
3
votes
3answers
125 views

Intersperse lines from two files

I have a data file ($file1) which contains two lines of data per individual. I need to intersperse a third line of data from another data file ($file2). So my input looks like: >cat $file1 bob ...
3
votes
3answers
147 views

How do I remove newline character at the end of file?

Let me clear about newline character: $ echo Hello > file1 ; cat file1 Hello $ echo -n Hello > file2 ; cat file2 Hello$ Here you can see that file1 has newline character at the end whereas ...
8
votes
3answers
425 views

Extract text between three single quotes

I have the following in a file description: ''' This rule forbids throwing string literals or interpolations. While JavaScript (and CoffeeScript by extension) allow any expression to ...
1
vote
2answers
62 views

Adding “:” time-formatting using awk

For an input file named Lab1: 034023 052030 034023 022130 044023 012030 034223 022030 034123 152030 024023 152030 AWK command awk 'gsub(/[0-9][0-9]/,"&:",$1) ...
4
votes
2answers
64 views

BSD sed: Replace only the Nth occurrence of a pattern

Using BSD sed; How can I perform the following substitution?: Before: hello hello hello hello hello hello After: hello world hello hello hello hello In other words; how can I replace only the ...
0
votes
2answers
67 views

Making csv file

I have two different textfiles.txt (textfile1.txt and textfile2.txt) with different data sets in them. Is it possible to compile two seperate .txt files as a single CSV file? UPDATE textfile1.txt ...
1
vote
5answers
57 views

Filtering out inputs to textfile

I am using and Raspberry Pi (running raspbian) and I have a shellscript continuously outputting values into textfile via this command: wavemon -d |grep -i signal >> textfile.txt The output is ...
1
vote
3answers
54 views

Duplicate file, replace character, then concatenate

I have a file that looks like this: i36aasf5i7538i123 i47982i16537i1256 i1647i6458i3457 i1856i8456i43865 I want to make a copy of the file, in which the first i in every line is replaced by an o. I ...
1
vote
2answers
44 views

Print smallest integer from file using awk custom function?

awk function looks like this in a file name fun.awk: { print small() } function small() { a[NR]=$0 smal=0 for(i=1;i<=3;i++) { if( a[i]<a[i+1]) smal=a[i] else smal=a[i+1] } return smal } ...
1
vote
4answers
189 views

Move lines 2 3 after line 5 using awk

I have file content as: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line awk 'NR==2 && NR==3 {f=$0;next} NR ==5 ...
0
votes
1answer
37 views

Consecutively copy two lines and skip the third using awk

With fairly simply awk: awk '(NR%3)' awk.write for this file: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line I have output as: ...
3
votes
2answers
98 views

Find common rows on two gzipped files

I've got two big gzip files. I need to find the common rows. Zgrep (gzip) does this to a large extent like zgrep -f <unzipped file> <zipped file> but I can't find any switches to do zgrep ...
4
votes
3answers
72 views

Bash split line into multiple lines?

I want to know if there is an easy way to split a line in a file into multiple lines, such as this: I have A B C 1 2 3 4 And I want to get something like this, based on the first string of the ...
0
votes
0answers
26 views

Creating a mini text-editing program

I often copy-paste text from pdfs into other applications, and wish there was an easy offline way of editing the text's characteristics. For example, this website removes line breaks, while this ...
1
vote
1answer
37 views

Remove spaces in file under certain criteria

I want to clean up some files, and make make the way in which they are written more uniform. So, my input might look something like this: $a$h$l )r ^9 ^5 l \ urd The thing is, some spaces are ...
2
votes
2answers
56 views

Remove characters if they do not follow specified patterns

I want to clean up some files, and make make the way in which they are written more uniform. So, my input looks something like this: $a$h$l )r ^9 ^5 l \ urd The thing is, some spaces are ...
-2
votes
2answers
82 views

How do you remove newline character in the middle of file?

File structure: Col1|Col2|Col3|Col4 123|456|abcdefg Tel.: 689346589934, xyz lckhasfkl|abc I want to remove the newline character from 'col3' only.Please suggest
6
votes
4answers
218 views

CSV columnar reformat with SED (or anything other coreutil)

Given the following made-up segment from an output file from hashdeep: 7241,11111111111111111111111111111111,\01-data\file1 1237241,22222222222222222222222222222222,\01-data\file2 ...
0
votes
0answers
34 views

Fetch data from a CSV file based on multiple conditions

I'm trying a write a generic shell script which should extract data from a CSV file based on multiple conditions (generic) and write it to another CSV file. Sample CSV file: ...
2
votes
2answers
66 views

Extracting subset from fasta file

I have a fasta file which looks like this: >chr1 ACGGTGTAGTCG >chr2 ACGTGTATAGCT >chrUn ACGTGGATATTT >chr21 ACGTTGATGAAA >chrX GTACGGGGGTGG >chrUn5 TGATAGCTGTTG I just want to ...
3
votes
5answers
58 views

scanning a sequence and outputting a sub sequence

I am looking for a one liner for this if it is possible. I have a sequence like this CCGGTCTCTTCCGGTTCTGTCTTTTCGCTGG I want to get the output where I scan the sequence base by base and then take 20 ...
1
vote
3answers
38 views

Transpose paramter values from a file and Generate parameter files from it

There's a file containing a table with the parameter names as the first line, values in the following lines: How to perform the following using awk, sed or any shell scripting utility: generate a ...
1
vote
1answer
48 views

substitute from beginning of line with 1 of multiple patterns to the end of another line with 2nd pattern

I have a file like this: data data data $globaltext blah gibberjabber somemorestuff etc ); data data data $otherjunk yada gibberish etc etc more etc ); My desired result is this file: data data ...
4
votes
4answers
202 views

Get lines matching a pattern in one file and put them into a second file matching the same pattern

Let's say I have 2 files each containing lines which start with a 'b' character and I only want to merge these lines in the same order they appear in the first file. First File (1.txt) b 12 32 b 23 ...
1
vote
1answer
27 views

Dynamically computed string replacement with standard tools

Is there a UNIX mechanism for replacing matched strings dynamically--with a function of the matched string? E.g., say I want to replace URL matches with their URL-encoded counterparts or a convert ...
3
votes
3answers
46 views

Symmetric Difference Pipe?

I have a command that produces a lists of strings followed by newlines, a, and a file containing a list of strings followed by newlines, b.txt. I need a command that calculates the symmetric ...
1
vote
3answers
140 views

Number of occurrences in a text file where first character after 12th pipe is equal to 1?

I am fairly new to regular expressions and am looking for a sed/awk/grep/wc command to find the following in a pipe delimited text file the number of occurrences of the digit 1 after the 12th pipe. ...
1
vote
2answers
140 views

File manipulation; Insert a “;” every 12th, 44th and 60th character in every line of a text file

I have to make a csv-file from a textfile which contains data without any delimiters. Looks like: 2015021DUMMY001EAN1377689456777678945GERDE0002345 2015031DUMMY002EAN1345647876867867867TURTR0154565 ...
1
vote
3answers
47 views

remove files based upon diff of two text files

I have created two .txt files that look like: 5fba026dcf271c6fa174b9fc2fc1fefadf1370f9 ./.changeFiles/change/addImage1.change bb352fba5807f4d03ea11abb00e411b00ce38673 ...
3
votes
2answers
82 views

Merge and Append columns for more than 2 files

The files are tab separated. I have "N" number of files whose contents are as shown in the screenshot above. I want to merge them and append the 5th column. The first 4 columns are the same. I ...
1
vote
2answers
35 views

Return line conditonally on next line

I am trying to return a list of IP Addresses for computers that have a windows OS (for my security course). The output is in this format Nmap scan report for 192.168.xx.xxx Host is up (0.066s ...
1
vote
1answer
47 views

Simulate sed read command with awk

I want to stimulate use of sed: sed '3r awk.scr' awk.script with awk.scr: a b c d e f and awk.script as: hello there is hello i'am there is hello sdfdf dfdfdf aads 23213 3 434 Using awk ...
2
votes
2answers
60 views

How to duplicate a file a number of times while embedding an index in each file

I have a file named, for example, ascdrgi.txt, with the following contents: tiger lion cat I want to duplicate this file a (variable) number of times by changing the last character of the filename ...
3
votes
2answers
181 views

merge duplicate rows in columns

Given a file like this: x y y z x x x y z z y x x x z y y z z y x x x x x x x x I would like the output to be: x y+ z x x+ y z+ y x+ z y y z+ y x+ x+ Is that possible to do with awk or perl in a ...
1
vote
3answers
212 views

String cutting in Unix

How could I transform this text? Input file format: MOD MAXT:TID=TID190,MT=VCU,MAXT=100; Desired output format: MOD MAXT:TID,MT,MAXT MOD MAXT:TID190,VCU,100;
2
votes
2answers
58 views

How do I list all lines after two subsequent patterns are matched up to when they don't

I have files that are used to install files with specific owner:group and permissions. I'll ignore permissions for clarity and the details of installing files. Once an owner & group are set to a ...
-1
votes
1answer
43 views

row to column conversion in linux

I have a script which gives me output like this 10.230.146.10: Terminal ID Gol 10.230.146.10: Far End ID pol 10.230.146.10: Number of E1s 4E1s 10.230.146.10: Packet ...
1
vote
3answers
158 views

Read specified lines in file - BASH

I would like to read specified lines in file iptables. I want to read lines started with : and ends with space. File iptables # Generated by iptables 04:13:50 2015 *filter :INPUT ACCEPT [0:0] ...
1
vote
4answers
39 views

Cost efficiently pair each line with lines of another file

I have a very huge file (~10Gb data) which contains data in below format - '1','1' '2','2' '3','3' '4','4' '5','5' '6','6' '7','7' '8','8' '9','9' '10','10' and ...
2
votes
1answer
44 views

Extract URL from non-formatted text

I've found only examples on extracting substrings from formatted text like HTML files, but in my case I need to output a list of URLs for example: ... https://twitter.com/user1/status/xyza ...
4
votes
4answers
111 views

Extracting strings from between special characters

I have a large text file created from combining lots of html files. cat *.html > all_html_files.txt Inside the text file are specfic strings that I want to extract to another text file. For ...
4
votes
1answer
118 views

In POSIX sed, does the period (dot) match a newline in a multiline pattern space?

In GNU sed it works. For example, this matches two blank lines in a multiline pattern space (created with N) /^\n$/ Is this standard?
1
vote
2answers
14 views

bash regex to find and save string from file

I'm a noob, so thanks for your assistance. Need some bash help... My file, hi.html, has many lines of text. I need to hunt down the line that starts with Hello abc123 blah blah blah and then save ...
1
vote
1answer
42 views

Awk split and conditional printing

I have an ugly data file that I need to clean up - Date User Name Item Id Title Quantity Price ...