Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
3answers
35 views

vlookup work in awk on linux

Hi I have two files containing data: file1: 4,abc 3,xyz 5,fut 6,gfd file2: gfd,2.3 xyz,4.5 abc,6.7 fut,3.2 i wish to create a file3 as output file using linux scripting commands: 4,6.7 3,4.5 ...
10
votes
1answer
226 views

How to sort a file based on a section of a field

I would like to sort a file : D104HN-D104HA 8.320 4.521 1.69e+05 -- D104HN-D104HB* 8.320 2.823 2.93e+05 -- A90HB#-A90HA 1.655 4.207 7.12e+05 -- ...
1
vote
2answers
21 views

Copy plaintext lines from file and append them to same file with mid-line prefix or remove mid-line prefix

Making an ugly script that compiles a bunch of hosts files into one big list. Most lists will have two entries for a URL, one with www. and one without like this: 127.0.0.1 facebook.com 127.0.0.1 ...
-1
votes
1answer
52 views

Replace non-ASCII characters with space in a file

I am using the following command to replace the non-ASCII characters, single quotes and non printable characters: sed -i -e "s/'//g" -e's/'//g' -e's/[\d128-\d255]//g' -e's/\x0//g' filename ...
0
votes
3answers
67 views

How to print the difference of two text files using shell scripting?

I have two text files containing a list of URLs. I want to compare the two files and print out the differences. I know that we can use 'diff' but here's the problem. The urls on one file are full e.g. ...
1
vote
5answers
262 views

To remove extra space from line using shell command

This is my file, TLRUIDA CBdms Status DP 6/1/1 DC 6/1/5 0 Y TLRUIDA CBdms Status DP 6/2/1 DC 6/2/5 0 Y TLRUIDA CBdms Status ...
-2
votes
2answers
37 views

Print every fifth occurence

I have following configuration in the file. But i have such many configs in the file. I have to print a word which are enclosed in [] brackets. I want to print the word summary_sai_verbose only once ...
1
vote
1answer
31 views

Searching and replacing a windows style path using sed

My task is to replace multiple entries for \\\\\\\\server\\\path1\\\\path2\\\\ with \\\\server\\path1\\path2\\ in myfile.txt (removing the superfluous backslashes). I have tried using sed for this ...
0
votes
1answer
36 views

How can I get many addresses from many trajectories with bash script without error?

I have many trajectories data. 500k. I try to get adresses of these coordinates. I use bash command in linux. #!/bin/bash filename="$1" while read -r line do uid=$(echo $line | cut -f1 -d,) ...
3
votes
5answers
157 views

Command line method to find repeated word typos in English text

Updated: Added third, multi-line sample to test solutions with From the command line, is there a way to check a file of English text and find repeat-word typos, in order to help correct them? To ...
1
vote
2answers
42 views

How to select a row with latest date and time

I have a text file (space as delimiter) as shown below: Date Time---------OtherFields-------Source IP Destination IP 2014-11-24 12:58:59.290 1.2.3.4 ...
3
votes
1answer
14 views

Searching for specific user group membership with an exact match

A user has multiple group memberships that contain spaces. Example: mydomain\staff mydomain\staff admin mydomain\staff super User groups can be returned using: id -Gn username | grep -i -o ...
1
vote
2answers
31 views

Cat all files in a folder including filename by using a for loop?

With this for loop which I execute directly in the shell, I am able to cat the contents of all files in a folder, with the value beside of it: $ for f in *; do echo -e "\t"$f"\n"; cat $f; done ...
1
vote
2answers
29 views

How do I extract only alphanumeric characters from a given text file and print them?

I have a text file that looks like the text that is pasted below. I want to extract to a file any characters that are alphanumeric in nature, and ignore everything else. What is the easiest way to ...
1
vote
3answers
118 views

Read a file line by line and if condition is met continue reading till next condition

I have a file foo.txt test qwe asd xca asdfarrf sxcad asdfa sdca dac dacqa ea sdcv asgfa sdcv ewq qwe a df fa vas fg fasdf eqw qwe aefawasd adfae asdfwe asdf era fbn tsgnjd nuydid hyhnydf gby asfga ...
0
votes
2answers
22 views

command to put hyphen in the second column if the space is given between a word [closed]

Suppose you have a file called test.txt with data like: you wel come how nice what do ing How can I add a hyphen in the second column if the space is given between a word?
0
votes
4answers
69 views

How can I replace a character in all the .php files inside a folder on OS X?

I need to replace the following characters: ă with a â with a î with i ș with s ț with t The replacement must be case sensitive. For example if a letter is upper-case then it must be replaced ...
1
vote
3answers
26 views

separate lsof output by column

I am using this command below and trying to separate the columns I only want to get the PID to use it in my python script. I can easily get this line by line but then how to separate into columns in ...
1
vote
2answers
29 views

How do I find a substring in a command line?

I have a few files that contains a similar entry like as follows: {"result":[]} {"result":[{"alternative":[{"transcript":"I ...
4
votes
5answers
150 views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
5
votes
2answers
141 views

Remove hyphenation with sed

I have a simple xml file containing some hyphenated words over page breaks. The input is something like ba bla bla hyphe-</page> <page>nated bla bla bla and the output should look ...
1
vote
2answers
28 views

Text processing - Building a slurm topology.conf file from ibnetdiscover output

First things first: no knowledge of either slurm or Infiniband is required - this is a purely text processing problem. Second - I'm aware of ib2slurm - the code is somehow broken and quite possibly ...
5
votes
6answers
128 views

How to remove symbols from a column using awk

I have data like this: chr1 134901 139379 - "ENSG00000237683.5"; chr1 860260 879955 + "ENSG00000187634.6"; chr1 861264 866445 - "ENSG00000268179.1"; chr1 879584 894689 - ...
0
votes
1answer
13 views

How to remove a row if the difference between two columns is less than 2000

I have a dataset that looks as follows: chr1 HAVANA gene 69091 70008 . + . gene_id "ENSG00000186092.4"; transcript_id "ENSG00000186092.4"; gene_type "protein_coding"; gene_status ...
1
vote
1answer
23 views

How to Load data to one table from Multiple excel files?

I have to load data from multiple excel file into one table , Please help me to achieve the solution. We are getting excel file from different retail shop which contain multiple column , each file is ...
1
vote
1answer
56 views

split file into multiple pieces

Problem: given a file samplein, it can be split up into multiple pieces as follows: $ cat samplein START Unix Linux START Solaris Aix SCO $ awk '/START/{x="F"++i;}{print > x}' samplein $ ls F* ...
0
votes
1answer
32 views

extract text from txt a file [duplicate]

I have a a file containing about 20 million sentences, how can I extract 2 million sentences out from it? I thought about using the split command like this "split -l 2000000 sub2016", but then it ...
3
votes
3answers
133 views

printing a block of text that matches a pattern

Using just shell script, how to search a text file and list all whole blocks of lines that has inside some text (simple grep criteria). The text file has blocks of lines separated by ...
0
votes
1answer
29 views

Output number of words, characters, or lines in an .doc or .docx document from command line

I have a folder full of .doc or .docx files. I'd like to know how many words--or, failing that, characters or lines--are in each file without opening each one of them. (Use case: I'm a TA who needs to ...
2
votes
2answers
63 views

How to sort data stream by 2 column names (column number may vary)?

I get a stream of data from an API with which look like this: redID blueID whiteID 1 22 2 44 15 41 2 15 15 31 2 14 What I need to do, is to sort ...
1
vote
2answers
60 views

Extract Values from Formatted Text

Is there an easy way to extract variables from a text file? E.g. given the output from ab: This is ApacheBench, Version 2.3 <$Revision: 1638069 $> Copyright 1996 Adam Twiss, Zeus Technology ...
2
votes
2answers
31 views

How to alter a column in a row in a tab delimited text file if it's duplicated

Lets say I have these many rows chr1 66999638 66999638 SGIP1 1 + chr1 66999251 66999251 SGIP1 1 + chr1 33545778 33549778 AZIN2 1 + chr1 8376144 8380144 ...
5
votes
4answers
257 views

convert table to ini file using bash arrays

I have a challenge to take an output file ("inventory.list") in this format: hostname1.env1.domain | abc | environment1 hostname2.env1.domain | abc | environment1 hostname3.env2.domain | abc ...
0
votes
2answers
43 views

How to Pivot the data by awk in unix shell script?

I'm trying to pivot a file using awk. This is an example of my input File: P2P,P2P,MP,DL,AIR,UP,HR,UPE,GSM,S,PP,1,NA,0.2,0.2 P2P,P2P,MP,DL,AI,UP,HR,UPE,GSM,F,PP,4,NA,0.2,0.8 ...
2
votes
2answers
61 views

Selective extraction of data

I have the following block of data in a txt file : Informatica(r) PMCMD, version [9.5.1 HotFix4], build [313.0217], SOLARIS 64-bit Copyright (c) Informatica Corporation 1994 - 2014 All Rights ...
3
votes
3answers
122 views

Intersperse lines from two files

I have a data file ($file1) which contains two lines of data per individual. I need to intersperse a third line of data from another data file ($file2). So my input looks like: >cat $file1 bob ...
3
votes
3answers
144 views

How do I remove newline character at the end of file?

Let me clear about newline character: $ echo Hello > file1 ; cat file1 Hello $ echo -n Hello > file2 ; cat file2 Hello$ Here you can see that file1 has newline character at the end whereas ...
8
votes
3answers
420 views

Extract text between three single quotes

I have the following in a file description: ''' This rule forbids throwing string literals or interpolations. While JavaScript (and CoffeeScript by extension) allow any expression to ...
1
vote
2answers
62 views

Adding “:” time-formatting using awk

For an input file named Lab1: 034023 052030 034023 022130 044023 012030 034223 022030 034123 152030 024023 152030 AWK command awk 'gsub(/[0-9][0-9]/,"&:",$1) ...
4
votes
2answers
64 views

BSD sed: Replace only the Nth occurrence of a pattern

Using BSD sed; How can I perform the following substitution?: Before: hello hello hello hello hello hello After: hello world hello hello hello hello In other words; how can I replace only the ...
0
votes
2answers
66 views

Making csv file

I have two different textfiles.txt (textfile1.txt and textfile2.txt) with different data sets in them. Is it possible to compile two seperate .txt files as a single CSV file? UPDATE textfile1.txt ...
1
vote
5answers
56 views

Filtering out inputs to textfile

I am using and Raspberry Pi (running raspbian) and I have a shellscript continuously outputting values into textfile via this command: wavemon -d |grep -i signal >> textfile.txt The output is ...
1
vote
3answers
54 views

Duplicate file, replace character, then concatenate

I have a file that looks like this: i36aasf5i7538i123 i47982i16537i1256 i1647i6458i3457 i1856i8456i43865 I want to make a copy of the file, in which the first i in every line is replaced by an o. I ...
1
vote
2answers
44 views

Print smallest integer from file using awk custom function?

awk function looks like this in a file name fun.awk: { print small() } function small() { a[NR]=$0 smal=0 for(i=1;i<=3;i++) { if( a[i]<a[i+1]) smal=a[i] else smal=a[i+1] } return smal } ...
1
vote
4answers
189 views

Move lines 2 3 after line 5 using awk

I have file content as: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line awk 'NR==2 && NR==3 {f=$0;next} NR ==5 ...
0
votes
1answer
37 views

Consecutively copy two lines and skip the third using awk

With fairly simply awk: awk '(NR%3)' awk.write for this file: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line I have output as: ...
3
votes
2answers
98 views

Find common rows on two gzipped files

I've got two big gzip files. I need to find the common rows. Zgrep (gzip) does this to a large extent like zgrep -f <unzipped file> <zipped file> but I can't find any switches to do zgrep ...
4
votes
3answers
67 views

Bash split line into multiple lines?

I want to know if there is an easy way to split a line in a file into multiple lines, such as this: I have A B C 1 2 3 4 And I want to get something like this, based on the first string of the ...
0
votes
0answers
26 views

Creating a mini text-editing program

I often copy-paste text from pdfs into other applications, and wish there was an easy offline way of editing the text's characteristics. For example, this website removes line breaks, while this ...
1
vote
1answer
37 views

Remove spaces in file under certain criteria

I want to clean up some files, and make make the way in which they are written more uniform. So, my input might look something like this: $a$h$l )r ^9 ^5 l \ urd The thing is, some spaces are ...