Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

0
votes
1answer
32 views

count number of lines provided to Python script before and after task

Input: {"case":"0901","emailed":"yes","vote":1,"accepted":"no"}, {"case":"0908","emailed":"yes","vote":8,1"accepted":"yes"}, {"case":"0911","emailed":"no","vote":10,1"accepted":"yes"}, {"case":"0090",...
1
vote
5answers
49 views

Concatenate Multiple lines into a single line until a pattern is found

With AWK and SED patterns i ended up with a file similar to | 9000 3000 1000 0000 0000 2000 1000 2000 | 5669000 518000 3000 16000 0000 28000 2241000 2841000 | 9020000 453000 520000 4000 2852000 ...
0
votes
2answers
44 views

Want to replace time column with the current time

I have data in file portals.csv like filename is portals.csv ip,time,name 1.1.1.1,2018-08-15 11:05:28:268813353,1.13.0-0007 1.1.1.2,2018-08-16 11:05:32:016469121,1.13.0-0007 1.1.1.3,2018-08-16 11:...
0
votes
2answers
33 views

How to remove nanosecond from file

How can I remove nanosecond from every line in a file. data looks like filename is test.csv ip,time,name 1.1.1.1,2018-08-17 15:05:52:016469121,1.13.0-0007 1.1.1.2,2018-08-17 15:05:52:016469121,1.13.0-...
1
vote
2answers
45 views

How to execute two grep commands on the same file? [duplicate]

I have a file. I want to select the lines that does not start with www. and does not contains slashes. I want to output the result into result.txt To achieve the first requirement: grep -v '^www\.' ...
-4
votes
0answers
20 views

Transpose / text process [duplicate]

I have a file with the lines like below. O2 C1 H4 C1 C1 N3 N3 C6 4.0703253804E-01 3.3703920725E-01 3.3646765114E-01 2.6225403436E-01 1.0014531178E-01 1.4247706185E-01 4.1553056004E-02 6....
0
votes
0answers
29 views

invalid char '@' in expression awk output

I have an awk command on ksh script, the command extract fields from a log file to then struct them in new one. ultimate=$(grep -oP 'Time:\K.*' $log2 | column -t | awk '{print $3,"/","'"$month"'","/",...
1
vote
1answer
12 views

preg_replace for replacing comment text in a file

In a file, I want to replace COMMENT AS `<some_comment>` with COMMENT '<some_comment>' That is, replace COMMENT AS with just COMMENT, and replace the backticks around the following ...
1
vote
3answers
32 views

How can I delete text before a pattern (--------------) in Linux

There is one file in which I am getting data after a pattern -------------- (14 dashes). I want to generate the file in which I will get the text after 14 dashes. Filename is portals.csv and Text in ...
0
votes
2answers
27 views

remove anything between angle bracket including it [duplicate]

I have a very big text file ~ 105 G it's includes a lot of <XXXX> (angle brackets including text in between). I want to remove those brackets and text in between.
3
votes
2answers
37 views

Display most recent line by key

I've got a file containing a bunch of fields: 12345 some values here 67890 other values maybe 12345 different values here ...etc. I'd like to display the most-recent line for each key. The output ...
1
vote
3answers
48 views

how to remove syntax with uniq characters from line

we have the following configuration file, env.gc.txt export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:{{hbase_log_dir}}/gc.log-`date +'%Y%m%d%H%M'`" We try to ...
2
votes
2answers
66 views

How to add properties in the end of the two first lines with double quote?

We want to add the following properties: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M in the two first lines that contain the word - HBASE_MASTER_OPTS Example before ...
-2
votes
2answers
26 views

find rows matching with another file and store them in a separate file

I have population file looks like this pop.file HG1W HG2W HG3W HG4W ... and another file contains individual IDs (there are 2 rows per individual, R1, and R2). ind.file HI.2613.007.Index_18....
-1
votes
3answers
44 views

how to print only part of each row that stars with a particular character

I have a file with over 10,000 rows: head samples https://genomequebec.mcgill.ca/nanuqMPS/readSetMd5Download/id/192170/type/READ_SET_FASTQ/filename/HI.2613.007.Custom_0022.ED9_SD2A27-1_180_R1....
0
votes
1answer
32 views

Awk of string followed by '\n'

I try to parse a plain text config file A.conf with the following content: bitrate 500000 use_can 1 I want to use the number behind bitrate to setup a CAN-interface with the appropriate bitrate. ...
2
votes
2answers
103 views

Concatenate multiple lines into “special-character”-delimited string

Say I have these Bash commands: $ sha="$(git log --all --format=format:%H -n 1 -- .npp.json)" $ git branch --contains "$sha" | tr -d " *" right now that might log something like: master dev ...
1
vote
1answer
43 views

Outputting context (-C) for grep produces massive files

Task: I'm using grep to search some text files, piping the results from one grep (excluding some lines) to another (matching some lines) + displaying some context using the -C parameter as shown ...
0
votes
4answers
59 views

Select lines based on lines above them

I have a list of items, from which I want to select the names of active items: item { status: "Active" properties { key_a: value } id: 42 name: "Foo" } item { status: "Disabled" ...
-1
votes
2answers
70 views

How to improve the specific columns count AWK

I have a big text file, it has 46 columns in total: INPUT Column1 Column2 ... Column46 I need to enumerate and count how many times repeat in themselves from column 17 until the 46. For example: ...
1
vote
1answer
41 views

How can I prepend lines that don't start with certain keywords?

I'm modifying a bunch of python files and I need to prepend every variable declaration with 'self.'. Here's an example of what I'm trying to do. From this: class testClass(parentClass): def ...
0
votes
4answers
34 views

How to fill the blank spaces in first column with value obtained from previous non-blank line?

Sample Input: James account note money Ruby account money Taylor account note Rob money Desired Output: James account James note James money Ruby account Ruby ...
-1
votes
2answers
35 views

i want to replace ^M\n part by space in a csv file using sed command in linux

I want to replace ^M\n part by space in a CSV file using sed command in Linux..... I used sed 's/\n/ /g' but it is only replacing \ and not the other part..... plz suggest something To be more ...
2
votes
1answer
53 views

Display a matching line and the closest preceding line matching another pattern

I have this requirement where show mac address table of a switch will list 1000 entries . file1: switch_1#show mac address table AA:AA:AA:AA:AA:AA port 47 BB:BB:BB:BB:BB:BB port 48 . . . DD:DD:DD:DD:...
1
vote
3answers
35 views

Delete string file after space

I have many log files generated like the bellow: Date;Time;Brand;User;Node;Location 01/01/01;666;blabla;George Armani;Springfield I want to delete the String Armani but my command is not working: ...
1
vote
2answers
18 views

Substitute numbers in a random IP address port with space

I have a file with IP and port numbers. I would like to take out the port number and leave the IP alone. xyz 10.93.10.13:58160). xyz 10.93.10.13:58161). xyz 10.18.104.181:12466). xyz 10.93.10.13:...
0
votes
1answer
33 views

How to delete from configuration file variable that not equal to 100

How to delete from configuration file the variable/s hadoop.tasklog.totalLogFileSize that are not equal to 100 example more file.conf hadoop.tasklog.totalLogFileSize=107 hadoop.tasklog....
1
vote
1answer
46 views

Switch columns in CSV using awk? [duplicate]

I have the exact same question as this one here. Only difference is I have 3 columns only and I am trying to move the 1st column to the end. The original file looks like this: col1,col2,col3 2,2015-...
2
votes
4answers
38 views

Match if a column partially matches another column

I am trying to compare columns of the same file. The table looks like this: G A AA GG CC TT CT C T CC TT GG GG AA What I want to do is to compare the first and second column with the rest, ...
0
votes
2answers
21 views

Replace integers from specific words from a file

I am having a file xyz.sh JAVA_OPTS="-Xmx3072M" RESOLVED_HEAP_SIZE="2048M" RESOLVED_OFF_HEAP_SIZE="256M" Expected - I want to replace -Xmx3072M with -Xmx4096M but is not compulsory that value ...
1
vote
2answers
31 views

Extract multiple occurrences in the same line of a string between two delimiters

This one is tricky... input has multiple occurrences per line over several lines. many "id=" lines exist, but I only want ones starting with ai-c " id=" Example input (two long lines): d-fl ai-c" ...
1
vote
3answers
344 views

Using bash to swap first and second columns in CSV

I'm using bash. I have a CSV file with two columns of data that looks roughly like this num_logins,day 253,2016-07-01 127,2016-07-02 I want to swap the first and second columns (making the date ...
4
votes
4answers
68 views

Speed up script that determines if all columns in a row are the same or not

I need to speed up a script that essentially determines whether or not all the "columns" for each row are the same, then writes a new file containing either one of the identical elements, or a "...
0
votes
2answers
61 views

awk + append lines before captured word only if lines are not defined in the file

The following awk syntax will add the 3 lines in the file before the line with the word - 'DatePattern': $ awk 'done != 1 && /DatePattern/ { print "log4j.appender.DRFA=org.apache.log4j....
2
votes
1answer
48 views

Merge multiple lines column by column based on column 1 - same file?

I'm trying to organise a CSV file with 30 columns and 1000 rows. I want to concatenate the rows column by column if the 1st column is the same, using a vertical line (|). Input: 3,XX,YY,123,1234,MM,-...
4
votes
3answers
226 views

How to merge first two lines of a csv column-by-column?

I have an excel file that I converted to csv. When converted, it looks like the following example (Please note that there are 100+ columns in the csv. This is a minified version): ,Product," ",Citty,...
3
votes
5answers
389 views

Insert multiline text before the first matching line in a file

we want to add the following lines in log4j file log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.MaxBackupIndex=100 log4j.appender.DRFA.MaxFileSize=10MB but these lines ...
-1
votes
2answers
32 views

sed + mark line in case of marched word

we want to mark with "#" the matched line by sed or perl line liner for example we want to mark all lines in file that include the DatePattern word log4j.appender.DRFA.DatePattern=.yyyy-MM-dd ...
1
vote
3answers
39 views

Printing the nth range with sed

I can print the braceted lines in the file a first [ third fourth ] sixth [ eighth ] tenth by doing % <a sed -n '/\[/,/\]/p' which prints [ third fourth ] [ eighth ] But what if I want only ...
0
votes
3answers
46 views

search for a pattern and always print the first line which contains the cn

I've a file which has the following output: The dn: can can have more rdcPositions. I only need the dn: which has an rdcPositions contains acme#6# The result should print the cn and also the ...
1
vote
2answers
35 views

alter awk variable based on match inside awk

I have a data file that I am preprocessing before sending it to gnuplot. It consists of a header line and multiple rows of data, with many columns of data types. The first three columns are always of ...
4
votes
3answers
236 views

Using awk to compare files and combine output from both files?

I am trying to compare two files using awk and I would like to print data from both files as output. The files I am comparing are as follows. File1: gene feature id fc ...
0
votes
1answer
19 views

Open file 1 and write to 3rd column [duplicate]

I need to open file 1 which has the following format: 20336 20336 20337 20336 20338 20336 20339 20336 20340 20336 20341 20336 . . . . . . 71643 20336 71644 20336 71645 20336 71646 20336 ...
1
vote
3answers
35 views

Open a file and write to 2nd column a certain number

I am trying to open file 1 which has the following format 20336 20337 20338 . . . 71644 71645 71646 and write to second column number 20336 so the new file will have the following format 20336 ...
3
votes
1answer
27 views

Extract .json from a text file with arbitrary text

I have an output from a program that give some arbitrary text, with .json stuff inside like: blablablabla blablab some more text blablablabla blablab some more text { "glossary": { "...
5
votes
2answers
384 views

how to convert text file to json string (replace newlines with “\n”)

we have the following file : more value.js "\n#\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with t ...
2
votes
2answers
50 views

Collect all of the sar data

Task: I can sar -u > tmp.csv and I can sar -r > tmp.csv But what I need is a big table of all the sa* files of all the options. sar -r -u -S -q > tmp.csv does not quite do the job ...
0
votes
3answers
48 views

math operation on a second column

Suppose my original file looks like this: N O D E O U T P U T THE FOLLOWING TABLE IS PRINTED FOR NODES BELONGING TO NODE SET NSET_OUTPUT NODE FOOT- NT11 NOTE 3 ...
0
votes
1answer
29 views

Sed conditional removal of EOL [duplicate]

For the following example, where we have "\" at the end, the \ and EOL character should be removed. line 1\ line 2 line 3 line 4\ line 5 Output should be line 1 line 2 line 3 line 4 line 5 Using ...
1
vote
2answers
41 views

Finding Greek from the command-line

How can I use perl on the command-line to print only lines containing Greek unicode characters? For example, only the second of the following lines hullo, world χαῖρε, ὦ κόσμε It should be ...