Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
2answers
243 views

Improve sed command to replace first instance of character and all following characters?

Input: <e1 name="file1" id="id1" anotherId="id2"> Desired output: file1 I can get what I need with this: echo '<e1 name="file1" id="id1" anotherId="id2">' | sed ...
6
votes
4answers
2k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
4
votes
4answers
227 views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
2
votes
3answers
87 views

Using sed to insert latex commands around headers of a document

I have a documen with headers in ALL CAPS, for example: NAME env - run a program in a modified environment and I want to insert \item[ before the header and ] \hfill \\ after a header ...
2
votes
5answers
827 views

How to remove duplicate letters using sed?

Using sed, how can I remove duplicate letters from HEADERS within a text file? NNAAMMEE nice - run a program with modified scheduling priority SSYYNNOOPPSSIISS nice [-n ...
2
votes
2answers
439 views

Convert output of tree command to json format

Is there a convenient way to convert the output of the *nix command "tree" to JSON format? Edit: I think I didn't describe my problem well enough. My goal is to convert something like: . |-- dir1 | ...
5
votes
2answers
251 views

Convert between Unicode Normalization Forms on the unix command-line

In Unicode, some character combinations have more than one representation. For example, the character ä can be represented as "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), ...
2
votes
2answers
390 views

How to remove an apostrophe ( ' ) from couple of columns of a .CSV file?

I have a .CSV file with 7 fields, and the 3rd and 4th columns of the file has a number starting with an apostrophe ( ' ). Please see the example below. col0,col1,col2,col3,col4,col5,col6, ...
2
votes
4answers
610 views

Extract lines from a text file with one column equal to the maximum

I have a text file (with header line) called file.txt. I am trying to extract the lines equal to the maximum value of a specific column (I do not know what the max value is): ID t1 q1 t2 q2 q3 1 f ...
1
vote
2answers
2k views

How to select, copy and paste some columns from different files and create one file from them?

I want to create and organize data in a file from a number of files by selecting parts of the columns of the given files. I have more than 10 file to copy the second, third and forth columns of each ...
1
vote
1answer
62 views

Why doesn't this unexpand seem to work?

It's my understanding that unexpand can replace multiple spaces with tabs. I have a file, x.x with contents 1 2 3 4 5 and I type expand x.x but the output is still 1 2 3 4 5 and ...
4
votes
1answer
260 views

Does the inverse command of cut exist?

First question, so if the format's not correct, sorry! I like using the cut command in Linux with the -c flag. However, I'm interested in finding a command that sort of does the set inverse of cut. ...
1
vote
1answer
268 views

Why does the paste command add line breaks? [duplicate]

I have 2 files: d and t. I would like to be able to combine these files so that the first line of file t is followed by a tab and then the first line of d. For shorter lines, paste t d seems to work ...
2
votes
2answers
79 views

selecting specific string in a field

I have this file: N104 F1810 N104 F3810 N104 F810 N104 F305 and a script to grep an input. Script: enter input read input cat file | grep $input when I input “810”, I get the ...
5
votes
4answers
1k views

SED : Delete 4 line above and below 5 line after pattern match

I have Hugh file with following details : define host{ use generic-host ; Name of host template to use host_name ServerA_172.29.16.102 ...
1
vote
2answers
111 views

Make the occurrences of a string bold

Now, this is a homework question, but I have simply no idea how to go about solving it. It'll be great if someone could guide me about how to go about it. I've been given a large string of Upper ...
1
vote
2answers
53 views

How can I merge multiple lines with spaces (blank line) separating them? [closed]

I'd like to merge the following multiple lines of output so that they form a single line: line 1: ...
1
vote
1answer
51 views

Find and Replace in wordpress postings

I have a wordpress blog with 93 posts on it, in which I want to replace [bash] with <pre> but how do I do it please? I need to go through all the posts changing [bash] to <pre> in each ...
1
vote
3answers
1k views

How to convert from one date format to other format?

I have one problem with my date format. I want to change from one format to the other and vice versa. My date formats are Format 1 YYYY-MM-DD Format 2 MM/DD/YYYY I want to change format 1 to ...
1
vote
3answers
148 views

how to calculate the average value of the column that have the row ID?

I want to calculate the average value of my data when they have the same ID (in my specific case, the same day but different time in a day). This is meant to calculate the daily average value of my ...
2
votes
2answers
3k views

Loop over a file and read values from two columns into variables

I have a file (data.dat) which contains 3 columns. I want to read in two of them with the shell script and fill the contents into a template file. The file looks like this: 12 180 2390 14 177 2210 16 ...
1
vote
1answer
50 views

how to clean free row/column spaces after the row of your data

I have a problem in cleaning my file where there are unnecessary free spaces below my last data row. This is created when I am copying and pasting my data from excel to text. I want my text file to ...
2
votes
2answers
177 views

Quickest way to find and replace a string in numerous HTML files

I have numerous HTML files all nested inside different folders contained in a single overall folder. In each of these HTML files I need to replace /contact/index.html With /contact/index.php ...
3
votes
3answers
138 views

Transforming text to tabular form

I have a text file with the following structure: aaa bbb ccc ddd eee fff 1 2 3 4 5 6 1.1 1.2 1.3 1.4 1.5 1.6 ggg hhh iii jjj kkk lll 7 8 9 10 11 12 2.1 2.2 2.3 2.4 2.5 2.6 and I want the ...
7
votes
2answers
424 views

How could I use Bash to find 2 bytes in a binary file, increase their values, and replace?

I'm trying to find two bytes inside binary file, then increase value of those two bytes and replace them inside file. Those two bytes are on positions 0x82-0x83. For now on I have successfully ...
0
votes
2answers
97 views

Count lines with alphabet in linux

I have a file with a number of lines in a file filename. I want to count how many lines start with character 'a', with 'b' and so on in one go. What command i should execute.?
2
votes
3answers
138 views

how to create a data file with a column of date from day1 to dayN?

I was wondering if I could generate a hourly or daily time step date in a column starting from some year in the past to some day in the near past or today. to be more clear, I want to create one ...
2
votes
5answers
700 views

vlookup function in unix

How to make something similar to Excel's vlookup function in Unix? excerpt from office website, VLOOKUP The V in VLOOKUP stands for vertical. Use VLOOKUP instead of HLOOKUP when your comparison ...
1
vote
1answer
48 views

how to look the value of thedata by lookup index? [duplicate]

I have list of data array which I want to look the values by its index (id) and store to the index id from the data array. I can do it in spreadsheet VALUE LOOKUP function, but I still have lots of ...
2
votes
2answers
148 views

Extract substring from 'list' of strings

I have a file with lines with IDs, and I'm extracting certain substrings from each line. I've seen this answer, which is great, but I don't know how to get it to execute on each line. I figure xargs ...
1
vote
2answers
120 views

how to multiply column of one file by column of anothe file?

I want your help in multiplying columns of one file by column of other another file where both files have the same number of columns and rows. I want the script to multiply the first column of the ...
1
vote
1answer
69 views

How do I modify this Perl solution so that it will substitute embedded double quotes with single quotes?

Previously in this question: "string having doublequotes in between apart from the enclosing quotes" @BernieReiter asked a follow-up question where he wanted to take CSV entries such as the following: ...
3
votes
5answers
167 views

Comparing Indices of two strings

I have two double quoted strings of same length, that look like this: "$con" - (consists of spaces and *'s): * ****** *** ** * ** and ...
1
vote
2answers
71 views

string having doublequotes in between apart from the enclosing quotes

I have a requirement. My CSV file has a string "Testurteil "sehr gut"" and I need to load this file (into redis) without removing the quotes, or I want it this way: Testurteil "sehr gut", even if it ...
2
votes
4answers
92 views

Adding values of the column that have the same data ID?

I want to prepare a uniform daily data from a mixed daily and sub-daily data by adding the sub-daily values and producing a uniform daily data. My sample input and desired out put data looks like ...
3
votes
1answer
1k views

Command to transpose (swap rows and columns of) a text file [duplicate]

Is there currently a generic command that will "pivot" input. e.g. #labeled.file name: bob title: code monkey name: joe title: pointy haired is converted to: name title bob code monkey joe ...
0
votes
3answers
88 views

selecting rows that have the same value [closed]

I have a problem in selecting rows that have the same value. My data is too huge to go row by row to do that. I want you guys to let me know scripts that can perform this. My data looks like as ...
1
vote
2answers
565 views

how to select, copy columns of different file to form new files

I have four files each contain daily station data. file 1: prec file 2: minT file 3: maxT file 4: wind Here prec, minT, maxT and wind are files that store values of prec, minT, maxT and wind for ...
4
votes
1answer
102 views

Looking for lines which is in one file but not in other using Unix and Awk

I have 2 files with 7 fields and I want to print the lines which are present in file1 but not in file2 based on field1 and field2. Logic: I want to print all the lines, where there is a particular ...
2
votes
3answers
218 views

changing column separators in a file

I have a large file, that I need to process and after writing some scripts that don't seem to work properly, I have discovered that a small subset of the lines in the file are actually space separated ...
2
votes
3answers
234 views

How to grep the last number on a line in a bash script

I have output from the smartctl -a command; one of the lines looks like this: 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 301 I've placed this into a variable (if that's the correct term) ...
4
votes
3answers
239 views

Grouped sorting of continuous paragraphs (separated by blank line)?

I think I'm pretty experienced now in sorting by columns; however, I haven't found anything so far how to sort continuous rows. Supposing we have a text file that looks like this: (very simplified, ...
3
votes
3answers
114 views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
3
votes
1answer
191 views

Advanced Find and Replace Program

I have code in which I need to do a mass find and replace (within only one file). I could easily do this except for that I need a free program (Preferably already compiled for Ubuntu so I don't have ...
0
votes
1answer
90 views

Removing lines based on tags for different files and merging into single file

I have to develope unix script in such way it should meet below scenario. I have 3 text files, 1.txt , 2.txt and 3.txt Data in those files as below. HDR|1|2|3||||||||||||||||||||||| ...
1
vote
2answers
111 views

Remove 2 and 3 newlines in a text file but not 1

I'm developing on a max osx 10.8 and have no preferences to which tool to use probably sed but it doesn't matter. Update after comment: The input file uses windows new lines if that's complicates ...
1
vote
2answers
82 views

Remove complete line in text files based on Tags

For example : 01.txt, 02.txt, 03.txt and 04.txt has following values. HDR|A|B|C|D|F||||||| DTL|1|2|3|4|5||||||||||| DTL|5|6|7|8|9||||||||||| TLR|A|B|C|D|F||||||| I have to remove complete lines ...
5
votes
5answers
920 views

How to append data of multiple files into single file?

For example: I have 01.txt and 02.txt files and need to append data of these files into new file new.txt. It should append data based on 01.txt followed by 02.txt files. Before this I have to delete ...
1
vote
1answer
47 views

Pipeline each line of a file to a specific variable in a second command

I want to execute the following instruction: basex -bword=ENTRY consulta.xq But for every line of a SampleText file (plain text mainly). The SampleText contains the following lines: hello evening ...
4
votes
1answer
561 views

Sorting a file based on one column using Unix and Awk

I need to sort the input file according the 6th column, which is the score. Input File: Sc2/80 20 . A T 86 Pass N=2 F=5;U=4 Sc2/80 20 . A C 80 Pass N=2 F=5;U=4 Sc2/60 55 . G T 90 Pass N=2 F=5;U=4 ...