1
vote
1answer
30 views

insert value at the end of every record based on unique IDs

I have 2 files , one with Delimited and other one is Positional file. These two files shares same unique ID's as shown below but both the files contains different information. del.txt ...
4
votes
3answers
105 views

Make a directory for each row within the given text file?

I'm trying to write a bash shell script that will read the data from a given (as an argument) file, and for each row in the file, make a sub-directory under the given (as an argument) directory. The ...
1
vote
2answers
60 views

Editing a Java properties file with Bash

I have the following Java properties file that is maintained by humans. # file.input = /very/old/name # file.input = /old/name file.input = /current/name file.input.default = /default/name ...
1
vote
2answers
47 views

Bash: split list of files into different files?

I have several files in a directory like: data1.dat data2.dat .... data24678.dat To run another code, I need to have a text file with the names of the files but I need to split in 50 files per ...
3
votes
1answer
54 views

Find and Replace Recursive

I want to modify every file in a directory recursively. I want to replace: <?php with <?php /** * * Copyright (C) MyCompany, Ltd. - All Rights Reserved * Unauthorized copying of ...
2
votes
3answers
57 views

How to optimize this oneliner? (if second column empty, put the line in another file)

I have a little "oneliner": while read ONELINE; do if [[ -n "`echo $ONELINE | awk '{print $2}'`" ]]; then echo "$ONELINE" >> LIST-ok.txt else echo "$ONELINE" >> ...
5
votes
1answer
97 views

counting multiple patterns in a single pass with grep?

I've written a grep loop to iteratively count DNA trinucleotides within a gzipped DNA fasta file containing DNA sequences e.g. declare -a tri=(AAA AAC AAG AAT CAA .. etc) for i in ${tri[@]} do ...
3
votes
4answers
80 views

Replace all newlines to space except the last

How can I replace all newlines with space except the last newline. I can replace all newline to space using tr but how I can do it with some exceptions?
1
vote
1answer
256 views

grep to extract a substring from a huge string

I am extracting certain information from a website using curl command. I need to extract a certain subsection of the string using grep command. For example, using curl command I am getting the HTML ...
2
votes
1answer
32 views

Count and merge consecutive patterns

I'm searching for a short snippet to find, count and merge consecutive duplicates using standard tools or a common scripting language. Say our input is: 1 2 2 2 7 22 a b b c c c c c d dd 2 2 c c ...
0
votes
2answers
64 views

Using AWK to delete line and output result

I have a text file and manage to find the line through output comparison. With the book title and author name. awk -F ':' '/KungFu Feet|Chuck Norris/' test.txt Kungfu Feet:Chuck Norris:12:1:1 Is ...
3
votes
5answers
90 views

Join two files, matching on a column, with repetitions

How can I get two files A and B, and out put a result like this: File A: 001 Apple, CA 020 Banana, CN 023 Apple, LA 045 Orange, TT 101 Orange, OS 200 Kiwi, AA File B: 01-Dec-2013 01.664 001 ...
1
vote
1answer
59 views

removing data from txt file

I have a comma-delimited giant file that was exported from a database. The file has this format Date,n1,n2,n3,n4,n5,n6,color,type,reference 10-Jan-2000,344,13,21,2,55,3,"rose garden" ,3444,32 ...
2
votes
3answers
42 views

To assign variable from a file separated by delimiter

I have a text file with datas as below separated by delimiter: 1000:Tom:user1 1002:Jerry:user2 1003:Victor:user3 1004:Jason:user4 I need to run a while loop (any loop) to fetch details from first ...
5
votes
1answer
99 views

How to append multiple lines to a file without last newline?

I have a very long string that is split in chunks. I want to append them to a file without putting the newline character using bash. Example: First append cat >> abc.sh << EOL echo "bla ...
2
votes
2answers
80 views

Show a sliding window of output from a program

I want to run a command (rsync -av) and only display 10 lines of output at any time. I have found a similar post, but I don't want to redirect it to a file. I want the output to be displayed ...
1
vote
2answers
56 views

Mailing out a customised content based on specific search in a logfile

Using below test.sh, I am getting result in dump.txt as below: test.sh: #!/bin/bash #"Maxl-script" zone starts..... essmsh -l admin password -s localhost -i << EOF spool on to 'dump.txt'; ...
0
votes
2answers
72 views

How to Lookup the values of an index values from a big data file?

I have a data file (data_array.txt in the example) and index file, index.txt, at which I want to extract the data from the data file that have the same ID in the index file and store in to a new file, ...
1
vote
5answers
263 views

Cut and transpose some lines of a file

I have 1344 rows of data column. I want to form 32 row by 42 columns from my input data file. I want to cut the first 42 row and paste to one raw of 42 column data and continue down likewise. My data ...
9
votes
4answers
100 views

Script to extract selected entries from a bibtex file

I have a large bibtex file with many entries where each entry has the general structure @ARTICLE{AuthorYear, item = {...}, item = {...}, item = {...}, etc } (in some cases ARTICLE might be a ...
4
votes
3answers
121 views

Parsing string token with bash

I have this output (from a previous command): aaa something bbb someother ccc blabla For each line, I would like to create a file whose name is the first token, and whose ...
0
votes
1answer
103 views

Copy file while replacing text in it

I'm writing a script to publish a webapp. While copying files, I must replace a placeholder current_date in a file with the current date. I would start with something like this to define the date ...
4
votes
2answers
90 views

Remove duplicates from basenames in two files

I have two long lists of filenames, and I want to merge these two by excluding duplicated filenames in the sense of basename. For example, I have something like ../data_folderA/file_1 ...
-1
votes
2answers
132 views

How to manipulate output from commands with shell scripts [closed]

Simple question, having a ls -l command it makes a list on the command prompt and if we want to manipulate it we pipeline it with awk command, for instance if we want to to show specific files. Is ...
4
votes
4answers
289 views

how do I trim each line in bash?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into?
5
votes
3answers
204 views

removing redundancy from output columns

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns A B B A A C A D C A D A B C C B A B and B A ...
1
vote
3answers
100 views

Extracting mail addresses

I have a file that contains lines like this: user=<xxxxxxx@asd.com.tr>, user=<yyyyyyy@asd.com.tr>, user=<zzzzzzz@asd.com.tr>, user=<ccccccc@asd.com.tr>, ...
2
votes
5answers
105 views

Read contents in a line in bash

I have a file with contents in this order which is separated by space 4096 1230 4560 1024.... n How can i read each number and pass it on to another command, eg divide each number by 1024.
3
votes
1answer
116 views

Simple Bash Script

I have a simple problem, but unfortunately I don't know where to even start (I'm just starting out). So, what I want to do is ultimately increase my vocabulary. I got the idea to strip the most ...
1
vote
3answers
199 views

compare files line by line and create new one bash programming

I have two text files. File 2 has logs over 1,000,000. File 1 has IP addresses line by line. I want to read file 2 lines and search these lines in file 1, I mean: file 1: 34.123.21.32 45.231.43.21 ...
3
votes
2answers
182 views

How to find frequency of occurrences of strings contained in a file?

I have a file that contains a list of URLs of the form EDIT http://www.google.com/absd/siidfs/kfd837382$%^$&, www.google.com, google.com yahoo.com/list/page/jhfjkshdjf... I ...
4
votes
3answers
380 views

using sed to replace pattern with hash values

I want to search the file and replace specific pattern with its hash (SHA1) values. For example, let file.txt has the following content: one S56G one two three four five V67X six and I want to ...
1
vote
1answer
595 views

C shell script : how do I compare two strings?

I'm a C shell user. I've two files. file 1: A B C D E file 2: A C B D E I want to compare only A-A, B-C, C-B, D-D, E-E using a for loop. If the strings match, then print "match". The output ...
3
votes
2answers
162 views

Bash-script. Shift seconds

Maybe somebody can help me. In bash I don't know how to do that. I need to do a bash-script. At stdin I have .srt file of subtitles in this format: num HH:MM:SS,SSS --> HH:MM:SS,SSS text line 1 ...
1
vote
1answer
113 views

Binary search for a failing line

git has bisect run to figure out which revision an error was introduced. I have a big file (100GB) and at least one of the lines is bad, but the program I have to check it with will not tell me which ...
3
votes
4answers
518 views

Align text to center with padding on both sides

Is there an easy way to align text to the center with padding on both sides, with the column width being the longest line from the input? For example, this: aaa bbbb c ddddd ee will turn into this ...
1
vote
2answers
305 views

Printing fields using awk

< pool.sam awk ' /./ {printf $1} {printf $7+1,"\t"} {printf $3,"\t"} { if($2 !=16) {print "\t", "+";} else {print "\t","-";} {printf $4,"\t" ,length($10)+$4, "\t", "1"} }' I am ...
1
vote
5answers
298 views

Print nth line before the matched line, Matching line and nth line from the matched line

I want Print nth line before the matched line, Matching line and nth line from the matched line where "n" is greater than 2. Here's an example of my data file (the line numbers below are not part of ...
1
vote
1answer
100 views

Parsing the headers of sequence file

I have a multiple sequence file as >abc|d017961 sequence1...... >cdf|rhtdm9 sequence2...... >ijm|smthr12 sequence3...... >abc|d011wejr sequence4...... >stg|eethwe77 sequence5...... ...
1
vote
2answers
77 views

Split out data from a column

So at the moment I have some input email addresses and would like to separate the alias and the domain, then copy them to the end of the line. For example, from a line that has ...
2
votes
3answers
570 views

Comparing files line by line in awk with delimiter

file1: abc|123|check def|456|map ijk|789|globe lmn|101112|equator file2: abc|123|check def|456|map ijk|789|equator lmn|101112|globe EXPECTED OUTPUT: ijk|789|equator lmn|101112|globe Current ...
1
vote
2answers
244 views

Improve sed command to replace first instance of character and all following characters?

Input: <e1 name="file1" id="id1" anotherId="id2"> Desired output: file1 I can get what I need with this: echo '<e1 name="file1" id="id1" anotherId="id2">' | sed ...
6
votes
4answers
2k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
4
votes
4answers
234 views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
2
votes
2answers
79 views

selecting specific string in a field

I have this file: N104 F1810 N104 F3810 N104 F810 N104 F305 and a script to grep an input. Script: enter input read input cat file | grep $input when I input “810”, I get the ...
2
votes
2answers
3k views

Loop over a file and read values from two columns into variables

I have a file (data.dat) which contains 3 columns. I want to read in two of them with the shell script and fill the contents into a template file. The file looks like this: 12 180 2390 14 177 2210 16 ...
3
votes
5answers
167 views

Comparing Indices of two strings

I have two double quoted strings of same length, that look like this: "$con" - (consists of spaces and *'s): * ****** *** ** * ** and ...
2
votes
3answers
236 views

How to grep the last number on a line in a bash script

I have output from the smartctl -a command; one of the lines looks like this: 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 301 I've placed this into a variable (if that's the correct term) ...
0
votes
1answer
90 views

Removing lines based on tags for different files and merging into single file

I have to develope unix script in such way it should meet below scenario. I have 3 text files, 1.txt , 2.txt and 3.txt Data in those files as below. HDR|1|2|3||||||||||||||||||||||| ...
1
vote
2answers
84 views

Remove complete line in text files based on Tags

For example : 01.txt, 02.txt, 03.txt and 04.txt has following values. HDR|A|B|C|D|F||||||| DTL|1|2|3|4|5||||||||||| DTL|5|6|7|8|9||||||||||| TLR|A|B|C|D|F||||||| I have to remove complete lines ...