awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

7
votes
3answers
507 views

awk does not end

I have the following code in a file named awktest1.awk: #!/bin/awk -f BEGIN{print "start"} {print $2, "\t", $5} END{print "end"} employee.txt where employee.txt contains the following data: 100 ...
7
votes
2answers
6k views

Counting number of times each IP address appears in log file

I have a file in the format as follows: $ cat file.txt 27.33.65.2 27.33.65.2 58.161.137.7 121.50.198.5 184.173.187.1 184.173.187.1 184.173.187.1 What's the best way to parse the file file.txt into ...
7
votes
5answers
2k views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
7
votes
3answers
3k views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
7
votes
4answers
6k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
7
votes
2answers
5k views

A shell script for joining two files

I want to write a shell script that get two files A and B, and get a result like this: File A: user_a tel_a addr_a user_b tel_b addr_b File B: process_1 user_a process_2 user_a process_3 user_b ...
7
votes
7answers
599 views

Removing numeric values in certain columns whilst keeping minus signs?

I put my hands up to being a complete beginner, so apologies if I'm doing this wrong. Say I have the following data frame that continues indefinitely horizontally and vertically with negative numbers ...
7
votes
3answers
242 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
7
votes
4answers
369 views

Matching and pasting to line

So I have 2 very big text files, consisting of lines like so: First: Robert:Dillain:Other:Other:Other Julian:Brude:Other:Other:Other Megan:Flikk:Other:Other:Other Samantha:Minot:Other:Other:Other ...
7
votes
4answers
821 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
7
votes
3answers
15k views

Awk printf number in width and round it up

I need to printf a number out, but with given width and rounded (with awk!) %10s I have this and somehow I need to connect the %d but everything I do, ends up with too much parametres for awk (...
7
votes
1answer
16k views

Sorting file by second column

How can I manipulate a two columns tab-separated text file by sorting by the first element of the second column (only if the first column element is the same)? Exemple: Input File 1 A 1-2 A 6-...
7
votes
4answers
429 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
7
votes
3answers
716 views

How to use shell to derive an IPv6 address from a MAC address?

We know that we can use the MAC address to create an interface identifier, e.g. for a link-local IPv6 address which should be unique in the Network. The image shows the way to do this: My ...
7
votes
4answers
387 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
7
votes
1answer
4k views

How to grep all lines where starting line ends with a pattern and ending line ends with another?

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
7
votes
2answers
7k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
7
votes
5answers
676 views

How to programmatically detect awk flavor (e.g. gawk vs nawk)

I'm using a command-line application which is essentially a collection of bash shell scripts. The application was written to run on BSD/OSX and also on Linux. One of the scripts relies on awk. It ...
7
votes
3answers
4k views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
7
votes
2answers
380 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
7
votes
2answers
6k views

Use awk to insert a line after N output

I have a text file filled with a bunch of data. I need to pull out lines that contain specific strings. I accomplished this with awk using the following: awk '/pattern1|pattern2|pattern3/ {print;}' ...
7
votes
1answer
867 views

Why are capital letters included in a range of lower-case letters in an awk regex?

$ echo ABC | awk '$0 ~ /^[a-b]/' ABC $ echo ABC | awk '$0 ~ /^[a-a]/' $ echo ABC | awk '$0 ~ /^a/' $ You see. /[a-b]/ captures A, but /[a-a]/ or /a/ doesn't. Why?
6
votes
8answers
8k views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
6
votes
6answers
5k views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
6
votes
5answers
400 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
6
votes
6answers
3k views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
6
votes
5answers
636 views

How to abbreviate /path/to/file to /p/t/file

I'm looking for an elegant one-liner (eg, awk) that will shorten a string of a Unix path using the first character of each parent/intermediate level, but the full basename. Easier to show by examples:...
6
votes
4answers
1k views

How to add a header and/or footer to a sed or awk stream?

I have a bunch of output going through sed and awk. How can I prefix the output with START and suffix the answer with END? For example, if I have All this code on all these lines and all these ...
6
votes
4answers
8k views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
6
votes
3answers
447 views

Print random lines respecting the order of the source file

I have a large text file (~500K lines) with short sentences (couple of words long). Additionally, there is some XML markup in most of the lines. Finally, the text file has been sorted before the ...
6
votes
5answers
313 views

Strategy to extract movies's name from this ununiform dataset?

I am working on a movie database problem to improve regular expressions, this is the problem I'm running into. My dataset looks like this: Movie Name (variable space and tabs) year Movie1(can ...
6
votes
4answers
46k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
3answers
577 views

Is there a perl or awk solution for this problem?

I have an input file (input.txt) like below. id1 id2 name weight 53723848 12651711 timburnes 1.36667 53530214 12651711 timburnes 1.51191 53723848 53530214 timburnes 1.94 764157 ...
6
votes
1answer
498 views

What is a *.sorted command and why doesn't it work in my bash?

In a Bash Guide for Beginners (sec. 3.2.1.2. Local variables) there is an example command which is supposed to list all of the local variables, that are not in enviromental variables, or so I think: ...
6
votes
8answers
7k views

can we print the last word of each line in linux using sed command?

suppose , if there is a file consisting of following lines , if they are 12345 567 7878 66 er3 t45t y6y46y 4y6 y656y y5y 46y6 65y7 y66uyuy yy46y6y The output has to look like: 66 ...
6
votes
5answers
5k views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
6
votes
4answers
444 views

Delete column that includes a specific text

I was looking for an option to delete a column with a specific text like "ddd" aaa bbb ccc ddd eee fff 1 2 3 4 5 6 2 3 4 5 6 0 So that the output looks likes this: aaa bbb ccc ...
6
votes
3answers
434 views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
6
votes
4answers
421 views

Find 2nd Occurrence of string from the end of file

I have a log file that looks like this, Another thousand lines above this I 10/03/15 12:04AM 42 [Important] 4th to last I 10/03/15 04:31AM 42 (534642712) [1974,2106,258605080,0,0,32817,30711] I 10/...
6
votes
5answers
366 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
6
votes
3answers
233 views

How can I display 3 lines of text at a time onto 1 line?

I have a log file that displays data 3 lines at a time. Like this: 1 data 2 data 3 data 1 data 2 data 3 data 1 data 2 data 3 data I'd like to take each 3 lines and ...
6
votes
3answers
36k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
6
votes
2answers
11k views

Compare two file columns

I have long text files with space-delimited fields: cat file1.txt Id leng sal mon 25671 34343 56565 5565 44888 56565 45554 6868 23343 23423 26226 6224 77765 88688 87464 6848 66776 23343 63463 ...
6
votes
3answers
4k views

AWK: wrap lines to 72 characters

$ awk 'length > 72' {HOW TO PRINT THE LINEs IN PCS?} msg ie I want it to add \n after 72 chars and continue, so initially you may need to remove all single \ns and the add them. It may be easier ...
6
votes
4answers
341 views

Count number of a substring repetition in a string

I have a file which contains a gene sequence such as: ...
6
votes
3answers
3k views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
6
votes
3answers
3k views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...
6
votes
2answers
10k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
6
votes
3answers
17k views

Count records matching pattern with Awk

So I was told to do last > lastloggedin which creates a file that shows the classes last login since the last system reboot, and now I am asked to write an Awk script which is named myawk that ...
6
votes
2answers
3k views

Replace string with sequential index

Can someone suggest an elegant way to accomplish this? Input: test instant () test instant () ... test instant () //total 1000 lines output should be: test instant1 () test ...