102
Removing the first n lines (or bytes) can be done in-place using dd (or alternatively using loop devices). There will be no temp file and no size limit; however, it's dangerous since there will be no track of progress, and any error will leave you with a broken file.
Create a sample file with 1000 lines:
$ seq 1 1000 > 1000lines.txt
$ head -n 3 1000lines....
55
If you have enough space to compress the file, you can try this:
gzip file && zcat file.gz | tail -n +300000001 | gzip > newFile.gz
That will first gzip the original input file (file) to create file.gz. Then, you zcat the newly created file.gz, pipe it through tail -n +300000001 to remove the first 3M lines, compress the result to save disk space ...
25
You can do it with losetup, as an alternative to the dd method described here. Again, this method is dangerous all the same.
Again, the same test file and sizes (remove lines 1-300 from 1000 lines file):
$ seq 1 1000 > 1000lines.txt
$ stat -c %s 1000lines.txt
3893 # total bytes
$ head -n 300 1000lines.txt | wc -c
1092 # first 300 lines bytes
$ echo $((...
19
If you're on Linux and the filesystem is ext4 or xfs, and the purpose of this move is to free up space on the file system (otherwise you can leave the file as is and just seek into it when reading it), you can use the collapse range feature of the fallocate system call or command line utility.
The only problem is that it can only collapse full blocks, so you ...
12
Another vote for custom program if you really DO need the task. C or any powerful enough dynamic language like Perl or Python will do. I won't write out the source here, but will describe algorithm that will prevent data loss while you move data around:
Read your big file from the end counting line-breaks. After gathering some pre-defined amount of lines ...
9
GNU Sed:
sed 's/;./\U&/g' file
For every character following a semicolon (;.), we make it uppercase with the \U special sequence. The g flag substitutes all occurrences in a line.
If GNU Sed is not available, a POSIX compliant alternative is to use Ex.
printf '%s\n' '%s/;./\U&/g' '%p' | ex file
The substitute command is the same, but all lines ...
8
Not awk or sed, but perl:
perl -C -pe 's/;(.)/;\u$1/g'
The -C option turns UTF-8 i/o depending on or off depending on your locale environment variables (LC_ALL, etc); if you want it to assume UTF-8 input and output unconditionally, change it to -CSD.
Notice that Unicode capitalization is tricky. That will turn ;ihsan into ;Ihsan instead of the correct ;...
8
printf %.1s @{1..20} $'\n'
the shell expands the braces first, this is called "Brace Expansion".
@{1..20} into @1 @2 @3 ... and so on
Then the first byte of each parameter will be output, including the last argument $'\n' consisting of one byte - the newline character
7
With ksh93:
tail -n +300000001 < file 1<>; file
The 1<>; operator is a ksh93-specific variation on the standard 1<> operator (which opens in read+write mode without truncation), that truncates the file after the command has returned at the position the command left its stdout at if that command was successful.
With other shells, you can ...
7
With zsh:
printf '%s\n' ${(l[20][@])}
(using the l left-padding parameter expansion flag. You could also use the right padding one here).
Of course, you don't have to use printf. You could also use print or echo here which do add a \n by default. (printf '%s\n' "$string" can be written print -r -- "$string" or echo -E - "$string&...
6
When you need to use the two forms of quotes ("') in the expression, things get tricky. For one, in your original attempt the shell identifies this 's/value: [' as a quoted string: the latter quote is not preserved.
In these cases, rather than having a headache, you can simply put the Sed commands in a file. Its contents won't be subject to the shell ...
5
Here's one way, using perl:
perl -C -pe 's/;(.)/";" . uc($1)/eg' file
Since you don't show any accents in your input file, I used this for testing:
$ cat file
John Doe;john Doe is ...;he lives in ...
Mike Nelson;mike Nelson works for ...;he makes ...
Émilie du Châtelet;émilie du Châtelet;works for ...;she makes ...
Marcy William;marcy's mother is ....
4
The regex you’re using to match the lines has two start-of-line anchors (^), so the second one will only match actual ^ symbols. If you remove the ^ before [[:space:]], the expression will match, but too much:
# number_of_cars_are=$number_of_cars_are
# LOG "number_of_cars_are=$number_of_cars_are"
LOG " matches .* so it’s accepted. I would ...
4
The problem with your sed call is two-fold
You have literal single quotes as part of your regular expression ['"] in a sed program enclosed in single quotes. This will not work as single quotes inside single quotes cannot be escaped. You can represent them as \x27, however.
The ... (.*) ... syntax is extended regular expression syntax for having ( and )...
3
The limitation of this problem is the amount of storage wherever that is located. Significant RAM is not required since fundamentally you can simply read one byte from wherever your file is stored and then either write or not write that byte [character] out to a new file wherever that may reside. Where the infile and outfile reside can be in totally ...
3
# if you need the space after comma
$ seq 10 | perl -pe 's/.+/"$&"/; s/\n/, / if !eof'
"1", "2", "3", "4", "5", "6", "7", "8", "9", "10"
# if space isn't required
$ seq 10 | sed 's/.*/"&"/' | paste -sd,
"1","2",&...
3
If the profiles are blank-separated, then you can use awk in paragraph mode ex.
$ awk -v RS= -v p='[uat]' '$1 == p' yourfile
[uat]
region = us-east-2
output = json
3
In GNU sed,
sed -E '2s/^(([^|]*\|){2})[^|]*/\1changed/' input
The first “2” (2s) identifies the line number.
The second “2” ({2}) identifies how many columns
to skip over and leave the same.
If you wanted to change the 17th field, you would use {16}.
This command will write the modified file to the standard output.
You should probably run it with output ...
answered Sep 17 at 6:07
G-Man Says 'Reinstate Monica'
17.5k12 gold badges49 silver badges100 bronze badges
3
using Raku (formerly known as Perl6)
An advantage of the Perl6/Raku project is that it's designed to handle Unicode gracefully, from the ground up. Thanks to @terdon for posting a nice test file:
~$ raku -pe 's:g/ \;(.) /;{$0.uc}/;' terdon_uni.txt
John Doe;John Doe is ...;He lives in ...
Mike Nelson;Mike Nelson works for ...;He makes ...
Émilie du Châtelet;...
3
With GNU awk for the 3rd arg to match() and gensub():
$ awk 'match($0,/(\s*value:\s*)(.*)/,a){$0=a[1] "\"" gensub(/[\047"]/,"","g",a[2]) "\""} 1' file
- param:
name: Command
type: String
value: "/bin/echo"
- param:
name: Args
type: String
value: "Hello World"
- param:
...
2
The file that is causing this behavior is vte.sh in /etc/profile.d/:
One aspect of VTE configuration is the use of /etc/profile.d/vte.sh.
The VTE uses this script to override the PROMPT_COMMAND in order to
feed itself additional information via terminal control codes. In
particular, this script is used to tell the VTE the current directory
of the shell. ...
2
The reason for the problem is that you use "shell glob" (a.k.a "wildcard") patterns where a regular expression pattern is needed.
The part /test/test123/*.txt actually means "/test/test123, followed by zero or more / characters, followed by any single character, followed by txt".
This pattern is not what you want, and nowhere in ...
2
You should use an XML editor to edit XML. It can handle changes to the layout of the XML schema within the file, unexpected comments, etc. My preferred one is xmlstarlet
Example file, x.xml
<root>
<examples/>
<example path='/test/test123/test12345.txt'/>
<examples/>
</root>
Command, which updates the attribute path in any ...
2
You get the error because of mismatched single quotes.
The quoted string ends at the second quote. The shell finds unquoted >/<example and interprets this as redirection.
Additionally your sed command is incomplete. The trailing separator / is missing.
I also fixed the sed commands to match the input data which has been edited.
If you want to have ...
2
Using @terdon's sample file and GNU sed:
$ sed -e 's/;\(.\)/;\U\1/g' file
John Doe;John Doe is ...;He lives in ...
Mike Nelson;Mike Nelson works for ...;He makes ...
Émilie du Châtelet;Émilie du Châtelet;Works for ...;She makes ...
Marcy William;Marcy's mother is ...;Marcy travels a lot...
Άσπα Κυριάκου;Άσπα's brother is ...; άσπα likes fish
$
2
Think of Towers of Hanoi. Sort of.
First, find the start of line 3 million and 1
Then:
create a new, empty file
repeat {
read a decent number of blocks from the end of the old file
append the blocks to the end of the new file
truncate the old file by that many blocks
} until when you get to the start of line 3 million and 1.
You should now have a file ...
2
What about using vim for in-place editing?
Vim is already capable of reasoning about lines:
vim -c ":set nobackup nowritebackup" -c ":300000000delete" -c ":wq" filename
Explanation:
vim will execute the various command passed to the -c switches as if they where passesed in an interactive session.
So:
we disable backup copy ...
2
I created a tool that may be of use to you: hexpeek is a hex editor designed for working with huge files and runs on any recent POSIX-like system (tested on Debian, CentOS, and FreeBSD).
One can use hexpeek or an external tool to find the 300-millionth newline. Then, assuming that X is the hexadecimal zero-indexed position of the first octet after the 300-...
2
Perl in "paragraph mode":
$ perl -00 -ne 'print if /\[default\]/' file
[default]
region = us-east-1
output = json
or
$ perl -00 -ne 'print if /\[prod\]/' file
[prod]
region = us-east-1
output = json
You could make this into a shell function by adding these lines to your shell's initialization file (e.g ~/.bashrc):
searchawsconfig(){
perl -...
2
I use this in a script:
printf '%150s\n' | tr ' ' '@'
Only top voted, non community-wiki answers of a minimum length are eligible
