Questions tagged [data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
37 views

awk - arithmetic operation, use different formatting in case field contains a string instead of a number

I am trying to subtract data sets from each other in a single file. However when the script hits a value of na the output is incorrect. Input file: # <header> 2002 328 0.1345 2002 ...
1
vote
1answer
71 views

Is it safe to copy a directory while the contents are constantly changed?

I want to backup a shared directory with lots of smalls files, which are constantly changed by my users (add/delete/edit). As usual, I use tar and filter the content through a compression program like ...
0
votes
1answer
32 views

Sort on substring of line (not using a delimiter)

I need some help understanding how the sort command is working in the following examples: cat testfile The input data: thisfilehasduplicates thissnowhasfallen thisduckhasdied thishallwasfull ...
0
votes
1answer
18 views

Making use of second (ext3) hard drive

I have two hard drives. Both are FireCudas, "half ssd, half hdd". One has 1TB of space and the other, with my Manjaro on it, has 480GB. Up until now (for 2 years) i have not used the second drive, ...
1
vote
0answers
24 views

Wipe disk with bad sectors

I'm trying to wipe a bad disk using dd with the following command sudo dd if=/dev/zero of=/dev/sda status=progress, but it exits prematurely and outputs the following: 13160186368 bytes (13 GB, 12 ...
1
vote
1answer
45 views

Metered network

My mobile ISP handles a data limit of 12,5GB, and I'm using Manjaro Linux. I'm asking myself how a "metered" connection will save data. In the NetworkManager, there are three different options under ...
0
votes
1answer
33 views

Data sanitisation question - regular expression for removing combinations of trailing spaces and commas

I have address strings which were (due to poor initial programming) allowed far too much entry of free text. In the course of cleaning up this inherited mess, I have noticed that at the end of the ...
0
votes
0answers
50 views

Trying to save a 2012 macbook pro failing disk with ddrescue

I have a Macintosh HD disk that failed suddenly (system would not boot anymore - diskutil complained about a problem with the B-tree. Tried various diskutil commands in the past without success) which ...
0
votes
0answers
6 views

Sniffing to Signal Coming From a Controller to Raspberry Pi

I am working on a project on Raspberry Pi to send commands to a controller. I am using ttyUSB0 port of the Rpi with RS485 to communicate with controller. Normally, there is a dos program which does ...
5
votes
1answer
248 views

how to sort based on numbers on column with seperator on bash

how to sort these data by fifth column PREPAID|085319338982|Replacement Card|2016-05-31 15:15:29|218BKSAT||14121837 | 0 PREPAID|08128352315|Replacement Card|2016-05-31 15:15:33|218KKSAT||14121842 | 0 ...
0
votes
1answer
31 views

Scrub filesystem to check all used sectors are readable

I'd like to have a way to periodically check that I can indeed read all the data on my drives, so I can re-write any files and/or investigate further if there is an issue. Both ZFS and Btrfs have a ...
-2
votes
1answer
32 views

How change the mount point on an external hdd that has data on it, without deleting any data?

I need to change the mountpoint from "/media" to "/mnt" on an external hdd that has data that can not be deleted. I read that I can create a directory using "sudo mkdir", but that is all I have ...
-2
votes
1answer
80 views

Cloning a disk to a larger disk without overwriting

If I clone a hdd to another (larger) hdd that already has data on it, will the older data get deleted or will new data get added on top of old data? I need to move data from 2 smaller hard drives to ...
3
votes
1answer
155 views

ZFS silent corruption

I am trying to find answers to some questions on how ZFS works: does it detect silent corruption via checksums as soon as data is changed (and differs from checksum), automatic in a way (then if ...
1
vote
2answers
126 views

Manipulate Data in Linux

I have data in 1 file example: a1b1:online xxxx:offline wxyz:...
0
votes
0answers
35 views

Slicing and dicing sqldata dumps

I have a sqldump (.sql) data file that is quite large (30GB) and would like to use sed, awk or grep to print each string of characters between a set of parentheses (). My goal is to convert it into a ....
0
votes
2answers
82 views

How to remove duplicate lines inside a text file & get the number of the removed lines?

I know that with awk '!seen[$0]++' filename > output.txt it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information ...
1
vote
1answer
113 views

lvm data recovery after reinstall

I have an SSD and a HDD on my pc. SSD is for system and HDD is for storing data. I changed my os from debian to ubuntu but I forgot to backup the lvm on my HDD. Of course I lost my data. I've tried ...
0
votes
0answers
24 views

DDRescue Rescue Rate Concern

I'm in the process of using DDRescue on a 896 GB partition of my 1 TB Toshiba HD. After almost 5 days it's rescued only 7% of my data (according to terminal input). I paused the program for the first ...
0
votes
2answers
73 views

Replace User Name on one file with User Name plus Company from output of another file

I have 2 text files. One is the list of user names along with the company they work on the same line but separated by a tab space. The second is a output from a weekly generated report that includes ...
0
votes
2answers
399 views

df -h results different than VGdisplay / LVdisplay

Im trying to remove the added disk to the VM, as the disk space is not needed anymore. However, it looks to be that the VM has in the VG almost all the data used what is available there. However, i'...
0
votes
3answers
54 views

Grep/Awk for Value Between

So I have a text file with lines like this: http://sg.ovies.m/.ss/ (Status: 200) [Size: 128] http://sg.mo.v/.dev/ (Status: 200) [Size: 12328] http://som.b/.hal/ (Status: 200) [Size: 1238] http://m.cb/...
12
votes
5answers
3k views

Check data integrity after copying thousands of files [duplicate]

I copied thousands of files into an exFAT MicroSD card. The number of files and bytes is identical, but how do I know whether the data is corrupt or not? It would be good if the JackPal Android ...
4
votes
1answer
365 views

What are data integrity / bit-rot protection options on CentOS 7?

I have a 2 disk CentOS 7 machine build that I need data integrity / bitrot protection on. How can I achieve this? Note from my reading btrfs,zfs and DM-Integrity does not seem to be options. Btrfs ...
0
votes
1answer
89 views

How to move physical disks content (rsnapshot content in there) to another physical disk?

Situation: simple linux system, with external removable usb media (rdx), single primary partition with ext4 filesystem on them for backups. Using some external usb disk media (rdx disks) for ...
1
vote
1answer
120 views

Data Cleaning Using Shellscript

I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV. json2csv -i Downloads/30....
0
votes
0answers
43 views

HAproxy / rewrite local ip to external ip

I have a local cluster and an external client on the internet. The solution is communicating on several ports, but though a HAproxy upstream LB. The cluster handles many other solutions on the local ...
3
votes
1answer
298 views

How to copy partitions from one device to another, including boot partitions?

Basic Explanation How could I copy partitions sdh1 though shd4 from my 600gb HDD (sdh) to my 1 TB SSD (sdg)? Which are my Windows partitions and still keep it bootable? I used GPT. Below is a ...
1
vote
0answers
18 views

I Node and Data Retrieval

Is there a way possible to open a data block without the file if we know the i node number that it's linked to ? Is it possible to open the content in data block without the file ?
25
votes
2answers
4k views

Purpose of cp -x (stay on file system)?

If I wanted to stay on the same file system, couldn't I just specify an output path for the same file system? Or is it to prevent accidentally leaving the current file system?
1
vote
0answers
24 views

How to see the file revisions in Tux3 and NILFS?

Tux3 and NilFS are file systems made for file history/revisions. How can I see these revisions? On Windows, there is a file history manager and Shadow Copies (enabled by default only on Windows 7 ...
0
votes
0answers
18 views

How to delete backed files only from original storage device?

I have a backup of an SD card but have not yet deleted the original files to free up space. Now, they got mixed up with new files. How do I delete all files that are already backed up (same file name ...
4
votes
0answers
146 views

Is there a standard text mode tool to render heatmaps for arbitrary numeric matrix data?

For the curiosity's sake I created a script to visualize Git history week/hour-of-day basis timestamps in order to collect some stats. The following script #!/bin/bash # Declare the global heatmap ...
0
votes
0answers
38 views

Deactivating disk cache for specific device or partition? (play 2160p video off SD card without sacrificing RAM space other programs need)

I would like to deactivate the disk cache for /dev/mmcblk0p1 in order to preserve more RAM for other system parts while playing 2160p video off /dev/mmcblk0p1. 2160p videos ravenously wipe the RAM ...
3
votes
1answer
74 views

Generating dead LBA list from gddrescue mapfile/logfile?

I ultimately would like to know which files in images generated by (g)ddrescue are affected by damage. gddrescue (command ddrescue) is a tool for recovering data from damaged media. When specifying ...
0
votes
1answer
150 views

Can dvdisaster recover optical discs better than gddrescue?

gddrescue is more universal. But both tools are just limited by hardware restrictions. If a CD's sector is unreadable, the drive usually refuses to let the computer know about the subcode. For ...
0
votes
1answer
17 views

List changed files and folders compared to disk image

Let's call the image Disk.img, taken from /dev/sda. How do I list every file and folder with differences? (Let's assume the time stamps of the files are incorrect.)
0
votes
1answer
211 views

Differentiate bad logical and physical blocks? (list seperately)

The tool badblocks can give a list of unreadable LBA's, including logical errors I guess. How can I differentiate between logical (soft) bad blocks and physical (hard) bad blocks? List logical and ...
0
votes
1answer
85 views

Recover Data while installing Liniux by option : “Erase disk and install Linux Mint”

I tried to intsall the Linux Mint by selecting the option by option : "Erase disk and install Linux Mint" assuming that the installation will be on C drive (where windows was installed ) but the ...
0
votes
1answer
1k views

Data getting overwritten with same timestamp in InfluxDB

There are some data in a csv file and it is getting overwritten if the timestamp is similar, for example: ip,time,name 1.1.1.1,2018-08-31 11:12:32,python 1.2.1.2,2018-08-31 11:05:28,oracle 1.3.1.3,...
0
votes
1answer
521 views

Understanding direct and indirect blocks?

Why can't we use use indirect blocks instead of direct blocks. Indirect blocks can store much more memory addresses which results in more data blocks. So why waste those 10 direct blocks in Inode ...
-1
votes
1answer
59 views

Error Displaying last 10 minutes log with awk [duplicate]

I am using below command to display log data for last 10 mins. It works fine if the month stays same however when month is changed it doesn't show any data. awk -v d1="$D1" -v d2="$D2" '$0 > d1 &...
2
votes
5answers
1k views

Grep specific data from a string [duplicate]

Here is the data: Valid from: Fri Nov 24 12:11:28 EST 2017 until: Sun Nov 24 12:11:28 EST 2019 Valid from: Fri Feb 24 13:21:19 EST 2017 until: Thu Feb 24 13:31:19 EST 2022 Valid from: Tue Dec 20 12:...
0
votes
2answers
42 views

Segregate Data in Log files

I want to segregate error types in my log file. I am unable to make a logic for it. Suppose i have these different types of errors in a log file. Now i want to extract those types only. Like in below ...
3
votes
1answer
647 views

Use Variable Date in Sed Command

I want to use sed command as below: $ sed -n '/MONTH_TODAY DATE_YEST/,/MONTH_TODAY DATE_TODAY/p' \ /home/crmiib/TESTUSERLOG/XYZ/user.log I can print logs using: $ sed -n '/Jul 27/,/Jul 28/p' /...
0
votes
1answer
58 views

Combine files into one [duplicate]

If I have a folder with the following file names: cluster_sizes_0.txt cluster_sizes_1.txt cluster_size_2.txt etc. Each file contains a single column of values. Is there a command within linux such ...
1
vote
1answer
147 views

Two consecutive OPs after pipe or two jq OPs in one run?

I have to extract data from a slightly mis-formatted JSON string, hence I first pass it through sed & awk. What I have is a command like: `sed 's/},/},\n/g' test.json |awk '/"characater"/ { gsub("...
2
votes
3answers
427 views

extract data from JSON string

I've got to extract a license plate digit and it's associated confidence from a JSON string that looks like: { "response": { "container": { "id": "0df307bc-06b2-45cf-b7ff-ce07fd04e04d", ...
2
votes
1answer
27 views

Prepare Live-Stick WITH extra data on 2nd partition

I have a USB Stick of 64 GB. I have a debian ISO. I can make myself a bootable USB stick with sudo cp debian.iso /dev/sdb Problem 1: I cannot copy various files to the stick preventing them to be ...
2
votes
1answer
60 views

Verify disk integrity without reading data?

There is an application called “HDDscan” for Windows: http://hddscan.com/ It can also verify disks by reading them in a way they only get transferred into the internal buffer of the drive, and ...