Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data]

The tag has no usage guidance.

2
votes
1answer
47 views

What are data integrity / bit-rot protection options on CentOS 7?

I have a 2 disk CentOS 7 machine build that that I need data integrity / bitrot protection on. How can I achieve this? Note from my reading btrfs,zfs and DM-Integrity does not seem to be options. ...
0
votes
1answer
17 views

How to move physical disks content (rsnapshot content in there) to another physical disk?

Situation: simple linux system, with external removable usb media (rdx), single primary partition with ext4 filesystem on them for backups. Using some external usb disk media (rdx disks) for ...
1
vote
1answer
69 views

Data Cleaning Using Shellscript

I want a combined query suitable for selecting rows where lang = en i.e. language is equal to English and removing the other rows while I convert this file from JSON to CSV. json2csv -i Downloads/30....
0
votes
0answers
16 views

HAproxy / rewrite local ip to external ip

I have a local cluster and an external client on the internet. The solution is communicating on several ports, but though a HAproxy upstream LB. The cluster handles many other solutions on the local ...
3
votes
1answer
64 views

How to copy partitions from one device to another, including boot partitions?

Basic Explanation How could I copy partitions sdh1 though shd4 from my 600gb HDD (sdh) to my 1 TB SSD (sdg)? Which are my Windows partitions and still keep it bootable? I used GPT. Below is a ...
1
vote
0answers
12 views

I Node and Data Retrieval

Is there a way possible to open a data block without the file if we know the i node number that it's linked to ? Is it possible to open the content in data block without the file ?
24
votes
2answers
3k views

Purpose of cp -x (stay on file system)?

If I wanted to stay on the same file system, couldn't I just specify an output path for the same file system? Or is it to prevent accidentally leaving the current file system?
1
vote
0answers
17 views

How to see the file revisions in Tux3 and NILFS?

Tux3 and NilFS are file systems made for file history/revisions. How can I see these revisions? On Windows, there is a file history manager and Shadow Copies (enabled by default only on Windows 7 ...
0
votes
0answers
17 views

How to delete backed files only from original storage device?

I have a backup of an SD card but have not yet deleted the original files to free up space. Now, they got mixed up with new files. How do I delete all files that are already backed up (same file name ...
4
votes
0answers
74 views

Is there a standard text mode tool to render heatmaps for arbitrary numeric matrix data?

For the curiosity's sake I created a script to visualize Git history week/hour-of-day basis timestamps in order to collect some stats. The following script #!/bin/bash # Declare the global heatmap ...
0
votes
0answers
17 views

Deactivating disk cache for specific device or partition? (play 2160p video off SD card without sacrificing RAM space other programs need)

I would like to deactivate the disk cache for /dev/mmcblk0p1 in order to preserve more RAM for other system parts while playing 2160p video off /dev/mmcblk0p1. 2160p videos ravenously wipe the RAM ...
2
votes
1answer
26 views

Generating dead LBA list from gddrescue mapfile/logfile?

I ultimately would like to know which files in images generated by (g)ddrescue are affected by damage. gddrescue (command ddrescue) is a tool for recovering data from damaged media. When specifying ...
0
votes
1answer
44 views

Can dvdisaster recover optical discs better than gddrescue?

gddrescue is more universal. But both tools are just limited by hardware restrictions. If a CD's sector is unreadable, the drive usually refuses to let the computer know about the subcode. For ...
0
votes
1answer
14 views

List changed files and folders compared to disk image

Let's call the image Disk.img, taken from /dev/sda. How do I list every file and folder with differences? (Let's assume the time stamps of the files are incorrect.)
0
votes
1answer
30 views

Differentiate bad logical and physical blocks? (list seperately)

The tool badblocks can give a list of unreadable LBA's, including logical errors I guess. How can I differentiate between logical (soft) bad blocks and physical (hard) bad blocks? List logical and ...
0
votes
0answers
16 views

Back up and sync to mirror disk or image file

Let's assume a computer with 1 TB SSD and 2 TB HDD and an empty external 4 TB HDD for backup purposes. Now, to prevent data loss if one of the drives fails, a disk image or a copy of each file can be ...
0
votes
1answer
27 views

Recover Data while installing Liniux by option : “Erase disk and install Linux Mint”

I tried to intsall the Linux Mint by selecting the option by option : "Erase disk and install Linux Mint" assuming that the installation will be on C drive (where windows was installed ) but the ...
0
votes
0answers
34 views

Executing a script after a TCP connection closes

I'm trying to figure out a way to execute a given script (a file) after a TCP connection closes, and if possible filter out the interface it was going through. Now, what i want to do with this data is ...
0
votes
0answers
11 views

Output data directly to MTP/FTP storage?

My phone (only MTP data access) has a MicroSD card inserted into it Now, I wish to save a ddrescue disk image directly onto it while the phone is connected to the PC.That disk is not damaged at all, ...
0
votes
1answer
188 views

Data getting overwritten with same timestamp in InfluxDB

There are some data in a csv file and it is getting overwritten if the timestamp is same, for example: ip,time,name 1.1.1.1,2018-08-31 11:12:32,python 1.2.1.2,2018-08-31 11:05:28,oracle 1.3.1.3,2018-...
0
votes
1answer
177 views

Understanding direct and indirect blocks?

Why can't we use use indirect blocks instead of direct blocks. Indirect blocks can store much more memory addresses which results in more data blocks. So why waste those 10 direct blocks in Inode ...
-1
votes
1answer
33 views

Error Displaying last 10 minutes log with awk [duplicate]

I am using below command to display log data for last 10 mins. It works fine if the month stays same however when month is changed it doesn't show any data. awk -v d1="$D1" -v d2="$D2" '$0 > d1 &...
2
votes
5answers
530 views

Grep specific data from string [duplicate]

Here is the data: Valid from: Fri Nov 24 12:11:28 EST 2017 until: Sun Nov 24 12:11:28 EST 2019 Valid from: Fri Feb 24 13:21:19 EST 2017 until: Thu Feb 24 13:31:19 EST 2022 Valid from: Tue Dec 20 12:...
0
votes
2answers
26 views

Segregate Data in Log files

I want to segregate error types in my log file. I am unable to make a logic for it. Suppose i have these different types of errors in a log file. Now i want to extract those types only. Like in below ...
3
votes
1answer
196 views

Use Variable Date in Sed Command

I want to use sed command as below: $ sed -n '/MONTH_TODAY DATE_YEST/,/MONTH_TODAY DATE_TODAY/p' \ /home/crmiib/TESTUSERLOG/XYZ/user.log I can print logs using: $ sed -n '/Jul 27/,/Jul 28/p' /...
0
votes
1answer
40 views

Combine files into one [duplicate]

If I have a folder with the following file names: cluster_sizes_0.txt cluster_sizes_1.txt cluster_size_2.txt etc. Each file contains a single column of values. Is there a command within linux such ...
1
vote
1answer
84 views

Two consecutive OPs after pipe or two jq OPs in one run?

I have to extract data from a slightly mis-formatted JSON string, hence I first pass it through sed & awk. What I have is a command like: `sed 's/},/},\n/g' test.json |awk '/"characater"/ { gsub("...
2
votes
3answers
124 views

extract data from JSON string

I've got to extract a license plate digit and it's associated confidence from a JSON string that looks like: { "response": { "container": { "id": "0df307bc-06b2-45cf-b7ff-ce07fd04e04d", ...
2
votes
1answer
20 views

Prepare Live-Stick WITH extra data on 2nd partition

I have a USB Stick of 64 GB. I have a debian ISO. I can make myself a bootable USB stick with sudo cp debian.iso /dev/sdb Problem 1: I cannot copy various files to the stick preventing them to be ...
2
votes
1answer
33 views

Verify disk integrity without reading data?

There is an application called “HDDscan” for Windows: http://hddscan.com/ It can also verify disks by reading them in a way they only get transferred into the internal buffer of the drive, and ...
0
votes
2answers
698 views

Removing EFI System Partition from data disk

I have one SSD and one HDD in my computer. I use the SSD as the drive which holds the OSes (Ubuntu and Windows). I installed them on my SSD in BIOS/legacy mode (MBR format) and I still have an ESP on ...
2
votes
0answers
103 views

Home NAS: Best setup to ease data recovery in case of hardware failure? Should I get rid of HW raid 1?

I'm not quite sure how to formulate the question here. I built myself a low energy NAS with an Intel N4200 with 8GB RAM. For the hard drives I'm using two WD red in a Raidon GR3660-B3 which is running ...
1
vote
1answer
53 views

Secure temporary storage

I'm trying to find a way to securely store data for the duration of a session - specifically it's for passwords on a jump box - so the user only needs to enter the password for a given target once in ...
-1
votes
1answer
170 views

Problems with external hard disk

I can't overcome a problem I have with my hard disk! I don't know if it is the right section, but the problem I have started suddenly while I was copying files on Debian. The problem is that my ...
0
votes
2answers
137 views

Using if and shell variables inside an awk script

I am trying to get data from a file that is like this: 6 6 1 0 0.1166667E+02 0.4826611E-09 0.4826611E-09 0.3004786E-09 0.5000000E-15 1.000000000000000E-004 CAR system-001 ...
0
votes
1answer
148 views

Handle if statements and variables with awk

I am trying to get data from a file that is like this: 6 6 1 0 0.1166667E+02 0.4826611E-09 0.4826611E-09 0.3004786E-09 0.5000000E-15 1.000000000000000E-004 CAR system-001 ...
-2
votes
3answers
183 views

Ubuntu 18.04 with a PC tracking mechanism is still a “free software”? [closed]

I read this article that basically says the Ubuntu 18.04 will include a mechanism that collects the following basic data on a user: Which version of Ubuntu you’re installing (including which flavour) ...
0
votes
0answers
55 views

Does Mac OSX keep a log of network connection addresses and data transfers statistics for all of my browsing activity?

I'm trying to find logs of host addresses my computer has connected with over time as well as the corresponding counts of sent/received bytes for each of these connections. For example, the nettop ...
3
votes
0answers
279 views

Do rsync --checksum and diff usually have different outputs or is my hdd broken?

I set up a NanoPi Neo 2 with OpenMediaVault (Debian Jessie) as its OS and have an hdd attached to it (the OS is on an SD card). When I now copy my data from my external hard drive onto my system with ...
0
votes
0answers
20 views

Forcing a clean uninterrupted baud when receiving data via serial port

I made a project with a micro-controller that allows me to send data to the PC where the low nibbles of data always follow the same sequence. For example, My data is sent like this: x0 x1 x2 x3 x4 ......
0
votes
0answers
79 views

Access/modify/change time for tar files in data domain

So I have implemented a data domain retention policy for tar files. After tar file is created: touch -at date_in_future file.tar Lets say my date_in_future is 60 days, so I will not be able to ...
1
vote
1answer
73 views

Pull multiple rows of data from 2 columns in several locations

I have some really poorly formatted data files, and unfortunately the format of the files produced cannot be altered. I was using GAWK to parse the data into a manageable format but now I've been ...
2
votes
1answer
2k views

Ubuntu 16.04 LTS: Value too large for defined data type

I am trying to run Partiview software which is a visualization tool for particle data sets. I have run a large simulation that includes millions of particles and the resultant data file that I am ...
1
vote
1answer
58 views

Transpose a 3D tensor reprensented by files

I have a 3D tensor of dimension MxNxD data object. And this data object is decomposed and stored in different files. Initially, each slice/matrix of dimension MxN (first two dimension) is saved in a ...
4
votes
2answers
4k views

diff two large CSV files (each 90GB) and output to another csv

This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g. file 1: ID Date ...
2
votes
0answers
238 views

SMART data of SanDisk Extreme Pro

I have RHEL 6.8 with smartmontool 5.43 on Dell OptiPlex 9020 with SanDisk Extreme Pro 240 GB SSD as OS disk used for quite disk i/o intensive work. I am concerned about durability of SSD. After one ...
3
votes
0answers
434 views

*REAL* zerofree equivalent for btrfs

is there any zerofree equivalent for btrfs ? and please don't give me sfill or cat /dev/zero >/tmp/file, this is not the same thing at all, be cause Zerofree finds the unallocated, non-zeroed blocks, ...
1
vote
1answer
57 views

I need to delete . in my data

I have a data file with 4 columns. 4th column is real variable with 5 decimals followed by 0. I need to remove the "." in this column. I tried using sed command in R system(paste('sed s/./""/g <"...
0
votes
1answer
95 views

Gnuplot fitting with Gauss result in different surfaces, why?

I have a question related to the surface generation with gnuplot, or lets say with the fitting procedure. I have different curves which does not follow a mathematical formula but I need to use these ...
3
votes
2answers
119 views

Which command produces more data per second?

Which command produces more data per second? This could be useful to quickly fill a file with garbage data or to test data transfer rates. So far, I found that "/dev/zero" is the quickest one. $ cat /...