Skip to main content
The 2024 Developer Survey results are live! See the results

Questions tagged [data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
2 answers
63 views

Many of my text files are suddenly missing

I am using Manjaro linux with a Samsung SSD 840 Pro. Ive noticed my Documents folder is suddenly empty. After rebooting, most of the text files I had saved are no longer there. The problem is some of ...
Rachel1983's user avatar
0 votes
1 answer
65 views

How can I (safely) refresh data on an HDD? [duplicate]

I have an HDD (for simplicity, let's assume it has a single partition). I would like to refresh the data on my HDD: Read all data, and write the same data, possibly but not necessarily at the exact ...
einpoklum's user avatar
  • 9,662
0 votes
0 answers
22 views

How to forward received data towards network namespace?

My goal is to forward received data to another pc. I have three devices (A, B and C) connected into the same network. A uses pyzmq to publish data in which B is subscribed. Then, B must send this ...
Raúl Parada Medina's user avatar
6 votes
2 answers
951 views

What is the best place for storing of semi-short-lived data produced by a bash script?

UPDATE Came up with an even better solution: Base directory /var/tmp Because think about it: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard | /var/tmp | Temporary files to be preserved ...
futurewave's user avatar
0 votes
0 answers
128 views

Mounting an old JBOD array with two disks DOS formatted

I have a couple of 500GB disks with data inside that I would like to get back. The two disks were mounted inside a Lacie disk, part of a JBOD array and formatted FAT32. I don't have the external disk ...
Joe Mauri's user avatar
0 votes
0 answers
42 views

Rsync transactions?

I use rsync to move changes in my data between two computers. I do backup of data dir from PC2 to usb stick and than I insert usb stick to PC1 and run rsync to update data dir on PC1 according to data ...
youni's user avatar
  • 26
0 votes
1 answer
104 views

Synchronize Data between two locations

I have to locations with a lot of data. Both locations have a slow Internet connection. So it's impossible to sync the Data between both locations via Internet. So the idea is to use a sneaker net to ...
user39063's user avatar
  • 201
1 vote
2 answers
295 views

How to know if files inside an encrypted ZFS dataset are actually encrypted or not?

When you make any change on a zfs dataset, changes are not applied to already existant data. So, if you find a ZFS dataset that says it has encyption ON, there is a way to check if an individual file ...
Héctor's user avatar
  • 278
0 votes
2 answers
1k views

XFS version 4 vs 5 (RHEL 7 to 8)

In RHEL 7.9 I formatted my large /data volume with XFS 4.5. In RHEL 8.8 XFS is version 5.0. XFS v5 in RHEL 8 can mount an XFS v4 file system created by RHEL 7.9, however RHEL 7.9 cannot mount an XFS ...
ron's user avatar
  • 7,284
0 votes
2 answers
1k views

Bash manage space separated list

I have a rather complex shell script which processes several list of values which are read from a file as space separated values e.g. SET1="value1 value2 value3" for i in ${SET1}; do ... ...
symcbean's user avatar
  • 5,839
0 votes
1 answer
1k views

how to troubleshoot slow usb drive?

TLDR: How do I troubleshoot a slow usb 3.1 device plugged into a laptop. ISSUE: When I copy (tried GUI and terminal) the first few .iso files copy almost instantly 300mb/s+, but then the 3rd/4th start ...
Xueshe's user avatar
  • 21
1 vote
0 answers
104 views

How important is LUKS reserved block count for external drive?

I have LUKS FDE on my external data HDD but recently almost filled it up. When cleaning it up I noticed that the available and free counters don't match. Read that LUKS reserves 5% of total capacity ...
BharathYes's user avatar
0 votes
0 answers
38 views

Strange data loss

My work laptops come with Ubuntu 20.04 and a very strange thing happened to a colleague of mine (I'm asking here because I think what happened doesn't depend on the distribution): he was working with ...
llantones's user avatar
0 votes
1 answer
241 views

Linux completely frozen

My linux system just froze and I can't even switch to tty. I can't force reboot it because it is moving a partiton (a HDD with no system files) with really important data which I can't afford to lose. ...
Leaves's user avatar
  • 1
0 votes
1 answer
259 views

Can I backup my machine by just doing "cp -R / /pathToSSD"

I want to completely backup my computer. I got a 1TB drive. Normally backing up is a whole process and I find it would be so easier to just copy the filesystem into the drive then in the event of my ...
AskedSuperior's user avatar
0 votes
0 answers
464 views

data recovery issue using foremost tool

i am trying to recover data using foremost data recovery tool in my ubuntu os foremost recovers most known file formats by default like png, jpg, pdf and others, but i mainly need to recover files ...
karim's user avatar
  • 1
0 votes
1 answer
270 views

fpsync syntax error

[oraadmin@sgwmsdbsprd03 ~]$ fpsync -v -n 8 -o "-lptgoD -v --numeric-ids -e ssh -p 222" /app/test/ [email protected]:/app/testing 1661915613 Info: Run ID: 1661915613-41320 1661915613 ===&...
Heinthant Oo's user avatar
2 votes
1 answer
52 views

What is the use/purpose of struct map_segment

I was looking through source code of the Linux kernel (version v2.6.38) and I stumbled upon an undefined struct map_segment defined in exec_domain: struct exec_domain { const char *name; ...
Sasha Kobrusev's user avatar
0 votes
2 answers
53 views

Extracting just GPS location to new files named their date stamp from one huge log file

So, i've huge (over 100k records) log file, and need to extract all GPS locations based on their datestamp. ./production.log.109.gz:I, [2022-02-10T10:00:59.703529 #25190] INFO -- : #<Event::...
Daniel's user avatar
  • 3
-1 votes
1 answer
278 views

Is there a standard location where executables dependencies are stored?

When I list the contents of the bin folders on my system, I often only see the executable files. Some of the executables require additional files such as data and configuration files in order to run. ...
ATL_DEV's user avatar
  • 147
0 votes
2 answers
700 views

read raw data from disk and convert

please help to find a way to convert the data as follows: I read from disk using dd utility dd if=/dev/sdb skip=8388608 count=560 iflag=skip_bytes,count_bytes |hexdump -C and I am getting ...
markmi's user avatar
  • 1
-1 votes
1 answer
75 views

average of nth line from mutiple files into one average master file

I have 3 files which contain 8 lines of numerical values and text. I'm trying to take the average of each rows across all three files and print a new file with those averages. The three example files ...
Lucas Djeu's user avatar
0 votes
0 answers
210 views

Is it overkill to shutdown a VM before taking an LVM snapshot for backup purposes?

As part of a backup strategy for hundreds of VMs built on LVM volumes, I am considering: Shutdown the VM. Take a snapshot of the LVM volume containing user and application data, including MySQL and ...
Pawel Herman's user avatar
1 vote
2 answers
740 views

How to calculate the size of data transferred between two machines in Ubuntu?

I have two computers. The first one is as a workstation. It runs the deep learning code and connects an AP by a wired connection. The second one is as a robot's computer. It sends the video and other ...
Redhwan's user avatar
  • 111
0 votes
1 answer
939 views

Copying an entire directory structure and/or skeleton

How do we copy an entire directory template/structure or skeleton ? As nothing in use instruction: Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY. -a, --archive same as -dR -...
cantik memekmu's user avatar
0 votes
1 answer
660 views

Copy data from AWS S3 bucket to Linux SFTP

I am writing bash script to copy data from AWS S3 bucket to Linux SFTP. #!/bin/bash #!/usr/bin/expect set CURRENT_DATE [lindex $argv 0] CURRENT_DATE=`date +"%Y%m%d"` cd /home/ubuntu/test/...
Saga's user avatar
  • 1
1 vote
1 answer
381 views

Does a process' kernel virtual memory contain process specific data?

I don't quite understand this picture from the CS:APP book. It shows how the kernel virtual memory of a process has a region different from other processes. Does this mean, that the kernel in the ...
seb's user avatar
  • 13
7 votes
1 answer
12k views

How secure is blkdiscard?

How secure is blkdiscard when using it on a ssd? I tried blkdiscard /dev/sda on a ssd running Ubuntu and after executing the command the OS doesn't survive a reboot and simply plugging the disk into a ...
sjaak's user avatar
  • 604
0 votes
1 answer
34 views

Pasting files and concatating files together

I have various years and months of data for numerous site locations. I have computed the average for all months for each needed year. Each Month/Year has its own its own file. Not all sites have data ...
stormctr2's user avatar
2 votes
0 answers
297 views

How to check the accumulated data written by each process

I have pihole running on a single board computer and i'm trying to reduce as much as possible data being written to the sd card. I already have logs and pihole database running in a ramdisk but awk '/...
Miguel's user avatar
  • 21
-1 votes
4 answers
140 views

Combining different terms in third column into one row according to sample code

I have some genomic data with three columns. Column 1 is samples, Column 2 is cancer type, and Column 3 is mechanism. Each sample has one or more gene mechanisms assigned to it (e.g. 6 ecDNA) in the ...
Michelle Yang's user avatar
1 vote
1 answer
124 views

How can I separate these two columns in this csv file in Linux/Bash?

I am looking to separate these two columns, each into their own separate text files. This data is from a csv file on Kaggle that contains Titanic passenger data. The first column is the number of ...
Tyler Young's user avatar
0 votes
3 answers
124 views

How would I format filtered data to new file?

Hello all I'm trying to read data from file, and check for each state and increment total transactions counts from data: REDACTED The end result should be printed to a new file in this format This ...
puuyo's user avatar
  • 5
0 votes
1 answer
62 views

How to keep blank data or assign missing data when recompiuting data for unit shift

I have a series of data files with format below. I have a script to read just the data to the needed units (ppm to ppb) thus the *1000 of column $3. The issue which I am running into when awk runs and ...
stormctr2's user avatar
1 vote
1 answer
297 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
Bernardo Loureiro's user avatar
0 votes
0 answers
56 views

how to pass a string with spaces as argument in shell script [duplicate]

Below is my code to which i'm trying to pass a 3 argument like "client name with datacenter" hostname ip so, "client name with Datacenter" is the 1st argument likewise hostname and ...
Varun Dogra's user avatar
0 votes
1 answer
596 views

Nextcloud bulk loading new data with Linux command line

I have nearly 40,000 folders with over 187,000 files to add to a nextcloud server. I did quite a bit a research and finally find an answer which I give below. I added this here so others did not need ...
Anthony Petrillo's user avatar
0 votes
1 answer
128 views

Transpose large table

guys I am sorry if this has been asked before, but I will give it a try. I have a table (tab delimited text file) that is formatted like this Species1 Species2 Species3 Gene 1 Value ...
MartinOdegaard's user avatar
0 votes
0 answers
501 views

Move data between partition

I have a slow drive with a big chunk of data that I want to cut/paste in an other partition (of the same drive obviously, otherwise I shouldn't speak about partition but drives directly), but this ...
aaa's user avatar
  • 167
0 votes
0 answers
287 views

How can I make a dummy directory tree with dummy files that contain realistic data?

I'm looking for preferably a one-liner that drops a directory tree a few layers deep with files that contain "realistic" data, as in, not /dev/zero nor /dev/urandom. Data that noticeably ...
Daffy's user avatar
  • 425
0 votes
2 answers
948 views

Merging two CSVs based on matching first column (with header)

I have two CSV files. One of which contains a lot of data: main.csv Order No.,invoiceNumber,invoiceAmount,invoiceAmountNet 20001,20010,500,5 20002,20011,400,4 20003,20012,300,3 20004,20013,200,2 20005,...
Tom's user avatar
  • 3
1 vote
1 answer
242 views

Computations on matrix and avoiding division by zero

I want to compute average of values in data table Example data matrix S1 1 3 4 5 16 3 3 4 6 0 S2 1 6 4 5 16 3 4 6 6 0 S3 1 3 4 3 4 7 3 5 16 9 S4 ...
CN_229133's user avatar
  • 115
0 votes
2 answers
60 views

how can i pull data based on certain flags for rows of data and create a new file with them

I'm working with Upper Air Balloon data with a format of .... **254 12 1 JAN 2002 1 94703 72501 40.87N 72.87W 20 1116** 2 100 100 250 40 32767 3 **3 ...
stormctr2's user avatar
1 vote
1 answer
508 views

How to view all the list that can be viewed by update-alternatives?

Tabbing gives me ls data but i want something to be shown like x-session-manager , x-terminal-emulator and so on. How can i get list what can i set?
Machine Yadav's user avatar
0 votes
1 answer
567 views

How to I remove the all meta-data, identify objects, embedded-data, non-pdf/embedded binary, identity(IDs), embedded-character etc in PDF file?

I want convert the PDF to only TEXT(as unicode) & IMAGE based PDF. So the pdf document should consist only of images and text. I want to do this in a GNU/Linux environment, it would be better for ...
rojen's user avatar
  • 184
1 vote
1 answer
2k views

Is it possible to merge 2 hard drives into 1 big volume without losing data on of them?

I got 2 connected hard drives in my CentOS 7 dedicated server, though only one is being used (already have data on it), while the other just sits there unpartitioned and unused. I want to 'add' the ...
Joe's user avatar
  • 11
0 votes
0 answers
30 views

Reducing amount of data on remote server

I have a vague question.. which is hard to Google on, so using stackexchange might be more useful. The current situation is local computer: Ubuntu 16.04, not much storage space (~100 Gb) Remote ...
zwep's user avatar
  • 101
0 votes
1 answer
784 views

date tz converts with GMT and City Names with same GMT get differ result

I want convert custom string time on UTC between different time zones defined as GMT. I general all answers that is here ok. But some question is with this examples: 1.TZ='GMT-07' date --date='TZ="...
lexxai's user avatar
  • 1
0 votes
1 answer
141 views

I would like to analyze a TXT file, but the data is a 1 column array, and it needs to be a in a multi-column array

Is there a way for me to manipulate the data (i.e.; change some feature of, so that it looks like it should, a multi-column array) without having to manually go through it? There are thousands of rows ...
Ctat41's user avatar
  • 23
1 vote
1 answer
153 views

awk - arithmetic operation, use different formatting in case field contains a string instead of a number

I am trying to subtract data sets from each other in a single file. However when the script hits a value of na the output is incorrect. Input file: # <header> 2002 328 0.1345 2002 ...
stormctr2's user avatar