Questions tagged [data]
The data tag has no usage guidance.
173
questions
0
votes
0
answers
14
views
Rsync transactions?
I use rsync to move changes in my data between two computers. I do backup of data dir from PC2 to usb stick and than I insert usb stick to PC1 and run rsync to update data dir on PC1 according to data ...
- 26
0
votes
1
answer
32
views
Synchronize Data between two locations
I have to locations with a lot of data. Both locations have a slow Internet connection. So it's impossible to sync the Data between both locations via Internet. So the idea is to use a sneaker net to ...
- 81
1
vote
1
answer
25
views
How to know if files inside an encrypted ZFS dataset are actually encrypted or not?
When you make any change on a zfs dataset, changes are not applied to already existant data. So, if you find a ZFS dataset that says it has encyption ON, there is a way to check if an individual file ...
- 258
0
votes
2
answers
110
views
XFS version 4 vs 5 (RHEL 7 to 8)
In RHEL 7.9 I formatted my large /data volume with XFS 4.5.
In RHEL 8.8 XFS is version 5.0.
XFS v5 in RHEL 8 can mount an XFS v4 file system created by RHEL 7.9, however RHEL 7.9 cannot mount an XFS ...
- 5,806
0
votes
2
answers
148
views
Bash manage space separated list
I have a rather complex shell script which processes several list of values which are read from a file as space separated values e.g.
SET1="value1 value2 value3"
for i in ${SET1}; do
...
...
- 5,100
-1
votes
1
answer
271
views
how to troubleshoot slow usb drive?
TLDR: How do I troubleshoot a slow usb 3.1 device plugged into a laptop.
ISSUE: When I copy (tried GUI and terminal) the first few .iso files copy almost instantly 300mb/s+, but then the 3rd/4th start ...
- 1
1
vote
0
answers
50
views
How important is LUKS reserved block count for external drive?
I have LUKS FDE on my external data HDD but recently almost filled it up. When cleaning it up I noticed that the available and free counters don't match.
Read that LUKS reserves 5% of total capacity ...
- 133
0
votes
0
answers
37
views
Strange data loss
My work laptops come with Ubuntu 20.04 and a very strange thing happened
to a colleague of mine (I'm asking here because I think what happened
doesn't depend on the distribution): he was working with ...
0
votes
0
answers
22
views
Copying folders within archive and preserve metadata
Im using 7zip in ubuntu 22.04 and Im using this command
7z l -slt "/media/azeem/368A-5AA7/airdroid test folder made with winzip.zip"
airdroid test folder made with winzip.zip file is what I ...
0
votes
0
answers
65
views
Ubuntu shutdown during remote session
I've been having an issue with my remote Ubuntu desktop computer spontaneously shutting down during multi-thread processing tasks. Generally, I am connected via SSH (using argument -X) with a screen ...
- 11
0
votes
1
answer
186
views
Linux completely frozen
My linux system just froze and I can't even switch to tty.
I can't force reboot it because it is moving a partiton (a HDD with no system files) with really important data which I can't afford to lose.
...
- 1
0
votes
0
answers
44
views
(Manjaro) My HDD is giving me I/O errors, smartctl isn't working, and I can't mount my encrypted root
Before my HDD unmounted during operation, I made a symbolic link to two folders on my NVMe, not touching the HDD at all. Not sure how relevant that is, but it's what I can think of happening shortly ...
0
votes
1
answer
126
views
Can I backup my machine by just doing "cp -R / /pathToSSD"
I want to completely backup my computer. I got a 1TB drive. Normally backing up is a whole process and I find it would be so easier to just copy the filesystem into the drive then in the event of my ...
0
votes
0
answers
191
views
data recovery issue using foremost tool
i am trying to recover data using foremost data recovery tool in my ubuntu os
foremost recovers most known file formats by default like png, jpg, pdf and others, but i mainly need to recover files ...
- 1
0
votes
1
answer
165
views
fpsync syntax error
[oraadmin@sgwmsdbsprd03 ~]$ fpsync -v -n 8 -o "-lptgoD -v --numeric-ids -e ssh -p 222" /app/test/ [email protected]:/app/testing
1661915613 Info: Run ID: 1661915613-41320
1661915613 ===&...
2
votes
1
answer
46
views
What is the use/purpose of struct map_segment
I was looking through source code of the Linux kernel (version v2.6.38) and I stumbled upon an undefined struct map_segment defined in exec_domain:
struct exec_domain {
const char *name; ...
0
votes
2
answers
46
views
Extracting just GPS location to new files named their date stamp from one huge log file
So, i've huge (over 100k records) log file, and need to extract all GPS locations based on their datestamp.
./production.log.109.gz:I, [2022-02-10T10:00:59.703529 #25190] INFO -- : #<Event::...
- 3
-1
votes
1
answer
140
views
Is there a standard location where executables dependencies are stored?
When I list the contents of the bin folders on my system, I often only see the executable files. Some of the executables require additional files such as data and configuration files in order to run. ...
- 137
0
votes
2
answers
566
views
read raw data from disk and convert
please help to find a way to convert the data as follows:
I read from disk using dd utility
dd if=/dev/sdb skip=8388608 count=560 iflag=skip_bytes,count_bytes |hexdump -C
and I am getting
...
- 1
-1
votes
1
answer
75
views
average of nth line from mutiple files into one average master file
I have 3 files which contain 8 lines of numerical values and text. I'm trying to take the average of each rows across all three files and print a new file with those averages.
The three example files ...
- 27
0
votes
0
answers
141
views
Is it overkill to shutdown a VM before taking an LVM snapshot for backup purposes?
As part of a backup strategy for hundreds of VMs built on LVM volumes, I am considering:
Shutdown the VM.
Take a snapshot of the LVM volume containing user and application data, including MySQL and ...
5
votes
3
answers
2k
views
Finding an appropriate backup strategy (borgbackup, restic etc)
I am fully aware that questions like this one have been asked multiple times. However, since reading lots of documentation of various backup tools didn't answer all my questions, I think its fine to ...
- 143
1
vote
2
answers
532
views
How to calculate the size of data transferred between two machines in Ubuntu?
I have two computers.
The first one is as a workstation. It runs the deep learning code and connects an AP by a wired connection.
The second one is as a robot's computer. It sends the video and other ...
- 111
0
votes
0
answers
648
views
How do I access my files from PMBR disk
I was creating a partition on my external hard disk via the 'Disks' app in Linux mint and I accidentally changed the partition scheme. Now the disk is not recognised in either Windows or Ubuntu and in ...
adrew nelson
0
votes
1
answer
690
views
Copying an entire directory structure and/or skeleton
How do we copy an entire directory template/structure or skeleton ?
As nothing in use instruction:
Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.
-a, --archive same as -dR -...
0
votes
1
answer
442
views
Copy data from AWS S3 bucket to Linux SFTP
I am writing bash script to copy data from AWS S3 bucket to Linux SFTP.
#!/bin/bash
#!/usr/bin/expect
set CURRENT_DATE [lindex $argv 0]
CURRENT_DATE=`date +"%Y%m%d"`
cd /home/ubuntu/test/...
- 1
1
vote
1
answer
320
views
Does a process' kernel virtual memory contain process specific data?
I don't quite understand this picture from the CS:APP book.
It shows how the kernel virtual memory of a process has a region different from other processes.
Does this mean, that the kernel in the ...
- 13
7
votes
1
answer
8k
views
How secure is blkdiscard?
How secure is blkdiscard when using it on a ssd? I tried blkdiscard /dev/sda on a ssd running Ubuntu and after executing the command the OS doesn't survive a reboot and simply plugging the disk into a ...
- 564
0
votes
1
answer
34
views
Pasting files and concatating files together
I have various years and months of data for numerous site locations. I have computed the average for all months for each needed year. Each Month/Year has its own its own file. Not all sites have data ...
- 65
2
votes
0
answers
236
views
How to check the accumulated data written by each process
I have pihole running on a single board computer and i'm trying to reduce as much as possible data being written to the sd card. I already have logs and pihole database running in a ramdisk but
awk '/...
- 21
-1
votes
4
answers
140
views
Combining different terms in third column into one row according to sample code
I have some genomic data with three columns. Column 1 is samples, Column 2 is cancer type, and Column 3 is mechanism. Each sample has one or more gene mechanisms assigned to it (e.g. 6 ecDNA) in the ...
1
vote
1
answer
85
views
How can I separate these two columns in this csv file in Linux/Bash?
I am looking to separate these two columns, each into their own separate text files. This data is from a csv file on Kaggle that contains Titanic passenger data. The first column is the number of ...
- 11
0
votes
3
answers
112
views
How would I format filtered data to new file?
Hello all I'm trying to read data from file, and check for each state and increment total transactions counts from data:
REDACTED
The end result should be printed to a new file in this format
This ...
- 5
0
votes
1
answer
50
views
How to keep blank data or assign missing data when recompiuting data for unit shift
I have a series of data files with format below. I have a script to read just the data to the needed units (ppm to ppb) thus the *1000 of column $3. The issue which I am running into when awk runs and ...
- 65
1
vote
1
answer
190
views
Use GNU Parallel when file has a single (long) line
I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv.
This file has 544609657 characters, is about 545 megabytes, all in a single line.
...
0
votes
0
answers
54
views
how to pass a string with spaces as argument in shell script [duplicate]
Below is my code to which i'm trying to pass a 3 argument like "client name with datacenter" hostname ip
so, "client name with Datacenter" is the 1st argument
likewise hostname and ...
- 33
0
votes
1
answer
505
views
Nextcloud bulk loading new data with Linux command line
I have nearly 40,000 folders with over 187,000 files to add to a nextcloud server. I did quite a bit a research and finally find an answer which I give below. I added this here so others did not need ...
- 111
0
votes
1
answer
102
views
Transpose large table
guys
I am sorry if this has been asked before, but I will give it a try.
I have a table (tab delimited text file) that is formatted like this
Species1 Species2 Species3
Gene 1 Value ...
0
votes
0
answers
409
views
Move data between partition
I have a slow drive with a big chunk of data that I want to cut/paste in an other partition (of the same drive obviously, otherwise I shouldn't speak about partition but drives directly), but this ...
- 157
0
votes
0
answers
238
views
How can I make a dummy directory tree with dummy files that contain realistic data?
I'm looking for preferably a one-liner that drops a directory tree a few layers deep with files that contain "realistic" data, as in, not /dev/zero nor /dev/urandom. Data that noticeably ...
- 375
0
votes
2
answers
667
views
Merging two CSVs based on matching first column (with header)
I have two CSV files. One of which contains a lot of data:
main.csv
Order No.,invoiceNumber,invoiceAmount,invoiceAmountNet
20001,20010,500,5
20002,20011,400,4
20003,20012,300,3
20004,20013,200,2
20005,...
- 3
1
vote
1
answer
220
views
Computations on matrix and avoiding division by zero
I want to compute average of values in data table
Example data matrix
S1 1 3 4 5 16 3 3 4 6 0
S2 1 6 4 5 16 3 4 6 6 0
S3 1 3 4 3 4 7 3 5 16 9
S4 ...
- 105
0
votes
2
answers
53
views
how can i pull data based on certain flags for rows of data and create a new file with them
I'm working with Upper Air Balloon data with a format of ....
**254 12 1 JAN 2002
1 94703 72501 40.87N 72.87W 20 1116**
2 100 100 250 40 32767 3
**3 ...
- 65
-1
votes
1
answer
177
views
How to view all the list that can be viewed by update-alternatives?
Tabbing gives me ls data but i want something to be shown like x-session-manager , x-terminal-emulator and so on. How can i get list what can i set?
- 149
0
votes
1
answer
357
views
How to I remove the all meta-data, identify objects, embedded-data, non-pdf/embedded binary, identity(IDs), embedded-character etc in PDF file?
I want convert the PDF to only TEXT(as unicode) & IMAGE based PDF.
So the pdf document should consist only of images and text. I want to do this in a GNU/Linux environment, it would be better for ...
- 184
1
vote
1
answer
2k
views
Is it possible to merge 2 hard drives into 1 big volume without losing data on of them?
I got 2 connected hard drives in my CentOS 7 dedicated server, though only one is being used (already have data on it), while the other just sits there unpartitioned and unused. I want to 'add' the ...
- 11
0
votes
0
answers
19
views
Reducing amount of data on remote server
I have a vague question.. which is hard to Google on, so using stackexchange might be more useful.
The current situation is
local computer: Ubuntu 16.04, not much storage space (~100 Gb)
Remote ...
- 101
0
votes
1
answer
640
views
date tz converts with GMT and City Names with same GMT get differ result
I want convert custom string time on UTC between different time zones defined as GMT.
I general all answers that is here ok.
But some question is with this examples:
1.TZ='GMT-07' date --date='TZ="...
- 1
0
votes
1
answer
97
views
I would like to analyze a TXT file, but the data is a 1 column array, and it needs to be a in a multi-column array
Is there a way for me to manipulate the data (i.e.; change some feature of, so that it looks like it should, a multi-column array) without having to manually go through it? There are thousands of rows ...
- 23
1
vote
1
answer
142
views
awk - arithmetic operation, use different formatting in case field contains a string instead of a number
I am trying to subtract data sets from each other in a single file. However when the script hits a value of na the output is incorrect.
Input file:
# <header>
2002 328 0.1345 2002 ...
- 65