Zfs send & receive with encryption - how to retrieve data?


 
Thread Tools Search this Thread
Operating Systems Solaris Zfs send & receive with encryption - how to retrieve data?
# 1  
Old 03-26-2018
Oracle Zfs send & receive with encryption - how to retrieve data?

Good morning everyone,

I'm looking for some help to retrieve data in a scenario where I might have made a big mistake. I'm hoping to understand what I did wrong.

My system is made of two Solaris 11 Express servers (old free version for evaluation). The first if for data and the second is for backups.

On the first, I created zfs filesystems with encryption turned on (tank/Documents). To make things easy, I used "keysource=passphrase,file:///zfs_key", then I copied the file to the second (backup) server in the same path.

In order to do my backups, I used zfs send & mbuffer to send the whole zpool (all the zfs' filesystems). Normally, this would work fine for both encrypted and unencrypted volumes. Except the last time I did this, I did not mount the encrypted filesystem and I ran send & receive without getting any errors... That is, until I rebooted the backup server and tried to access the data (mount the filesystem).

For some reason I do not understand, I always get an "invalid key" error. The weird thing is the "keysource" in the backup system is still the same as the source and the "zfs_key" is the same. I thought that when you send&receive encrypted filesystem the "key" was automatically generated on the receiving system using the "keysource" mentioned here, but there seems to be something fundamentally different when the filesystem is not mounted. (For example scrub of encrypted zfs filesystem give errors when it is not mounted)

I would like to know where is the valid key in such a scenario? and/or what happened?

Thank you for giving me your opinion on the subject.

Best Regards,
# 2  
Old 03-27-2018
Hello,

I have found an old thread on zfs-discuss mailing list from 2012 :

Quote:
Roberto Waltman2012-Feb-18 05:12 UTChead link
[zfs-discuss] Cannot mount encrypted filesystems.
https ://thr3ads.net/zfs-discuss/2012/02/1839530-Cannot-mount-encrypted-filesystems

I do not know if anyone has found out the solution to this problem.

If this can help anyone help me better understand what happened and how to solve it, I would greatly appreciate it.

Best Regards,
# 3  
Old 05-22-2018
Hi rnd

Did you ever find a solution to this, I have a paraphrase issue and I know its correct even though the encryption says it isn't.

Cheers
# 4  
Old 05-23-2018
Hi,

I'm sorry to say that I have not found a solution for this problem yet.

I have also contacted Mr. Waltman that was having a similar problem mentioned in the mailing list link, but he told me he never received an answer nor modified library file to fix his problem also.

I tried contacting the Oracle dev (and Oracle support) to no success yet. I have not abandoned, but I was really hoping someone on this forum who had a similar problem found a solution...

I wish you good luck. If you find a solution, I would love to hear the answer.

Best Regards
# 5  
Old 05-23-2018
Can you confirm when everything is mounted on primary location, backup server receives the data and can be rebooted and import the zpool in question ?

A clean send / recv after everything is destroyed on backup location.

Just to mention, unrelated, i think that when you are using send/receive and mbuffer you are sending unencrypted filesystems over network.
Since you are using encryption on endpoints, i guess security is important so you trust that network Smilie

Regards
Peasant.
# 6  
Old 05-23-2018
@Peasant,

Thanks for your reply. Starting with the side comment, yes, I was aware that zfs send/receive on encrypted filesystem was, in fact, unencrypted and using mbuffer for such a thing needed to be on a "safe network". But it is a good reminder.

To answer the first question, I must say that when I had a primary and backup server, eveything worked fine initially. After I messed up the backup server and re-created the whole zpool, all the unencrypted data was mounted and worked perfectly. The problem was with the encrypted zfs filesystems. The encrypted ones would not mount anymore.

The reason is well explained in this mailing list, mostly the last answer from Darren J Moffat : https ://thr3ads.net/zfs-discuss/2012/02/1839530-Cannot-mount-encrypted-filesystems

Here is what is explained
Quote:
That should have failed because the keysource property is inherited from
slice_2/base. So you have found a bug and I can reproduce it.

The reason that should have failed is the source of where the keysource
comes from is used to determine which dataset to look at for the hidden
salt property. We know what that salt property should actually be in
your case because it is set on slice_2/base.

Unfortunately ''zfs set salt'' won''t work because salt
is read-only from
userland (so it doesn''t accidentally get overridden and cause the very
same symptoms you have!).

In theory you would assume that you could go back to having the
keysource inherited by running:
''zfs inherit keysource slice_2/base/bitsavers''

However that won''t work because of a protection we have in place to
again avoid yet another route into these same symptoms. It will fail
with an error message something like this:

cannot inherit keysource for ''slice_2/base/bitsavers'': use
''zfs key -c
-o keysource=...''

Using a hacked up libzfs that removes the check that ''zfs
inherit'' does
so I can get out of the situation and make the datasets accessible
again. So this is fixable so don''t abandon hope yet.

--
Darren J Moffat

In my case, it is similar, but basically I had something like this on my primary server
/tank/
/tank/shared/unencrypted/
/tank/shared/encrypted/
Also, on the primary server I had sourcekey which was a file like this:
/key/passfile
(Not very secure, I know).

Here is the fun part,
Instead of creating the same "tree" of zpool and zfs filesystem on the backup system, I just created the zpool. I expected everything would be created automatically when I would "zfs recv" the whole zpool on the backup server.

Also, to make this more fun, I pre-created the the folder and file :
/key/passfile <- a copy from the initial primary server
The fun actually lies in a bug that this creates. Normally, if you create the file system manually, you get prompted for a password, you type it, and the systems create a key based on your password and a SALT. The important bug here is doing it the way I did, the salt is automatically a bunch of zeros instead of a random salt. There are protection in the system to not accept salts that are all zeros. Hence, eveything is encrypted with a known key and salt, but unmountable because of this bug.

** This was my understanding of the bug, based on the mailing list **

So things worked for years, but I had forgotten that my first backup server was done manually and correctly. After I re-did it a second time, I did not take the time to double-check... (I know, my bad).. The worst part is the primary server got destroyed and I erased all the hard-drives. So I now only have the backup server... which means I have lost my data.

To make matter worse, I had "send files" on another external hard-drive, but my "photos" files got corrupted and I cannot restore this data. After a long while, I get an error and I cannot restore, even partially, my files...

I really hope I can find a solution or get some help with this issue.

Thank you for reading this very long post.

Best Regards,
# 7  
Old 06-06-2018
@rnd,

We had to bite the bullet and destroy the encrypted zpool due to time restrains.
Hope yours works out better and you get a workable solution.

All the best,
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Send/receive file through serial using minicom

i have connected with my board through serial interface using minicom and i am running a bash script, which should test ethernet (ping test), USB read/write, RS232 .. I have managed to test ethernet and USB read/write. I test ethernet with ping. I test USB read/write, using dd and verifying... (10 Replies)
Discussion started by: linuxmember
10 Replies

2. Solaris

ZFS receive error

trying to clone a zfs file system on the same system using zfs send -r root/branch@snapshot |zfs receive root/newbranch and get the following error :parent does not exist. How do I fix this? (1 Reply)
Discussion started by: os2mac
1 Replies

3. Solaris

zfs send receive performance issues

I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands zfs list zfs snapshot pool/u01@new zfs send pool/u01@new | zfs -F receive newpool/u01 Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies

4. Shell Programming and Scripting

Send Receive Mails

Hi All, I am writing one script to automate one long process. In this process we need to upload some input files and download some output files. So , I want to automate this upload and download by using mail functionality. I want to trigger this script when I am sending mail to server. I know... (0 Replies)
Discussion started by: NirajThakar
0 Replies

5. Shell Programming and Scripting

How to monitor send/receive bytes

Hello, I need to create a script to monitor sent/received packets for a period of time (the period of time will be a users input) and write the result to some txt file. Is there any command (don`t want to use any 3rd party sw) what I can use? I`m using Solaris 10. Thank you (14 Replies)
Discussion started by: msojka77
14 Replies

6. Programming

how can I send and receive data in client server socket programing

char name; printf ("Welcome to the server \n"); printf ("Enter user name: \n"); scanf ("%c", &name); how can client send name to server:what should be the code? int send ( int sid , const char ∗buffer Ptr , int len , int f l a g ) how can client receive ack from... (1 Reply)
Discussion started by: saiful_911
1 Replies

7. Shell Programming and Scripting

Writing a program to receive the GPS data and send to other server

Hi, I would like to write a program to receive the GPS data and then send the data via network to other program. All of the program is not write yet(include host and sender) All of the server OS is unix or linux Could you mind to give me some idea to do this? Thanks so much! Ken ... (2 Replies)
Discussion started by: kenlok
2 Replies

8. Programming

Send/Receive buffer size??

Dear friends, How do I find the TCP send and receive buffer size? (1 Reply)
Discussion started by: nagalenoj
1 Replies

9. UNIX for Dummies Questions & Answers

Can send but not receive email

Hi, One of the users in our company can send but not receive email. We are using SENDMAIL in conjunction with procmail. The funny thing is that all his sent email is in his /var/spool/mail but the email client does not pick anything up! He is using IMAP. Anyone see have any ideas? (2 Replies)
Discussion started by: mojoman
2 Replies

10. UNIX for Dummies Questions & Answers

I can send but cannot receive mail with unix?

Hi all, First post! I have just discovered that I can use unix to send mail to a mail address. I normally use entourage for my mail. This unix mail is very intriguing to me, but something is not working... I tried the search, but could not find the answer... This works: (in terminal)... (1 Reply)
Discussion started by: bjorn
1 Replies
Login or Register to Ask a Question