High level overview

We have in a `ppk` file and not in any of the other formats accepted by azure (`pem`, `pfx` or `byok`). The azure restrictions will apply when uploading the file manually to key-vault. You can choose to do it via azure cli like this:

az keyvault secret set vault-name $(keyVaultName) name $(secretKeyName) --file '$(file);

but this will not do the trick for you. It will upload the file but you will not be able to use it in ADF.

This is a 3 steps process:

1. Convert the ppk to pem format
2. Encode the pem file as base64
3. Upload the file with azure cli in 2 steps

Convert the ppk to pem format

If you are a windows user then get the tool puttygen and load your ppk key file with it. On the selection menu click conversions and convert it to a (private key) openssh key format. Be sure to add the `.pem` extension. If you are a linux user use the puttygen tool as well (after installing putty-tools), but in the console like this

puttygen privatekey.ppk -O private-openssh -o pemkey.pem

Encode the pem file as base64

The resulting pem file needs to be encoded as base64 in order to be used from ADF. This is a pretty standard encoding algorithm and you can find many tools for windows or linux to do it. For Windows there is the base64 encoder from Microsoft and for Linux the `base64` binary is part of the builtin toolbox.

Upload the file with azure cli in 2 steps

Now you have the base64 encoded file and you need to add it to the keyvault. Use azure cli like this:

az keyvault secret set vault-name $(keyVaultName) name $(secretKeyName) --file '$(file);

Some private keys are encrypted with a a passphrase. You can either upload the passphrase yourself as an additional key (which we did in out case), or you can use the pem upload command from azure keyvault like this:

sh az keyvault key import pem-file $(pemFile) vault-name $(keyVaultName) name $(secretKeyName) pem-passphrase $(passPhrase)

Now you can use it in ADF.

1. row_number()
2. monotonically_increasing_id()
3. rdd.zipWithIndex()
4. hash functions

They all have their pros and cons. What you choose depends on one or both points below:

1. whether it really gives you the uniqueness you hope to get
2. whether you like the properties of the ids you end up having

One would think that the first point is actually no point at all, since we kinda have an understanding what uniqueness is and it is easy to test weather we have achieved it or not. But it is slightly more complex than that. It goes back to the hashing collisions and the size of table you are operating on. So, let’s start with that right away.

Hash functions

There are many, many, many hash functions and you could theoretically use any of the if you had the implementation and register it as an udf. We are looking only for functions that will give us and integer as result and for those that are already in the spark library. This changes as new versions of the spark api come out and we are now at version 3.1.2.

Let’s list a few of the hash functions available for free in spark:

• hash
• xxhash64
• crc32
• sha, sha1 and sha2
• md5

Now looking at it, I wish the function xxhash64 had been available in spark 2.X.X since that would have been the weapon of choice on the solution I was working on a few years back. The past is the past and I wanna explain with what restriction you have to fight in choosing the right hash function.

Row Number

To be precise, this is the pyspark.sql.functions.row_number¹ window function which acts on a dataframe. Window functions are functions comming from the sql world which act on a smaller set of rows (a window of rows) within a bigger set of rows being processed. Let me explain.

If you provide null as partition (in sql you can omit it completely), it will act on a single row as the window and provide you with a nice sequence starting from 1.

SELECT row_number() OVER(ORDER BY NULL) as rownum, * from [db.]tablename 

Monothonically increasing id

Rdds ZipWithIndex