Automating upload to Azure Blob Storage
This is my latest effort in an attempt to offer myself slightly more robust backups of my personal website. I’m using PowerShell and Azure PowerShell to automate the process of zipping up a folder (actually my website’s application folder) and its associated MySQL database in to a zip file and finally, as well as storing the zip locally on-disk (yes, I know!…. wait for it!) uploading the file to Azure Blob Storage. Using some new functionality in .NET Framework 4.5 (zip files, yay) and Azure PowerShell to get the job done.
Here’s a quick snippet from the script:
|
1 |
Send-FileToAzureBlob -AzureStorageAccountName $StorageAccount -AzureStorageContainer $StorageContainer -FilePath $BackupDestinationLocation$ZipFileName -BlobName $BlobName |
I also ensure I’m optimising my use of Azure Storage by only retaining the latest 4 files in the target container and deleting anything older than 30 days on the local machine. Run once per week, this will give you one month’s worth of backups and help you sleep a little easier.
Even without the MySQL dump integration, this is a handy script for backing up a folder and all of its child contents then uploading it for safe keeping to Azure Storage….The script is a little verbose and heavily commented for my future self and there is some repetition. I could have handled the storage keys and storage contexts a little better between functions rather than repeating code but, well, I wasn’t about to re-architect it after adding another function with a few lines of repeated code.
Hopefully the script is self-explanatory.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 |
#===========# # Variables # #===========# # The folder (with trailing slash) location to be backed up. $BackupSourceLocation = "C:\Path\To\Folder To Zip\" # The destination for the backup zip file. We'll deal with the name of # the zip file using a date and time format to avoid clashes. $BackupDestinationLocation = "C:\Path\To\Archive Destination\" # MySQL Connection Information $MySQLDB = 'MySQLDatabase' $MySQLUser = 'MySQLUserAccount' $MySQLPassword = 'MySQLPassword' # Delete backups older than...days $ArchiveAge = 30 # Store a date format as a string for use in file & folder names. $FormatDate = (Get-Date -Format yyyyMMdd-HHmm).ToString() # Define the names of the SQL and ZIP files. $ZipFileName = $FormatDate + ".zip" $DBBackupFileName = $FormatDate + ".sql" #======================# # Azure Storage Backup # #======================# # Azure Storage Account $StorageAccount = "myfirststorageaccount" # This must already exist. Don't be lazy now. # Azure Storage Container (if this doesn't exist, it will be created.) $StorageContainer = "mystoragecontainer" #===============# # End Variables # #===============# #===========# # Functions # #===========# Function Compress-Folder { Param ( [Parameter(Mandatory=$true)] [ValidateScript({Test-Path $_ -PathType 'Container'})] [String] $SourceDirectory, [Parameter(Mandatory=$true)] [String] $ZipFilePath, [ValidateSet("NoCompression","Optimal","Fastest")] [String] $CompressionLevel = "Fastest" ) [Reflection.Assembly]::LoadWithPartialName("System.IO.Compression.FileSystem") | Out-Null $CompLevel = [System.IO.Compression.CompressionLevel]::$CompressionLevel [System.IO.Compression.ZipFile]::CreateFromDirectory($SourceDirectory, $ZipFilePath, $CompLevel, $false ) } Function Send-FileToAzureBlob { Param ( [Parameter(Mandatory=$true)] [string] $AzureStorageAccountName, [Parameter(Mandatory=$true)] [string] $AzureStorageContainer, [Parameter(Mandatory=$true)] [string] $FilePath, [Parameter(Mandatory=$true)] [string] $BlobName ) Try { # Grab the storage keys (passwords) for the Azure Storage account. $StorageKeys = Get-AzureStorageKey $AzureStorageAccountName -ErrorAction Stop } Catch { Write-Error "Well, this is embarrassing.`n" ` "The storage account you specified `"$AzureStorageAccountName`" doesn't seem to exist`n" ` "or I wasn't able to get the storage key (password) for it.`n" ` "Without this, I can't upload the file to Azure so I'll quit now.`n" ` "Sorry." Exit 1 } Try { # Get the storage context $StorageContext = New-AzureStorageContext $AzureStorageAccountName -StorageAccountKey $StorageKeys.Primary -ErrorAction Stop } Catch { Write-Error "Well, this is embarrassing.`n" ` "I wasn't able to create a context for the `"$AzureStorageAccountName`" storage account.`n" ` "Without this, I can't upload the file to Azure so I'll quit now.`n" ` "Sorry." Exit 1 } Try { Write-Host "Attempting to create a container called: `"$AzureStorageContainer`"" New-AzureStorageContainer -Context $StorageContext -Container $AzureStorageContainer -ErrorAction Stop -ErrorVariable $e } Catch { Write-Warning "The container `"$AzureStorageContainer`" already exists. Continuing..." } Try { Write-Host "Attempting to upload `"$FilePath`" to Storage Container `"$AzureStorageContainer`" in Azure Storage Account: `"$AzureStorageAccountName`"" Set-AzureStorageBlobContent -File $FilePath -Container $AzureStorageContainer -Context $StorageContext -Blob $BlobName -Force -Verbose -ErrorAction Stop -ErrorVariable $e } Catch { Write-Error "The attempt to upload `"$FilePath`" to `"$AzureStorageContainer`" in Azure Storage Account: `"$AzureStorageAccountName`" failed!" } } Function Optimize-AzureStorageContainer { Param ( [Parameter(Mandatory=$true)] [string] $AzureStorageAccountName, [Parameter(Mandatory=$true)] [string] $AzureStorageContainer, [Parameter(Mandatory=$true, HelpMessage="The number of most recent files to keep.")] [ValidateRange(1,6)] [int] $KeepLatest ) Try { # Grab the storage keys (passwords) for the Azure Storage account. $StorageKeys = Get-AzureStorageKey $AzureStorageAccountName -ErrorAction Stop } Catch { Write-Error "Well, this is embarrassing.`n" ` "The storage account you specified `"$AzureStorageAccountName`" doesn't seem to exist`n" ` "or I wasn't able to get the storage key (password) for it.`n" ` "Without this, I can't upload the file to Azure so I'll quit now.`n" ` "Sorry." Exit 1 } Try { # Get the storage context $StorageContext = New-AzureStorageContext $AzureStorageAccountName -StorageAccountKey $StorageKeys.Primary -ErrorAction Stop } Catch { Write-Error "Well, this is embarrassing.`n" ` "I wasn't able to create a context for the `"$AzureStorageAccountName`" storage account.`n" ` "Without this, I can't upload the file to Azure so I'll quit now.`n" ` "Sorry." Exit 1 } # Get a list of files, newest to oldest order and skip the top ([int]$KeepLatest) files # The idea here is to delete all but the newest 4 files, irrespective of how old they actually are. # You can fettle with this to delete anything older than 30 days as per the last few commands # in this script. $OldAzureFiles = Get-AzureStorageBlob -Context $StorageContext -Container $AzureStorageContainer | Sort-Object { $_.LastModified } -Descending | Select-Object -Skip $KeepLatest # Now we have a list of files, we can remove the older Azure Blobs. Foreach ($OldAzureFile In $OldAzureFiles) { Remove-AzureStorageBlob -Blob $OldAzureFile.Name -Container $AzureStorageContainer -Context $StorageContext -Force } } #===============# # End Functions # #===============# #===================# # Script Processing # #===================# # Dump the MySQL database to disk. $cmd = "& 'C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqldump.exe' -u $MySQLUser -p$MySQLPassword $MySQLDB > " + $BackupDestinationLocation + $DBBackupFileName Invoke-Expression $cmd | Out-Null # Zip up the source folder contents. Compress-Folder -SourceDirectory $BackupSourceLocation -ZipFilePath $BackupDestinationLocation$ZipFileName -CompressionLevel Optimal # Re-open and add the currently uncompressed MySQL database to the zip file. $zip = [System.IO.Compression.ZipFile]::Open($BackupDestinationLocation + $ZipFileName, "Update") [System.IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip,$BackupDestinationLocation + $DBBackupFileName,$DBBackupFileName,"Optimal") $FileCount = $zip.Entries.Count # Make some changes to the zip file by updating it # Cannot use a foreach as I get a collection has been altered error. For ($i=0; $i -le $FileCount-1; $i++) { If (($zip.Entries[$i].FullName.StartsWith("unwanted\folder1")) -or ($zip.Entries[$i].FullName.StartsWith("unwanted\folder2")) -or ($zip.Entries[$i].FullName.StartsWith("unwanted\folder3"))) { "Deleting " + $zip.Entries[$i].FullName $zip.Entries[$i].Delete() # Having removed an entry, the zip file automatically re-orders # so we decrement our counts by 1 to accommodate that. # Makes a mockery of the for loop! $i-- $FileCount-- } } # Dispose the object (saves the zip) $zip.Dispose() # Now the database SQL dump is in the zip file, delete it. Err, the database backup, not the zip file. Remove-Item $BackupDestinationLocation$DBBackupFileName # Now let's upload the complete zip to Azure Blob Storage. # The following assumes that the Add-AzureAccount & Select-AzureSubscription # cmdlets have ALREADY been run and therefore authentication info is saved. # See: https://azure.microsoft.com/en-gb/documentation/articles/powershell-install-configure/ # The name of the blob we want to create in Azure storage. [String] $BlobName = (Get-Item $BackupDestinationLocation$ZipFileName).Name # Upload the file to Azure Blob Storage. Send-FileToAzureBlob -AzureStorageAccountName $StorageAccount -AzureStorageContainer $StorageContainer -FilePath $BackupDestinationLocation$ZipFileName -BlobName $BlobName # Clean up the storage container to retain only the latest <n> files. # If a backup runs once per week, this means there will be 4 weeks # of zip files in the container at any one time. # Valid ranges are 1-6 as per the function above. Optimize-AzureStorageContainer -AzureStorageAccountName $StorageAccount -AzureStorageContainer $StorageContainer -KeepLatest 4 # Clean up old local files. Anything older than $ArchivAge is deleted from the $BackupDestination folder. $Files = Get-ChildItem $BackupDestinationLocation | Where {$_.LastWriteTime -le (Get-Date).AddDays(-$ArchiveAge)} If ($Files) { $Files | % { Remove-Item $_.FullName -Force } } |
– Lewis
Nice Post. Can we unzip the file in blob container using powershell. Please any idea for this
You won’t be able to unzip inside the container but i see no reason why you can’t automate the process to download the zip, decompress it and the iteratively upload each file, creating containers for folders where necessary.