February 24, 2019

Cohesity Unofficial PowerShell Function Library

In my previous post: Accessing the Cohesity REST API Using PowerShell, I showed you the hard way to access the API, by using the Invoke-RestMethod PowerShell cmdlet after manually manipulating the header, body and URL to build a REST API call. If you like, you can certainly reuse such code to build scripts, although even the most basic script will end up looking like pretty complicated code. In my opinion, it's easier to wrap such code into functions that can be called with minimal arguments, making your scripts faster to write and easier to read.

So, I built a group of functions and stored them in a script called cohesity-api.ps1. You can download a copy of the library by running the following command in PowerShell:


Invoke-WebRequest -Uri https://raw.githubusercontent.com/bseltz-cohesity/scripts/master/powershell/cohesity-api/cohesity-api.ps1 -OutFile cohesity-api.ps1


To use the functions you can simply source, or dot, the script.


. ./cohesity-api.ps1


This makes the functions available in the current PowerShell session (or in the current script). Let's review some of the basic functions it provides...

Authentication


The first function authenticates you to a Cohesity cluster.


apiauth -vip mycluster -username admin -domain local
Connected!


Notice that I haven't provided a password nor have I been prompted for one. this is because apiauth stores your password (encrypted) for future use, so that scripts can run unattended. The first time you attempt to connect to a cluster, you will be prompted for your password and it will be stored.

If your Cohesity (of Active Directory) password changes, the stored password will need to be updated. This can be done using the -updatepassword switch:


apiauth -vip mycluster -username admin -domain local -updatepassword
Enter password for admin at mycluster: *****
Connected!


Also note that -domain defaults to 'local', and positional arguments work fine, so to connect as local admin, you can simply type:


apiauth mycluster admin
Connected!



Making API Calls


Once you've been authenticated, you can make API calls. You can simply provide the HTTP verb (get, post, put, delete) and the tail of the URL of the API call. for example, to get a list of protection jobs, you can type:


$jobs = api get protectionJobs

$jobs | Format-Table

   id name                 environment    policyId                            viewBoxId parentSourceId sourceIds
   -- ----                 -----------    --------                            --------- -------------- ---------            
    7 VM Backup            kVMware        770535285385794:1544976774290:5             5              1 {29, 30}             
   12 Infrastructure       kVMware        770535285385794:1544976774290:5             5              1 {121, 36, 39, 33...} 
   35 Oracle               kOracle        770535285385794:1544976774290:377           5             64 {61}                 
  841 Generic NAS          kGenericNas    770535285385794:1544976774290:25            5             85 {86}                 
 6222 File-Based Backup    kPhysicalFiles 770535285385794:1544976774290:5             5             60 {98}                 
 8028 SQL Backup           kSQL           770535285385794:1544976774290:377           5             46 {31}                 
12886 Scripts              kView          770535285385794:1544976774290:25            5            102 {116}                
12889 _DELETED_Utils       kView          770535285385794:1544976774290:25            5            102 {117}                
18471 _DELETED_NAS2 Backup kGenericNas    770535285385794:1544976774290:25            5             85 {639, 641, 643, 64...
27443 GarrisonToVE1        kVMware        770535285385794:1544976774290:27442         5              1 {1709, 1708}         
29663 SQL Physical         kSQL           770535285385794:1544976774290:25            5             46 {1696}    
           

The URL of this API call is actually https://mycluster/irisservices/api/v1/public/protectionJobs. I provided protectionJobs to the api function and it tacked on https://mycluster/irisservices/api/v1/public/ in front of it.

Public vs. Internal API


Cohesity API calls are categorized as either public or internal. All are accessible to the user, but public calls are designed to be easier to use and perhaps less subject to change from version to version. However, there are operations that you just can't do without resorting to some internal APIs. The Internal calls are a bit more complex, but that shouldn't stop us from using them when we have to.

To make an internal API call using our api function, simply prepend a / infront of the URL tail, like so:


api get /backupjobssummary

backupJobSummary
----------------
@{jobDescription=; numSuccessfulJobRuns=117; avgRunTimeUsecs=55172566; minRunTimeUsecs=26007890; maxRunTimeUsecs=65136469...
@{jobDescription=; numSuccessfulJobRuns=73; avgRunTimeUsecs=178826855; minRunTimeUsecs=91026530; maxRunTimeUsecs=17669251...
@{jobDescription=; numSuccessfulJobRuns=349; numFailedJobRuns=2; numCancelledJobRuns=1; avgRunTimeUsecs=35299180; minRunT...
@{jobDescription=; numSuccessfulJobRuns=78; numFailedJobRuns=3; numCancelledJobRuns=1; avgRunTimeUsecs=4321777; minRunTim...
@{jobDescription=; numSuccessfulJobRuns=51; numFailedJobRuns=1; avgRunTimeUsecs=7610093; minRunTimeUsecs=5000368; maxRunT...
@{jobDescription=; numSuccessfulJobRuns=216; avgRunTimeUsecs=23021394; minRunTimeUsecs=7001213; maxRunTimeUsecs=506184963...
@{jobDescription=; numSuccessfulJobRuns=37; avgRunTimeUsecs=0; minRunTimeUsecs=0; numObjectsBackedUp=37; totalBytesReadFr...
@{jobDescription=; numSuccessfulJobRuns=6; avgRunTimeUsecs=0; minRunTimeUsecs=0; numObjectsBackedUp=6; totalBytesReadFrom...
@{jobDescription=}
@{jobDescription=; numSuccessfulJobRuns=9; avgRunTimeUsecs=37345742; minRunTimeUsecs=27006816; maxRunTimeUsecs=86040358; ...
@{jobDescription=; numSuccessfulJobRuns=3; avgRunTimeUsecs=31007165; minRunTimeUsecs=29007773; maxRunTimeUsecs=32007717; ...


Drilling Down on the Data


Let's have a look at some of the data we got back. Let's return to our public protectionJobs call.


$jobs = api get protectionJobs


Having a look at $jobs shows us the list of jobs we got back.


$jobs | ft

   id name                 environment    policyId                            viewBoxId parentSourceId sourceIds
   -- ----                 -----------    --------                            --------- -------------- ---------            
    7 VM Backup            kVMware        770535285385794:1544976774290:5             5              1 {29, 30}             
   12 Infrastructure       kVMware        770535285385794:1544976774290:5             5              1 {121, 36, 39, 33...} 
   35 Oracle               kOracle        770535285385794:1544976774290:377           5             64 {61}                 
  841 Generic NAS          kGenericNas    770535285385794:1544976774290:25            5             85 {86}                 
 6222 File-Based Backup    kPhysicalFiles 770535285385794:1544976774290:5             5             60 {98}                 
 8028 SQL Backup           kSQL           770535285385794:1544976774290:377           5             46 {31}                 
12886 Scripts              kView          770535285385794:1544976774290:25            5            102 {116}                
12889 _DELETED_Utils       kView          770535285385794:1544976774290:25            5            102 {117}                
18471 _DELETED_NAS2 Backup kGenericNas    770535285385794:1544976774290:25            5             85 {639, 641, 643, 64...
27443 GarrisonToVE1        kVMware        770535285385794:1544976774290:27442         5              1 {1709, 1708}         
29663 SQL Physical         kSQL           770535285385794:1544976774290:25            5             46 {1696}          
     

We can have a closer look at the first one by providing a list index:


$jobs[0]  

id                               : 7
name                             : VM Backup
environment                      : kVMware
policyId                         : 770535285385794:1544976774290:5
viewBoxId                        : 5
parentSourceId                   : 1
sourceIds                        : {29, 30}
excludeSourceIds                 : {19, 87, 24, 26...}
startTime                        : @{hour=23; minute=30}
timezone                         : America/New_York
incrementalProtectionSlaTimeMins : 60
fullProtectionSlaTimeMins        : 120
priority                         : kMedium
alertingPolicy                   : {kFailure}
indexingPolicy                   : @{disableIndexing=False; allowPrefixes=System.Object[]; denyPrefixes=System.Object[]}
qosType                          : kBackupHDD
environmentParameters            : @{vmwareParameters=}
uid                              : @{id=7; clusterId=770535285385794; clusterIncarnationId=1544976774290}
policyAppliedTimeMsecs           : 1548971018230
modificationTimeUsecs            : 1548971018230150
modifiedByUser                   : admin
creationTimeUsecs                : 1544977491464965


and drill down into nested attributes:


$jobs[0].startTime

hour minute
---- ------
  23     30


We can get properties from each job in the list:


$jobs | Select-Object -Property name, startTime 

name                 startTime
----                 ---------
VM Backup            @{hour=23; minute=30}
Infrastructure       @{hour=23; minute=40}
Oracle               @{hour=0; minute=0}
Generic NAS          @{hour=0; minute=10}
File-Based Backup    @{hour=0; minute=40}
SQL Backup           @{hour=0; minute=20}
Scripts              @{hour=0; minute=30}
_DELETED_Utils       @{hour=23; minute=50}
_DELETED_NAS2 Backup @{hour=21; minute=0}
GarrisonToVE1        @{hour=23; minute=10}
SQL Physical         @{hour=18; minute=0}


Or, we can enumerate over the list:


foreach ($job in $jobs){
    "$($job.name)`t($($job.environment))"
}

VM Backup (kVMware)
Infrastructure (kVMware)
Oracle Adapter (kOracle)
File-Based Backup (kPhysicalFiles)
SQL Backup (kSQL)
Scripts (kView)
GarrisonToVE1 (kVMware)
SQL Physical (kSQL)
Generic NAS (kGenericNas)



Query Strings


The last topic I want to cover in this article, before we move on to other operations like posts and puts, is query strings in get URLs. You can pass along parameters with your get calls like this:


api get protectionJobs?environments=kVMware


and we can use the ampersand to include multiple parameters:


api get protectionJobs?environments=kVMware&isActive=true 

isActive=true : The term 'isActive=true' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:45
+ api get protectionJobs?environments=kVMware&isActive=true
+                                             ~~~~~~~~~~~~~
+ CategoryInfo          : ObjectNotFound: (isActive=true:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException


Whoops! PowerShell interpreted the & as the call operator. We can escape the & with a back tick like this:


api get protectionJobs?environments=kVMware`&isActive=true


and that works fine, but when there are several & and other troublesome characters in the URL, it's easier to simply quote the entire URL:


api get 'protectionJobs?environments=kVMware&isActive=true'


We can even use spaces:



api get 'protectionJobs?names=VM Backup'



or using double quotes, we can use variable expansion in our URL:


api get "protectionJobs?environments=$environment&isActive=true"



What's Next


You may have noticed that everything we've done so far uses the get verb. We've yet to cover posts, puts and deletes, but we'll save those for the next few articles.



February 17, 2019

Accessing the Cohesity REST API Using PowerShell

I'm a long time scripter, so when I started working for Cohesity, one of the first things I did was start learning how to manage Cohesity through its REST API. Cohesity is a hyperconverged scale-out storage platform that serves as an integrated backup appliance and scale-out NAS platform designed to consolidate secondary storage workloads (backup, file, object, test/dev, analytics) onto a single, massively scalable data fabric that spans data centers and clouds.

Cohesity takes an API-first approach to software design, meaning that all functions exposed in the web and cli-based user interfaces are serviced by REST API calls in the background. That means that anything that can be done in the UI can be done from any programming language that can send HTTP messages.



Why Use The Cohesity REST API?


Over the past year or so, I've been writing scripts for customers to help automate Cohesity workflows for various reasons. For example, during initial setup, I've been asked to provide a way to register protection sources en masse, such as hundreds of NAS shares and physical servers. That's a great use case for a script. Other common use cases include:
  • Infrastructure Orchestration: Customers have automated self-service workflows for provisioning new VMs and need a way to automatically add the new VM to a protection job and role-based access lists
  • DevOps Orchestration: Customers want to automate the provisioning of database clones for test/dev environments
  • Web Portal Integration: Customers want to execute Cohesity workflows from other service catalog portals like ServiceNow or vRealize Automation

Whatever the use case, the good news is that everything that can be done in the Cohesity UI can be done via the REST API.

Why PowerShell?


PowerShell is the ubiquitous language in the Windows scripting world these days. So many of my example scripts are written in PowerShell. I also get a lot of call for scripts that run on linux platforms, so I also write examples in Python. PowerShell is available for linux, but I haven't met many linux admins who are keen on installing PowerShell on their systems. Python on the other hand is often already installed and ready to run.

PowerShell is an awesome language both for scripting but also interactive sessions which are great for demos as well as for interactive inspection of the objects returned by the REST API. The density and speed of code generation in PowerShell is I think unmatched by any other scripting language that I've come across.

Connecting to Cohesity (the hard way)


If the following looks a bit complicated, don't worry. I'll show you the raw steps to access the API just to form a basic understanding of what's going on. Later, we can wrap up these steps into easy function calls. Read on.

The first step in accessing the API is to authenticate with a Cohesity cluster and get an access token for use with subsequent API calls. As with any REST API call, we will need a header, body, and URL to construct our API call. In PowerShell, these will look like this:


$HEADER = @{'accept' = 'application/json'; 
            'content-type' = 'application/json'}

$BODY = ConvertTo-Json @{'domain' = 'local'; 
                         'username' = 'admin'; 
                         'password' = 'mypassword'}

$URL = 'https://mycluster/irisservices/api/v1/public/accessTokens'


You can see that the header specifies that we'll be sending and receiving JSON formatted data, and the body contains our credentials. We can then make our REST API call to get our access token:


$auth = Invoke-RestMethod -Method Post -Uri $URL -Header $HEADER -Body $BODY -SkipCertificateCheck

$auth | fl

accessToken : eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJkb21haW4iOiJMT0NBTCIsImV4cGlyYXRpb24tdGltZSI6IjE1NTA1MjkzNjYiLCJzaWRzL
              Whhc2giOiJ1VWdYNWJGa0NlY1F1UE15aVF0MS1mb0IwQndRcWJFRktQYzk3RHFPOGxBIiwidGVuYW50LWlkIjoiIiwidXNlci1zaWQiOiJTLTE
              tMTAwLTIxLTMzMDkwMTE0LTU3NDA5MzUtMSIsInVzZXJuYW1lIjoiYWRtaW4ifQ.WcdmFkyqYYWm2EhFe4LBT95ycMjRs2wyKeyIGrxuAnKN9C
              NbZAR0u3F3gsyjMo1u_iPxEsoR_Zl_JOLALXDrp400kek-rSCQcpnsmgutLQ8B5nVKKtAceDd1gFcSIeuE_X1ENppfYfw2_GlTW3vcBEe8DWjq
              ua1WY6JAZjJUJ-A
tokenType   : Bearer
privileges  : {ANALYTICS_MODIFY, ANALYTICS_VIEW, CLONE_MODIFY, CLONE_VIEW...}


As you can see, we received back an accessToken, tokenType, and a list of role-based access privileges granted to us by our login. We can then insert the tokenType and accessToken into our header, so that we can make authenticated API calls.


$HEADER = @{'accept' = 'application/json'; 
            'content-type' = 'application/json'; 
            'authorization' = $auth.tokenType + ' ' + $auth.accessToken}


Finally, we can set a new URL for our next API call, and get some real data from the Cohesity cluster. For example, we can get a list of protectionJobs:



$URL = 'https://mycluster/irisservices/api/v1/public/protectionJobs'

$jobs = Invoke-RestMethod -Method Get -Uri $URL -Header $HEADER -SkipCertificateCheck

$jobs | Format-Table

   id name                 environment    policyId                            viewBoxId parentSourceId sourceIds
   -- ----                 -----------    --------                            --------- -------------- ---------            
    7 VM Backup            kVMware        770535285385794:1544976774290:5             5              1 {29, 30}             
   12 Infrastructure       kVMware        770535285385794:1544976774290:5             5              1 {121, 36, 39, 33...} 
   35 Oracle               kOracle        770535285385794:1544976774290:377           5             64 {61}                 
  841 Generic NAS          kGenericNas    770535285385794:1544976774290:25            5             85 {86}                 
 6222 File-Based Backup    kPhysicalFiles 770535285385794:1544976774290:5             5             60 {98}                 
 8028 SQL Backup           kSQL           770535285385794:1544976774290:377           5             46 {31}                 
12886 Scripts              kView          770535285385794:1544976774290:25            5            102 {116}      
          

Let's take a closer look at the first job in the list.



$jobs[0] | fl

id                               : 7
name                             : VM Backup
environment                      : kVMware
policyId                         : 770535285385794:1544976774290:5
viewBoxId                        : 5
parentSourceId                   : 1
sourceIds                        : {29, 30}
excludeSourceIds                 : {19, 87, 24, 26...}
startTime                        : @{hour=23; minute=30}
timezone                         : America/New_York
incrementalProtectionSlaTimeMins : 60
fullProtectionSlaTimeMins        : 120
priority                         : kMedium
alertingPolicy                   : {kFailure}
indexingPolicy                   : @{disableIndexing=False; allowPrefixes=System.Object[]; denyPrefixes=System.Object[]}
qosType                          : kBackupHDD
environmentParameters            : @{vmwareParameters=}
uid                              : @{id=7; clusterId=770535285385794; clusterIncarnationId=1544976774290}
policyAppliedTimeMsecs           : 1548971018230
modificationTimeUsecs            : 1548971018230150
modifiedByUser                   : admin
creationTimeUsecs                : 1544977491464965


As you can see, we can extract all the details of the job.


DateTime Formats


You can see in the output above, that dates and times returned by the API are expressed in Unix Time (also known as POSIX time or UNIX Epoch time), usually in microseconds (since 00:00:00 Jan 1, 1970). In the example above, we can convert the creationTimeUsecs into a human-readable date in PowerShell like so:


$unixTime = ($jobs[0].creationTimeUsecs)/1000000

[datetime]$origin = '1970-01-01 00:00:00'

$origin.AddSeconds($unixTime).ToLocalTime()

Sunday, December 16, 2018 11:24:51 AM


and of course we can perform the conversion in the other direction:


[datetime] $dateString = 'Sunday, December 16, 2018 11:24:51 AM'

([Math]::Floor([decimal](Get-Date($datestring).ToUniversalTime()-uformat "%s")))*1000000

1544977491000000


OK, this is looking really complicated. I really don't want to be using a command like that in my day-to-day scripting. It's time to provide some functions to hide this kind of complexity.


Using a Function Library


Once I learned to do the basics as I've shown above, I developed a function library to make all these steps much easier. I built a number of functions and placed them in a file called cohesity-api.ps1 (we'll discuss this library in the next article).

To use the functions, we can source the library, then use its functions like so:



. ./cohesity-api.ps1

apiauth mycluster admin
Connected!

$jobs = api get protectionJobs

usecsToDate $jobs[0].creationTimeUsecs

Sunday, December 16, 2018 11:24:51 AM


As you can see, we've eliminated all of the complicated steps of managing headers, URLs, and date conversions by wrapping all that code into functions in the library. Now we can stop wasting time fussing with the mechanics of API calls and spend more of our time doing what we came here to do: learn to exercise the Cohesity API itself.

Where To Go From Here


Now that we've got a simple way to access the Cohesity REST API, what can you do with it? Well there are several documentation resources at your disposal. First, there is the on-board Swagger page that's hosted on every Cohesity cluster. You can access page by clicking the REST API link at the bottom of the Cohesity UI. This link takes you to: https://<mycluster>/docs/restApiDocs/browse/



The swagger interface shows you that each API call has its own URL and which HTTP verb to use. If you click on one of the items, it will expand to show you the various parameters and even has a 'Try it out' button to make API calls right in the interface.

If you need more comprehensive documentation, every cluster hosts an API doc page:

https://<mycluster>/docs/restApiDocs/bootprint/

and you can also find the API documentation on the Cohesity. com web site at:


Examples


I've written quite a few example scripts already. You can find them here:


I'll explain how I learned how to write these scripts in my next few articles. Coming soon! 

The Cohesity Unofficial Function Library


Take a read of my next article: Cohesity Unofficial PowerShell Function Library to learn the easy way to use the Cohesity API...