CI updating azure image gallery

Automation/IaC
, ,
1

Hello everyone,
I made a CI/CD pipeline wich updates custom Templates using packer on azure. Once a new imaghe is created and pushed to image gallery,
I then wrote a python script which will update nodes and layouts acordingly.
I imagined that images are synced during the cloud synchronization, but, after testing, i’m not sure anymore. The pipeline i’ve just launched never found the image , despite i launched a cloud refresh (daily). After cheking, the image didn’t show up. It appears half an hour after.
How should i achieve my goal ?

regards,

Matthieu

2

I think you would just need to do a refresh (short)

3

Hello Tyler,

Thank you for your response :slightly_smiling_face:

I assumed that short refresh was included in daily refresh :wink:

I will update my scripts and pipelines accordingly. Thank you

4

No worries, Daily sync is for things that don’t need to be updated as often. This includes things like Pricing data. Short sync should give you anything that is important to have relatively quickly like images, VMs, networks, SGs, etc.

5

I think my job didn’t wait enough for the cloud to be properly refreshed.
I actually call the refresh endpoint, then watch specific variables returned by the “zones” endpoint :
“statusDate”:“2024-08-28T15:23:48Z”,
“lastSync”:“2024-08-28T15:23:33Z”,
“lastUpdated”:“2024-08-28T15:23:48Z”,

As for now, i am waiting for the “lastupdated” field to be “later” than the datetime i launched the refresh. Is that correct ? What is the exact meaning of these fields ?

def refresh(APPLIANCE_FQDN,TOKEN,cloudId):
  logger.info("_______________refresh__________________________")
  url = f"https://{APPLIANCE_FQDN}/api/zones/{cloudId}/refresh"
  logger.info(f"with url {url}")
  payload = { "mode": "short" }
  headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": f"Bearer {TOKEN}"  }

  response = requests.post(url, json=payload, headers=headers)
  update_dt = datetime.now(timezone.utc)

  # logger.debug(response.text)
  url_info =  f"https://{APPLIANCE_FQDN}/api/zones/{cloudId}"
  update_info = requests.get(url_info, headers=headers)
  logger.debug(update_info.json())
  logger.debug(f"update request dt : {update_dt}")
  #Tant que l'heure à laquelle on a fait la demande de mise à jour est strictement supérieure à l'heure à laquelle les données ont été synchronisées
  logger.debug(f"Cloud {cloudId} has been updated {update_info.json()['zone']['lastUpdated']} and synced {update_info.json()['zone']['lastSync']} (duration {update_info.json()['zone']['lastSyncDuration']})")
  while update_dt > datetime.fromisoformat(update_info.json()['zone']['lastUpdated']):
    
    logger.info(f"Wait for cloud {cloudId} to be updated ...")
    time.sleep(5)
    update_info = requests.get(url_info, headers=headers)
    logger.debug(f"Cloud {cloudId} has been updated {update_info.json()['zone']['lastUpdated']} and synced {update_info.json()['zone']['lastSync']} (duration {update_info.json()['zone']['lastSyncDuration']})")

regards,

Matthieu

7

I would imagine what you are doing would work though I haven’t tested.

What I have done in the past is, after calling to sync the cloud, I call the zones endpoint to check the status.

If it’s set to ‘syncing’ then I sleep for a set amount of time and then check again. If it’s back to ‘ok’ I know it’s finished.

8

Hello Tyler,
i rewrote my function as i found your approach more elegant. Unfortunatly, it seems not to work properly. For instance, the last run of this function gave :

024-08-29 11:57:27,186 :: INFO :: root :: _______________refresh__________________________
2024-08-29 11:57:27,186 :: INFO :: root :: with url https://morpheus-dev.devops.groupe-bel.net/api/zones/7/refresh
2024-08-29 11:57:27,188 :: DEBUG :: urllib3.connectionpool :: Starting new HTTPS connection (1): morpheus-dev.devops.groupe-bel.net:443
2024-08-29 11:57:27,548 :: DEBUG :: urllib3.connectionpool :: https://morpheus-dev.devops.groupe-bel.net:443 "POST /api/zones/7/refresh HTTP/1.1" 200 None
2024-08-29 11:57:27,551 :: DEBUG :: urllib3.connectionpool :: Starting new HTTPS connection (1): morpheus-dev.devops.groupe-bel.net:443
2024-08-29 11:57:27,682 :: DEBUG :: urllib3.connectionpool :: https://morpheus-dev.devops.groupe-bel.net:443 "GET /api/zones/7 HTTP/1.1" 200 None
2024-08-29 11:57:27,684 :: DEBUG :: root :: {'zone': {'id': 7, 'uuid': '6b3bb780-63e5-479f-afdf-48c8379d907d', 'externalId': None, 'name': 'BEL-SUB-SHAREDSERVICES', 'code': 'azrpsharedservices', 'labels': [], 'location': 'AZURE EA', 'owner': {'id': 1, 'name': 'BEL'}, 'accountId': 1, 'account': {'id': 1, 'name': 'BEL'}, 'visibility': 'public', 'enabled': True, 'status': 'ok', 'statusMessage': None, 'statusDate': '2024-08-29T11:56:43Z', 'lastSync': '2024-08-29T11:56:28Z', 'nextRunDate': '2024-08-29T12:22:27Z', 'lastSyncDuration': 15417, 'costStatus': 'ok', 'costStatusMessage': None, 'costStatusDate': None, 'costLastSyncDuration': None, 'costLastSync': None, 'zoneType': {'id': 9, 'code': 'azure', 'name': 'Azure (Public)'}, 'zoneTypeId': 9, 'guidanceMode': 'off', 'agentMode': 'cloudInit', 'userDataLinux': None, 'userDataWindows': None, 'consoleKeymap': None, 'costingMode': 'off', 'serviceVersion': None, 'securityMode': 'off', 'inventoryLevel': 'off', 'timezone': None, 'apiProxy': None, 'provisioningProxy': None, 'networkDomain': {'id': 1, 'name': 'localdomain'}, 'domainName': 'localdomain', 'regionCode': 'francecentral', 'autoRecoverPowerState': True, 'scalePriority': 1, 'config': {'resourceGroup': '', 'configCmdbDiscovery': False, 'datacenterName': '', 'rpcMode': 'guestexec', 'cloudType': 'global', 'tenantId': '[MASKED]', 'subscriberId': '6b0ab7cb-4ff0-434c-8988-6f5b29c0f92c', 'importExisting': '', 'applianceUrl': '', 'clientId': '', 'clientSecret': None, 'accountType': '', 'networkServer.id': 'unmanaged', 'networkServer': {'id': 'unmanaged'}, 'securityServer': 'off', 'backupMode': 'internal', 'replicationMode': '-1', 'diskEncryption': '', 'encryptionSet': '', 'cspTenantId': '', 'cspClientId': '', 'cspClientSecret': None, 'azureCostingMode': 'standard', 'configManagementId': '3', 'configCmdbId': '', 'clientSecretHash': None, 'cspClientSecretHash': None}, 'credential': {'id': 12, 'name': 'AZURE_CLIENT_ID_EA_PROD', 'type': 'client-id-secret'}, 'imagePath': None, 'darkImagePath': None, 'dateCreated': '2023-12-22T13:37:53Z', 'lastUpdated': '2024-08-29T11:57:27Z', 'groups': [{'id': 1, 'name': 'azure', 'accountId': 1}], 'securityServer': None, 'networkServer': {'id': 4, 'name': 'BEL-SUB-SHAREDSERVICES', 'type': 'azure'}, 'stats': {'serverCounts': {'all': 0, 'host': 0, 'hypervisor': 0, 'containerHost': 0, 'vm': 0, 'baremetal': 0, 'unmanaged': 0}}, 'serverCount': 0}}
2024-08-29 11:57:27,684 :: DEBUG :: root :: Cloud 7 has been updated 2024-08-29T11:57:27Z and synced 2024-08-29T11:56:28Z (duration 15417)
2024-08-29 11:57:27,684 :: DEBUG :: root :: status is ok
2024-08-29 11:57:27,684 :: INFO :: root :: _____________getVirtualImageId_________________
2024-08-29 11:57:27,684 :: DEBUG :: root :: with url https://morpheus-dev.devops.groupe-bel.net/api/virtual-images?max=250&offset=0&filterType=Synced&imageType=vhd
2024-08-29 11:57:27,686 :: DEBUG :: urllib3.connectionpool :: Starting new HTTPS connection (1): morpheus-dev.devops.groupe-bel.net:443
2024-08-29 11:57:27,888 :: DEBUG :: urllib3.connectionpool :: https://morpheus-dev.devops.groupe-bel.net:443 "GET /api/virtual-images?max=250&offset=0&filterType=Synced&imageType=vhd HTTP/1.1" 200 None
[a huge dump of the result, without the expecting image]
2024-08-29 11:57:29,308 :: CRITICAL :: root :: template windows-2019 (24.8.29) not found.

workaround is to add a timer and/or a retry on gitlab, so it seems that the “status” message is ok before the cloud (and especialy images) is totaly synced.
Or my test is not good :

def refresh(APPLIANCE_FQDN,TOKEN,cloudId):
  logger.info("_______________refresh__________________________")
  url = f"https://{APPLIANCE_FQDN}/api/zones/{cloudId}/refresh"
  logger.info(f"with url {url}")
  payload = { "mode": "short" }
  headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": f"Bearer {TOKEN}"  }

  response = requests.post(url, json=payload, headers=headers)

  # logger.debug(response.text)
  url_info =  f"https://{APPLIANCE_FQDN}/api/zones/{cloudId}"
  update_info = requests.get(url_info, headers=headers)
  logger.debug(update_info.json())

  logger.debug(f"Cloud {cloudId} has been updated {update_info.json()['zone']['lastUpdated']} and synced {update_info.json()['zone']['lastSync']} (duration {update_info.json()['zone']['lastSyncDuration']})")
  logger.debug(f"status is {update_info.json()['zone']['status']}")
  while (update_info.json()['zone']['status'] != "ok"): 
    logger.info(f"Wait for cloud {cloudId} to be updated ...")
    time.sleep(5)
    update_info = requests.get(url_info, headers=headers)
    logger.debug(f"Cloud {cloudId} has been updated {update_info.json()['zone']['lastUpdated']} and synced {update_info.json()['zone']['lastSync']} (duration {update_info.json()['zone']['lastSyncDuration']})")
    logger.debug(f"status is {update_info.json()['zone']['status']}")

thank you so much for your help :wink: