Azure-in-bullet-points

Designing a Data Platform Solution

• Azure Storage
  • All services can be used using a REST API
  • Uses an auto-partitioning system that automatically load-balances your data based on traffic.
  • Billing is calculated by usage, not capacity.
  • Can have Azure Search to index & have fast results.
  • Storage functions:
    • Azure Blobs
      • VHDs and large blocks of data (e.g. images or documents)
      • Soft delete allows you to have a retention policy for files up to X days after deletion up to 365 days.
      • Entities
        • Page blobs
          • Lend themselves to storing random access files up to 8TB in size
          • Ideal for VHD storage for Virtual Machines.
        • Block blobs are for images or other documents and files up to 4TB in size.
        • Append blobs are similar in format to block blobs but allow append operations.
          • Ideal for auditing and logging applications such as log or monitoring data from many sources.
    • Azure Tables
      • Structured data (a NoSQL store)
        • Ideal for: User, device and service metadata, structured data.
        • Features include:
          • Schema-less entities with strong consistency.
          • No limits on number of table rows or table size.
          • Dynamic load balancing of table regions.
          • Best for Key/value lookups on partition key and row key.
          • Entity group transactions for atomic batching.
    • Azure Queues
      • For message passing in applications and for backlog processing
      • Ideal for: Data sharing, Big Data, Backups. Functions
      • Functions:
        • Decouple components and scale them independently.
        • Scheduling of asynchronous tasks.
        • Building processes/work flows.
        • No limits on number of queues or messages.
        • Message visibility timeout to protect from component issues.
        • UpdateMessage to checkpoint progress part way through.
    • Azure Files: Fully managed SMB 3.0 file shares
  • Storage Account
    • Stores & manages storage services.
      • 💡 Managed Disks for Virtual Machine disk storage does not need a storage account.

    • Types

      Type of Account	Services Supported	Types of Blobs supported
      General Purpose Standard	Blob, File, Queue services	Block blobs, Page blobs and Append blobs
      General Purpose Premium	Blob service	Page blobs
      Blob Storage (hot and cool access tiers)	Blob service	Block blobs and Append blobs

    • Storage account security
      • Access is controlled by Storage Account Keys
        • There are primary and secondary to allow for key refreshes whilst maintaining access.
        • If you have the key, you have access to the account and all the data in the account.
      • Container Security
        • Provided for blob storage at the time of creation; the container can be public, private or read-only.
          • Public read access policy
            • Controls what data is available anonymously for your container
            • Setting can be Container, Blob or Off.

            Setting	Individual blobs and their properties can be accessed	Blobs can be enumerated
            Container	✔️	✔️
            Blob	✔️	❌
            Off	❌	❌

        • 💡 For greater security:
          • Use Azure AD RBAC for
            • Management functions on Storage Account
            • Storage account keys
          • Shared Access Signatures and Stored Access Policies
            • A shared access policy adds the ability to revoke, expire or extend access.
        • Shared Access Signature
          • A URI that grants restricted access rights to containers, blobs, queues,and tables.
          • For delegations, not resources.
          • Access is for specified specified period of time, with a specified set of permissions.
          • All information is described in URI.
            • E.g. GET https://[account].blob.core.windows.net/pictures/profile.jpg?sv=2012-02-12&st=2009-02-09&se=2009-02-10&sr=c&sp=r&si=YWJjZGVmZw%3d%3d&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d
              • The signature is using version 2012-02-12 of the storage API; it allows read access is beginning from 02/09/09 to 02/10/09 to the container.
          • You can grant following:
            • Content: Reading and writing page or block blob content, block lists, properties, and metadata
            • Deleting: Deleting, leasing, and creating a snapshot of a blob
            • Listing: Listing the blobs within a container -CRUD:
              • Adding, removing, updating, and deleting queue messages
              • Querying, adding, updating, deleting, and upserting table entities
            • Metadata: Getting queue metadata, including the message count
            • Copying: Copying to a blob from another blob within the same account
        • Shared Access Policies
          • Settings defined in a server-stored policy can be changed and are reflected in the token without requiring a new token to be issued, but settings defined in the token itself cannot be changed without issuing a new token.
          • This approach also makes it possible to revoke a valid SAS token before it has expired.
            • To revoke a stored access policy, you can either delete it or rename it by changing the signed identifier.
          • ❗ A maximum of five access policies may be set on a container, table, or queue at any given time.
    • Storage account replication
      • Locally redundant storage (LRS)
        • Replicated three times across two to three facilities, either within a single region or across two regions
        • Do not protect from the failure of a single facility.
      • Zone-redundant storage (ZRS)
        • ZRS is replicated three times across two to three facilities, either within a single region or across two regions.
      • Geo-redundant storage (GRS)
        • Replicated three times within the primary region, and is also replicated three times in a secondary region hundreds of miles away from the primary region.
        • Replicates asynchronously
          • Means replica is eventually consistent and could possibly have older data if you access the replica before the replication**
      • Read access geo-redundant storage (RA-GRS)

        • Provides read-only access to primary location on top of GRS.

          Replication	LRS	ZRS	GRS	RA-GRS
          Data stored in multiple datacenters	No	Yes	Yes	Yes
          Data read from secondary & primary location	No	No	No	Yes
          No of copies of data stored in separate nodes	3	3	6	6

  • Storage Performance and Pricing
    • Standard
    • Premium
      • SSD disk support for VMs.
      • Using multiple disks gives your applications up to 256 TB of VM storage
      • Up to 80,000 I/O operations per second (IOPS) per VM, and a disk throughput of up to 2,000 megabytes per second (MB/s) per VM.
      • Only LRS redundancy
      • Applications that require consistent high performance and low latency such as SQL Server, Oracle, MongoDB, MySQL, and Redis can run happily in Azure Premium Storage.
      • More expensive: The storage is allocated on creation and is charged for the whole disk size rather than the data used.

    • Features:

      Disk Type	Description	💡 Suggested Use
      Azure Files	SMB 3.0 interface, client libraries and a REST interface access from anywhere to stored files.	Application lift and shift using native file system API, Windows File Shares.
      Azure Blobs	Client libraries, REST interface, for unstructured data stored in Block blobs	Streaming and random access, access application data from anywhere.
      Azure Disks	Client libraries, REST interface for persistent data accessed from a VHD,	Access to data inside a Virtual machine on a VHD, lift and shift file system API apps that write to persistent disks.

  • Blob Storage
    • VM disks:
      • Azure IaaS VMs can attach OS and Data Disks.
      • Disks can be premium storage (SSD based) or Standard storage (HDD).
      • Unmanaged disks:
        • Available in Standard and Premium tiers.
        • Any Azure VM size can have multiple standard disks attached.
        • A storage account can only store one type of disk.
        • Can have disk snapshots
        • ❗ Maximum of 20,000 IOPS per account, managed disk has not IOPS limit.
      • ❗ Each storage account has a limit of 20000 IOPS throughput.
        • 💡 To be able to provide maximum performance with multiple disks => those data disks across many storage accounts.
      • Managed disks
        • Simplifies the creation and management of Azure IaaS VM Disks.
          • Azure manages the storage accounts.
          • Only choose storage type and the disk size.
        • Other benefits:
          • Upgrade a standard disk to a premium disk and downgrade a premium disk to a standard disk.
            • ❗ Requires you to detach the disk from a VM before it’s upgraded or downgraded.
          • The ceiling of 20000 IOPS disappears without needing to manage additional storage accounts.
          • ❗ Allows up to 10000 VM disks per subscription.
            • In VM Scale Sets allows up to 1000 VMs per scale set using a Marketplace image.
          • Better reliability for Availability Sets
            • All the disks in a VM are stored in the same scale unit
            • Each VM’s disks will be stored in separate scale units
          • 99.9% availability
          • Granular access control
          • Managed disk snapshots
            • Stand-alone objects and can be used to create new disks.
            • 💡 You are only billed for the used size of the data on the disk.
              • E.g. if a 4 TB disk that holds 500 GB of data, the snapshot will be billed for 500 GB.
          • Images: Managed Disks support creating a managed custom image
          • Images vs. snapshots
            • An image is a copy of the VM and all disks attached.
            • A snapshot is a single disk copy.
            • A snapshot is not a suitable choice for the scenario with multiple striped data disks. No snapshot co-ordination is available.
          • Managed Disks and Encryption
            • By default Managed disks are encrypted by Azure Storage Service Encryption
              • Provides encryption at rest for disks, snapshots, and images
              • Also available at the VM level
                • In Windows it uses BitLocker Drive Encryption
            • Azure Key Vault integration is included which allows users to bring their own disk encryption keys.

        • Sizing and pricing

          Type	Allowed disks	Pricing
          Premium	P4 (32 GB) -> P50 (4 TB)	Pay for allocated size without transaction charges
          Standard	S4 (32 GB) -> S5 (4 TB)	Billed only for the number of transactions performed

      • Deployment
        • Through Portal, PowerShell or Azure CLI.
        • You can deploy as top-level disk resource that’ll be attached later or can create within VM creation template.
      • Azure Disk Encryption
        • Azure Disk Encryption is a capability that helps you encrypt your Windows and Linux IaaS VM disks.
        • Disk Encryption leverages the industry standard BitLocker feature of Windows and the DM-Crypt feature of Linux to provide volume encryption for the OS and data disks.
        • The solution is integrated with Azure Key Vault to help you control and manage the disk-encryption keys and secrets.
        • The solution also ensures that all data on the VM disks are encrypted at rest in your Azure storage.
  • Azure Files
    • Shared storage for applications using the standard SMB 3.0 protocol.
    • Access data
      • VM’s and cloud services: mounted shares via file I/O API’s.
      • On-premises: in a share via the File storage API.
      • E.g. on linux you run sudo mount
    • Typical uses
      • Migrating on-premises applications that rely on file shares.
      • Transfer file between vm in different subnet without vpn.
      • Storing shared application settings, for example in configuration files.
      • Storing diagnostic data such as logs, metrics, and crash dumps in a shared location.
      • Storing tools and utilities needed for developing or administering Azure virtual machines or cloud services.
    • Components
      • Storage Account
      • Share
        • SMB 3.0 file share in Azure.
        • All directories and files must be created in a parent share.
        • An account can contain an unlimited number of shares.
        • A share can store an unlimited number of files, up to the capacity limits of the storage account.
      • Directory: An optional hierarchy of directories.
      • File: A file in the share. A file may be up to 1 TB in size.
      • URL format: https://[account].file.core.windows.net/<share>/<directory/directories>/
        • E.g. http://[account].file.core.windows.net/logs/CustomLogs/Log1.txt
    • Azure File Sync
      • On-premises -> File share in Azure
        • Turns your file server into a cache of the Azure-based file share.
          • Centralizes your file shares in Azure Files, whilst maintaining the compatibility of an on-premises file server.
        • Any protocol installed on the Windows Server can access the file share, including SMB, NFS, and FTPS.
      • Components
        • Storage Sync Service
          • Azure resource created to host the Azure File Sync service.
          • Required since service can sync between multiple storage accounts, hence an additional resource is required to manage this.
          • A subscription can contain multiple storage sync services.
        • Sync Group
          • Defines and controls the hierarchy and topology of the files to be synced.
          • The sync group will contain Cloud and Server endpoints.
          • Async service can contain multiple sync groups.
        • Registered Server
          • Before adding a server endpoint to a sync group, the server must be registered with a storage sync service.
          • A server can only be registered to a single sync service.
          • Async service can host as many registered servers as you need.
        • Azure File Sync Agent
          • To register a server, you need to install the Azure File Sync Agent.
          • This is a small downloadable MSI package comprising three components:
            • FileSyncSvc.exe: Monitors changes on Server Endpoints, and for initiating sync sessions to Azure.
            • StorageSync.sys: A file system filter, which handles tiering files to Azure Files.
            • PowerShell Management cmdLets: PowerShell cmdlets for the Microsoft.StorageSync Azure resource provider.
        • Server Endpoint
          • Once registered, you can add a server to a Sync group, this then becomes a server endpoint.
          • A server endpoint is synonymous with a folder or a volume on the server that will cache the contents of the Azure File Share.
          • Cloud tiering is configured individually by server endpoint.
        • Cloud Endpoint
          • When added to a sync group, an Azure File Share is a cloud endpoint.
          • One Azure File Share can only be a member of one Cloud Endpoint and thereby can only be a member of one Sync Group.
          • Any files that exist in a cloud endpoint or a server endpoint before they are added to the sync group, automatically become merged with all other files in the sync group.
      • Features
        • Adding files to the File Share
          • Adding and removing files directly within the Azure file share.
          • Syncs every 24 hours as detection job runs every 24 hours.
            • Only from on-prem to azure
        • Cloud tiering
          • Caching mechanism to save space.
          • Flow:
            1. File is tiered
            2. The sync system filter replaces the local file with a pointer to the location if the Azure file share.
            3. When accessed locally the file is downloaded and opened for use, e.g.Hierarchical Storage Management (HSM).
        • Supported versions of Windows Server: Windows Server 2012 R2 and Windows Server 2016
        • Access control lists (ACL)
          • Supported and enforced on files held on Server endpoints.
          • Azure Files do not currently support ACLs.
        • NTFS compression
          • Fully supported, and Sparse files are fully supported but are stored in the cloud as full files, and any cloud changes are synced as full files on server endpoints.
        • Failover Clustering
          • Supported for File Server for General Use but not for Scale-out file server for application data.
          • ❗ Cluster Shared Volumes are not supported.
          • To function correctly, the sync agent must be installed on every node of a cluster.
        • Data Deduplication
          • Fully supported for volumes that do not have cloud tiering enabled.
        • Encryption solutions
          • Azure File Sync, is known to work with BitLocker Drive Encryption and Azure Rights Management Services.
          • NTFS Encrypted File System does not work with Azure File Sync.
• StorSimple
  • A physical device.
  • Creates workflows for migrating data to a cloud storage center or back on premise.
  • Combination of service device management tools.
  • On-premises hybrid storage array
  • Provides primary storage and iSCSI access to data stored on it.
    • iSCSI (Small Computer System Interface)
      • Storage networking standard for linking data storage facilities over TCP/IP
  • Manages communication with cloud storage
    • Helps to ensure the security and confidentiality of data
  • Includes
    • Solid state drives (SSDs)
    • Hard disk drives (HDDs)
    • Support for clustering and automatic failover.
    • Shared processor, shared storage, and two mirrored controllers
  • You can alternatively use StorSimple to create a virtual device that replicates the architecture and capabilities of the actual hybrid storage device.
    • The StorSimple virtual device (also known as the StorSimple Virtual Appliance) runs on a single node in an Azure virtual machine.
  • StorSimple provides a web-based user interface (the StorSimple Manager service), or you can use PowerShell CLI.
  • Security through encryption algorithms to protect data stored in or traveling between the components of StorSimple solution.
  • Features
    • Transparent integration
      • Uses Internet Small Computer System Interface (iSCSI) protocol to invisibly link data storage facilities.
        • Data that’s stored in the cloud, in the data center, or on remote servers, appears to be stored at a single location.
    • Reduced storage costs
      • Compression
      • Deduplication
        • Eliminates redundant versions of the same data (deduplication)
    • Simplified storage management
      • Provides system administration tools that you can use to configure and manage data:
        • Backup and restore functions from a Microsoft Management Console (MMC) snap-in.
        • Separate, optional interface to extend StorSimple management and data protection services to content stored on SharePoint servers.
    • Improved disaster recovery and compliance
      • Does not require extended recovery time. Instead, it restores data as it is needed.
        • Regular operations can continue with minimal disruption.
      • You can configure policies to specify backup schedules and data retention.
    • Data mobility
      • Data uploaded to Microsoft Azure cloud services can be accessed from other sites for e.g. recovery and migration purposes.
      • You can use StorSimple to configure StorSimple virtual devices on virtual machines (VMs) running in Microsoft Azure.
        • The VMs can then use virtual devices to access stored data for test or recovery purposes.
    • Data Tiering
      • Automatically tiers and classifies your data.
      • Based on how often you access it.
      • Data is always being shuffled between tiers as the mechanism learns about your usage patterns.
        • To enable quick access, it stores hot data
          • On SSD.
          • Locally
        • It stores occasionally used (warm data) data
          • on HDDs in the device or on servers at the datacenter.
        • Inactive data
          • Automatically migrats to the cloud.
      • Rearranges data and storage assignments as usage patterns change

Comparing Database Options in Azure

• Azure SQL Database
  • Database as a service
  • Predictable performance
    • Through comparison of DTU’s (Database Throughput Units)
    • DTUs describe capacity of tier and performance level.
    • Relative: E.g. Basic (B) 5, Standard (S2) 50, so standard is 10 times power of Basic.
  • High compatibility
    • A tabular data Stream (TDS) endpoint is provided for each logical server.
  • Simple management
    • In portal: Azure Management Portal -> Manage
    • REST API, PowerShell or Xplat CLI
  • Database Tiers
    • Three tier where every tier have different performance levels:
      • Basic/Standard model
        • Based on remote storage.
        • Uses Azure Premium Storage Disks, i.e. accessed over network.
      • Premium/Business Critical model
        • Uses AlwaysOn Availability Groups
        • Has local attached SSD storage
          • Provides higher IOPS and throughput
      • Basic (5 DTU) is ideal for small dbs, single active operation, dev/test, small scale apps
      • Standard (10 - 100 DTU) is ideal for multiple operations, workgroup or web apps
      • Premium (100 - 800 DTU) is ideal for high transaction volumes, large number of users, multiple operations, mission critical apps.
  • Elastic scale
    • Scaling it/out by simplified sharding.
      • Coordinates data movement between shards to split or merge ranges of data among different databases
    • Satisfies common scenarios such as pulling a busy tenant into its own shardING
    • Split-Merge service
      • Provided through a downloadable package
      • Customers can deploy as an Azure cloud service into their own subscription.
      • Two main parts:
        • An Elastic Scale library (SDK) for client applications to configure shards and access shards.
          • Direct transactions to the appropriate shard
          • Perform queries across multiple shards
          • Modify service tier for existing shards
        • The Elastic Scale features in Azure SQL Database that implements the any changes requested by your application.
    • Third-party managed SQL databases in Azure
      1. Azure Database for MySQL
         • MYSQL community with tools preinstalled like mysql.exe and phpMyAdmin.
      2. Azure Database for PostgreSQL
         • Both has common features:
           • You can run one or more databases with this instance.
           • High availability with no additional cost.
           • Predictable performance, using inclusive pay-as-you-go pricing.
           • Scale on the fly within seconds.
           • Secured to protect sensitive data at-rest and in-motion.
           • Automatic backups and point-in-time-restore for up to 35 days.
           • Enterprise-grade security and compliance.
         • Other options
           • Own VM’s running e.g. MySQL
           • ClearDB provides managed MySQL that you can create from Azure Marketplace.
  • SQL Availability
    • Azure Site Recovery: Only for VM’s
    • AlwaysOn
      • Available only in SQL servers
      • Azure SQL database implements it in its underyling infrastructure for Premium tier to achieve high availability but abstracts it away.
      • AlwaysOn availability groups
        • An availability group supports a failover environment for a discrete set of user databases, known as availability databases, that fail over together.
      • Always On Failover Cluster Instances
        • Leverages Windows Server Failover Clustering (WSFC) functionality
        • Provides local high availability through redundancy at the server-instance level failover cluster instance (FCI).
        • FCI (failover cluster instance)
          • Single instance of SQL Server that is installed across Windows Server Failover Clustering (WSFC) nodes and, possibly, across multiple subnets
    • Active geo-replication
      • Leverages the Always On technology of SQL Server.
      • Azure SQL Database feature.
        • Supported in Elastic Pools.
        • ❗ Not supported in managed instances.
        • 💡 Use auto-failover groups for them.
          • It also supports SQL database.
      • Multiple secondaries are supported as oppsed to auto-failover groups.
      • Allows you to create readable secondary databases of individual databases on a SQL Database server in the same or different data center (region).
      • Best practice configuration:
        • User => Azure Traffic Manager =>
          1. Ingress LB => Sql in Primay Logical Server
          2. Ingress LB => Sql in Secondary Logical Server
        • Capabilities:
          • Automatic Asynchronous Replication
          • Readable secondary databases
          • Planned / Unplanned failover
          • Multiple readable secondaries
          • Geo-replication of databases in an elastic pool
            • Each secondary database can separately participate in an elastic pool or not be in any elastic pool at all.
          • Configurable compute size of the secondary database
          • User-controlled failover and failback
          • Keeping credentials and firewall rules in sync
    • Auto-failover group
      • Allows you manage replication and failover of a group of
        • databases on a SQL Database server
        • or all databases in a Managed Instance to another region
      • It uses the same underlying technology as active geo-replication.
      • You can failover manually or you can delegate it to the SQL Database service based on a user-defined policy.
      • When working with single or pooled databases on a SQL Database server and you want multiple secondaries in the same or different regions, use active geo-replication.
      • You get URL (HA URL) for the DB.
      • Terminology and capabilities
        • Failover group
          • Group of databases that can either be:
            • SQL Database servers
            • Managed Instances
        • Primary & secondary hosts
          • Use Active-geo replication for multiple secondary hosts.
        • Adding databases in elastic pool to failover group
          • Multiple failover groups: You can configure multiple failover groups for the same pair of servers to control the scale of failovers. Each group fails over independently
            • ❗ Managed Instance does not support multiple failover groups
        • Grace period with data loss: By configuring GracePeriodWithDataLossHours, you can control how long the system waits before initiating the failover that is likely to result data loss.
      • Failover stragies
        • Automatic failover policy: Configured by default
        • Read-only failover policy:
          • Kicks in after a set period (hours) so that data is not last during a long failure.
          • Disabled by default
          • When disabled:
            • Performance of the primary is not impacted when the secondary is offline.
            • Read-only sessions will not be able to connect until the secondary is recovered
          • When enabled
            • Read-only traffic will be automatically redirected to the primary if the secondary is not available.
            • Performance gets lower in the primary
          • Use if you
            • cannot tolerate downtime for the read-only sessions.
            • are OK to temporarily use the primary for both read-only and read-write traffic at the expense of the potential performance degradation of the primary
        • Planned failover: Full synchronization without data loss
          • Use-cases:
            • Perform disaster recovery (DR) drills in production when the data loss is not acceptable
            • Relocate the databases to a different region
            • Return the databases to the primary region after the outage has been mitigated (failback).
        • Unplanned failover: Switches directly without any synchronization.
        • Manual failover: You can initiate forced or friendly failover (with full data synchronization).
  • Retention
    • Long-term retention (LTR): Atomatically retain backups in Azure Blob storage for up to 10 years
    • Azure SQL Database automatic backups: 7-35 days
• NoSQL services
  • Azure Storage
    • Massively scalable to support big data scenarios & small amount of data.
    • Billing is calculated by usage, not capacity.
    • Auto-partitioning with automatically load-balancing based on traffic.
    • Elastic & decoupled from application.
    • Access via REST API or SDKs: .NET, Java/Android, Node.js, PHP, Ruby, Python, PowerShell.
    • All resources in Storage can be protected from anonymous access and can be used in the Valet-Key pattern.
      • Valet-Key Pattern
        • Access resources using valet key (=key with granular & time-limited access for pre-defined operations/scopes).
        • E.g. SAS
    • Azure Table Storage
      • The Azure Table storage service stores large amounts of structured data.
      • Supports OData protocol.
      • Tables scale as demand increases.
      • Components
        • URL format: http://<storage account>.table.core.windows.net/<table>
        • Storage Account : All access to Azure Storage is done through a storage account.
        • Table
          • Collection of entities
          • Don’t enforce a schema on entities
            • which means a single table can contain entities that have different sets of properties.
          • The number of tables that a storage account can contain is limited only by the storage account capacity limit.
        • Entity : An entity is a set of properties, similar to a database row. An entity can be up to 1 MB in size.
        • Properties
          • Name-value pair.
          • Each entity can include up to 252 properties to store data
          • Each entity also has 3 system properties:
            • Partition key
            • Row key
            • Timestamp
      • Partitioning
        • Rate limit
          • ❗ A partition has a scalability target of 500 entities per second.
            • The throughput may be higher during minimal load on the storage node, but it will be throttled down when the node becomes hot or very active.
        • PartitionKey & RowKey properties
          • Can store up to 1 KB of string values.
          • Empty strings are also permitted; however, null values are not.
          • Clustered index sorting: Ascending PartitionKey then by ascending RowKey.
            • The sort order is observed in all query responses.
          • Each partition server can serve one or more partitions.
          • You should use more partitions, so that the Azure Table service can distribute the partitions to more partition servers.
• Azure Search
  • PaaS / Search as a Service over scoped text content.
  • Stores your data in an index that can be searched through full text queries.
    • Can be created in Azure Portal or with SDK/REST APIs.
    • Can be auto-generated from SQL Database or Cosmos DB.
  • Search is not only indexing!
    • Search needs to be smarter, have language understanding, understand uniqueness.
  • Advanced search behaviors
    • Type-ahead query suggestions based on partial term input
    • Hit-highlighting
    • Faceted navigation
    • Natural language support using linguistic rules for the specified language.
  • Pricing
    • Free: Shared with other Azure Search subscribers
    • Higher
      • Dedicated resources only used by your service
      • More search units & partitions
        • Search unit
          • How fast it indexes
          • Each partition and replica counts as one SU
          • E.g. 3 replicas + 3 partitions = 9 SU (as each partition have its own replica)
• Azure Cosmos DB
  • Multi-model database service.
  • Globally distributed: users access datacenters closer to them.
  • Four different APIs:
    • Document DB (SQL) API,
    • MongoDB API
    • Graph (Gremlin) API
    • Tables (Key/Value) API
  • The Azure Cosmos DB Data Migration tool is an open source solution that imports data to Azure Cosmos DB from a variety of sources, including: JSON files, MongoDB, SQL Server, CSV files, Azure Cosmos DB collections.
  • Consistency levels
    • Automatic data distribution across partitions.
    • Same partition key ensures data will be stored within same partition.
    • Consistent-prefix:
      • Data versions can be behind by are always ordered in a right way based on when they’re written.
    • Four consistency levels from stronger guarantees to better performance and availability:
    • Strong => Bounded Stateless => Session => Eventual
      • Strong
        • Write operation on primary database => it’s replicated to the replica instances.
        • The operation is only committed (and visible) on the primary after it has been committed and confirmed by ALL replicas.
        • Always read most recent copy from master r/w node.
      • Bounded Stateless
        • Reads honor consistent-prefix guarantee.
        • Established through 2 metrics: time/version.
          • E.g. goal: 90%>
        • Difference from Strong is you can configure how stale documents can be within replicas.
          • Staleness : Quantity of time (or version count) a replica document can be behind the primary document.
      • Session
        • Reads honor consistent-prefix guarantee.
        • Guarantess all queries are served by same server per session.
          • You always see the latest change you make.
        • All read & write are consistent & monotonic across primary and replica instances within a user session.
      • Eventual
        • Commits any write operation against the primary immediately.
        • Transactions are handled asynchronously and will eventually (over time) be consistent with the primary.
        • Most performant as you don’t wait for replicas to commit to finalize its transactions.
  • Consistency strategy
    • Stronger consistency -> Write is slower
    • Looser -> Performance is at peak but read can lag behind to older versions of data
• SQL Data Warehouse
  • Cloud-based Enterprise Data Warehouse (EDW)
  • Main difference from SQL is that it’s underlying infrastructure is that it’s a columnar storage.
    • Tables are still relational
  • Quickly run complex queries across petabytes of data.
    • Leverages Massively Parallel Processing (MPP).
  • E.g. use as a key component of a big data solution.
  • Supports PolyBase T-SQL queries
    • PolyBase is a technology that accesses and combines both non-relational and relational data, all from within SQL Server.
    • It allows you to run queries on external data in Hadoop/Spark or Azure blob storage.
  • 💡 Choose if you run many aggregated queries.
  • Data flow:
    • Ingest : Data orchestration and monitoring
    • Store : Big data store
    • Prep & train : Hadoop/Spark and Machine Learning
    • Model & serve : Data warehouse
• Azure Data Lake Store
  • Hyper-scale repository for big data analytic workloads.
  • Stores data of any size, type and ingestion speed in a one single place for operational and exploratory analytics.
  • It’s a distributed file system
    • File systems
      • Single machine: Fat NFS
      • Network file-systems: NFS, SMB
        • Distributed lock.
        • Different nodes can mount same drive without corruption.
      • Cluster file-systems: ZFS, ADFS, Oracle RAC
        • They respect each others lock.
        • Allows multiple machines to manipulate files within them concurrently.
        • Oracle RAC is active-active.
          • Machines share their caches.
          • They cross replicates to create illusion that they’re active nodes, but the reality is that there’s a single master node.
            • If master node fails -> mastership is delegated to another node.
      • Distributed file system
        • Connects machines.
        • Instead of everyone mounting a shared device -> Every machine has a local storage and distributed across others.
        • Allows distributing jobs so it can be distributed.
          • Parallel processing, e.g. Hadoop.
        • It’s not much about distributing files but parallel processing.
      • More, e.g. storj based on blockchain.
  • Can be accessed from Hadoop (available with HDInsight cluster) using the WebHDFS-compatible REST APIs.
    • WebHDFS => Web Hadoop File System for IaaS.
      • Used also by Kafka.
    • HDInsight
      • Compute layer of the data lake store.
      • Managed hadoop cluster
  • Includes security (e.g. OAuth, 777), manageability, scalability, realiability and availability.
  • Azure Data Lake Analytics
    • Biggest challange of Hadoop is that it’s a very big ecosystem.
      • So many components, hard to get stuff done.
    • Simplify big data analytics.
    • Can handle jobs of any scale instantly by setting the dial for how much power you need.
    • Pay only for your job when it is running, making it cost-effective.
    • The analytics service supports Azure Active Directory letting you manage access and roles, integrated with your on-premises identity system.
    • Includes U-SQL
      • Similiar to C# and SQL.
      • A language that unifies the benefits of SQL with the expressive power of user code.
      • Scalable distributed runtime enables you to efficiently analyze data in the store and across SQL Servers in Azure, Azure SQL Database, and Azure SQL Data Warehouse.
• Data integration
  • Azure Data Factory
    • Pipeline solution to schedule data-driven workflows.
      • Typically a pipeline is:
        • Connect & Collect / Ingest
        • Store: E.g. Azure Store
        • Transform & Enrich
          • Process in Spark, Data Lake Analytics and Machine Learning.
          • Prep & train with Azure Databricks.
        • Publish
          • Load into analytics engine (e.g. Azure SQL database) to be used by BI tools.
        • Monitor
          • Check failure/success rates
    • Handles
      • Extract-transfer-load
      • Data integration
        • Can ingest data from data stores.
        • Send output data to data stores.
        • E.g. file shares, FTP web, databases, SaaS services.
      • Hybrid extract-transfer-load
• Data Analysis Options
  • Azure Analysis Services
    • PaaS
    • Integrated with Azure data platform services.
    • You can mashup and combine data from multiple sources, define metrics, and secure your data in a single, trusted semantic data model.
    • Integrations
      • Data Sources
        • Cloud: E.g. SQL Database, SQL Data Warehouse, Data Lake, HDInsights/Spark…
        • On-premises: E.g. SQL Server / Oracle…
      • Client tools
        • Cloud: Power BI
        • On-premises: Third-Party. Power BI Desktop. Excel
    • Handles
      • Security
      • In-memory cache
      • Data modeling
      • Lifecycle management
      • Business logic & metrics
    • Compatible with many features already in SQL Server Analysis Services Enterprise Edition
      • Supports tabular models at the 1200 and 1400 compatibility levels
      • Partitions, row-level security, bi-directional relationships, and translations are all supported.
      • In-memory and DirectQuery modes are also available for fast queries over massive and complex datasets.
    • For developers, tabular models include the Tabular Object Model (TOM) to describe model objects.
      • TOM
        • Client library for SQL
        • Exposed in JSON through the Tabular Model Scripting Language (TMSL) and the AMO data definition language.
          • TOM is built on AMO.
            • Analysis Management Objects (AMO) is a library of programmatically accessed objects that enables an application to manage an Analysis Services instance.
            • E.g. AMO has data mining classes
        • Has classes for models, relationship, roles, annotations, cultures etc. to manage SQL analysis objects.
        • Structured in a tabular form.
        • Arranges data elements in vertical columns and horizontal rows. Each cell is formed by the intersection of a column and row.
  • HDInsight
    • Common use:
      1. Create HDInsight
      2. Schedule Jobs
      3. Delete HDInsight Cluster
    • Azure distribution of Apache Hadoop components
      • Framework for processing and analysis of big data sets on clusters.
      • Including Apache Hive, HBase, Spark, Kafka, Storm, R and many others.
        • Apache Spark is an open-source parallel processing framework that supports in-memory processing to boost the performance of big-data analytic applications.
    • Built on top of Azure Storage
  • Azure Data Catalog
    • A single, central place for all of an organization’s users to contribute their knowledge and build a community and culture of data.
      • It includes a crowdsourcing model of metadata and annotations.
        • Descriptive metadata supplements the structural metadata (such as column names and data types) that’s registered from the data source.
      • The data remains in its existing location, but a copy of its metadata is added to Data Catalog, along with a reference to the data-source location.
      • The metadata is also indexed to make each data source easily discoverable via search and understandable to the users who discover it.
    • Any user (analyst, data scientist, or developer) can discover, understand, and consume data sources.
      • Users can contribute to the catalog by tagging, documenting, and annotating data sources that have already been registered.
      • They can also register new data sources, which can then be discovered, understood, and consumed by the community of catalog users.

Monitoring & Automating Azure Solutions

• Azure Network Watcher
  • Inbuilt in Monitor -> Network
  • Features
    • Topology: e.g. VNet’s, subnets, VM’s, NIC’s
    • Variable Packet Capture: Captures TCP packages at NIC level as wireshark files.
    • IP Flow Verify: Troubleshoots NSG
    • Next hop: Troubleshoots route tables
    • Connection troubleshoot: Why it does not connect?
    • Diagnostics Logging
    • Security Group View
    • NSG Flow Logging
    • VPN Gateway Troubleshooting
    • Network Subscription Limits
    • Role Based Access Control
• Network Monitor
  • Troubleshooting blade.
  • Available for the following network resources: ExpressRoute, VPN Gateway, Application Gateway, Network Security Logs, Routes, DNS, Load Balancer, and Traffic Manager.
  • Available through Portal, CLI, PowerShell, Rest API, retrieved using Power BI or third-party logs.
  • Events are logged in storage accounts, ready to be sent to Event Hub or Log Analytics.
  • Metrics for performance measurements are collected over a period of time.
• Azure Security Center
  • Unified security management and advanced threat protection for workloads running in Azure, on-premises, and in other clouds.
  • Enables you to discover and assess the security of your workloads and to identify and mitigate risk.
    • E.g. network recommendations, network maps, internet facing endpoints without NSG’s.
  • In standard tier, Just-in-time (JIT) virtual machine (VM) access can be used to lock down inbound traffic to your Azure VMs, reducing exposure to attacks while providing easy access to connect to VMs when needed.
    • When a user requests access to a VM, Security Center checks that the user has Role-Based Access Control (RBAC) permissions that permit them to successfully request access to a VM. If the request is approved, Security Center automatically configures the Network Security Groups (NSGs) to allow inbound traffic to the selected ports and requested source IP addresses or ranges, for the amount of time that was specified. After the time has expired, Security Center restores the NSGs to their previous states. Those connections that are already established are not being interrupted, however.
• Azure Monitor
  • Metrics
    • Also called performance counters
    • Tasks
      • Track the performance of your resource (such as a VM, website, or logic app) by plotting its metrics on a portal chart and pinning that chart to a dashboard.
      • Perform advanced analytics or reporting on performance or usage trends of your resource.
      • Get notified of an issue that impacts the performance of your resource when a metric crosses a certain threshold.
      • Archive the performance or health history of your resource for compliance or auditing purposes.
      • You can choose to stream this information to an event hub.
        • Doing so allows you to route them to Azure Stream Analytics for near-real-time analysis.
    • There is no need to set up additional diagnostics for metrics, nor opt-in for the data.
    • Frequency of one minute.
    • Some metrics available can also have name-value pair attributes
      • Known as dimensions
      • Allows further segmentation
    • Users to access up to 30 days of history for each metric.
      • You can also choose to achieve metrics to storage if you need to retrain them for longer than the archive period.
        • You configure it in settings for the resource.
• Azure Advisor
  • It analyzes your resource configuration and usage telemetry. It then recommends solutions to help improve the performance, security, and high availability of your resources while looking for opportunities to reduce your overall Azure spend.
  • Across subscriptions
  • You can apply filters to display recommendations for specific subscriptions and resource types.
  • The recommendations are divided into four categories:
    • High Availability : To ensure and improve the continuity of your business-critical applications.
    • Security : To detect threats and vulnerabilities that might lead to security breaches.
    • Performance : To improve the speed of your applications.
    • Cost : To optimize and reduce your overall Azure spending.
• Azure Service Health
  • Provides personalized guidance and support when issues in Azure services affect you
  • Helps you prepare for upcoming planned maintenance.
  • Alerts you and your teams via targeted and flexible notifications.
  • Service Health tracks three types of health events:
    • Service issues : Problems in the Azure services that affect you right now.
    • Planned maintenance : Upcoming maintenance that can affect the availability of your services in the future.
    • Health advisories: Changes in Azure services that require your attention.
      • E.g. Azure features are deprecated
      • E.g. you exceed a usage quota.
  • Data is presented at
    • Azure Portal: Personalized Service Health Dashboard
    • Azure Monitor: Service Health Alerts
    • http://status.azure.com : General Health Overview of All Azure Services
• Operations Management Suite – Log Analytics
  • Log Analytics is a service in Operations Management Suite (OMS) that monitors your cloud and on-premises environments to maintain their availability and performance.
  • Deprecated. Now it’s Azure Logs in Portal.
  • In portal:
    • Log searches
    • Graphical views of your most valuable searches
    • Solutions which provide additional functionality and analysis tools.
  • Log Analytics Agent
    • Can be installed on:
      • Azure VMs using the Azure Log Analytics VM extension for Windows and Linux
      • For machines in a hybrid environment using setup, command line, or with Desired State Configuration (DSC) in Azure Automation.
    • Outbound connection over TCP port 443
      • If no connection to internet:
        • The Log Analytics gateway is an HTTP forward proxy that supports HTTP tunneling using the HTTP CONNECT command.
        • Agents sends their data to gateway
      • Sends data to log analytics workspace -vThe Log Analytics retention settings allow you to configure a minimum of 31 days (if not using a free tier) up to 730 days.
      • Collects custom logs, IIS logs, performance counters, syslog, windows event logs.
• Azure Diagnostic Agent
  • Collection of diagnostic data on a deployed application
  • Supports azure Cloud Service (classic) Web and Worker Roles, Virtual Machines, Virtual Machine scale sets, and Service Fabric.
  • Collects: Performance counter metrics, Application logs, Windows Event logs, .NET EventSource logs, IIS Logs, Manifest based ETW logs, Crash dumps (logs), Custom error logs, Azure Diagnostic infrastructure logs.
  • Data storage
    • The extension stores its data in an Azure Storage account that you specify.
    • You can also send it to Application Insights.
    • Another option is to stream it to Event Hub, which then allows you to send it to non-Azure monitoring services.
    • You also have the choice of sending your data to Azure Monitor metrics time-series database
• Application Insights
  • Application Performance Management (APM) service for web developers building and managing apps on multiple platforms.
  • Use it to monitor your live web application.
  • It will automatically detect performance anomalies.
  • It integrates with your DevOps process, and has connection points to a variety of development tools.
  • It can monitor and analyze telemetry from mobile apps by integrating with Visual Studio App Center and HockeyApp.
• Power BI
  • Turn your data processing efforts into analytics and reports that provide real-time insights into your business.
  • Works with cloud-based or on-premises data.
  • Has a multitude of Azure connections available.
  • Shape and refine your data to build customized reports.
  • Products: Power BI Desktop, Power BI Pro, Power BI Premium, Power BI Mobile, Power BI Embedded, Power BI Report Server
  • Azure SQL Database and Power BI
    • You can start with a straightforward connection to an Azure SQL Database, and create reports to monitor the progress of your business.
    • E.g: “Get Data dialog”: Within the same Query you can connect to your Azure SQL Database, your Azure HDInsight data source, and your Azure Blob Storage (or Azure Table Storage), then select only the subsets within each that you need, and refine it from there.

Backup

• Azure Backup
  • Simple and cost-effective backup as a service (BaaS) solution, that gives you trusted tools on-premises with rich and powerful tools in the cloud.
  • Data is stored in a Storage Account.
    • Azure Backup Server
      • Can be installed on Azure or on-premises and can back up VM’s.
      • Inherits much of the workload backup functionality from Data Protection Manager (DPM).
      • Backs up in Site Recovery vault.
      • Can back up File servers, SQL Server, Hyper-V, Exchange Server
  • In a seamless portal experience with Azure Site Recovery, gives you cost-efficiency and minimal maintenance, consistent tools for offsite backups and operational recovery, and unified application availability and data protection.
  • There are three primary options for backing up to Azure Backup:
    1. Azure Backup / Restore of On-Premises Files & Folders
       • Ideal for backing up Files & Folders only
       • Steps:
         1. Deploy the Azure Backup Agent (Azure Recovery Services Agent) on the VM guests running on-premises Hyper-V / SCVMM / Vmware / Physical infrastructure.
         2. Configure Azure Backup from within the VM guest.
         3. Configure the integration with Azure Backup Vault.
         4. Run the Backup job from within the VM guest.
            • Runs on 443 HTTPS over Public Internet or ExpressRoute with Public Peering
         5. Files & Folders backup will be stored in Azure Backup Vault, and can be restored from there.
    2. Azure Backup / Restore of On-premises running full workloads (OS, Sysvol and Applications)
       • Better for backing up full system workloads (OS, system state, applications – consistent)
         • Steps:
           1. Deploy the Azure Backup Server (or System Center DPM 2012 R2 or 2016) on the on-premises Hyper-V / SCVMM / Vmware / Physical infrastructure.
           2. Configure Azure Backup Server backup policies, backup storage (2-tier) and deploy agents to your workloads.
           3. Configure the integration with Azure Backup Vault.
           4. Run the Backup job from within the Azure Backup Server console.
              • Runs on 443 HTTPS over Public Internet or ExpressRoute with Public Peering
           5. VM workloads (system state, OS, applications,…) backup will be stored in Azure Backup Vault, and can be restored from there.
    3. Azure VM Backup / Restore to Azure Backup Vault
       • Best for when you want to backup Azure VMs to Azure Backup Vault
         • Steps:
           • Deploy the Azure Backup Extension, or select Azure Backup in the VM configuration.
           • Configure Azure Backup backup policies, in the Azure platform.
           • Configure the integration with Azure Backup Vault.
           • Run the Backup job from within the Azure Platform.
             • Runs on 443 HTTP over Azure Backbone Infrastructure.
           • Azure VMs will be backed up as full VM snapshots, and can be restored from within the Azure Portal.
  • Hybrid Backup Encryption
    • Allows for end-to-end encryption of the backup platform:
      • It starts with a passphrase for the Azure Recovery Services Agent installation.
      • The next layer is the Backup Data itself, which gets encrypted in transit.
      • Once the data is stored in Azure Backup Vault, it gets encrypted at rest as well.
  • Monitoring
    • * Azure Backup Monitoring with Log Analytics
      • Azure Backup monitoring is possible from Log Analytics.
      • Out of Log Analytics, one can get a detailed view on the backup statistics, the amount of data that is being consumed, successful and failed jobs and alike.
    • Azure Backup Reports with Power BI
      • Azure Backup also allows for reporting integration with Microsoft Power BI.
  • Linux Application Consistent Azure Backup
    • Taking backups of Azure VMs running Linux OS is fully supported, for Azure supported Linux Operating Systems.
    • To allow for application consistent backups, you need to run a pre- and post- backup script. The VM Snapshot will be your VM Backup, which gets stored in the Backup Vault using an incremental update process
  • Microsoft System Center Data Protection Manager (DPM)
    • Underlying infrastructure of Azure Back-up
    • Backs up both on-prem servers/VM’s & cloud VM’s
    • Can store back-up data to:
      • Disk: For short-term storage DPM backs up data to disk pools.
      • Azure: DPM data stored in disk pools can be backed up to the Microsoft Azure cloud using the Azure Backup service
      • Tape
    • Features:
      • Application-aware backup: Application-aware back up of Microsoft workloads, including SQL Server, Exchange, and SharePoint.
      • File backup: Back up files, folders and volumes for computers running Windows server and Windows client operating systems.
      • System backup: Back up system state or run full, bare-metal backups of physical computers running Windows server or Windows client operating systems.
      • Hyper-V backup: Back up Hyper-V virtual machines (VM) running Windows or Linux. You can back up an entire VM, or run application-aware backups of Microsoft workloads on Hyper-V VMs running Windows.
    • Microsoft Azure Backup Server (MABS)
      • Can be installed on Azure or on-premises and can back up VM’s.
      • Inherits much of the workload backup functionality from Data Protection Manager (DPM).
      • Backs up in Site Recovery vault.
      • Can back up File servers, SQL Server, Hyper-V, Exchange Server
• Azure Site Recovery
  • Azure as your Disaster/Recovery Datacenter Site
    • Built to provide a datacenter disaster recovery solution for your VM workloads.
      • Recovers from
        • Hyper-V
        • VMware
        • Physical hosts
        • AWS VMs
    • Replication-based failover to Azure Virtual Machines
    • Failover & Failback
    • Application-consistent failover
  • Ideal as a Virtual Machine “Lift & Shift” migration tool:
    • An ideal tool for performing VM lift & Shift operations of your workloads.
    • Full machine-state replication to an Azure VM
    • Perfect for test/dev scenarios
    • Zero-data loss during migration
  • Replication policies
    • Recovery Point Objective (RPO) is the maximum time of acceptance for data loss.
    • Recovery Time Objective (RTO) in ASR means the period when a failover starts to the time the process completes and a virtual machine is running in Azure.
    • App-consistent snapshots capture disk data, all data in memory, and all transactions in process.
  • Run a failover
    • Recovery Plans > recoveryplan_name. Click Failover
    • You can alternatively run failover from Replicated Items
    • Select a Recovery Point to failover to
      • Latest
        • Processes all data that’s sent to Site Recovery service.
        • Creates a recovery point for each virtual machine
          • Used by VM during failover.
        • Lowest data loss (RPO, recovery point objective)
          • As VM created after failover has all the data that has been replicated to Site Recovery service when the failover was triggered.
      • Latest processed
        • Fails over all virtual machines of the recovery plan to the latest recovery point that has already been processed by Site Recovery service.
        • If you are doing failover of a recovery plan, you can go to individual virtual machine and look at Latest Recovery Points tile to get this information.
        • Low recovery time (RTO, recovery time objective)
          • As no time is spent to process the unprocessed data.
      • Latest app-consistent
        • Fails over all virtual machines of the recovery plan to the latest application consistent recovery point that has already been processed by Site Recovery service.
        • When you are doing test failover of a virtual machine, time stamp of the latest app-consistent recovery point is also shown.
        • If you are doing failover of a recovery plan, you can go to individual virtual machine and look at Latest Recovery Points tile to get this information.
      • Latest multi-VM processed
        • This option is only available for recovery plans that have at least one virtual machine with multi-VM consistency ON.
        • VMs that are part of a replication group failover to the latest common multi-VM consistent recovery point.
        • Other virtual machines failover to their latest processed recovery point.
      • Latest multi-VM app-consistent
        • Only available for recovery plans that have at least one virtual machine with multi-VM consistency ON.
        • Virtual machines that are part of a replication group failover to the latest common multi-VM application-consistent recovery point.
        • Other virtual machines failover to their latest application-consistent recovery point.
      • Custom
        • If you are doing test failover of a virtual machine, then you can use this option to failover to a particular recovery point.
        • ❗ Only supported when failing over to Azure.
• Azure Backup vs Azure Site Recovery
  • Azure Backup is used to protect and restore data at a more granular level.
    • E.g. if some files become corrupted, you can use Azure Backup to restore them

  • Azure Site Recovery is used to replicate the configuration and data of a system to another datacenter, then you would use.

    Concept	Explanation	Backup	Disaster Recovery (DR)
    Recovery point objective (RPO)	The amount of acceptable data loss if a recovery needs to be done.	Wide variability in their acceptable RPO. E.g. VM backups usually 1 day, DB backups 15 minutes.	Low RPOs. The DR copy can be behind by a few seconds or a few minutes.
    Recovery time objective (RTO)	The amount of time that it takes to complete a recovery or restore.	Larger RPO -> the amount of data needed to process is much higher -> leads to longer RTOs. E.g. days to restore data from tapes, depending on the time it takes to transport the tape from an off-site location.	Smaller RTOs because they are more in sync with the source. Fewer changes need to be processed.
    Retention	How long data needs to be stored	For scenarios that require operational recovery (e.g. data corruption, inadvertent file deletion, OS failure), backup data is typically retained for 30 days or less. 💡 From a compliance standpoint, data might need to be stored for months or even years. Backup data is ideally suited for archiving in such cases.	Needs only operational recovery data, which typically takes a few hours or up to a day. 💡 Because of the fine-grained data capture used in DR solutions, using DR data for long-term retention is not recommended.

    Product	Use cases
    Azure Backup	• Accidental deletion • Patch Testing • Alternative location recovery • Security • Long-term data retention
    Azure Site Recovery	• Disaster Recovery with quick failover • Migration • Dev/Test Environment (e.g. test failover)

Automation

• Azure Automation
  • SaaS
  • Provides a way for users to automate the manual, long-running, error-prone, and frequently repeated tasks that are commonly performed in a cloud and enterprise environment
  • Automate processes using runbooks or automate configuration management using Desired State Configuration, in Azure, other cloud services, or on-premises.
  • Schedule processes to be automatically performed at regular intervals
  • Trigger through Azure Portal, Webhooks, PowerShell or Alerts.
  • Automation sandboxes
    • Runbooks that you run in Azure are executed on Automation sandboxes.
    • They are hosted in Azure PaaS virtual machines.
    • They provide tenant isolation for all aspects of runbook execution – modules, storage, memory, network communication, job streams, etc.
    • This role is managed by the service and is not accessible from your Azure or Azure Automation account for you to control.
    • Hybrid Runbook Worker (HRW) roles
      • To automate the deployment and management of resources in your local datacenter or other cloud services, after creating an Automation account, you can designate one or more machines to run the Hybrid Runbook Worker (HRW) role.
      • Each HRW requires the Microsoft Management Agent with a connection to a Log Analytics workspace and an Automation account.
        • Log Analytics is used to bootstrap the installation, maintain the Microsoft Management Agent, and monitor the functionality of the HRW
      • The delivery of runbooks and the instruction to run them are performed by Azure Automation.
  • Automation Account
    • You first create an Automation Account object.
    • It includes following objects to be used with runbooks:
      • Certificates
        • Contains a certificate used for authentication from a runbook or DSC configuration or add them.
      • Connections
        • Contains authentication and configuration information required to connect to an external service or application from a runbook or DSC configuration.
      • Credentials
        • It is a PSCredential object
        • It contains security credentials such as a username and password required to authenticate from a runbook or DSC configuration.
      • Integration modules
        • They are PowerShell modules included
        • Make use of cmdlets within runbooks and DSC configurations.
      • Schedules
        • Schedules start or stop a runbook at a specified time, including recurring frequencies.
      • Variables
        • Contain values that are available from a runbook or DSC configuration.
      • DSC Configurations
        • PowerShell scripts that describes how to configure an operating system feature or setting or install an application on a Windows or Linux computer.
      • Runbooks
        • They are a set of tasks that perform some automated process in Azure Automation based on Windows PowerShell.
    • Authentication
      • Role-based access control is available with Azure Resource Manager to grant permitted actions to an Azure AD user account and Run As account, and authenticate that service principal.
      • When you create authentication account, it comes with two authentication entities:
        • A Run As account
          • Creates a service principal in Azure Active Directory (Azure AD) and a certificate.
          • Assigns the Contributor role-based access control (RBAC), which manages Resource Manager resources by using runbooks.
        • A Classic Run As account.
          • This account uploads a management certificate, which is used to manage classic resources by using runbooks.
        • You can add your own connection with a certificate or as service principal.
        • For more access to VM you can add extensions to the automation account to e.g. ssh into machine and run a script.
  • Cross-Cloud: Azure Automation is configured across regions.
  • Azure Automation allows management and configuration of Azure systems, on-premises running systems, systems in AWS, Google Cloud, or any other 3rd party hosting data center.
    • On-premises: Use Azure Automation Hybrid Worker
    • Or run Azure Automation Agent in e.g. AWS, Google Cloud or any third party hosted data center.
  • Azure Automation lives in Azure and can run tasks on different platforms but are triggered from Azure Automation.
  • Configuration Management
    • Typical flow: You provision/manage infrastructure through bootstrap agents that customizes VMs.
    • Some of the infrastructure automation tools that you can use in Azure:
      • Chef
        • Automation platform that helps define how your infrastructure is configured, deployed, and managed.
        • Additional components includes:
          • Chef Habitat for application lifecycle automation
          • Chef InSpec helps automate compliance with security and policy requirements.
        • Chef Clients are installed on target machines, with one or more central Chef Servers that store and manage the configurations.
      • Puppet
        • Handles the application delivery and deployment process.
        • Agents are installed on target machines to allow Puppet Master to run manifests that define the desired configuration of the Azure infrastructure and VMs.
        • Puppet can integrate with other solutions such as Jenkins and GitHub for an improved devops workflow.

This site is open source. Improve this page.