Category: Fabric Capacity

Azure, Business Intelligence - BI, Fabric Administrator, Fabric Capacity, Logic Apps, Microsoft Fabric, Power BI

Microsoft Fabric: Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps

Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps

A while back, I published a blogpost explaining how to use Azure LogicApps to automate scaling Microsoft Fabric F capacities under the PAYG (Pay-As-You-Go) licensing option. Some of my followers reported an issue with their Capacity Admin settings when running the LogicApp solution. The issue was that their capacity admins disappeared after they had run the LogicApps to upscale or downscale the capacity. After some investigation, I found out what the problem was. At the same time, some of my other followers suggested a fix which involved hardcoding the admins into the solution. While this would work in some cases, it is not a practical solution in the long run, as the admin settings may evolve over time. This makes the solution hard to maintain and unreliable. Back then, I suggested using the APIs and an HTTP action in a new LogicApps solution. This blog is the continuation of the previous blog and a quick and easy fix that ensures the automation runs smoothly with minimal to no manual work or maintenance afterwards. I have also published a tutorial video on YouTube explaining the process from the beginning (which was already covered in my previous blog, so I do not explain it here again) which you can watch here:

A Reminder of the Previous Solution

I suggest you read my blog about automating Fabric capacity scaling with Azure LogicApps as it provides a step-by-step guide to implement the solution. But if you have already implemented, or you are just after the fix, jump to the next section. The following image shows how the original solution works:

Automate Scaling Microsoft Fabric F Capacities with Azure LogicApps
Automate Scaling Microsoft Fabric F Capacities with Azure LogicApps

Here is a quick explanation of how it works:

  1. The Trigger runs the workflow automatically every hour.
  2. The Read a Resource, which is an Azure Resource Manager operation, reveals information about a resource that, in our implementation, is a Microsoft Fabric Capacity.
  3. A condition to check the Status of the capacity. If the capacity is Paused (the condition is true), then do nothing. This is needed as this method only works when the capacity is Active.
  4. If the capacity is Active (the condition is false), then check the current time to see if it is between 2pm and 4pm. This is the timeframe for which we want to upscale the capacity.
  5. If the condition is True, then upscale the capacity to F8 using another ARM operation: Create or update a resource.
  6. If the condition is False, then set the capacity SKU to F2.

The solution works fine if you do not have any Capacity Admin settings either on Azure Portal or on Admin Portal on Microsoft Fabric. But in many cases, we indeed have capacity admins. Let’s see what the issue is.

The Problem

The issue arises when we add some capacity administrators; that are wiped after running the above solution in its current implementation. The following image shows the Capacity Admin settings on both portals:

Fabric Capacity Admins on Azure Portal and Fabric Admin Portal
Fabric Capacity Admins on Azure Portal and Fabric Admin Portal

The reason if that the Create or update a resource also updates the properties of the resource with the ones we define in LogicApps. Therefore, if we do not add any capacity admins, we literally empty the existing capacity admins. Let’s run the solution again to understand why this is happening. The following image shows my capacity admins are wiped out after running my LogicApp workflow:

Capacity Admins deleted after running LogicApps
Capacity Admins deleted after running LogicApps

The issue is also demonstrated in the tutorial video on YouTube:

Continue reading “Microsoft Fabric: Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps” →
Azure, Fabric Administrator, Fabric Capacity, Microsoft Fabric, Power BI

Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps

Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps

In the first blog post of this series, I explained that we can Pause and Resume a Microsoft Fabric capacity from Azure Portal. In the second blog and its accompanying YouTube video, I showed you how to automate the Pause and Resume actions in Azure LogicApps so the capacity starts at 8:00 AM and stops at 4:00 PM. While I have already mentioned in those posts, it is worthwhile to mention again that these methods only make sense for PAYG (Pay-As-You-Go) capacities and NOT the Reservation capacities. While the method works fine, you may need more fine-tuning.

Managing operational costs becomes crucial for businesses leveraging Microsoft Fabric capacities when the holiday season approaches. This presents a unique challenge of maintaining efficiency while reducing unnecessary expenses, especially during Christmas when business operations might slow down or pause entirely.

In this post and video, I will extend the discussions from my previous blog and demonstrate how to optimise your Azure Logic Apps to manage Microsoft Fabric capacity during the Christmas holidays.

Extending the Logic Apps Workflow

Existing Setup Recap

In earlier discussions, we’ve explored using Azure Logic Apps to manage Fabric capacity effectively from 8:00 AM to 4:00 PM on regular business days and pausing operations afterwards. This setup ensures that we’re not incurring costs when the capacity isn’t needed, particularly from 4:00 PM to 8:00 AM the next morning, and throughout the weekends. I encourage you to check out my previous post for more information. This is how the existing solution looks like in Azure LogicApps:

Automating Microsoft Fabric Capacity with Azure LogicApps
Automating Microsoft Fabric Capacity with Azure LogicApps

Incorporating Holiday Schedules

The key to extending this setup for the Christmas period lies in integrating specific holiday schedules into your existing workflows using Workflow Definition Language which is used in Azure Logic Apps and Microsoft Flow. The following expression determines if the current date (in New Zealand Standard Time) falls within the period from December 25th of the current year to January 2nd of the next year:

and(
    greaterOrEquals(
        int(
            formatDateTime(
                convertFromUtc(
                    utcNow(), 
                    'New Zealand Standard Time'
                ), 
                'yyyyMMdd'
            )
        ), 
        int(
   concat(
    formatDateTime(
     utcNow()
     , 'yyyy'
     )
    , '1225'
    )
   ) 
    ), 
    lessOrEquals(
        int(
            formatDateTime(
                convertFromUtc(
                    utcNow(), 
                    'New Zealand Standard Time'
                ), 
                'yyyyMMdd'
            )
        ), 
        int(
   concat(
    add(
     int(
      formatDateTime(
       utcNow()
       , 'yyyy'
       )
      )
     ,1
     )
    , '0102'
    )
   )
  )
)

The following section explains how the expression works.

Continue reading “Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps” →
Copilot, Fabric Capacity, Microsoft Fabric, Power BI

Use Copilot in Power BI Desktop to Create Measures from Numeric Columns

I have been thinking about a mechanism to generate measures from numeric columns on Power BI data models. Of course, we can use Tabular Editor, but it requires some scripting, which is all right. However, the more advanced our requirements get, the more complex the C# script. In real-world development scenarios, it does not make sense to blindly create measures for all numeric columns, such as the key columns used to define relationships between tables, making C# scripting a bit more complex.

In this blog and accompanying YouTube video, I explain using Copilot within Power BI Desktop to create measures from numeric columns. This feature represents a significant advancement in Power BI’s capabilities as of April 2024, enabling data analysts and BI professionals to streamline parts of their data analysis tasks.

Prerequisites

As explained in a previous post here, we first need to enable Copilot on the Fabric Portal. Please note that Copilot in Power BI Desktop requires either Power BI Premium Capacity or AT LEAST an F64 Fabric Capacity. Unfortunately, Copilot is NOT available on PPU, Embedded capacities, Fabric capacities smaller than F64 and Fabric Trial (FT) capacities.

We also need to have the latest version of Power BI Desktop installed on our machine. With that, let’s begin.

YouTube Video

Here is the video on YouTube where I explain the same thing in less than 5 min. But if you are after more details, continue reading.

Introduction to Power BI and Copilot

As Power BI evolves, it incorporates more sophisticated AI-driven capabilities that simplify various aspects of data analytics. The integration of Copilot in Power BI Desktop enhances user interaction with data in many ways. Our focus on this blog is specifically using Copilot to create simple yet crucial measures based on numeric columns that previously required manual effort.

Use Copilot for Measure Creation

Using Copilot is straightforward and demonstrates impressive intelligence in its operational logic. The following steps explain how to do so:

Continue reading “Use Copilot in Power BI Desktop to Create Measures from Numeric Columns” →
Fabric Capacity, Microsoft Fabric, Power BI, Power BI Desktop, Power BI Service, Power Query, Self-Service BI

Incremental Refresh in Power BI, Part 3: Best Practices for Large Semantic Models

Incremental Refresh in Power BI, Best Practices for Large Semantic Models

In the two previous posts of the Incremental Refresh in Power BI series, we have learned what incremental refresh is, how to implement it, and best practices on how to safely publish the semantic model changes to Microsoft Fabric (aka Power BI Service). This post focuses on a couple of more best practices in implementing incremental refresh on large semantic models in Power BI.

Note

Since May 2023 that Microsoft announced Microsoft Fabric for the first time, Power BI is a part of Microsoft Fabric. Hence, we use the term Microsoft Fabric throughout this post to refer to Power BI or Power BI Service.

The Problem

Implementing incremental refresh on Power BI is usually straightforward if we carefully follow the implementation steps. However in some real-world scenarios, following the implementation steps is not enough. In different parts of my latest book, Expert Data Modeling with Power BI, 2’nd Edition, I emphasis the fact that understanding business requirements is the key to every single development project and data modelling is no different. Let me explain it more in the context of incremental data refresh implementation.

Let’s say we followed all the required implementation steps and we also followed the deployment best practices and everything runs pretty good in our development environment; the first data refresh takes longer, we we expected, all the partitions are also created and everything looks fine. So, we deploy the solution to production environment and refresh the semantic model. Our production data source has substantially larger data than the development data source. So the data refresh takes way too long. We wait a couple of hours and leave it to run overnight. The next day we find out that the first refresh failed. Some of the possibilities that lead the first data refresh to fail are Timeout, Out of resources, or Out of memory errors. This can happen regardless of your licensing plan, even on Power BI Premium capacities.

Another issue you may face usually happens during development. Many development teams try to keep their development data source’s size as close as possible to their production data source. And… NO, I am NOT suggesting using the production data source for development. Anyway, you may be tempted to do so. You set one month’s worth of data using the RangeStart and RangeEnd parameters just to find out that the data source actually has hundreds of millions of rows in a month. Now, your PBIX file on your local machine is way too large so you cannot even save it on your local machine.

This post provides some best practices. Some of the practices this post focuses on require implementation. To keep this post at an optimal length, I save the implementations for future posts. With that in mind, let’s begin.

Best Practices

So far, we have scratched the surface of some common challenges that we may face if we do not pay attention to the requirements and the size of the data being loaded into the data model. The good news is that this post explores a couple of good practices to guarantee smoother and more controlled implementation avoiding the data refresh issues as much as possible. Indeed, there might still be cases where we follow all best practices and we still face challenges.

Note

While implementing incremental refresh is available in Power BI Pro semantic models, but the restrictions on parallelism and lack of XMLA endpoint might be a deal breaker in many scenarios. So many of the techniques and best practices discussed in this post require a premium semantic model backed by either Premium Per User (PPU), Power BI Capacity (P/A/EM) or Fabric Capacity.

The next few sections explain some best practices to mitigate the risks of facing difficult challenges down the road.

Practice 1: Investigate the data source in terms of its complexity and size

This one is easy; not really. It is necessary to know what kind of beast we are dealing with. If you have access to the pre-production data source or to the production, it is good to know how much data will be loaded into the semantic model. Let’s say the source table contains 400 million rows of data for the past 2 years. A quick math suggests that on average we will have more than 16 million rows per month. While these are just hypothetical numbers, you may have even larger data sources. So having some data source size and growth estimation is always helpful for taking the next steps more thoroughly.

Practice 2: Keep the date range between the RangeStart and RangeEnd small

Continuing from the previous practice, if we deal with fairly large data sources, then waiting for millions of rows to be loaded into the data model at development time doesn’t make too much sense. So depending on the numbers you get from the previous point, select a date range that is small enough to let you easily continue with your development without needing to wait a long time to load the data into the model with every single change in the Power Query layer. Remember, the date range selected between the RangeStart and RangeEnd does NOT affect the creation of the partition on Microsoft Fabric after publishing. So there wouldn’t be any issues if you chose the values of the RangeStart and RangeEnd to be on the same day or even at the exact same time. One important point to remember is that we cannot change the values of the RangeStart and RangeEnd parameters after publishing the model to Microsoft Fabric.

Continue reading “Incremental Refresh in Power BI, Part 3: Best Practices for Large Semantic Models” →