Azure Web App deployment slots are used to help roll out new versions of an app without downtime or cold start activation. New version is typically deployed to a staging slot, then after testing and final verification it gets swapped into a production slot. During the swap operation the Web App’s worker process may get restarted in order for some settings to take effect. Even though the swap does not proceed until the restarted worker process comes back online on every VM instance, it may still not be enough for application to be completely ready to take on production traffic. This post explains how you can use the recently enabled Application Initialization Module to completely warm up your application prior to swapping it into production.
First of all it is necessary to explain the sequence of actions that happens when a staging slot is swapped into production. When the Swap button is clicked in Azure Portal or a corresponding management API is called:
- The App Settings and Connection Strings that are marked as “Slot” are read from the Production slot and applied to the site in the Staging slot. That causes the site’s worker process to be restarted for those changes to take effect and become visible as process environment variables;
- Then the site in the staging slot gets warmed up. To warm up the site an HTTP request is made to the root directory of the site to every VM instance where site is supposed to run. The warm up request has a User-Agent header set to “SiteWarmup”;
- After warm up has completed the host names for the sites in production and staging slots get swapped. Now the site that has been warmed up in the staging slot starts getting production traffic and the site that used to be in the production slot is now in the staging slot
- The site that is now in the staging slot gets updated with the App Settings and Connection Strings associated with the staging slot. That causes restart of that site, but it is not in production slot any more so restart is harmless.
Sometimes hitting the site’s root URL is not enough to completely warm up the application. For example it maybe necessary to hit all important routes in an ASP.NET MVC app or to pre-populate the in-memory cache. That is where the Application Initialization Module can help.
Let’s use a simple example to demonstrate how a Web App can be warmed up in the deployment slot during the swap operation. First let’s create a site and a staging deployment slot:
Next let’s set some slot settings on the App and its staging slot. These slot settings will cause the App’s worker process to restart during swap.
For the actual app code I used two simple PHP files: index.php and warmup-cache.php. The index.php is served when site’s root URL is requested. The warmup-cache.php is my “cache warmup” code that takes long time to run (emulated by sleep() command). In real application that can be the script that makes database queries to fill up the cache.
Finally I also have a web.config file which configures AppInit module:
<system.webServer>
<applicationInitialization >
<add initializationPage="/warmup-cache.php" hostName="appinit-warmup.azurewebsites.net"/>
</applicationInitialization>
<system.webServer>In the applicationInitialization section I can specify multiple URL paths that need to be requested in order to warm up my application. In my case I only need to hit one URL. Also notice that I can specify the host name to use for the warm up requests (this is optional and if not specified the “localhost” will be used as a host name).
The following steps are just for verification/debugging purposes. There is no need to perform them when using AppInit module. In fact enabling them for your production site may considerably slow it down.
To confirm that the warmup-cache.php is actually requested during the swap I will use Failed Request Tracing. It can be enabled from Azure Portal:
However, that will trace only failed requests. I need it to trace all requests. For that I add the following section to the web.config file:
<tracing>
<traceFailedRequests>
<clear/>
<add path="*">
<traceAreas>
<add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,Rewrite,iisnode" verbosity="Verbose" />
</traceAreas>
<failureDefinitions statusCodes="200-600" />
</add>
</traceFailedRequests>
</tracing>The important attribute here is statusCodes which specifies that requests with status codes between 200 and 600 should be logged.
Now let’s run the swap command.
During the swap operation the application is restarted and the first warm up request is made to the root directory of the web app. That starts the AppInit module which makes a request to warmup-cache.php URL and waits until it completes. Only after that request competes the swap operation proceeds to the next step and swaps the host names so that the warmed up site starts getting production traffic. Because of that the swap operation takes longer time to complete.
After the swap completed we can analyze the Request Tracing logs to confirm that the warmup-cache.php URL has been hit prior to the swap. Note that we need to get those log files from the site in the production slot now.
Looking through the logs we can see the following:
- The first warm up request is made to the web site. Notice the user agent value set to “SiteWarmup”:
- Another log file shows that the appInit module has started and made an HTTP request to the warmup-cache.php around the same time. That request has the host name I specified in the web.config file. Also the user-agent is different.
As expected that request took around 30 seconds.
That simple example demonstrates how to use IIS Application Initialization Module with Azure Web App deployment slots to ensure that the application in the slot is completely warmed up.
Great article, thanks!
One question, though: is targeting a specific host name safe? Could that result in not all of your instances being warmed up – given you have more than one?
Mickaël. You could inject deployment params into your deployment for each environment you have. These could flow through to your web.config and host name in the initializationPage section meaning the host names will be relevant to the einvironment being warmed up.
Thanks for the details. This has been very helpful. I have one follow up question.
After warm up when the host names are swapped, what happens to the traffic that is currently in the production slot? We are deploying a web app with very high traffic and want to make sure that the users do not have a bad experience when the swapping is done. Do some requests get dropped during the swapping process?
Hi Vishnu,
Are you able to swap the traffic without any failures to the existing traffic in production?
Hi Rusany,
Thanks for the very helpful article!
We have tried this out and found out that during swapping, warm up requests are sent to the wrong slot.
We are sure of this because we did some logging to db in the warmup page controller and the logs definitely belonged to the production code, not the development code.
Could there be a bug?
Hi Selim, that should not happen and if you see that happening, then this sounds like a bug. Can you let me know your site name where you can repro this problem? Just send me a private message here http://ruslany.net/contact/
Hi Vishnu,
No requests should be dropped during swap. If there are any requests that are currently being processed by the site in production slot then will still complete.
nice article about azure web app
I’m trying to find a solution to warming up when swapping to production so I’ve been reading trough your article but there are some strange things going on.
I enabled the logging for all the status codes so I can see my warmup requests and they seem to respond with a 301 status code right away. Instead my site goes already in production and it’s down for a couple of minutes after the first request.
Here’s the log: https://gist.github.com/nickdima/d01e3a6114eee1895d40
Any ideas?
I have been trying to install the applicationInitialization on our site.
It does work, but I need to hit some pages that are behind a login page.
In my /Cache URL request, I make a WebRequest to /Login, get the CSRF token, make a login request, get the cookie. Then I make 20 more requests with a WebRequest to the private pages.
The problem here is I get the following error:
Unexpected Error in Application: System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions 127.0.0.1:80
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
Also, the website will give me a YSOD, then a 502 from the server.
After a while, the site becomes available but does did not warm up appropriately. Definitely crashed during my Cache warm up.
In order to get the correct path to hit, I use the following:
return Request.Url.Scheme + System.Uri.SchemeDelimiter + Request.Url.Host + (Request.Url.IsDefaultPort ? string.Empty : “:” + Request.Url.Port);
But I suspect there might be a problem with this?
I get the following:
http://localhost
Like Mickaël mentioned on your blog, I am afraid to use the “hostName” property in the WebConfig as it would not warm up my instances properly.
Is there a way to achieve this thing without any problems?
Thanks!
Hi,
is it possible to create a “warm-up” for Java services? I am using an Azure API Service with Tomcat.
Regards
Sebastian
Hi,
Do the initialization pages get hit when deploying to a single instance? I am trying to test this in my test/dev website but warm up never runs.
Thanks,
Carlos
Hi Carlos, the initialization page should be hit even if web app is on a single instance.
Hi RuslanY
I don’t know if this is a dumb question, but before the sites are swapped, are the requests already made to the production slot allowed to complete? I’m thinking of a situation where a request takes a few seconds, maybe calling an API or processing a photo. What happens here?
I am also getting issues with scaling and warm up, I don’t want to set hostname since I could be running multiple instances and I don’t know if the one needing warm up would be hit. Does anyone have a solution for this? It seems that localhost is forbidden and not getting hit.
I’m having SingalR clients all disconnected on slot swaps. Is there a way to prevent that? Or maybe a way to fire an action on the replaced slot to inform SignalR clients to request a new connection?
Hi Ruslan,
We have several web apps that we would like to have warmed up and swapped almost at the same time. For instance, we have a frontend website and several api websites that should be available in production at the same time and with minimum downtime.
We haven’t found a smart way to do this with slots, so right now we are using the Switch-AzureRmWebAppSlot cmdlet in following way:
1. Call Switch-AzureRmWebAppSlot with ApplySlotConfiguration for all 5 sites async/in parallel.
2. When all 5 cmds are completed successfully, we issue the cmd CompleteSlotSwap for all 5 sites async/in parallel.
The first step takes around 4 secs. for all 5 websites and the second step takes around 90 secs.
This indicates to me that the ApplySlotConfiguration will complete successfully before the sites are actually warmed up. Likewise, as we start the CompleteSlotSwap right after all 5 sites have completed ApplySlotConfiguration, we are actually trying to complete the swap while the slot is warming up.
Is that assumption correct?
Would we have to implement a manual warmup request to each site ourselves before calling CompleteSlotSwap or can we query the azure management rest api somehow to know when the warmup has completed?
Do you have another (and better) way to support our scenario with slot swap for multiple websites in batch?
Thanks.
I’m having SingalR clients all disconnected on slot swaps. Is there a way to prevent that? Or maybe a way to fire an action on the replaced slot to inform SignalR clients to request a new connection?
Hey is there a way to validate the XML and check for errors before applying it to my instance? Does Azure automatically check?
How do we do this with web app for containers?
When I swap slots for a multi container webapp (using compose), I notice that the service is up, but the underlying containers haven’t started up. Every deployment basically causes a brief outage while the containers start up. I haven’t found an easy way to check if the underlying containers are up and then swap the slot.
Hi,
I am trying to use the warmup feature but I can’t warm up most of my endpoints since they are behind oauth. I am trying to connect to them using an internal HttpClient with a test token but I get an exception:
Unexpected Error in Application: System.Net.Sockets.SocketException (0×80004005): An attempt was made to access a socket in a way forbidden by its access permissions 127.0.0.1:80
What address/port I have to use to connect while swapping?
Thanks
Just note, you WILL lose session state during the swap, so if you’re using in-proc session, those people may be affected.
So, I actually thought I’d found the answer with “Swap with preview” via PowerShell but it doesn’t appear to work as expected.
I am using the PowerShell command to initiate the swap with preview:
$ParametersObject = @{targetSlot = "[slot name – e.g. “production”]"}
Invoke-AzResourceAction -ResourceGroupName [resource group name] -ResourceType Microsoft.Web/sites/slots -ResourceName [app name]/[slot name] -Action applySlotConfig -Parameters $ParametersObject -ApiVersion 2015-07-01
and then following that with the command to swap:
$ParametersObject = @{targetSlot = "[slot name – e.g. “production”]"}
Invoke-AzResourceAction -ResourceGroupName [resource group name] -ResourceType Microsoft.Web/sites/slots -ResourceName [app name]/[slot name] -Action slotsswap -Parameters $ParametersObject -ApiVersion 2015-07-01
Reading the preview section in the docs would lead me to believe that this would do the multi-phase swap but if I look at my activity logs after doing this I see that the slot is warmed up in the second request, not the first. As a result, the first PowerShell command finishes quickly whereas the second does not; the opposite of what I expected.
Can you shed some light on this please? Or is there another way?
The idea of the swap with preview is that during the first step the configuration settings from the production slot are applied to the staging slot. After that you can manually browse to the site to make sure it works, or run some automated tests on it, etc. You also can put some load on it and hit all the important URLs in order to warm it up. Basically you warm it up using any method you prefer. After that you complete the swap. We will do the automated warm up sequence during this step anyway, but it should be very fast since the site is already warmed up.
Is there any other possibility that restart of azure web app happens other than slot setting? Our app always restarts on swap but can’t find a reason why.
The webapp in the staging slot will most likely restart during swap. This is necessary in order to apply the production configuration settings to it before it gets swapped into production slot. The webapp in the production slot should never restart.
According to this article, the Application Initialization module will fail when the web site requires SSL: https://support.microsoft.com/en-us/help/2843964/application-initialization-module-fails-when-web-site-requires-ssl
Does this apply to the “TLS/SSL settings” > “HTTPS Only” (On/Off) setting also? I.e. if we set “HTTPS Only” to “On” then warmup requests will not work? (The ARM template property is “httpsOnly”: true)
Andreas, no this does not apply to the HTTPS Only setting. The AppInit warm up should still work if you use this setting.
I haven’t found any documentation of what the success criteria are. If my status endpoint returns a response immediately but with an HTTP status of 500 InternalServerError, will that instance be put in service or will app initialization try again?
Tom – by default as long as web app returns some response with any status code – the swap will proceed. You can change that behavior by using the app setting described in Azure App Service Deployment Slots Tips and Tricks.
I can confirm appInit is running by enabling Tracing Failed requests. however, site is not warming up. It’s a sitecore site and I am specifying the home page. Could the issue be relatad to appInit not rendering the whole page ?
It maybe that the sitecore page that is hit during warmup does not involve executing the server side code? Maybe it just responds with http 301 or 304 status code? You can also try to specify the specific warm up URL and expected status code as described in Azure App Service Deployment Slots Tips and Tricks.
Ruslan
Our site takes some time to load and setting the warmup path and the status code fails my swap. I believe the startup watcher times out before we’re up and running and able to provide a 200. Does the warm up path get hit repeatedly until until the required status code is returned or will it try to access the path one time and then fail if 200 isn’t provided? If it only tries to execute the warm up path once, is there a way to specify multiple warm up attempts before failing?
Hi Evan, the timeout for the warmup path is 90 seconds. It is retried 5 times and if all 5 times the requests time out then the swap fails. If it takes so long time for an application to start up then maybe swap with preview would be a better option? That way you will initiate the first step of the swap which will apply production settings on the site in the staging slot and will restart the site. Then you can make http requests to all the warm up paths on that site. After you’ve made sure it is up and running then you complete the swap. The warmed-up site will be moved to production slot without restarting again.
Hi Ruslan!
We got an .NET Core 3.1 API running on two P2V2 instances. We have the application warmup in place as you described. By writing to the logging we can see the warmup being initialized. We warmup the following:
– Identity Sever 4 (generating tokens)
– Azure Key Vault (load encryption keys)
– Redis and Memory Cache (pre-load caches)
– Entity Framework Core (making a few queries)
However when we are under heavy peak/burst load and we are scaling out, there are still a few requests taking more then 30 seconds making the app unstable and queues filling up.
I thoroughly tested the effects of warmup by logging request/execution times and I see that the first request takes many seconds and the subsequent test-calls are finished ms.
So this makes me doubt the Azure infrastructure (PaaS) limiting something here?
Hi Eric,
For the cases when first request takes many seconds – do you see the prior warmup requests before that? Or the URLs are never warmed up? Also when you talk about many seconds – how many is it – tens or hundreds?
Hello,
we currently encounter the problem that only three of our four instances get the call for the swap-endpoint we defined and then the swap just fails. I checked the http-logs in Kudu and can see that three of the instances get the “GET /swap” from “SiteWarmup” while the fourth instance just has “GET /” from AlwaysOn and HostnameSyncPinger during that time. So that instance is up and serving requests, but didn’t get any for the Swap-Endpoint.
Any idea how we could solve this or who to talk to to get this solved?
Thx
Sebastian
Great article.
Also one doubt regarding the swap operation
while we are swapping from stage to prod
how about the files in the root folder will they also be swapped like the App_data folder under the wwwroot of both the webapp slots staging and prod.
Thanks,
Midhun
The swap operation is essentially a DNS swap between “primary” and “deployment” slots … the file systems are 100% independent of each other and after your deployment slot is “swapped”.
For simple example, I create a deployment slot and its filesystem … deploy my updated code to that slot … later, swap the staging slot with primary webapp … afterwards, my updated code is now on the primary slot … and my previous software version is now under the “staging” deployment slot.
Also, for the questions about restarts as part of the swap operation … in my experience, it restarts the endpoints while swapping … only in cases were the “WebApp configuration settings” themselves change.
Otherwise, the swap operation can (in theory) be done with NO impact on traffic … as existing traffic to either the deployment slot and/or primary instance is allowed to complete … then redirected by utilizing DNS to the “new” destination.
Hi Ruslan,
Does either the App initialisation or WEBSITE_SWAP_WARMUP_PING_PATH work with Azure functions? We’re finding that when we swap slots on our functions app, were getting a brief 503 unavailable and I haven’t been able to get either of App Initialisation or WEBSITE_SWAP_WARMUP_PING_PATH working for Azure functions.