Rewrite

This is a post about code refactoring, the long journey of removing technical debts in the system I have built at work.

Since late 2015, I have been working on a system which is supposed to be extensible and scalable to support our branches in different countries. With the frequent change of user requirements and business strategies, mistakes and system-wide code refactoring are things that I can’t avoid.

Today, I’m going to list down some key lessons and techniques I learned in this one year plus of journey of clearing technical debts.

…we are not living in a frozen world of requirements and changes. So you can’t and shouldn’t avoid refactoring at all.

The risk in avoiding would definitely bring up messy codes, and tough maintenance of your software, you can leave it behind but somebody else would suffer.

— Nail Yuce, Author of the redbook “Collaborative Application Lifecycle Management”

Databases Organization

Originally, we have three branches in three countries. The branches are selling the same product, laptop. It’s business decision that these three branches should behave individually with different names so that customers generally can’t tell that these three branches are operated by one company.

Let’s say these three branches have names as such, Alpha, Beta, and Gamma. Instead of setting up different databases for the branches, I put the tables of these three branches in the same database for two reasons:

  1. Save cost;
  2. Easy to maintain because there will be only one database and one connection string.

These two points turn out to be invalid few months after I designed the system in such a way.

I’m using Azure SQL, actually separating databases won’t incur higher cost because of the introduction of Elastic Database Pool in Microsoft Azure. It’s also not easy to maintain because to put three business entities in one database, I have two ways to do it.

  1. Have a column in each table specifying the record is from/for which branch;
  2. Prefix the table names.

I chose the 2nd way. Prefix the table names. Hence, instead of a simple table name such as Suppliers, I now have three tables, APSupplier, BTSupplier, and GMSupplier, where AP, BT, and GM are the abbreviation of the branch names.

This design decision leads to the second problem that I am going to share with you next.

Problem of Prefixing Table Names

My senior once told me that experience was what made a software developer valuable. The “experience” here is not about technical experience because technology simply moves forward at a very fast pace, especially in web development.

The experience that makes a software developer valuable is more about system design and decision making in the software development process. For example, now I know prefixing table names in my case is a wrong move.

TooYoungTooSimple
Experience helps in building a better system which is not “too simple and naive”.

There are actually a few obvious reasons of not prefixing.

For those who are using Entity Framework and love intellisense feature in Visual Studio IDE, they will know my pain. When I’m going to search for Supplier table, I have to type the two-letter branch abbreviation first then search for the Supplier table. Hence in the last one year, our team spent lots of man hours going through all these AP, BT, and GM things. Imaging the company starts to grow to 20 countries, we will then have AP, BT, GM, DT (for Delta), ES (for Epsilon), etc.

To make things worse, remember that three branches are actually just selling the laptops with the similar business models? So what would we get when we use inheritance for our models? Many meaningless empty sub-classes.

public abstract class BaseSupplier
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string PersonInCharge { get; set; }
    public string Email { get; set; }
    public bool Status { get; set; }
}

public class APSupplier : BaseSupplier { }

public class BTSupplier : BaseSupplier { }

public class GMSupplier : BaseSupplier { }

So if we have branches in 20 countries (which is merely 10% of the total number of countries in the world), then our software developers’ life is going to be miserable because they need to maintain all these meaningless empty classes.

Factory Design Pattern and Template Methods

However, the design above actually at the same time also makes our system to be flexible. Now, imagine that one of the branches requests for a system change which requires addition of columns in its own Supplier table, we can simply change the corresponding sub-class without affecting the rest.

It-Is-Not-Bad-If-You-Know-How-To-Use_It
Design Patterns are good if we know how to use them properly. (Image Source: Rewrite)

This leads us to the Factory Design Pattern. Factory Design Pattern allows us to standardize the system design for each of the branch in the same system while at the same time allowing for individual branch to define their own business models.

public abstract class SupplierFactory
{
    public static SupplierFactory GetInstance(string portal)
    {
        return Activator.CreateInstance(Type.GetType($"Lib.Factories.Supplier.{portal}SupplierFactory")) as SupplierFactory;
    }

    protected abstract BaseSupplier CreateInstanceOfSupplier();

    protected abstract void InsertSupplierToDatabase(BaseSupplier newSupplier);

    public abstract IQueryable RetrieveSuppliers();

    public async Task AddNewSupplierAsync(SupplierManageViewModel manageVM)
    {
        ...
        var newSupplier = CreateInstanceOfSupplier();
        newSupplier.Name = manageVM.Name;
        newSupplier.PersonInCharge = manageVM.PersonInCharge;
        newSupplier.Email= manageVM.Email;
        newSupplier.Status = manageVM.Status;
        InsertSupplierToDatabase(newSupplier);
    }

    ...
}

For each of the branch, then I define their own SupplierFactory which inherits from this abstract class SupplierFactory.

public class AlphaSupplierFactory : SupplierFactory
{
    private AlphaDbContext db = new AlphaDbContext();

    public override IQueryable RetrieveSuppliers()
    {
        return db.APSuppliers;
    }

    protected override void InsertSupplierToDatabase(BaseSupplier newSupplier)
    {
        db.APSuppliers.Add((APSupplier)newSupplier);
    }

    ...
 }

As shown in the code above, firstly, I no longer use the abbreviation for the prefix of the class name. Yup, having abbreviation hurts.

Secondly, I have also split the big database to different smaller databases which store each branch’s info separately.

The standardization of the workflow is done using Template Method such as AddNewSupplier method shown above. The creation of new supplier will be using the same workflow for all branches.

Reflection

public static SupplierFactory GetInstance(string portal)
{
    return Activator.CreateInstance(Type.GetType($"Lib.Factories.Supplier.{portal}SupplierFactory")) as SupplierFactory;
}

For those who wonder what I am doing with the Activator.CreateInstance in the GetInstance method, I use it to create an instance of the specified type that type’s default constuctor with portal acts as an indicator on which sub-class the code should pick using reflection with the Type.GetType method. So the values for portal will be Alpha, Beta, Gamma, etc. in my case.

This unfortunately adds one more burden to our development team to pay attention to the naming convention of the classes.

Fluent API: ToTable

All these unnecessary complexity is finally coming to an end after my team found out how to make use of the Fluent API. Fluent API provides several important methods to configure entities which help to override Code First conventions. ToTable method is one of them.

ToTable helps mapping entity to the actual table name. Hence, now we can fix our naming issues in our databases with the codes below in each of the branch’s database context class.

protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
    base.OnModelCreating(modelBuilder);

    modelBuilder.Entity().ToTable("APSuppliers");
    ... // mappings for other tables
 }

With this change, we can furthermore standardize the behavior and workflow of the system for each of our branch. However, since we only start to apply this after we have expanded our business to about 10 countries, there will be tons of changes waiting for us if we are going to apply Fluent API to refactor our codes.

Technical Debts

I made a few mistakes in the beginning of system design because of lacking of experience building extensible and scalable systems and lack of sufficient time and spec given to do proper system design.

This naming issue is just one of the debts we are going to clear in the future. Throughout the one year plus of system development, the team also started to realize many other ways to refactor our codes to make it more robust.

As a young team in a non-software-development environment, we need to be keen on self-learning but at the same time understand that we can’t know everything. We will keep on fighting to solve the crucial problems in the system and at the same time improving the system to better fit the changing business requirements.

I will discuss more about my journey of code refactoring in the future posts. Coming soon!

I-Was-Such-An-Idiot-Back-Then.png
Reading my old codes sometimes makes me slap myself. (Image Source: Rewrite)
Advertisements

Never Share Your Secrets (Secret Manager and Azure Application Settings)

secret-manager-tool-azure-app-service-2

It’s important to keep app secrets out of our codes. Most of the app secrets are however still found in .config files. This way of handling app secrets becomes very risky when the codes are on public repository.

Thus, they are people put some dummy text in the .config files and inform the teammates to enter their respective app secrets. Things go ugly when this kind of “common understanding” among the teammates is messed up.

i-made-a-mistake-cannot-be-reversed
The moment when your app secrets are published on Github public repo. (Image from “Kono Aozora ni Yakusoku o”)

Secret Manager Tool

So when I am working on the dotnet.sg website, which is an ASP .NET Core project, I use the Secret Manager tool.It offers a way to store sensitive data such as app secrets in our local development machine.

To use the tool, firstly, I need to add it in project.json as follows.

{
    "userSecretsId": "aspnet-CommunityWeb-...",
    ...
    "tools": {
        ...
        "Microsoft.Extensions.SecretManager.Tools": "1.0.0-preview2-final"
    }
}

Due to the fact that the Secret Manager tool makes use of project specific configuration settings kept in user profile, we need to specify a userSecretsId value in the project.json as well.

After that, I can start storing the app secrets in the Secret Manager tool by entering the following command in the project directory.

$ dotnet user-secrets set AppSettings:MeetupWebApiKey ""

Take note that currently (Jan 2017) the values stored in the Secret Manager tool are not encrypted. So, it is just for development only.

As shown in the example above, the name of the secret is “AppSettings:MeetupWebApiKey”. This is because in the appsettings.json, I have the following.

{
    "AppSettings": {
        "MeetupWebApiKey": ""
    },
    ...
}

Alright, now the API key is stored in the Secret Manager tool, how is it accessed from the code?

By default, appsettings.json is already loaded in startup.cs. However, we still need to add the following bolded lines in startup.js to enable User Secrets as part of our configuration in the Startup constructor.

public class Startup
{
    public Startup(IHostingEnvironment env)
    {
        var builder = new ConfigurationBuilder()
            .SetBasePath(env.ContentRootPath)
            .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
            .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);
            
        if (env.IsDevelopment())
        {
            builder.AddUserSecrets();
        }

        builder.AddEnvironmentVariables();

        Configuration = builder.Build();
    }
    ...
}

Then in the Models folder, I create a new class called AppSettings which will be used later when we load the app secrets:

public class AppSettings
{
    public string MeetupWebApiKey { get; set; }

    ...
}

So, let’s say I want to use the key in the HomeController, I just need to do the following.

public class HomeController : Controller
{
    private readonly AppSettings _appSettings;

    public HomeController(IOptions appSettings appSettings)
    {
        _appSettings = appSettings.Value;
    }

    public async Task Index()
    {
        string meetupWebApiKey = _appSettings.MeetupWebApiKey;
        ...
    }
    
    ...
}

Azure Application Settings

Just now Secret Manager tool has helped us on managing the app secrets in local development environment. How about when we deploy our web app to Microsoft Azure?

For dotnet.sg, I am hosting the website with Azure App Service. What so great about Azure App Service is that there is one thing called Application Settings.

Screen Shot 2017-01-29 at 11.19.42 PM.png
Application Settings option is available in Azure App Service.

For .NET applications, the settings in the “App Settings” will be injected into the AppSettings at runtime and override existing settings. Thus, even though I have empty strings in appsettings.json file in the project, as long as the correct values are stored in App Settings, there is no need to worry.

Thus, when we deploy web app to Azure App Service, we should never put our app secrets, connection strings in our .config and .json files or even worse, hardcode them.

Application Settings and Timezone

Oh ya, one more cool feature in App Settings that was introduced in 2015 is that we can change the server time zone for web app hosted on Azure App Service easily by just having a new entry as follows in the App Settings.

WEBSITE_TIME_ZONE            Singapore Standard Time

The setting above will change the server time zone to use Singapore local time. So DateTime.Now will return the current local time in Singapore.

References

If you would like to read more about the topics above, please refer to following websites.

Front-end Development in dotnet.sg

yeoman-bower-npm-gulp

The web development team in my office at Changi Airport is a rather small team. We have one designer, one UI/UX expert, and one front-end developer. Sometimes, when there are many projects happening at the same time, I will also work on the front-end tasks with the front-end developer.

In the dotnet.sg project, I have chance to work on front-end part too. Well, currently I am the only one who actively contribute to the dotnet.sg website anyway. =)

Screen Shot 2017-01-29 at 12.49.23 AM.png
Official website for Singapore .NET Developers Community: http://dotnet.sg

Tools

Unlike the projects I have in work, dotnet.sg project allows me to choose to work with tools that I’d like to explore and tools that helps me work more efficiently. Currently, for the front-end of dotnet.sg, I am using the following tools, i.e.

  • npm;
  • Yeoman;
  • Bower;
  • Gulp.

Getting Started

I am building the dotnet.sg website, which is an ASP .NET Core web app, on Mac with Visual Studio Code. Hence, before I work on the project, I have to download NodeJs to get npm. The npm is a package manager that helps to install tools like Yeoman, Bower, and Gulp.

After these tools are installed, I proceed to get a started template for my ASP .NET Core web app using Yeoman. Bower will then follow up immediately to install the required dependencies in the web project.

screen-shot-2017-01-28-at-9-03-10-pm
Starting a new ASP .NET Core project with Yeoman and Bower.

From Bower with bower.json…

Working on the dotnet.sg project helps me to explore more. Bower is one of the new things that I learnt in this project.

To develop a website, I normally make use of several common JS and CSS libraries, such as jQuery, jQuery UI, Bootstrap, Font Awesome, and so on. With so many libraries to manage, things could be quite messed up. This is where Bower comes to help.

Bower helps me to manage the 3rd party resources, such as Javascript libraries and frameworks, without the need to locate the script files for each resources myself.

For example, we can do a search of a library we want to use using Bower.

Screen Shot 2017-01-28 at 9.44.47 PM.png
Search the Font Awesome library in Bower.

To install the library, for example Font Awesome in this case, then with just one command, we can easily do it.

$ bower install fontawesome

The libraries will be installed in the directory as specified in the Bower Configuration file, .bowerrc. By default, the libraries will be located at the lib folder in wwwroot.

screen-shot-2017-01-28-at-10-08-44-pm
Downloaded libraries will be kept in wwwroot/lib as specified in .bowerrc.

Finally, to check the available versions of a library, simply use the following command to find out more about the library.

$ bower info fontawesome

I like Bower because checking bower.json into the source control ensures that every developer in the team has exactly the same code. On top of that, Bower also allows us to lock the libraries to a specific version. This will thus prevent some developers to download some different version of the same library from different sources themselves.

…to npm with package.json

So, now some of you may wonder, why are we using Bower when we have npm?

Currently, there are also developers supporting the act to stop using Bower and switch to npm. Libraries such as jQuery, jQuery UI, and Font Awesome, can be found on npm too. So, why do I still talk about Bower so much?

Screen Shot 2017-01-28 at 11.30.58 PM.png
Searching for packages in npm.

For ASP .NET Core project, I face a problem on referring to node_module from the View. Similar as Bower, npm will position the downloaded packages in a local folder also. The folder turns out to be node_module, which is on the same level as wwwroot folder in the project directory.

As ASP .NET Core serves the CSS, JS, and other static files from the wwwroot folder which doesn’t have node_module in it, the libraries downloaded from npm cannot be loaded. One way will be using Gulp Task but that one is too troublesome for my projects so I choose not to go that way.

Please share with me how to do it with npm in an easier way than with Bower, if you know any. Thanks!

Goodbye, Gulp

I first learnt Gulp was when Riza introduced it one year ago in .NET Developers Community Singapore meetup. He was then talking about the tooling in ASP .NET Core 1.0 projects.

Riza Talking about Gulp.png
Riza is sharing knowledge about Gulp during dotnet.sg meetup in Feb 2016.

However, about four months after the meetup, I came to a video on Channel9 announcing that the team removed Gulp from the default ASP .NET template. I’m okay with this change because using BundleMinifier to do bundling and minifying of CSS and JS now without using Gulp because using bundleconfig.json in BundleMinifier seems to be straightforward.

Screen Shot 2017-01-28 at 11.59.18 PM.png
Discussion on Channel 9 about the removal of Gulp in Jun 2016.

However, the SCSS compilation is something I don’t know how to do it without using Gulp (Please tell me if you know a better way. Thanks!).

To add back Gulp to my ASP .NET Core project, I do the following four steps.

  1. Create a package.json with only the two compulsory properties, i.e. name and version (Do this step only when package.json does not exist in the project directory);
  2. $ npm install --save-dev gulp
  3. $ npm install --save-dev gulp-sass
  4. Setup the generated gulp.js file as shown below.
var gulp = require('gulp');
var sass = require('gulp-sass');

gulp.task('compile-scss', function(){
    gulp.src('wwwroot/sass/**/*.scss')
        .pipe(sass().on('error', sass.logError))
        .pipe(gulp.dest('wwwroot/css/'));
})

//Watch task
gulp.task('default', function() {
    gulp.watch('wwwroot/sass/**/*.scss', ['compile-scss']);
})

After that, I just need to execute the following command to run gulp and changes made to the .scss files in the sass directory will trigger the Gulp Task to compile the SCSS to corresponding CSS.

$ gulp

There is also a very detailed online tutorial written by Ryan Christiani, the Head Instructor and Development Lead at HackerYou, explaining each step above.

Oh ya, in case you are wondering what is the difference between –save and –save-dev in the npm commands above, I like how it is summarized on Stack Overflow by Tuong Le, as shown below.

  • –save-dev is used to save the package for development purpose. Example: unit tests, minification.
  • –save is used to save the package required for the application to run.

Conclusion

I once heard people saying that web developers were the cheap labour in software development industry because they are still having the mindset that web developers just plug-and-play modules on WordPress.

After working on the dotnet.sg project and helping out in front-end development at work, I realize that web development is not an easy plug-and-play job at all.

Azure Blob Storage and File API

Azure Blob Storage - Azure SDK - ASP .NET MVC - Entity Framework - HTML5

When my applications were hosted on Windows Azure Virtual Machines (VM), we stored the images uploaded via our web applications in the hard disks of the VMs (except the temporary disk). However, when we started load balancing, we soon encountered a problem that the uploaded images were only found in one of the VMs. So we needed to find a centralized storage for those images.

Recently, when we are using Azure PaaS (aka Cloud Service), even without load balancing, we already encounter the same issue. That is simply because the hard drives used in Cloud Service instances are not persistent. Hence, a persistent file storage on the cloud is needed.

IaaS vs. PaaS
IaaS vs. PaaS

Blob Storage

Azure Blob Storage, according to Azure Documentation, is a service for storing large amount of unstructured data that can be accessed everywhere via HTTP or HTTPS. Hence, it is an ideal tool that we can use as the persistent image cloud storage.

There are two types of blob, Page Blob and Block Blob. Page Blob is commonly used for storing VHD files for VMs because it is optimized for random read and write operations.

For most of the files uploaded, it’s recommended to store as Block Blobs because large files will be split into smaller blocks and then uploaded concurrently. Hence, Block Blob is designed to give us faster upload and better throughput, which is great for image upload.

The maximum size for a Block Blob is 64 MB. Hence, if the uploaded file is more than 64 MB, we must upload it as a set of blocks; otherwise, we will receive status code 413 (Request Entity Too Large). For my web applications, there is no need for uploading an image which is more than 5MB most of the time. Hence, I can just limit the size of images before the user uploads them.

HttpPostedFileBase imageUpload;
...
if (imageUpload.ContentLength > 0 && imageUpload.ContentLength <= 5242880)
{
    //warn the user to resize the image
}

Let’s Try Uploading Images

I’m going to share how to upload more than one image to the Azure Blob Storage from an ASP .NET MVC 5 application. If you are going to upload just one image, simply remove the for loop and change List to just DBPhoto in the codes below.

First of all, I create a class to handle upload to Azure Storage operation.

public class AzureStorage
{
    public static async Task UploadAndSaveBlobAsync(
        HttpPostedFileBase imageFile, CloudBlobContainer container)
    {
        string blobName = Guid.NewGuid().ToString() + 
            Path.GetExtension(imageFile.FileName);

        CloudBlockBlob imageBlob = container.GetBlockBlobReference(blobName);
        using (var fileStream = imageFile.InputStream) 
        {
            await imageBlob.UploadFromStreamAsync(fileStream);
        }

        return imageBlob;
    }
}

So, in my controller, I have the following piece of code which will be called when an image is submitted via web page.

[HttpPost]
[ValidateAntiForgeryToken]
public async Task Create(
    [Bind(Include = "ImageUpload")] PhotoViewModel model)
{
    var validImageTypes = new string[] { "image/jpeg", "image/pjpeg", "image/png" };
    
    if (ModelState.IsValid) 
    {
        if (model.ImageUpload != null && model.ImageUpload.Count() > 0)
        {
            var storageAccount = CloudStorageAccount.Parse 
                (WebConfigurationManager.AppSettings["StorageConnectionString"]);

            var blobClient = storageAccount.CreateCloudBlobClient();
            blobClient.DefaultRequestOptions.RetryPolicy = 
                new LinearRetry(TimeSpan.FromSeconds(3), 3);  

            var imagesBlobContainer = blobClient.GetContainerReference("images");
            foreach (var item in model.ImageUpload) 
            { 
                if (item != null) {
                    continue;
                }
                
                if (validImageTypes.Contains(item.ContentType) && 
                    item.ContentLength > 0 && item.ContentLength <= 5242880)
                {
                    var blob = await AzureStorage.UploadAndSaveBlobAsync(item, imagesBlobContainer);
                    DBPhoto newPhoto = new DBPhoto(); 
                    newPhoto.URL = blob.Uri.ToString();
                    db.DBPhoto.Add(newPhoto); 
                } 
                else 
                {
                    // Show user error message 
                    return View(model); 
                }
            }
            db.SaveChanges();
            ... 
        } 
        else
        {
            // No image to upload
        } 
    }
    return View(model);
}

In the code above, there are many new cool things.

Firstly, it is the connection string to Azure Blob Storage, which I store in StorageConnectionString in web.config. The format for secure connection string is as follows.

DefaultEndpointsProtocol=https;AccountName=;AccountKey=;
Retrieve the access keys to the Storage Account.
Retrieve the access keys to the Storage Account.

Secondly, it’s LinearRetry. It is basically a retry policy which states how many times the program will retry and how much time needed between retries. In my case, it will only wait for 3 seconds after each try up to 3 tries.

Thirdly, I get the URL of the image on the Azure Blob Storage via blob.Uri.ToString() and store it into the database table. The URL will be used later for displaying the image as well as deleting the image.

Fourthly, I actually check to see if model.ImageUpload has null entries. This is because if I submit the form without any image to upload, model.ImageUpload has one entry. Not zero, but one. The only one entry is actually null. So if I don’t check to see whether the entry in model.ImageUpload is null, there will be an exception thrown.

The controller has such a long code. Luckily the code needed in the model and view is short and simple.

For the model PhotoViewModel, I have the following.

public class PhotoViewModel
{
    ...
    
    [Display(Name = "Current Images")]
    public List AvailablePhotos { get; set; }
}

For view, it is easy to allow selecting multiple files in the same view page. The “multiple = “true”” is to make sure more than one file can be selected in the File Explorer. You can omit this attribute if you only want at most one file being selected.

@Html.LabelFor(model => model.ImageUpload, new { style = "font-weight: bold;" })
@Html.TextBoxFor(model => model.ImageUpload, new { type = "file", multiple = "true" })
@Html.ValidationMessageFor(model => model.ImageUpload)

Image Size and HttpException

The image upload function looks fine. However, when images having size larger than a certain size is uploaded, HttpException will be thrown.

There is no way that having exception would be fun too! (Image Credit: Tari Tari)
There is no way that having exception would be fun too! (Image Credit: Tari Tari)

In order to prevent DOS attacks which upload huge files to the server, IIS by default only allows files which have size less than 4MB to be uploaded. Hence, although I earlier put a check to prevent image larger than 5MB to be uploaded, the exception will still be thrown if an image of size between 4 to 5MB is uploaded.

What if we just change the if clause above to allow only at most 4MB of image being uploaded? This won’t work because the exception is already thrown before the if condition is reached.

Then, can we just increase the IIS limit from 4MB to, let’s say, 100MB or something bigger? Sure. This can work. However, it still doesn’t stop someone uploads something bigger than the limit. Also, it makes attackers easier to exhaust your server with big files. Hence, expanding the upload size restriction is not really a full solution.

If you are interested, there are many good articles online discussing about this problem. I highlight some interesting ones below.

  1. Use HttpModule to Handle File Uploads;
  2. Use RIA (Rich Internet Application) Services in Silverlight (Seriously, we are talking about Silverlight in year 2015?);
  3. SubStatusCode = 13 in IIS 7;
  4. Catch the Exception in Global.asax.

I don’t really like the methods listed above, especially the 3rd and 4th options. It’s already too late to inform the user when the exception is thrown. Could we do something at client side before the images are being uploaded?

Luckily, we have File API in HTML 5. It allows to loop through the files in JavaScript to check their size. So, after the submit button is clicked, I will call a JavaScript method to check for the size of the images before they are being uploaded.

function IsFileSizeAcceptable() {
    if (typeof FileReader !== "undefined") {
        var filesBeingUploaded = document.getElementById('ImageUpload').files;
        for (var i = 0; i < filesBeingUploaded.length; i++) {
            if (filesBeingUploaded[i].size >= 4194304) { // Less than 4MB only
                alert('The file ' + filesBeingUploaded[i].name + ' is too large. Please remove it from your selection.');
                return false;
            }
        }
    }
    return true;
}
File API is currently supported in major modern browsers. (Image Credit: http://caniuse.com/#feat=fileapi)
File API is currently supported in major modern browsers. (Image Credit: http://caniuse.com/#feat=fileapi)

Remove from Azure Blob Storage

It’s normal that files uploaded to storage will be removed later. So how are we going to implement this feature in our ASP .NET MVC 5 application?

First of all, I added the following code to my AzureStorage.cs.

public static async Task DeleteBlobAsync(Uri blobUri, CloudBlobContainer container)
{
    string blobName = blobUri.Segments[blobUri.Segments.Length - 1];
    CloudBlockBlob blobToDelete = container.GetBlockBlobReference(blobName);

    await blobToDelete.DeleteAsync(); 
}

Secondly, I just pass in the Azure Storage URL of the image that I would like to remove and then call the DeleteBlobAsync method.

Uri blobUri = new Uri();
await AzureStorage.DeleteBlobAsync(blobUri, imagesBlobContainer);

Then the image will be deleted from the Azure Storage successfully.

Global.asax.cs and Blob Container

In order to have my application to create a blob container automatically if it doesn’t already exist, I add a few lines in Global.asax.cs as follows.

var storageAccount = CloudStorageAccount.Parse(
    WebConfigurationManager.AppSettings["StorageConnectionString"]);
var blobClient = storageAccount.CreateCloudBlobClient();
var imagesBlobContainer = blobClient.GetContainerReference("images");
if (imagesBlobContainer.CreateIfNotExists())
{
    imagesBlobContainer.SetPermissions(new BlobContainerPermissions
        {
            PublicAccess = BlobContainerPublicAccessType.Blob
        });
}

Write a Console Program to Upload File to Azure Storage

So, how is it done if we are developing a console application, instead of web application?

Windows Azure Storage NuGet Package needs to be installed first.
Windows Azure Storage NuGet Package needs to be installed first.

The codes below show how I upload an html file from my local hard disk to Azure Blob Storage. Then I can share the Azure Storage URL of the file to my friends so that they can read the web page.

Similar to what I do in web application, this is how I connect to the Storage account via https.

var azureStorageAccount = new CloudStorageAccount(
    new StorageCredentials("", ""), true);

This is how I access the container.

var blobClient = new CloudBlobClient(azureStorageAccount.BlobStorageUri, azureStorageAccount.Credentials);
var container = blobClient.GetContainerReference("myfiles");

Then the next thing I do is just upload the local file to Azure Storage by specifying the file name, content type, etc.

CloudBlockBlob blob = container.GetBlockBlobReference("mysimplepage.html");
using (Stream file = System.IO.File.OpenRead(@"C:\Users\ChunLin\Documents\mysimplepage.html")) 
{
    blob.Properties.ContentType = "text/html"; 
    blob.UploadFromStream(file); 
}

Yup, that’s all. =)

Pricing

Hosting your files on cloud storage is sure convenience. However, Azure Blob Storage is not free. The following table shows the current pricing of Azure Block Blob Storage in South East Asia region. To get the latest pricing details, please visit Azure Storage Pricing page.

Azure Standard Block Blob Storage in SEA Pricing
Azure Standard Block Blob Storage in SEA Pricing

Summer 2015 Self-Learning Project

This article is part of my Self-Learning in this summer. To read the other topics in this project, please click here to visit the project overview page.

Summer Self-Learning Banner

Role Management and Social Network Login

ASP .NET MVC - Entity Framework - Facebook - Google - Twitter

Often, we need to specify the resources users in our web application are allowed to access. For example, the sales report can only be seen by managers. The control panel can only be accessed by admin of the company.

Individual User Account

In Visual Studio 2013, when we first create an ASP .NET MVC5 project, we will always have the option to choose authentication mode. One of the available modes is Individual User Account.

Individual User Account is the default Authentication method.
Individual User Account is the default Authentication method.

Individual User Account offers two channels for users to log in.

Firstly, user can register on the web application by entering email and password. The application will then create an account with the password hashed and stored in the database. Next time, the user can just log in with email and password which will be verified by the ASP .NET Identity.

Secondly, user can also register and log in with external service, such as Facebook, Twitter, and Google+. Interestingly, no password will be stored in our database for this method. Instead, the user will be authenticated by signing in to the external service.

Login to our ASP .NET web application via Twitter.
Login to our ASP .NET web application via Twitter.

Identity and Entity Framework 6 Code First

When an ASP .NET MVC 5 web application with Individual User Account as Authentication is created, a new ASP .NET Identity Provider using EF6 Code First will be added to the project as well.

Calling Code First "Code-Based Modeling" is more suitable.
Calling Code First “Code-Based Modeling” is more suitable. (Reference)

Code First APIs will create new database if no existing database attached to the web application. Code First will map our entity classes with the database using default conventions. Hence, with the Code First approach, the developers can focus on the domain design and later have the database tables created according to the entity classes.

Because of Code First, in the first run of the application which has no database attached to it, EF6 will automatically create a database. If we have attempted to access any Identity functionality, there will be following 5 tables created automatically.

  • AspNetRoles
  • AspNetUserClaims
  • AspNetUserLogins
  • AspNetUserRoles
  • AspNetUsers

Role Based Security

Besides AspNetUserClaims table, the other four tables will be used in the role based security in our ASP .NET web application.

AspNetUsers table stores the profile information of a user, such as Email, Password, and Phone Number. To add more fields to the table, simply add the new fields in ApplicationUser class in IdentityModels.cs.

public class ApplicationUser : IdentityUser
{
    ...

    public string FirstName { get; set; }
    public string LastName { get; set; }
    public DateTime DateOfBirth { get; set; }
}

To create new role, we can do the following in the Seed() method in Configuration.cs, as suggested in an online tutorial about ASP .NET Identity.

using Microsoft.AspNet.Identity;
using Microsoft.AspNet.Identity.EntityFramework;
...

internal sealed class Configuration : DbMigrationsConfiguration<ApplicationDbContext>
{
    ...

    protected override void Seed(ApplicationDbContext context)
    {
        var roleManager = 
            new RoleManager<IdentityRole>(new RoleStore<IdentityRole>(context));
       
        //Create Role Admin if it does not exist
        if (!roleManager.RoleExists("Admin"))
        {
            roleManager.Create(new IdentityRole("Admin"));
        }
    }
}

To add a user to one or many roles, we can do the following. Hence, we can assign roles to new user upon registration.

var roleManager = 
    new RoleManager<IdentityRole>(new RoleStore<IdentityRole>(context));
var roles = roleManager.Roles.ToList();

foreach(var role in roles) { 
    var isInRole = await UserManager.IsInRoleAsync(userId, role); 
    if(!isInRole) 
    { 
         await UserManager.AddToRoleAsync(userId, role); 
    }
}

So, when user is accessing a page which is allowed for members having a certain role, we first need to check if the user is logging in with the following code.

if (Request.IsAuthenticated)
{
    ...
}

Inside the IF statement, we can continue to check if the user is having a certain role, as shown in the following code.

if (Request.IsAuthenticated && User.IsInRole("Admin"))
{
    ...
}

Alternatively, if we only allow the page to be accessed by Admin user, then we can use AuthorizeAttribute.

[Authorize(Roles="Admin")]
public ActionResult Report()
{
    . . .
}

Facebook OAuth2 Authentication

As said earlier, Individual User Account allows user to log in to the web application via external service, such as Facebook, as well. Before we can use the Facebook OAuth2 authentication, we need to register as a Facebook developer (Instruction here). I have already registered as a Facebook developer few years ago, so I just start directly from the Facebook Developers page.

First of all, we will click on the “Add a New App” button to begin. Then we will choose “Website” as our platform.

Adding a new app in Facebook Developers.
Adding a new app in Facebook Developers.

Secondly, we will key in name of our web application before we can create a new Facebook App ID. After that, we will select a category for our app.

Entering app name.
Entering app name.

Thirdly, we have to provide the URL of our website. Fortunately, Facebook allows us to key in non-https localhost URL. =)

Yup, tell them about our site!
Yup, tell them about our site!

After that, we just scroll up to the top of the page and then click on the “Skip Quick Start” button. It will then bring us to a page with more details about the new Facebook App that we have just created.

Facebook App ID and App Secret can be found in the Dashboard of our app.
Facebook App ID and App Secret can be found in the Dashboard of our app.

With the App ID and App Secret, we can now put in these values to the sample codes in Startup.Auth.cs to activate Facebook login. Yup, now user can just log in to our web application with their Facebook account!

After logging in, user still need to enter their email address in order to finish the new user registration process on our website. Without doing this step, both the AspNetUserLogins and AspNetUsers tables in our database will have no record of this user.

Once the user finishes the registration, we will be able to see their info in both of the tables mentioned above. The AspNetUserLogins table will keep data such as Login Provider (Facebook), Provider Key (a reference key to Facebook users table), and UserId (which is a reference key to AspNetUsers table).

Interestingly, as Facebook says, “(The web app) may still have the data you share with them” even though we unlink the app from our Facebook account.

Link with Google

To enable user to log in to our ASP .NET website using Google account, we will head towards the Google Developers Console to configure.

In the first step, we need to give a name to our project. Next, we can just click on the “Create” button to add the project to the console.

Adding a new project in Google Developers Console.
Adding a new project in Google Developers Console.

After the project is created, we will proceed to the Credentials under the APIs & Auth section.

"You do not have sufficient permissions to view this page." What?
“You do not have sufficient permissions to view this page.” What?

If you encounter issue on viewing the Credentials page because it kept complaining “You do not have sufficient permissions to view this page”, please switch to use another browser which has no Google account already signed in. For my case, I use the new browser from Microsoft, Edge.

Create new Client ID.
Create new Client ID.

Click on the “Create New Client ID” button under OAuth. It will then ask for Application Type. For our case, it will be the default option, “Web application”.

Select application type.
Select application type.

Do you notice the little warning there saying we need to provide a Product Name? So after that, we will be brought to the Consent Screen page to fill in our Product Name. In the same page, we can also key in URL to our homepage, product logo, Google+ page, privacy policy, and ToS.

After saving the updates on Consent Screen page, we will be prompted to key in two important information: Authorized JS Origins and Authorized Redirect URIs. For local testing purpose, it accepts non-https localhost URL as well.

After that, we should receive a Client ID for our web application.

Google Client ID and Client Secret.
Google Client ID and Client Secret.

Before going back to Visual Studio, we will proceed to the APIs section under the APIs & Auth. There, we can enable the Google+ API.

Enabling Google+ API.
Enabling Google+ API.

Same as Facebook, with the Client ID and Client Secret, we can now put in these values to the sample codes in Startup.Auth.cs to activate Google login. Yup, now user can just log in to our web application with their Google account!

Interestingly, I am not able to access the Credentials page after this again. =P

Logging In with Twitter

To get the Consumer Key and Consumer Secret from Twitter, we first need to login to the Twitter Apps.

According to Twitter, we must add mobile phone number to our Twitter profile before creating an application. For the Callback URL field, although it is optional, we have to put in our localhost URL (for testing environment) first. Otherwise, we will receive 401Unauthorized Error. Also, Twitter considers “localhost” as invalid in URL, so we have to use “127.0.0.1” instead.

After creating the new app, we will be given the Consumer Key and Consumer Secret that we can use to put in our Startup.Auth.cs.

Twitter Customer Key and Customer Secret.
Twitter Customer Key and Customer Secret.

More External Services Providing Login

If you would like to read more about allowing user to login to your ASP .NET website with 3rd party services, I would like to suggest a few articles to you.

Customizing Association Form

As mentioned earlier, we can modify the AspNetUsers table to store other profile information of a user by adding new fields in ApplicationUser class in IdentityModels.cs.

public class ApplicationUser : IdentityUser
{
    ...

    public string FirstName { get; set; }
    public string LastName { get; set; }
    public DateTime DateOfBirth { get; set; }
}
Association Form
Association Form

For external login, we need to update the fields to the Association Form as well so that no matter where the user comes from, we will always capture the same set of user info.

Firstly, in the AccountViewModels.cs, we need to add the three new fields to the ExternalLoginConfirmationViewModel.

public class ExternalLoginConfirmationViewModel
{
    [Required]
    [Display(Name = "Email")]
    public string Email { get; set; }

    [Required]
    [Display(Name = "First Name")]
    public string FirstName { get; set; }

    [Required]
    [Display(Name = "Last Name")]
    public string LastName { get; set; }

    [Required]
    [Display(Name = "Date of Birth")]
    public DateTime DateOfBirth { get; set; }
}

Then we will update the Views accordingly to enable user to key in those info.

In AccountController.cs, we will then add in logic to ExternalLoginConfirmation HttpPost method to store data of the three new fields into the AspNetUsers table.

var user = new ApplicationUser {
    ...
    FirstName = model.FirstName,
    LastName = model.LastName,
    DateOfBirth = model.DateOfBirth
};

If you are still not clear about what I am writing here, please read a more detailed tutorial written by Rick Anderson about adding new fields to the Association Form.

Summer 2015 Self-Learning Project

This article is part of my Self-Learning in this summer. To read the other topics in this project, please click here to visit the project overview page.

Summer Self-Learning Banner

Entity Framework and Database

By using Entity Framework, we can save a lot of time on writing SQL ourselves because Entity Framework, a Microsoft-supported ORM for .NET, is able to generate the SQL for us.

I started to use ADO .NET when I was building .NET web applications in my first job. I learnt about how to call stored procedures with ADO .NET. I witnessed how my colleague wrote a 400-line SQL to complete a task which we normally will choose to do it in C#. I also realized the pain of forgetting to update the stored procedure when the C# code is already different.

After that, my friend introduced me Entity Framework when I was working on my first ASP .NET MVC project. Since then, I have been using Entity Framework because it enables me to deliver my web applications faster without writing (and debugging) any SQL myself. I read a very interesting article comparing between Entity Framework and ADO .NET. The author also acknowledged that the performance of Entity Framework was slower than hand-coded ADO .NET. He emphasized that, however, Entity Framework did maximize his productivity.

How I react when I read a 400-line stored procedure submitted by my colleague.
How I react when I read a 400-line stored procedure submitted by my colleague.

What Is Happening in Database with Entity Framework?

The SQL generated by Entity Framework is believed to be pretty good. However, it’s still nice to be aware of what SQL is being generated. For example, I have the following code to retrieve Singapore weather info.

using (var db = new ApplicationDbContext())
{
    var forecastRecords = db.SingaporeWeathers.ToList();
}

In Visual Studio, I can just mouse-over “SingaporeWeather” to get the following query.

SELECT 
    [Extent1].[RecordID] AS [RecordID], 
    [Extent1].[LocationID] AS [LocationID], 
    [Extent1].[WeatherDescription] AS [WeatherDescription], 
    [Extent1].[Temperature] AS [Temperature], 
    [Extent1].[UpdateDate] AS [UpdateDate]
FROM [dbo].[SingaporeWeathers] AS [Extent1]

If I have the following code which retrieves only records having temperature greater than 37, then I can use ToString().

using (var db = new ApplicationDbContext())
{
    var query = from sw in db.SingaporeWeathers where sw.Temperature > 37 select sw;
    Console.WriteLine(query.ToString());
}
SELECT
     [Extent1].[RecordID] AS [RecordID],
     [Extent1].[LocationID] AS [LocationID],
     [Extent1].[WeatherDescription] AS [WeatherDescription]
     [Extent1].[Temperature] AS [Temperature],
     [Extent1].[UpdateDate] AS [UpdateDate]
FROM [dbo].[SingaporeWeathers] AS [Extent1]
WHERE [Extent1].[Temperature] > cast(37 as decimal(18))

I am using DBContect API, so I can just use ToString(). Alternatively, you can also use ToTraceString(), which is a method of ObjectQuery, to get the generated SQL.

SQL Logging in Entity Framework 6

It is a great news for developer when Entity Framework is announced to have SQL Logging feature added For example, to write database logs to a file, I just need to do as follows.

using (var db = new ApplicationDbContext())
{
    var logFile = new StreamWriter("C:\\temp\\log.txt");
    db.Database.Log = logFile.Write;
    var forecastRecords = db.SingaporeWeathers.Where(x => x.Temperature > 37).ToList();
    logFile.Close();
}

Then in the log file, I can see logs as follows.

...
Closed connection at 6/6/2015 10:59:32 PM +08:00
Opened connection at 6/6/2015 10:59:32 PM +08:00
SELECT TOP (1) 
    [Project1].[C1] AS [C1], 
    [Project1].[MigrationId] AS [MigrationId], 
    [Project1].[Model] AS [Model], 
    [Project1].[ProductVersion] AS [ProductVersion]
FROM ( SELECT 
    [Extent1].[MigrationId] AS [MigrationId], 
    [Extent1].[Model] AS [Model], 
    [Extent1].[ProductVersion] AS [ProductVersion], 
    1 AS [C1]
    FROM [dbo].[__MigrationHistory] AS [Extent1]
    WHERE [Extent1].[ContextKey] = @p__linq__0
) AS [Project1]
ORDER BY [Project1].[MigrationId] DESC
-- p__linq__0: 'MyWeb.Migrations.Configuration' (Type = String, Size = 4000)
-- Executing at 6/6/2015 10:59:32 PM +08:00
-- Completed in 70 ms with result: SqlDataReader

Closed connection at 6/6/2015 10:59:32 PM +08:00
Opened connection at 6/6/2015 10:59:32 PM +08:00
SELECT 
    [Extent1].[RecordID] AS [RecordID], 
    [Extent1].[WeatherDate] AS [WeatherDate], 
    [Extent1].[WeatherDescription] AS [WeatherDescription], 
    [Extent1].[WeatherSecondaryDescription] AS [WeatherSecondaryDescription], 
    [Extent1].[IconFileName] AS [IconFileName], 
    [Extent1].[Temperature] AS [Temperature], 
    [Extent1].[UpdateDate] AS [UpdateDate]
FROM [dbo].[Weathers] AS [Extent1]
WHERE [Extent1].[Temperature] > cast(37 as decimal(18))
-- Executing at 6/6/2015 10:59:33 PM +08:00
-- Completed in 28 ms with result: SqlDataReader
...

So, as you can see, even the Code First migration related activity is logged as well. If you would like to know what are being logged, you can read an article about SQL Logging in EF6 which was written before it’s released.

Migration and the Verbose Flag

Speaking of Code First migration, if you would like to find out the SQL being generated when Update-Database is executed, you can add a Verbose flag to the command.

Update-Database -Verbose

Navigation Property

“I have no idea why tables in our database don’t have any relationship especially when we are using relational database.”

I heard from my friend that my ex-colleague shouted this in the office. He left his job few days after. I think bad codes and bad design do anger some of the developers. So, how do we do “relationship” in Entity Framework Code First? How do we specify the foreign key?

I quit!
I quit!

In Entity Framework, we use the Navigation Property to represent the foreign key relationship inside the database. With Navigation Property, we can define relationship between entities.

If we have a 1-to-1 Relationship between two entities, then we can have the following code.

public class Entity1
{
    [Key]
    public int Entity1ID { get; set; }
    public virtual Entity2 Entity2 { get; set; }
}

public class Entity2
{
    [Key, ForeignKey("Entity1")]
    public int Entity1ID { get; set; }
    public virtual Entity1 Entity1 { get; set; }
}

By default, navigation properties are not loaded. Here, the virtual keyword is used to achieve the lazy loading, so that the entity is automatically loaded from the database the first time a property referring to the entity is accessed.

However, there are people against using virtual keyword because they claim that lazy loading will have subtle performance issue in the application using it. So, what they suggest is to use the include keyword, for example

dbContext.Entity1.Include(x => x.Entity2).ToArray();

By specifying the ForeignKey attribute for Entity1ID in Entity2 class, Code First will then create a 1-to-1 Relationship between Entity1 and Entity2 using the DataAnnotations attributes.

For 1-to-n Relationship, we then need to change the navigation property, for example, in Entity1 class to use collection as demonstrated in the code below.

public class Entity1
{
    [Key]
    public int Entity1ID { get; set; }
    public virtual ICollection<Entity2> Entity2s { get; set; }
}

Finally, how about n-to-m Relationship? We will just need to change the navigation property in both Entity1 and Entity2 classes to use collection.

public class Entity2
{
    [Key]
    public int Entity2ID { get; set; }
    public virtual ICollection<Entity1> Entity1s { get; set; }
}

Together with the following model builder statement.

protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
    modelBuilder.Entity<Entity2>()
        .HasMany(e2 => e2.Entity1s)
        .WithMany(e1 => e1.Entity2s)
        .Map(e12 => 
            {
                e12.MapLeftKey("Entity1ID");
                e12.MapRightKey("Entity2ID");
                e12.ToTable("Entity12");
            });
}

The code above is using Fluent API which won’t be discussed in this post.

Database Context Disposal

When I first used Scaffolding in MVC 5, I noticed the template of controller class it generates look something as follows.

public class MyController : Controller
{
    private MyContext db = new MyContext();
    
    protected override void Dispose(bool disposing)
    {
        if (disposing) 
        {
            db.Dispose(); 
        } 
        base.Dispose(disposing);
    }
}

Before using Scaffolding, I have always been using the Using block, so I only create database context where I have to, as recommended in a discussion on StackOverflow. Also, the Using block will have the Dispose() be called automatically at the end of the block, so I don’t need to worry about forgetting to include the Dispose() method to dispose the database context in my controller.

Azure SQL: Database Backup and Restore

Before ending this post, I would like to share about how DB backup and restore is done in Azure SQL Database.

First of all, Azure SQL Database has built-in backups and even self-service point in time restores. Yay!

For each activate databases, Azure SQL will create a backup and geo-replicate it every hour to achieve 1-hour Recovery Point Objective (RPO).

If there is a need to migrate the database or archive it, we can also export the database from Azure SQL Database. Simply click on the Export button in the SQL Databases section of Azure Management Portal and then choose an Azure blob storage account to export the database to.

Finally, just provide the server login name and password to the database and you are good to go.

Export DB from Azure SQL Database.
Export DB from Azure SQL Database.

Later, we can also create a new database using the BACPAC file which is being generated by the Export function. In the Azure Management Portal, click New > Data Services > SQL Database > Import. This will open the Import Database dialog, as shown in the screenshot below.

Create a new database in Azure SQL Database by import BACPAC file.
Create a new database in Azure SQL Database by import BACPAC file.

Okai, that’s all for this post on Entity Framework, database, and Azure SQL Database. Thank you for your time and have a nice day!

Summer 2015 Self-Learning Project

This article is part of my Self-Learning in this summer. To read the other topics in this project, please click here to visit the project overview page.

Summer Self-Learning Banner

PageSpeed, Bundling, Minification, and Caching

After deploying our web application, it’s not the end of the story. There are still many tasks to do. One of them is to analyze and optimize our new website. Normally, I start with testing my website in Google PageSpeed Insights to understand about the website performance issues.

Oops, my new website only gets 57/100.
Oops, my new website only gets 57/100.

Unfortunately, as shown in the screenshot above, my new website seems having a poor performance. Before finding out the ways to improve the performance, I would like to share what the 3 passed rules are.

The Things Done Right

Improve Server Response Time

Yup, the first thing that affects our web page loading is of course the server response speed. Google recommended to keep the server response time under 200ms.

My homepage loads about 600ms but PageSpeed Insights still let me pass. =)

Avoid Landing Page Redirects

Redirects will increase the loading time of the web page because there will be more HTTP request-response cycle.

In the old days when responsive design was still not popular, redirect is the way we used to bring users from desktop view to the mobile view of our website. So, when users first came in from “www.example.com”, they would be brought to “www.example.com/mobile/index.aspx”.

However, sometimes we still need to do redirects in our website, especially when we have a new web page to replace the old one. We may do redirects also when we are changing the domain name of our website.

There are two major types of HTTP Redirection.

  • 301 (Permanent Redirect): The requested resource has been assigned a new permanent URI and any future references to the resource should be done using the new URI;
  • 302 (Temporary Redirect): The requested resource resides temporarily under different URI.

Hence, 301 Redirect is able to pass the PageRank of the old page to the new page. However, for 302 Redirect, due to the fact that it’s a temporary redirect, the old page will still retain its PageRank and the new page will not accumulate any. So for mobile view redirect, we should use 302 Redirect, as recommended by Google also.

So, how do we do 301 Redirect and 302 Redirect in ASP .NET?

By default, Response.Redirect uses 302 Redirect. This is because the common use of Response.Redirect is just to move users to another page after a postback. Of course, we can change it to use 301 Redirect as follows.

Response.Redirect(newUrl, false);
Response.StatusCode = 301;
Response.End();

In ASP .NET 4.0, we have an alternative way of doing 301 Redirect which is using HttpResponse.RedirectPermanent method.

Besides, we can also do 301 Redirect in URL Rewrite in web.config.

<rewrite>
    <rules>
        <rule name="Redirect to www" stopProcessing="true">
            <match url=".*" />
            <conditions>
                <add input="{HTTP_HOST}" pattern="^domain.com$" />
            </conditions>
            <action type="Redirect" url="http://www.domain.com/{R:0}" redirectType="Permanent" />
        </rule>
    </rules>
</rewrite>

Prioritize Visible Content

One of the key principles of prioritizing visible content is to make sure the above-the-fold content is able to be loaded first.

What does “Above-the-fold Content” (ATF) mean? I found some very interesting explanation about it. ATF Content typically means the content above where the newspaper is folded horizontally. In web design context, ATF Content refers to any content we immediately see without scrolling when the page is first loaded.

Hence, in order to make sure ATF Content to be loaded first, we must have an inline part of the CSS which is responsible for styling the ATF Content.

Thanks, ASP .NET 4.5!

In ASP .NET 4.5, there are two new techniques, Bundling and Minification, introduced to help improving the web page loading time. Hence, a web project done in ASP .NET 4.5 with good back-end design should have all the 3 rules above passed in Google PageSpeed Insights.

The Things Can Be Considered Fixing

Minify CSS and JS

Here I will just use CSS as reference. Minification of JS is actually similar to minifying CSS.

Normally we will have more than one CSS references. If we would like to reference all of them, then we will make one HTTP request for each of them.

<link href='/Content/style1.css' rel='stylesheet' />
<link href='/Content/style2.css' rel='stylesheet' />
...
<link href='/Content/stylen.css' rel='stylesheet' />

In ASP .NET 4.5, we can use Bundling+Minification feature to make the loading time of CSS files shorter.

Bundling is able to improve the load time by reducing the number of requests to the server. Minification will reduce the size of requested CSS files. For example, if we want to reference all the CSS files in the “Content” folder, we will do it as follows. Doing so will tell ASP .NET to scan the “Content” folder, bundle and minify the CSS files within it. A single HTTP response with all of the CSS content will then be sent back to the browser.

<link href='/Content/css' rel='stylesheet' />

By default, CSS files are sorted alphabetically. If there are reset.css and normalize.css, the two files will go before any other files.

Alternatively, we can also do it in C# to tell ASP .NET which CSS files need to be in bundle as well as their ordering.

bundles.Add(new StyleBundle("~/Content/css").Include(
    "~/Content/bootstrap.css",
    "~/Content/animate.css",
    "~/Content/font-awesome.css",
    "~/Content/site.css"));

By default, Bundling will select “.min” file for release when both “style1.css” and “style1.min.css” exist. In debug mode, then it will select non-“.min” version. This is important because Bundling seems to have problem with bootstrap.css.

Bundling doesn’t work well with @import, @keyframes, @-webkit-keyframes that are in the CSS files. Hence, sometimes Bundling+Minification will return some error messages as follows when you view the source of /Content/css on browser.

/* Minification failed. Returning unminified contents.
    (3272,1): run-time error CSS1019: Unexpected token, found '@import'
    (3272,9): run-time error CSS1019: Unexpected token, found 'url("https://fonts.googleapis.com/css?family=Roboto:300,400,500,700")'
    (3278,83211): run-time error CSS1019: Unexpected token, found '@-webkit-keyframes'
    (3278,83255): run-time error CSS1035: Expected colon, found '{'
    (3278,83406): run-time error CSS1019: Unexpected token, found '@keyframes'
    ...
    (3672,3): run-time error CSS1062: Expected semicolon or closing curly-brace, found '0%'
    (4246,11): run-time error CSS1036: Expected expression, found ';'
*/

So one of the solutions suggested on Stack Overflow is to include both bootstrap.css and bootstrap.min.css in the same “Content” folder so that Bundling+Minification will not try (and then fail) to minify bootstrap.css itself. Instead, it will just pickup the existing bootstrap.min.css. However, we don’t need to change anything to the C# code above, so it still looks the same.

bundles.Add(new StyleBundle("~/Content/css").Include(
    "~/Content/bootstrap.css",
    "~/Content/animate.css",
    "~/Content/font-awesome.css",
    "~/Content/site.css"));

One disadvantage of doing so is that if there is any changes done to the bootstrap.css, we have to manually minify it and update the bootstrap.min.css because Minification will not do minification for us but it will just pickup the bootstrap.min.css.

For JavaScript, we will do the same to add multiple JS files into the bundle.

bundles.Add(new ScriptBundle("~/bundles/jquery").Include(
    "~/Scripts/jquery-{version}.js",
    "~/Scripts/jquery-ui.js",
    "~/Scripts/jquery.datetimepicker.js"));

The {version} wildcard above will automatically pick the appropriate version of jQuery available in the “Scripts” folder. Hence, we can happily use NuGet to update jQuery to a newer version without changing our C# code.

In the ASP .NET MVC template, there are actually two JavaScript bundles, “~/bundles/jquery” and “~/bundles/jqueryval”. The reason of doing Bundle Partitioning is to make sure the page doesn’t get the resources that it doesn’t need.

Minify HTML

To minify HTML and Razor views, there are a few options.

One is to use Meleze.Web NuGet Package. It will process the HTML and Razor views on the fly.

Another one is using ASP .NET HTML Minifier which can be included as part of the build process so that the files are minified before being deployed in Visual Studio. The simple command tool is open source and the project can be found on Github.

Enable Compression

It’s great that nowadays all the modern browsers support and automatically negotiate Gzip compression for all HTTP requests. Hence, enabling Gzip compression is able to reduce the size of transferred response.

IIS supports Static Compression and Dynamic Compression. Static Compression is used to compress static resources such as CSS, JS, images. It will compress each file once and then save the compressed file to disk. In IIS 8, Static Compression is enabled by default.

A way to check if the response is Gzipped.
A way to check if the response is Gzipped.

Enabling Dynamic Compression will help to compress the dynamic content and thus always gives us more efficient use of bandwidth (with higher CPU usage). I will not talk about Dynamic Compression here because Google PageSpeed Insights currently emphasizes more on the static file compression. So by using IIS 8, this shouldn’t be an issue.

The reason why I have this compression issue is because of the external files used are not Gzipped. However, I will leave it there first.

The Things That Should Be Fixed

Leverage Browser Caching

Fetching resources over the network is slow. Hence, it is always a good idea to cache our static resources by setting an expiry date in the HTTP headers for static resources.

Bundling in ASP .NET will help to set the HTTP Expire Header of CSS and JS files one year from when the bundle is created.

Bundles set the HTTP Expires Header one year from when the bundle is created.
Bundles set the HTTP Expires Header one year from when the bundle is created.

Alternatively, we can do so in web.config in IIS 7 and above, as shown in the sample below.

<configuration>
    <system.webServer>
        <staticContent>
            <clientCache httpExpires="Fri, 1 Jan 2016 00:00:00 GMT" cacheControlMode="UseExpires" />
        </staticContent>
    </system.webServer>
</configuration>

With this setting, all static content, such as CSS, JS, and images, will now have an expires HTTP header set to next year (2016).

Optimize Image and Eliminate Render-Blocking CSS/JS

I won’t discuss both of them here because to me they are not really good suggestions, especially the CSS Delivery Optimization recommendation from Google.

The document says that “If the external CSS resources are small, you can insert those directly into the HTML document, which is called inlining. Inlining small CSS in this way allows the browser to proceed with rendering the page.” This will actually split the CSS code into multiple files and eventually make the whole project harder to maintain.

For the images, I think there is no need to compress them. Doing image browser caching should be enough to make the web page loads fast already.

So, I will just skip these two parts.

Summer 2015 Self-Learning Project

This article is part of my Self-Learning in this summer. To read the other topics in this project, please click here to visit the project overview page.

Summer Self-Learning Banner