Some Turkish symbols have display issues

[sathomas]

Some of the characters in İçeriğe erişim are more bold than the others. Perhaps we're missing something in the Open Sans font we're using.

[nicolaasmatthijs]

Assigning to @sathomas for investigation

[sathomas]

One thing I notice right away is that we're only serving Web Open Font Format (.woff) fonts. That format isn't supported on the Android browser before the latest version 4.4 (KitKat). Since only a whopping 1.1% of Android devices are running 4.4, we should probably add TrueType support as well. If the Turkish issue is related, we can combine the two issues. If it turns out that Turkish problem is something different, I'll create a separate issue for Android font support.

[sathomas]

BTW, any reason for not serving the Open Sans font directly from Google web fonts instead of hosting it ourselves? Google's font loader is pretty sophisticated about picking the best format for the user's browser (e.g. although .woff works everywhere but Android, Windows systems usually render .svg better), plus we'd automatically get the latest version should any of the fonts get updated.

[sathomas]

It does indeed appear to be an issue with the font. The version of Open Sans that we're serving only includes character sets for (some) Latin languages (234 glyphs). The complete Open Sans font from Adobe has more extended character sets, including some for Cyrillic and Greek languages (for a total of 938 glyphs).

Alas, I'm ignorant of the Turkish language, but I would guess it includes characters from Latin languages not fully represented in our current subset (e.g. South Eastern European) or perhaps some characters borrowed from Cyrillic or Greek. Updating to the full font would probably address this issue, but there isn't a complete guarantee. (For example, the full font from Adobe includes the entire basic Greek character set, but not the Polytonic or Coptic characters.) In any case, it's presumably the best we can do.

Note that the upgrade will definitely increase the size of our assets. The 10 font variations we're currently hosting take up about 250 KB. Moving to the full font would increase that to about 1 MB. (TrueType fonts are about 10x larger, so if we add .ttf variants, poor little Android smartphones could get a whopping 10 MB of fonts. That's more than twice the browser cache size on older Android versions, so performance will definitely suffer. The only good news is that TTF is highly compressible, so as long as we're gzipping on the server, bandwidth and latency won't take quite as much of a hit.)

If we need to support Android 2.x devices (with 4 MB browser cache), we might want to consider an adaptive approach where we only serve extended language versions when needed (based on HTTP Accept-Language and then accounting for user preferences). FWIW, Android 2.x is currently about 25% of the Android market.

[sathomas]

Re: my last comment

Actually, we can use the css :lang() pseudo-selector to load the appropriate fonts; no magic HTTP Accept-Language parsing required. The trick will be mapping the languages to the appropriate fonts. I wonder if crowdin can help with that?

Also, we'll want to verify that Android 2.x only loads font assets that are actually needed by the page content. I'm pretty sure that's the case, but not certain. I'd hate to go to the trouble only to find out that Android was opportunistically pre-loading all fonts referenced in the style sheets.

[sathomas]

Another option: Continue to use the existing Open Sans subset for Western European languages but fall back to a standard font stack (e.g. "Arial, Helvetica, sans-serif") for others.

[SamPeck]

Open Sans is almost identical to Droid Sans, is there any way we can tap
into Andriod's native font?
http://en.wikipedia.org/wiki/Open_Sans

[sathomas]

Sure. We could put Droid Sans at the head of our font stack. We could also switch from Open Sans to Droid Sans for all browsers. Droid Sans Pro includes the same character sets as the full Adobe Open Sans. Android browsers would avoid having to download any fonts, while all other browsers would see the same behavior as today, albeit with a (slightly) different visual result.

Note, though, that Droid Sans Pro is only available in two weights, Regular and Bold. Open Sans is available in ten weights (add Light, SemiBold, and ExtraBold, plus italic versions of all). We're currently serving all 10 weights of Open Sans, though it's not clear how many we're actually using (or intend to use).

Also, for some inexplicable reason, Google Web Fonts only offers the non-Pro version of Droid Sans, and that version omits the extra character sets needed, e.g., for Turkish. I want to research this a bit more. Maybe there's some licensing issue that would affect us as well.

[sathomas]

A quick check shows that we're currently using all 10 weights/styles of Open Sans. That probably rules out Droid Sans as an option, except possibly as a last resort for Andoid only..

[nicolaasmatthijs]

@sathomas : I don't remember exactly as to why we have taken a local copy of the Font instead of serving from Google directly. I'm assuming it was related to production builds and being able to concatenate everything together to reduce the total number of requests.

However, I'm not against serving this from Google, especially if it can help the issues we're seeing. Can you investigate what the page size implications are when switching to serving from Google web fonts ?

Also, can you explain how the css :lang pseudoselector would work in practice? Would we load the correct font files based on the lang attribute we add to the html tag on the fly? This would presumably also remove the ability to concatenate everything and cause additional HTTP requests, in which case the Google option becomes more attractive again.

[mrvisser]

I'm assuming it was related to production builds and being able to concatenate everything together to reduce the total number of requests.

We did try to inline, then backed out because there was too much duplication and the static assets were huge, so we went back to not inlining. So this very well may be the case. I haven't heard any consideration of using the google-hosted stuff though.

[sathomas]

Will definitely take a look at using Google.

There are a couple of ways I can think of for using :lang, but the idea is along the lines @nicolaasmatthijs describes. For the CSS rules which specify a font family, include an additional rule based on the language.

Here's a really quick example that I haven't thought through completely (so it may be a really bad idea). But it shows the general idea.

• Default all fonts to the generic sans-serif font. That pretty much guarantees that all the characters render consistently.

  body {
      font-family: sans-serif;
  }
  

• For languages that can be supported with the basic character set we're currently using, add a rule that specifies Open Sans.

  body::lang(en), body::lang(fr), ... {
      font-family: "Open Sans", sans-serif;
  }
  

There are plenty of other variations (e.g. "Open Sans" for Western European, "Open Sans Extended" for other languages). In theory, the browser will only download fonts that it actually needs, but that (among other things) should be tested.

[sathomas]

Browser Support of Font Formats

Nearly all browsers support Web Open Font Format .woff. The only significant exception (relevant to OAE) is Android prior to version 4.4. (Details at caniuse.com.) Earlier versions of Android (which currently make up about 99% of Android devices in use) support the TrueType .ttf format.

Font Format(s) Provided by Google Web Fonts

When a web page requests fonts hosted by Google Web Fonts, the service provides different font formats based on the browser's user agent string. The algorithm for selecting the format(s) appears to be:

• For Android browsers, provide TrueType .ttf fonts.
• For Internet Explorer, provide Embedded OpenType .eot fonts with a fallback to Web Open Font Format .woff.
• For all other browsers, provide Web Open Font Format .woff.

Character Sets

The OAE currently uses a basic subset of the full Open Sans character set. That basic subset contains 234 glyphs and generally supports only Western European languages. The full Open Sans character set contains 938 glyphs and supports many additional languages.

The OAE currently hosts local copies of the Open Sans fonts and delivers those fonts from its own server. Were OAE to switch to accessing the fonts via Google Web Fonts, it could still elect to use only the basic subset. It could also use the full character set or any of a number of other subsets and combinations.

Font Sizes

The OAE style sheets currently reference all five available Open Sans weights: Light (300), Normal (400), Semi-Bold (600), Bold (700), and Extra-Bold (800). The exact fonts in use on any OAE page will depend on which CSS styles are active on that page, but it would be possible for a single page to use all five weights and their italic (technically, oblique) styles. The following data assumes all 10 font faces are in use.

The following table lists the file sizes (in KB) of the different font formats for both the basic and full character sets. The table shows both native TrueType file sizes and TrueType files compressed with gzip. Other font formats have native compression, so gzip has little to no effect on their file sizes. (Note: while gzip compression reduces bandwidth and latency, it does not affect memory requirements, e.g. browser cache consumption, once assets are delivered to the browser.)

The table includes data for Embedded OpenType .eot as it represents a potential optimization (about 20% reduction in file size) for Internet Explorer users and because it is the default format that Google Web Fonts delivers to IE.

              Basic     Full
.eot           224       842
.ttf           481      2173
.ttf (gzip)    260      1139
.woff          261      1100

From the table, changing from the basic subset to the full character set increases the font size by about a factor of 4. Uncompressed TrueType fonts are about twice the size of other font formats.

[sathomas]

Font Recommendations and Options

Here are some of the main options available. Various permutations of the approaches below are also possible.

Note: In the following "consistent character set" means that all characters on a page are rendered in the same font for any given user. It does not mean that all users see the same font.

1. Use Google Web Fonts instead of local hosting and reference the full Open Sans character set.

Advantages:

• Outsources font maintenance and configuration to Google
• Open Sans fonts for all users
• Consistent character set for all users

Disadvantages:

• Potentially unnecessary bandwidth and latency for Western European users (additional ~750KB)
• Significant browser cache consumption for older Android browsers

2. Locally host Open Sans .woff and .ttf formats and provide full character set only to non-Western European users

Advantages:

• Open Sans fonts to all users
• Consistent character set for all users
• Minimizes bandwidth and latency for all users

Disadvantages:

• OAE responsible for font maintenance and configuration
• Additional burden of matching CSS files to crowdin.net data
• Modest browser cache consumption for Western European users on older Android browsers
• Significant browser cache consumption for non-Western European users on older Android browsers

3. Use Google Web Fonts basic character subset only, defaulting non-Western European users to sans-serif

Advantages:

• Outsources font maintenance and configuration to Google
• Consistent character set for all users
• Zero bandwidth and latency for non-Western European users
• No browser cache consumption for non-Western European users

Disadvantages:

• Open Sans only for Western European users
• Additional burden of matching CSS files to crowdin.net data
• Modest browser cache consumption for Western European Users on older Android browsers

4. Use Google Web Fonts basic character subset, locally hosted extended character set, and default older Android devices to sans-serif

Advantages:

• Open Sans to all users except non-Western European users on older Android devices
• Consistent character set for all users
• Minimizes bandwidth and latency for all users
• No browser cache consumption for non-Western European users on older Android devices

Disadvantages:

• OAE responsible for font maintenance and configuration
• Additional burden of matching CSS files to crowdin.net data
• Modest browser cache consumption for Western European users on older Android browsers

---

As an additional optimization if OAE locally hosts Open Sans, it can rename the web font version(s) (e.g. to "Open Sans Web") and add the standard name ("Open Sans") to the front of the font stack. Users that happen to have Open Sans installed locally will then use their local copy instead of the web font, eliminating the network bandwidth and latency of the font download.

[simong]

Do we need all the different weights? I had a quick browse through the app and I couldn't see more than 2 opensans font at page load and then 3 extra that got lazily loaded. Having to fetch all 10 fonts on page load seems like an absolute worst-case scenario. Doing a bunch of work that will need to be maintained (ie: every time we add a new language) seems a bit cumbersome for something that seems like an edge-case?

I'd like to see the impact of simply using Google's font CDN as:

• it would cut bandwidth
• speed up asset downloading (different domain etc)
• it seems like their would be quite some users who might have it cached [1]
• it might be "good enough"?

[1] http://www.google.com/fonts/specimen/Open+Sans#stats

[simong]

I forgot to mention that it also supports ~800 characters rather than the ~250 we currently support

[nicolaasmatthijs]

I was very concerned about having to download megabytes of fonts on the first page load. However, if the browsers lazy load them, I see no reason for not using Google's font CDN. Lets go ahead and switch to that.

[simong]

There is some indication that IE9 downloads all the @font-faced fonts as per Paul Irish [1] but I don't have a VM to try it out in. IE10 doesn't seem to suffer from this problem. You can try it out at [2]

[1] http://www.paulirish.com/2009/fighting-the-font-face-fout/#update2011
[2] http://dl.dropboxusercontent.com/u/39519/webfontsdemo/loadtest.html

[sathomas]

Just as a note, it's not always the case that the browser will completely optimize font loading. For example, on the OAE home page Safari 7.1 loads five of the ten fonts:

Even though only three font weights (200, 400, 600) are used on the page:

[nicolaasmatthijs]

@simong : I'm actually seeing that IE9 and IE10 both download all the @font-faced fonts on page load. I'll give @sathomas's solution to verify that this is indeed the case.

[nicolaasmatthijs]

I think we need this to be // to make sure that OAE and the downloaded font are using the same protocol and avoid unsafe content messages.

[simong]

When I run the production build on this branch, it doesn't seem to import the google's font CSS although it is in the minified file ( http://cl.ly/code/1S120h2s3X3F )

We'll probably also want to add shelljs to the fileExclusionRegExp in Gruntfile.js

[simong]

FWIWI: It's not loading the google CSS file because the @import rule is not the first rule in the (minified) file

[sathomas]

Here are the CSS selectors that specify a font weight other than normal (400) or bold (700):

In oae.base.css:

    h1, h2, h3, h4, h5, h6: 600
    .nav-list .nav-header: 500
    .modal-header h3: 200
    .modal-body h4: 200
    .badge: 600
    ul.as-list li.as-result-item em: 600

In oae.components.css:

    .oae-clip .oae-clip-content > button, .oae-clip .oae-clip-content ul li button: 600
    .oae-clip .oae-clip-content h1: 600
    .oae-tile h3, .oae-tile h3 a: 600
    .oae-listitem h3: 500

In oae.error.css:

    #error-content h1: 200
    #error-content #error-quote-container #error-quote: 600

In bootstrap.css:

    .lead: 200
    blockquote p: 300
    .navbar .brand: 200
    .carousel-control: 100
    .hero-unit: 200

In oae.index.css:

    #index-content: 200
    #index-content h1, #index-content h2: 200

In manageaccess.css:

    #manageaccess-modal label[for="manageaccess-share-role"]: 200

[sathomas]

The current styles do tend to use 600 (semibold) weight a fair bit, and there is a definite difference between it:

and true bold (700):

A font weight of 200 is also commonly used. Since there is no 200 weight for Open Sans, browsers are presumably substituting the closest (300) weight. Again, there is a definite difference between 300

and normal (400):

It does seem like we don't need to include the extra bold (800) variant when requesting the fonts from Google

[sathomas]

I think this is about the best we can do without either:

• redesign to eliminate the use of 300 and/or 600 weight
• self-hosting the fonts and taking on the burden of coordinating CSS with translations

In the worst case (Android browser, Western European user) we're adding about an extra 600 KB to the page load

[nicolaasmatthijs]

@sathomas: Is there more work you'd like to do here or is this ready to be merged?

[sathomas]

Ready for merging. If we do decide to modify the design or take on responsibility of managing fonts with translations, we can re-visit. Also, keep an eye out for Vietnamese translations. If/when that happens, we'll have to add the Vietnamese character set.

[nicolaasmatthijs]

@sathomas : Can you submit this PR against the bootstrap branch?

[sathomas]

Closed in favor of #3556