Got Thickness, Slant, and Width data by scraping website. Put sliders in Google Font App.

To scrape the contents of the website I first got all the elements with attribute ‘gf-font-style’ using this code.

The value of the attribute was a string which contained the font family, weight and style information. I parsed the string to extract these as three separate variables. This function helped for parsing the string also.

I then built up a list of functions which I could put in the console, and it would extract the weight, family and style of all the fonts on the page. I did this on each page for all the thicknesses, widths, and slants, and so I have all this data saved in arrays.

I built sliders in the extension to filter the data by thickness, width and slant using this new data.


Filter sliders working

Number style filter working.

Now want to fix up the styling a little and it will be ready for next push.

Have put in a switch so that I can apply the changes to all pages, or just the one.

Finally, I need to put an option to persist the changes or not. So like whenever a tab is opened, those fonts will show.

This can be a checkbox.

To persist the changes, every time a tab is opened, it will need to apply the styles. So it is pretty straightforward. The function to apply the styles already exists on the contents page. I just need to save the font family, weight, and style somewhere (background page) and then apply them whenever a new page opens.

So I will have an object in the background page. Each object within it will be identified by a selector. Screen Shot 2017-03-23 at 10.27.37 PM

Details page fully working, and trying some web scraping

The details page is fully working. App reloads to details view if that was the last view before the popup closes. To do this I had to move more states to the applicationState object, since this is the object that is sent to the content page, and so the showDetails value is stored in it now, so the content page can update the popup on that.

Have hidden overflow of the detail variants text. So before it was moving to a new line if it couldnt fit. I resolved this by adding the following CSS:

  • text-overflow: ellipsis;
  • white-space: nowrap;
  • overflow: hidden;

The nowrap stops it moving to a new line, and the overflow:hidden hides any text tht goes over the div. Ideally I want any text that overflow to be truncated with a … , which is what the text-overflow:ellipsis is supposed to do, but for some reason that is not working. So I have just left it for now. The nowrap was an important enough find and will do for now.

Am now loading full set of Roboto fonts on load, because they are needed by material UI. When switching to the detail page, I noticed some text in the apply div was going bolder and causing a flash of unstyled fonts. This only happened in the Roboto detail, so bascially when the Roboto variants were donwloaded, the buttons and other icons were updating too. So have preloaded the full set of Roboto weights now, so these load correctly on start. Material UI didnt specify which weights were needed, as long as nowhere I could see, so I have just included them all now. Makes a much smoother transtion between the detail and main view, since the applyDiv, which is shared in both views, no longer reloads, so it looks a lot cleaner. 

Cleaned up the code more, and removed functions am not using. Moved the Google Filter Data (categories, varitans, languages, e.t.c.) to a separate file and am importing them on load, so the App.js looks cleaner.

Have pushed this new version to the store.

Will now work on scraping the Google Fonts page to get updated filters sets for weights, width and slant, and also to get the font pairs.

Things remaining to do on the UI is get the ellisis working on the font overlflow.

How the require method loads modules in Node.

Spent a while trying to crawl the google fonts page. It is actually not that simple because the fonts on the page are rendered using javascript, so if we donwload and then crawl the page, the fonts wont be included in the page source.

I tried this initially using requestJs, but this only gets the page source so it wasnt working. I also tried using a headless browser, phantomJs, to render the page, however the Google Fonts page blocks the headless browsers as it realises it is not a real browser. I also tried a python scrpaer, pyQt4, but the tutorial seemed outdated and i couldnt get it to work.

What make it touch also is the fonts only load on scroll. And the google font page will only ever have upto 20 font tiles in the DOM at any one time. It removes and adds the font tags from the DOM as your scroll. However the <style> tags for the different fonts remain.

I was testing at the end ways to extract the fonts from the page using the inspect console. Will try more on this tomorrow.

Background script added

12.30 pm : Fixed some bugs in the web font loading.

Loading the Lobster font seems to be giving errors. I have disabled that font for now.

Fixed the errors finally. Was loading the Lobster font as Lobster:400, which was givng errors on the font loader. So for all am now loading all regular variants as just the family name with no :400 attached. This seems to have fixed the issue. Also got rid of most of the Warnings

Lets add the favourites to a background scripts

6.45 pm Have added the background script so that the favourites are shared across all popups. Want to clean up the popup script a little now. To try and sort the render cycle so there is not a flash of the page before filtering. So that on open it renders the correct list straight away. It is especially slow for the favourites filter.

8.30 – cleaned up the script. It seems to be loading faster. ready for second version deploy.

Made the apply div show on all pages. Updated the styling a little.

Tomorrow need to get the details page to redisplay when the popup reopens if that was the last used screen.

Once I get that going, I can start building a scraper for the google fonts site to get the information that will allow people to filter the fonts based on slant, width, and thickness. And also to get the information for recommended pair fonts.

Once I get that, I can add those filters and data to the extension. Then it will be ready for final release.


Lazy loading Google Font App

I got the extension to open up quickly. Basically it took ages to open the popup because it was trying to render 818 font items before it opened. I set it up so it only loads 20 items initiallaly, and then the Waypoint scroll event will cause the rest of the items to be loaded all in one go once the user starts scrolling. It gives a good performance for that.

Updated the select fields so they restore the old value they had a when a user opens and closes the popup.

Also the previous states is being fully restored when the user closes and opens the popup.

There is currently a bug with the favourites sign. It is not turning red if the favourites are true!

Lazy loading the fonts, and updating the applyStyles function on the content page

Updated the styling on the chrome verion.

Also got the page updates to work well. Basically every selector has its own separate style tag injected onto the content page. If the extension wants to update the style for that selecotor on the page, the content page first see if that style element exists, since the style tags id contains the selector. If it does exist it deletes the style tag, then creates a new one for the same selector.So if we are updating all headers so

So if we are updating all headers so selectors are [h1, h2, h3, h4, ..] then a separate style tag is created for each selector. That way say if later we want to update the h1 selector only, we can easily delete its style tag and then update it. Also if we have multiple selectors in the style tag, the * selector seems to be able to override them, as oppose to if they are separate, they have more power/specificity over hte * selector.

Also noticed the extension popup was taking a while to open. I discovered the reason is because it is trying to prerender the entire list of fonts before opening. With 818 fonts, it takes a while for it to do that.

So I have discovered a workaroudn where I will only load the first 10 fonts initially, but I still need to figure out how to get the lazyloading working for the fonts also, or just how to render the rest of the fonts once the popup has opened.

I thought the slowness was from the api request to Google to get the list of fonts, but it was not that in the end.

So once I get this fast loading OK, I will be ready to share with Dennis to get started on the logos and tiles for the chrome store.

At that point I can start building the details page.


Google Font App Styling and Updating the Keyword Finder parser and options

Spent half the day on the Goolge Extension. All functionality fixed and styling updated. I figured I need to set the colors on the theme instead of updating the colors on the components separately. Once I do this it will be ready for first release. Want to share with Dennis at that point so he can start building nice icons and banners for it.

Spent second hald of the say on the keyword extension. It is much better now. I improved the parser lots so it sticks less words together, and is not getting text from the html attributes as before. Also I put it to display words that appear only 3 times or more, to filter out buggy words that still get through. Have also set minimum length to two letters to filter out more buggy words.

Set the maximum results length to 40 words, to stop people pages freexing increase they are trying to highlight hundreds of keywords. The Mark.js can cause the page to freeze if it is trying to highlight hundreds of keywords on the page.

Have pushed it to google. Am happy to share it now. Will go over it once more in the morning, before sharing with a few people.