downloading folders from google drive.

I wanted to download some course material on RL shared by the author via Google drive using the command line.  I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.

Insight into progressive web apps

Some notes from a Meetup on PWAs in January 2016. I feel quite knowledgable on PWA but I wanted to learn more on implementing service worker. I ended up adding some research and collecting some great resources.

However I ended up getting more detailed materials on the service worker based on google's developers docs. Also the resources have been expanded.





Service worker


Service Workers Its just a simple JavaScript file that sits between you and the network
– It runs in another thread
– It has no access to DOM
– It intercepts every network request (including cross domain)

Entry point: self.caches (in service worker) or window.caches (on page)

 

Registering a Service Worker

• Works with promises
• Re-registration works fine

In main.js
navigator.serviceWorker.register('/sw.js').then(function(reg){
console.log('regsitered');
}.catch(function(err)){
console.log('Boo!');
});

Setting an interception scope

The default scope is where the sw file is, but you can control that

navigator.serviceWorker.register('/sw.js',{scope: '/my-app/'});

It will then control /my-app/ and its subdirectories

On install


  • Add initial resources (the application shell) to the cache
  • Cache has a name
  • Array of resources to cache
  • Mechanism to get a resource by path (a map)

console.log('Service Worker Registered!');
// This function build an array of urls,
// fetch them, and store the responses in the cache,
// example: key: 'main.js' value: 'alert(3)'

var cacheName = 'app-shell-cache-v1';
var filesToCache = ['/', '/index.html', ...];
self.addEventListener('install', event => {
 event.waitUntil(
   caches.open(cacheName).then(cache => {
       return cache.addAll(filesToCache);   //load app shell into the cache.
   }).then(() => {
     return self.skipWaiting();
   })
 );
});

The install should happen in the background in case there is a previous version of the service worker running. If the install fails the old service worker will be left intact.


On activate

Update cache - remove outdated resources. Cache should be versioned. If the sum off all caches is to big for an origin point is too big they may be reclaimed. So we should ensure th remove old data. This is done more easily if we use versioned caches.

self.addEventListener('activate',e => {
 e.waitUntil(
   caches.keys().then(keyList => {
     return Promise.all(keyList.map(key => {
       if (key !== cacheName) return caches.delete(key);
     }));
   }));
 return self.clients.claim();
});


On fetch

Retrieve from cache with network fallback
Allows to intercept page loading
Can get page from the cache or from network,
Handle offline and 404 with exception

self.addEventListener('fetch', event => {
 event.respondWith(
   caches.match(event.request)
    .then(response => {
      return response || fetch(event.request); //return cached else fetch
   })
 );
});

How this is handled in practice depends on resources and their rate of change. So the shell might be fetched from cache first
news might be fetched from the network and fall back to the cache if offline.

Serving files from the cache


Cache falling back to network


As above

self.addEventListener('fetch', function(event) {
 event.respondWith(
   caches.match(event.request)
   .then(function(response) {
     return response || fetch(event.request);
   })
 );
});

Network falling back to cache


Frequently updated data with fallback to cache - say for news where we have an older feed.

self.addEventListener('fetch', function(event) {
 event.respondWith(
   fetch(event.request).catch(function() {
     return caches.match(event.request);
   })
 );
});

Cache then network


For resources that update frequently and are not versioned in the shell
E.g. (articles, avatars, social media timelines, game leader boards)
Requires 2 requests - one to cache and one to the network.

Note this code goes in the main script not the SW as it is … reactive

var networkDataReceived = false;
var networkUpdate = fetch('/data.json')
.then(function(response) {
 return response.json();
}).then(function(data) {
 networkDataReceived = true;
 updatePage(data);
});


Next we look for the resource in the cache. This will usually respond faster than the network request. We use the cached data to provide a quick response. If the network provides newer data we update again. If cache fails we try to get from the net


caches.match('/data.json').then(function(response) {
 return response.json();
}).then(function(data) {
 if (!networkDataReceived) {
   updatePage(data);
 }
}).catch(function() {
 return networkUpdate;
})

Generic fallback


Here is a version with a generic fallback to an offline mode if network fails

self.addEventListener('fetch', function(event) {
 event.respondWith(
   caches.match(event.request).then(function(response) {
     return response || fetch(event.request);
   }).catch(function() {
     return caches.match('/offline.html');
   })
 );
});


Progressive web app use a manifest to setup an icon on mobile.


In html:

<link rel="manifest" href="/manifest.json">

Sample WebApp Manifest:

{
 "name": “Tovli",
 "short_name": “TovliWeb",
 "start_url": ".",
 "display": "standalone",
 "background_color": "#fff",
 "description": “Feel better today",
 "icons": [{
"src": "images/homescreen48.png",
"sizes": "48x48",
"type": "image/png"
 }]
}

Cache storage limits


Browser
Limitation
Notes
Chrome and Opera
No limit.
Storage is per origin not per API
Firefox
No limit.
Prompts after 50 MB
Mobile Safari
50MB.

Desktop Safari
No limit.
Prompts after 5MB
Internet Explorer (10+)
250MB.
Prompts after 10MB

The PWA Checklist

• Site is served over HTTPS (localhost permitted)
• Pages are responsive on tablets & mobile devices
• Site works cross-browser
 • Each page has a URL
• Page transitions don't feel like they block on the network
 • The start URL (at least) loads while offline
 • Metadata provided for Add to Home screen
 • First load fast even on 3G
 • See the full checklist

PWA with Vue


vue pwa app

npm install -g vue-cli
vue init pwa my-project
cd my-project
npm install
npm run dev

once done use NPM to ...

Run the app in development mode:
npm run dev

Build for production (uglify, minify etc):
npm run build

Run unit test with karma+ mocha + karma-webpack:
npm run unit

Run end to end test with night watch:
npm run e2e

Resources


Comments

Popular posts from this blog

Moodle <=< Mediawiki SUL integration - first thoughts

downloading folders from google drive.