This blog is where I share insights from any number of adventures in data analysis.
I will cover best practices from modeling and analysis projects; share tips on using new tools; outline new projects; recount war stories from Wikipedia and other FOSS projects I contribute to, and discuss challenges on information retrieval challenges, natural language processing tricks and game theoretic insights, portfolio analysis, social network analysis.
I wanted to download some course material on RL shared by the author via Google drive using the command line. I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.
Get link
Facebook
Twitter
Pinterest
Email
Other Apps
Automating web app development with Polymer and Yeoman
Get link
Facebook
Twitter
Pinterest
Email
Other Apps
Yeoman lets you configure and
stamp out sophisticated boilerplate
projects from the command line.
In a digital marketing agency the data science team may be asked to provide for each client's campaign fairly similar media reports and explanatory analytics dashboards for both external clients and internal clients. For longer term project we may be asked to also provide predictive analytics. The data comes from advertisers Google, Facebook, Taboola, OutBrain, phone tracking metrics API, outcomes are channels via segment into say Google Analytics which has both an API and polymer components. Usually there will be additional data science products prediction special segment data, funnels, market research, attribution charts which and long term data in BigQuery which has an API as well. Some vendors don't have an API so to access their data it is exported into a CSV and placed into google sheets which has an API and a Polymer component phone tracking. It takes too much work and time to code all these dashboards unless the campaign is long term. But when one uses a generator like Yeoman and incrementally add each part the creation of much of these dashboards can be automated reducing time and complexity for creating these solutions.
A second use case for using this combo is creating html5 banners. Often campaigns require creating many simail banners especially if you want to scientifically optimise your creative using a fractional factorial experimental design. Here is the Buildabanner Yeoman generator
Automation
Polymer together with Yeoman can help kick start new web project with an opinionated fully baked tooling infrastructure. Each new edition of polymer has many changes and different tooling. Yeoman is not very well documented and challenging to integrate into the increasingly automated build formats most CLI use today. So while these have fairly steep learning curves which may make it difficult to justify the return on time invested for tooling etc. But together if you have many similar projects planned, or are building a self-serve system Yeoman and polymer may be just the right fit.
Yeoman is an automation tool for creation of a web project. The more structured your projects are the more Yeoman is going to save you time. It also shares with web components the notion of compatibility which can help support complexity. However to make a Yeoman generator bullet-proof may require long term support and fixing bugs which occur on other people's systems.
The two have been combined in the Polymer CLI, though currently you may be interested in the following resources if this is a project you wish to automate. It also allows teams to concentrate knowledge into a generator which will more readily supports additional automation via scripts build tools and a split production and development pipeline as well as CI down the line.
Another issue common to work with large boiler plate projects - few people know what all the boiler plate is doing, how it can be tested or changed. So consider that you should document the project thoroughly.
Resources on accelerating Polymer projects with Yeomen
I cover these because Yeoman is easy to get started with but you soon end up interacting with Bower, NPM as well as other tools which boils down to a steep learning curve.
The Polymer the Javascript library that teaching the browser new tricks. Polymer has had strong ties to Yeoman project - perhaps since the polymath Addy Osmani is on both development teams. Yeoman is described as "The web's scaffolding tool for modern web apps".
Here is an article from 2013 on creating polymer projects using Yeoman, a year later he release the following video:
+Addy Osmani - Building a Polymer app with Yeoman 2014
YOLOmer! Polymer and Yeoman for lighting fast dev
When Polymer 1.0 was released it was introduced with a couple of starter kit projects one for new users and another for power users. This project included some fairly sophisticated use of tooling to provide a plethora of configurable features such as offline support using service workers. At the polymer summit in 2015. Rob Dobson introduced a polymer Yeoman generator that stamps out a psk project
+Rob Dobson - End to End with Polymer from The Polymer Summit 2015
In 2016 polymer introduced routing and layouts components and behaviors. These were released with a Polymer CLI a command line tool based on Yeoman that provides unified install and access to many tools that a used with Polymer. However, the PSK2 project was not updated to work with these and eventually a much simpler starter project was recommended.
These were introduced by Rob Dobson in 2016 in a couple of Polycast episodes 52 and 53 focused on the PSK2.
Rob Dobson - How to build a CLI generator -- Polycasts #53
Polymer CLI Generators 101
Polymer CLI allows us to generate components. There are many types of components and creating a working environment with a demo and tests takes lots of work and research. We are also at the cusp between polymer 1 and polymer 2 where es2015 rules supreme.
So what Let's build some generators for custom components in Polymer. This is one use case where working with generators can have a significant payoff in time savings
Polymer 2 preview elements
es6 polymer element element
psk2 with es6 support
an element in psk2 with es6 support
a style element
a behaviour
d3 element etc (integration with libraries that can play with polymer) What are the requirements for a lib so it can be integrated with polymer's data binding model.
SUL is Wikipedia's Single User Login system The goal is to use it to authenticate moodle users there is one caveat - what happen if the moodle user has no account. Moodle and MediaWiki account creation require different information. Since we'd like to use MediaWiki's standard which is highly permissive. It is necessary to change moodle's requirements. Challenges Moodle does not explicitly define an object for setting registration requirements. This needs to be accomplished by admins via edit a number of files manually. errors will place the users in a limbo/blocked state... This is implemented differently on different version of moodle. This will break if the changes are overwritten by a system is upgrade. Directions Ideally Moodle should have a registration policy object which allows the admin to define which fields are required and if they need to be unique . For example Moodle complains if different users share an email adr
I wanted to download some course material on RL shared by the author via Google drive using the command line. I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.
AWS CloudFormation Pros and Cons So I'm building a PAAS product that does ML based optimisations and that means doing work in the cloud. The ML is a neat feature but without the basic product nothing will happen and to bootstrap this project on AWS I tried to make use of CloudFormation a service that automates creation and destruction of service stacks. Based on a week's worth of experimenting with CloudFormation I will try to answer the question: "Is learning CloudFormation worth the effort?" Despite the rant CloudFormation support creation, updating and deletion of entire stacks of services. SAM is built on top of CloudFormation and It has a visual editor. The way CloudFormation is described, is that you can copy paste snippets to create resources and build a library of reusable components. This is a simplistic point of view. In reality you need to bring properties, specify dependencies, and introduce signalling mechanisms to ensure your template works. T
Comments
Post a Comment