downloading folders from google drive.

I wanted to download some course material on RL shared by the author via Google drive using the command line.  I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.
Post a Comment

AWS CloudFormation Pros and Cons

AWS CloudFormation Pros and Cons

So I'm building a PAAS product that does ML based optimisations and that means doing work in the cloud. The ML is a neat feature but without the basic product nothing will happen and to bootstrap this project on AWS I tried to make use of CloudFormation a service that automates creation and destruction of service stacks. Based on a week's worth of experimenting with CloudFormation I will try to answer the question:
"Is learning CloudFormation worth the effort?"
Despite the rant CloudFormation support creation, updating and deletion of entire stacks of services. SAM is built on top of CloudFormation and It has a visual editor.

The way CloudFormation is described, is that you can copy paste snippets to create resources and build a library of reusable components. This is a simplistic point of view. In reality you need to bring properties, specify dependencies, and introduce signalling mechanisms to ensure your template works.
The samples typically lack security features so when you secure your stack, assuming you know want, you'll need to dig deep into the documentation.  That's when you may find that code from GitHub or even the documentation is often broken. To say that CloudFormation templates in YAML are poorly documented is an understatement. They frequently contain blocks of complex scripts and JSON specs. These are often encoded using string processing commands, defeating the notion of having a template.

The Pros:

  • Infrastructure as code is an automation win .
  • Requires to integrate provisioning  into CI/CD.
  • Bridge between architect and implementer.
  • Formalise manual provisioning.
  • There is a tool to convert an existing setup to a template.
  • Lots of template and snippets available.
  • Can integrate and enhance work with other devops toolsets.
  • Working with CF this may quickly enhance your knowledge of the command line
  • CF templates are a part of most AWS tutorial and workshop. 
    • But there is a caveat - the CF templates are rarely even glossed over. They setup an architecture described using blocks diagram. If you want to look under the hood prepare to open a can of worms.

The Cons:

  • Steep learning curve.
    • The Templates
      • Using Template Intrinsic Functions
      • Referencing other templates
        • Outputs
        • Resources/Params
      • Pseudo Params
    • The python Helper Scripts
      • ConfigSets
      • Commands
      • signal init patten
      • install test pattern
    • Integrating with Git, CI/CD
    • Resilience - building resources in multiple availability zones etc. 
    • Security - implementing AWS recommendations. 
  • Knowledge of provisioning AWS services is a prerequisite to use CloudFormation in a serious way. (Both general and specific knowledge is needed).
  • StackOverFlow may be of little help in Q2 of 2019.
  • Like Puppet, Ansible, Chef, Vagrant, a deep knowledge of the linux and its command line and Configuration management is a prerequisite to just read the file.
  • AWS platform specific and is a AWS platform lock.
  • The Visual Editor in cloud formation is a waste of space as it only generates the top level place holders.
  • Many CloudFormation samples are broken indicating that the AWS services and their dependencies have frequent breaking changes.
  • Lack of guidance on how to keep CF code clear and clean. (If there is I never found it).
  • Your CF code is up and running - great you have created dozens of security holes....
Learning CloudFormation is pretty much like Alice's rabbit hole. You need to jump in and go all the way through and it is unclear where you will end up once you do. If you take a "crash course", gaping holes in your understanding will make reading and working with samples CF code nearly impossible.

The many broken sample templates indicate that AWS does not dog food their CF samples code using a CI/CD pipeline. AWS workshop frequently state that their session follows best practices via a CF  template. Good luck trying to read it, to see how this is done.

Unzipping the resource specification for a single region results in 330 files. Some services have just one, others likeApiGateWay and Ec2 have as many as 50 spec files. Most services have a just a few. You wont work from the spec - unless you are a tool developer but it presents a view of the scope and complexity of CloudFormation.

The bottom line - learning CF is worth the effort for Developers and DevOps if and only you have serious experience with AWS and are committed to the AWS platform for multiple projects.

Further reading:


Comments

Post a Comment

Popular posts from this blog

Moodle <=< Mediawiki SUL integration - first thoughts

Image
SUL is  Wikipedia's  Single User Login system The goal is to use it to authenticate moodle users there is one caveat - what happen if  the moodle user has no account. Moodle and MediaWiki account creation require different information. Since we'd like to use MediaWiki's standard which is highly permissive. It is necessary to change  moodle's requirements. Challenges Moodle does not explicitly define an object for setting registration requirements.  This needs to be accomplished by admins via edit a number of files manually.  errors will place the users in a limbo/blocked state... This is implemented differently on different version of moodle. This will break if the changes are overwritten by a system is upgrade. Directions Ideally Moodle should have a registration policy  object which allows the admin to define which fields are required and if they need to be unique . For example Moodle complains if different users share an email adr
Read more

downloading folders from google drive.

I wanted to download some course material on RL shared by the author via Google drive using the command line.  I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.
Read more

Insight into progressive web apps

Image
Some notes from a Meetup on PWAs in January 2016. I feel quite knowledgable on PWA but I wanted to learn more on implementing service worker. I ended up adding some research and collecting some great resources. However I ended up getting more detailed materials on the service worker based on google's developers docs. Also the resources have been expanded. https://www.meetup.com/The-Future-is-Javascript/events/246346861/ Service worker Service Workers Its just a simple JavaScript file that sits between you and the network – It runs in another thread – It has no access to DOM – It intercepts every network request (including cross domain) Entry point: self.caches (in service worker) or window.caches (on page)   Registering a Service Worker • Works with promises • Re-registration works fine In main.js navigator.serviceWorker.register('/sw.js').then(function(reg){ console.log('regsitered'); }.cat
Read more