Posts

Showing posts with the label database

downloading folders from google drive.

I wanted to download some course material on RL shared by the author via Google drive using the command line.  I got a bunch of stuff using wget a folder in google drive was a challenge. I looked it up in SO which gave me a hint but no solution. I installed gdown using pip and then used: gdown --folder --continue https://drive.google.com/drive/folders/1V9jAShWpccLvByv5S1DuOzo6GVvzd4LV if there are more than 50 files you need to use --remaining-ok and only get the first 50. In such a case its best to download using the folder using the UI and decompress locally. Decompressing from the command line created errors related to unicode but using the mac UI I decompressed without a glitch.

SQL - Selection v.s. Projection

Selection and projection are two high level processes taking place when SQL queries are executed. Selection is choosing some records (rows) from a table and leaving others out. e.g. rows having name='oren'. Projection is the choosing some columns from each record and leaving others out. e.g. only name. So the select keyword performs projection while the where and keyword performs selection. Clearly the choice of using the keyword select for projection (choosing columns) rather than choosing rows, is an unfortunate flaw in the design of SQL, but this oversight is too well established to be fixed.

SQL Dojo

Image
TLDR: Imagine just before your DS interview - you are NEO your coach is Morpheus, and you will be practice SQL in rapidly changing schemas. Now here is a little project I thought up: Despite any number of excellent SQL based projects I have created I tend to get rusty in SQL as I don't use it on a regular basis. I decided it might be worthwhile  to setup a virtual space to practice, hence the dojo. The dojo lets a student practice analytical sql primarily queries analysts use. Ultimately I'd like to to use it in an agile manner as an LMS with a minimal UI.  This would require creating a story for each query and a test that the query returns a good answer. Also to make things interesting the tasks should be related and proceed from easy to more challenging and cover a number of techniques like filtering, aggregation and subqueries. However, initially I want to have things up and running quickly and to collect questions and answers that reflect how to do create view...

My first BigQuery DWH

Image
BigQuery is Google's Analytics Database Some notes about a project that has taken up lots of time recently. It was a media attribution dashboard for a client running several hundred campaigns. A POC version of the project had been created manually using spread sheets and we had to provide a drop in a replacement ASAP . I took up the task of migrating a spreadsheet based BI to a more robust script and SQL based platform able to handle the rapidly aggregating data which would soon overpower the spreadsheet's models. A secondary challenge was that the entire system were analyzing was under development and would change daily. Despite it's lacks as a classical database (missing triggers and in schema protections) I choose BigQuery for its scale-ability and ease of integration. Despite its limitations it soon felt like a perfect fit for this type of project. Data collection Data is currently acquired daily via API from various platforms: for example Google AdWords an...

Finding your friends on Wikipedia

Image
A segment of a social network (Photo credit: Wikipedia ) In my opinion, the most notable missing features of Wikipedia is the ability to network socially . Lack of social features conspire to minimise participants'  social capital (the expected collective or economic benefits derived from the preferential treatment and cooperation between individuals and groups). Lack of networking capabilities is in line with a pervasive attitude that ownership of contributions should be suppressed, external authority ignored and that knowledge matters in the project. Accordingly social costs of participation are growing. Participants in the project have great difficulty in acquiring social capital. The stakes of establishing world opinion tend to outweigh social stakes. Accordingly there is a growing gender gap, under-representation of cultural minorities and disrespect of real world authority. Currently users will try to establish social capital by listing on their user page their m...

Tips on working with SQL Fiddle

Image
C-3PO vs. Data (137/365) (Photo credit: JD Hancock ) Trying to get help with SQL ?  Chances are that you could use the open-source web application  mysqlfiddle.  Usually questions asked on the web leave much to be desired in terms of precision. The good news is that with Sqlfiddle you can set up a copy of the relevant sql schema, add some sample data,  add your best attempt and and share these with the people you are chatting or corresponding with. Next follows an example environment I set up to explore on collecting editor sessionstatistics for wikipedia's undocumented revision_user table. Working with a full environment was not practical. Working with the sqlfiddle actually ended up as a big time saver since testing the queries on a live database would have taken hours. http://sqlfiddle.com/#!2/d2f02/22 It sometimes seems that setting up a Database environment is a bit of an overkill - especially when it is just to ask a question. However if you have a wor...

Popular posts from this blog

downloading folders from google drive.

Random Thoughts on Linear Regressions

SQL - Selection v.s. Projection