I was playing around with Sagemaker notebook and EMR functionality recently. I was looking for some ready made cloudformation examples which bootstrapped a sagemaker notebook which was provisioned with a connection to an EMR cluster. This AWS blog looked interesting. I was able to grab the cloudformation template, but was…

Here are the steps to install python 3 on docker. First download the appropriate docker image: docker pull centos 2. Start the docker container: docker run -it -name boto3-centos centos The -name parameter ensures that the container has the name boto3-centos . You could name it whatever you find reasonable …

As I mentioned in part 2 of this story, the async-parallel version is actually not really async. Really the speedup is more due to using the multithreading facilities of asyncio (bascially invoke your blocking sync call in loop.run_in_executor(...) ) rather than actually concurrently running async tasks. So the search was…

In Part 1 of this story , I shared with you how I managed to get a nice speedup of s3 processing (with a running example of s3 copy) using multiple threads and finetuning the level of parallelism. However can we do better? In terms of tuning code for processing…

AWS S3 is the go to cloud based object storage widely used in the industry today. But as you become a power user of s3, some of the standard ways of copying , deleting , listing objects show their limitations. For example: How does one copy objects from one s3…

As a software engineer and a semi regular computer science guy, I like to take various software challenges that get presented at work and dig a little bit deeper. One such issue was that of exploring context free grammars. We sometimes need to express a CFG and then use a…

I happened to research on how arbitrary boolean expressions can be parsed at runtime to compute true / false values. Since I wanted to implement this in Scala, I did some research and found this interesting nugget in StackOverflow. It was a way to idiomatically expressing the grammar in Scala…

In part 1 I discussed the use case of assigning session id to closely occurring events generated by a particular user’s activity. We discussed the various options available out of the box from spark and why there is a gotcha with each option. …

I have been looking at all kinds of interesting use cases for data processing for events. Then recently came upon this interesting problem: Lets say you have a series of events emitted as user clicks from a web application. Simplistically the event object would look like this: { "userid"…